Package | Description |
---|---|
org.apache.hadoop.conf |
Configuration of system parameters.
|
org.apache.hadoop.crypto.key |
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.
|
org.apache.hadoop.filecache | |
org.apache.hadoop.fs |
Implementations of
AbstractFileSystem for hdfs
over rpc and hdfs over web. |
org.apache.hadoop.fs.adl |
Supporting classes for metrics instrumentation.
|
org.apache.hadoop.fs.adl.oauth2 |
public interface to expose OAuth2 authentication related features.
|
org.apache.hadoop.fs.azure |
A distributed implementation of
FileSystem for reading and writing files on
Azure Block Storage. |
org.apache.hadoop.fs.azure.metrics |
Infrastructure for a Metrics2 source that provides information on Windows
Azure Filesystem for Hadoop instances.
|
org.apache.hadoop.fs.ftp | |
org.apache.hadoop.fs.impl |
This package contains implementation classes for use inside
filesystems.
|
org.apache.hadoop.fs.obs |
Package for supporting
HuaweiCloud
Object Storage Service (OBS) as a backend filesystem in Hadoop.
|
org.apache.hadoop.fs.permission |
This package provides support for HDFS permission and ACL.
|
org.apache.hadoop.fs.s3a |
S3A Filesystem.
|
org.apache.hadoop.fs.s3a.auth |
Authentication and permissions support.
|
org.apache.hadoop.fs.s3native |
A distributed implementation of
FileSystem for reading and writing files on
Amazon S3. |
org.apache.hadoop.fs.viewfs |
ViewFileSystem and ViewFileSystemOverloadScheme classes.
|
org.apache.hadoop.ha | |
org.apache.hadoop.hdfs |
A distributed implementation of
FileSystem . |
org.apache.hadoop.hdfs.client |
This package provides the administrative APIs for HDFS.
|
org.apache.hadoop.hdfs.server.aliasmap | |
org.apache.hadoop.hdfs.server.common.blockaliasmap.impl | |
org.apache.hadoop.hdfs.server.federation.store.driver |
The state store uses modular data storage classes derived from
StateStoreDriver to handle querying, updating and deleting data records.
|
org.apache.hadoop.hdfs.server.federation.store.driver.impl |
Implementations of state store data providers/drivers.
|
org.apache.hadoop.hdfs.server.namenode | |
org.apache.hadoop.hdfs.web.oauth2 |
OAuth2-based WebHDFS authentication.
|
org.apache.hadoop.io |
Generic i/o code for use when reading and writing data to the network,
to databases, and to files.
|
org.apache.hadoop.io.compress | |
org.apache.hadoop.ipc |
Tools to help define network clients and servers.
|
org.apache.hadoop.mapred | |
org.apache.hadoop.mapred.join | |
org.apache.hadoop.mapred.lib | |
org.apache.hadoop.mapred.pipes | |
org.apache.hadoop.mapreduce | |
org.apache.hadoop.mapreduce.lib.aggregate | |
org.apache.hadoop.mapreduce.lib.chain | |
org.apache.hadoop.mapreduce.lib.db | |
org.apache.hadoop.mapreduce.lib.input | |
org.apache.hadoop.mapreduce.lib.jobcontrol | |
org.apache.hadoop.mapreduce.lib.join | |
org.apache.hadoop.mapreduce.lib.output | |
org.apache.hadoop.mapreduce.lib.output.committer.manifest |
Intermediate manifest committer.
|
org.apache.hadoop.mapreduce.lib.partition | |
org.apache.hadoop.mapreduce.security | |
org.apache.hadoop.mapreduce.tools | |
org.apache.hadoop.mapreduce.v2.hs | |
org.apache.hadoop.metrics2.sink |
Builtin metrics sinks
|
org.apache.hadoop.net |
Network-related classes.
|
org.apache.hadoop.security | |
org.apache.hadoop.security.alias |
Provides the hadoop credential provider API.
|
org.apache.hadoop.security.authorize |
Support for service-level authorization.
|
org.apache.hadoop.security.http |
Filters for HTTP service security.
|
org.apache.hadoop.security.token |
Support for delegation tokens.
|
org.apache.hadoop.service |
Support for services.
|
org.apache.hadoop.service.launcher |
This package contains classes, interfaces and exceptions to launch
YARN services from the command line.
|
org.apache.hadoop.tools |
Command-line tools for MapReduce.
|
org.apache.hadoop.util |
Common utilities.
|
org.apache.hadoop.util.functional |
Support for functional programming within the Hadoop APIs.
|
org.apache.hadoop.yarn.applications.distributedshell |
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.
|
org.apache.hadoop.yarn.client | |
org.apache.hadoop.yarn.client.api | |
org.apache.hadoop.yarn.client.api.impl | |
org.apache.hadoop.yarn.client.cli | |
org.apache.hadoop.yarn.conf |
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.
|
org.apache.hadoop.yarn.event |
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.
|
org.apache.hadoop.yarn.logaggregation |
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.
|
org.apache.hadoop.yarn.logaggregation.filecontroller |
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.
|
org.apache.hadoop.yarn.security |
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.
|
org.apache.hadoop.yarn.security.admin |
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.
|
org.apache.hadoop.yarn.security.client |
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.
|
org.apache.hadoop.yarn.service.client |
Package org.apache.hadoop.yarn.service.client contains classes
for YARN Services Client API.
|
org.apache.hadoop.yarn.sharedcache | |
org.apache.hadoop.yarn.util |
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.
|
org.apache.hadoop.yarn.util.timeline |
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.
|
Modifier and Type | Method and Description |
---|---|
Configuration |
Configurable.getConf()
Return the configuration used by this object.
|
Configuration |
Configured.getConf() |
Modifier and Type | Method and Description |
---|---|
void |
Configuration.addResource(Configuration conf)
Add a configuration resource.
|
static void |
Configuration.dumpConfiguration(Configuration config,
String propertyName,
Writer out)
Writes properties and their attributes (final and resource)
to the given
Writer . |
static void |
Configuration.dumpConfiguration(Configuration config,
Writer out)
Writes out all properties and their attributes (final and resource) to
the given
Writer , the format of the output would be, |
void |
Configurable.setConf(Configuration conf)
Set the configuration to be used by this object.
|
void |
Configured.setConf(Configuration conf) |
void |
Configuration.writeXml(String propertyName,
Writer out,
Configuration config)
Write out the non-default properties in this configuration to the
given
Writer . |
Constructor and Description |
---|
Configuration(Configuration other)
A new configuration with the same settings cloned from another.
|
Configured(Configuration conf)
Construct a Configured.
|
Modifier and Type | Method and Description |
---|---|
Configuration |
KeyProvider.getConf()
Return the provider configuration.
|
Modifier and Type | Method and Description |
---|---|
abstract KeyProvider |
KeyProviderFactory.createProvider(URI providerName,
Configuration conf) |
static KeyProvider |
KeyProviderFactory.get(URI uri,
Configuration conf)
Create a KeyProvider based on a provided URI.
|
static List<KeyProvider> |
KeyProviderFactory.getProviders(Configuration conf) |
static org.apache.hadoop.crypto.key.KeyProvider.Options |
KeyProvider.options(Configuration conf)
A helper function to create an options object.
|
Constructor and Description |
---|
KeyProvider(Configuration conf)
Constructor.
|
Modifier and Type | Method and Description |
---|---|
static void |
DistributedCache.addLocalArchives(Configuration conf,
String str)
Deprecated.
|
static void |
DistributedCache.addLocalFiles(Configuration conf,
String str)
Deprecated.
|
static void |
DistributedCache.createAllSymlink(Configuration conf,
File jobCacheDir,
File workDir)
Deprecated.
Internal to MapReduce framework. Use DistributedCacheManager
instead.
|
static FileStatus |
DistributedCache.getFileStatus(Configuration conf,
URI cache)
Deprecated.
|
static long |
DistributedCache.getTimestamp(Configuration conf,
URI cache)
Deprecated.
|
static void |
DistributedCache.setArchiveTimestamps(Configuration conf,
String timestamps)
Deprecated.
|
static void |
DistributedCache.setFileTimestamps(Configuration conf,
String timestamps)
Deprecated.
|
static void |
DistributedCache.setLocalArchives(Configuration conf,
String str)
Deprecated.
|
static void |
DistributedCache.setLocalFiles(Configuration conf,
String str)
Deprecated.
|
Modifier and Type | Method and Description |
---|---|
Configuration |
FilterFileSystem.getConf() |
Modifier and Type | Method and Description |
---|---|
static boolean |
FileUtil.copy(File src,
FileSystem dstFS,
Path dst,
boolean deleteSource,
Configuration conf)
Copy local files to a FileSystem.
|
static boolean |
FileUtil.copy(FileSystem srcFS,
FileStatus srcStatus,
FileSystem dstFS,
Path dst,
boolean deleteSource,
boolean overwrite,
Configuration conf)
Copy a file/directory tree within/between filesystems.
|
static boolean |
FileUtil.copy(FileSystem srcFS,
Path[] srcs,
FileSystem dstFS,
Path dst,
boolean deleteSource,
boolean overwrite,
Configuration conf) |
static boolean |
FileUtil.copy(FileSystem srcFS,
Path src,
File dst,
boolean deleteSource,
Configuration conf)
Copy FileSystem files to local files.
|
static boolean |
FileUtil.copy(FileSystem srcFS,
Path src,
FileSystem dstFS,
Path dst,
boolean deleteSource,
boolean overwrite,
Configuration conf)
Copy files between FileSystems.
|
static boolean |
FileUtil.copy(FileSystem srcFS,
Path src,
FileSystem dstFS,
Path dst,
boolean deleteSource,
Configuration conf)
Copy files between FileSystems.
|
static AbstractFileSystem |
AbstractFileSystem.createFileSystem(URI uri,
Configuration conf)
Create a file system instance for the specified uri using the conf.
|
static FileSystem |
FileSystem.get(Configuration conf)
Returns the configured FileSystem implementation.
|
static FileSystem |
FileSystem.get(URI uri,
Configuration conf)
Get a FileSystem for this URI's scheme and authority.
|
static AbstractFileSystem |
AbstractFileSystem.get(URI uri,
Configuration conf)
The main factory method for creating a file system.
|
static FileSystem |
FileSystem.get(URI uri,
Configuration conf,
String user)
Get a FileSystem instance based on the uri, the passed in
configuration and the user.
|
static String |
StorageType.getConf(Configuration conf,
StorageType t,
String name)
Get the configured values for different StorageType.
|
static URI |
FileSystem.getDefaultUri(Configuration conf)
Get the default FileSystem URI from a configuration.
|
static FileContext |
FileContext.getFileContext(AbstractFileSystem defFS,
Configuration aConf)
Create a FileContext with specified FS as default using the specified
config.
|
static FileContext |
FileContext.getFileContext(Configuration aConf)
Create a FileContext using the passed config.
|
static FileContext |
FileContext.getFileContext(URI defaultFsUri,
Configuration aConf)
Create a FileContext for specified default URI using the specified config.
|
FileSystem |
Path.getFileSystem(Configuration conf)
Return the FileSystem that owns this Path.
|
static Class<? extends FileSystem> |
FileSystem.getFileSystemClass(String scheme,
Configuration conf)
Get the FileSystem implementation class of a filesystem.
|
protected static FileSystem |
FileSystem.getFSofPath(Path absOrFqPath,
Configuration conf) |
static TrashPolicy |
TrashPolicy.getInstance(Configuration conf,
FileSystem fs)
Get an instance of the configured TrashPolicy based on the value
of the configuration parameter fs.trash.classname.
|
static TrashPolicy |
TrashPolicy.getInstance(Configuration conf,
FileSystem fs,
Path home)
Deprecated.
Use
TrashPolicy.getInstance(Configuration, FileSystem) instead. |
static LocalFileSystem |
FileSystem.getLocal(Configuration conf)
Get the local FileSystem.
|
static FileContext |
FileContext.getLocalFSFileContext(Configuration aConf) |
static FileSystem |
FileSystem.getNamed(String name,
Configuration conf)
Deprecated.
call
FileSystem.get(URI, Configuration) instead. |
void |
TrashPolicy.initialize(Configuration conf,
FileSystem fs)
Used to setup the trash policy.
|
abstract void |
TrashPolicy.initialize(Configuration conf,
FileSystem fs,
Path home)
Deprecated.
Use
TrashPolicy.initialize(Configuration, FileSystem) instead. |
void |
LocalFileSystem.initialize(URI name,
Configuration conf) |
void |
FilterFileSystem.initialize(URI name,
Configuration conf)
Called after a new FileSystem instance is constructed.
|
void |
FileSystem.initialize(URI name,
Configuration conf)
Initialize a FileSystem.
|
void |
RawLocalFileSystem.initialize(URI uri,
Configuration conf) |
static boolean |
Trash.moveToAppropriateTrash(FileSystem fs,
Path p,
Configuration conf)
In case of the symlinks or mount points, one has to move the appropriate
trashbin in the actual volume of the path p being deleted.
|
static FileSystem |
FileSystem.newInstance(Configuration conf)
Returns a unique configured FileSystem implementation for the default
filesystem of the supplied configuration.
|
static FileSystem |
FileSystem.newInstance(URI uri,
Configuration config)
Returns the FileSystem for this URI's scheme and authority.
|
static FileSystem |
FileSystem.newInstance(URI uri,
Configuration conf,
String user)
Returns the FileSystem for this URI's scheme and authority and the
given user.
|
static LocalFileSystem |
FileSystem.newInstanceLocal(Configuration conf)
Get a unique local FileSystem object.
|
void |
ChecksumFileSystem.setConf(Configuration conf) |
static void |
FileSystem.setDefaultUri(Configuration conf,
String uri)
Set the default FileSystem URI in a configuration.
|
static void |
FileSystem.setDefaultUri(Configuration conf,
URI uri)
Set the default FileSystem URI in a configuration.
|
Constructor and Description |
---|
Trash(Configuration conf)
Construct a trash can accessor.
|
Trash(FileSystem fs,
Configuration conf)
Construct a trash can accessor for the FileSystem provided.
|
Modifier and Type | Method and Description |
---|---|
static Configuration |
AdlFileSystem.propagateAccountOptions(Configuration source,
String accountName)
Propagates account-specific settings into generic ADL configuration keys.
|
Modifier and Type | Method and Description |
---|---|
protected AzureADTokenProvider |
AdlFileSystem.getCustomAccessTokenProvider(Configuration conf)
This method is provided for convenience for derived classes to define
custom
AzureADTokenProvider instance. |
void |
AdlFileSystem.initialize(URI storeUri,
Configuration originalConf)
Called after a new FileSystem instance is constructed.
|
static Configuration |
AdlFileSystem.propagateAccountOptions(Configuration source,
String accountName)
Propagates account-specific settings into generic ADL configuration keys.
|
Modifier and Type | Method and Description |
---|---|
abstract void |
AzureADTokenProvider.initialize(Configuration configuration)
Initialize with supported configuration.
|
Modifier and Type | Method and Description |
---|---|
void |
NativeAzureFileSystem.initialize(URI uri,
Configuration conf) |
Constructor and Description |
---|
WasbFsck(Configuration conf) |
Constructor and Description |
---|
AzureFileSystemInstrumentation(Configuration conf) |
Modifier and Type | Method and Description |
---|---|
void |
FTPFileSystem.initialize(URI uri,
Configuration conf) |
Modifier and Type | Method and Description |
---|---|
Configuration |
AbstractFSBuilderImpl.getOptions()
Get the mutable option configuration.
|
Constructor and Description |
---|
OBS(URI theUri,
Configuration conf) |
Modifier and Type | Method and Description |
---|---|
static FsPermission |
FsPermission.getUMask(Configuration conf)
Get the user file creation mask (umask)
UMASK_LABEL config param has umask value that is either symbolic
or octal. |
static void |
FsPermission.setUMask(Configuration conf,
FsPermission umask)
Set the user file creation mask (umask)
|
Modifier and Type | Method and Description |
---|---|
protected software.amazon.awssdk.auth.credentials.AwsCredentials |
TemporaryAWSCredentialsProvider.createCredentials(Configuration config)
The credentials here must include a session token, else this operation
will raise an exception.
|
Constructor and Description |
---|
S3A(URI theUri,
Configuration conf) |
SimpleAWSCredentialsProvider(URI uri,
Configuration conf)
Build the credentials from a filesystem URI and configuration.
|
TemporaryAWSCredentialsProvider(Configuration conf)
Construct from just a configuration.
|
TemporaryAWSCredentialsProvider(URI uri,
Configuration conf)
Constructor: the URI will be null if the provider is inited unbonded
to a filesystem.
|
Constructor and Description |
---|
AssumedRoleCredentialProvider(URI fsUri,
Configuration conf)
Instantiate.
|
Modifier and Type | Method and Description |
---|---|
void |
NativeS3FileSystem.initialize(URI uri,
Configuration conf)
Deprecated.
Always fail to initialize.
|
Modifier and Type | Method and Description |
---|---|
void |
ViewFileSystem.initialize(URI theUri,
Configuration conf)
Called after a new FileSystem instance is constructed.
|
Constructor and Description |
---|
ViewFileSystem(Configuration conf)
Convenience Constructor for apps to call directly.
|
ViewFs(Configuration conf) |
Modifier and Type | Method and Description |
---|---|
HAServiceProtocol |
HAServiceTarget.getHealthMonitorProxy(Configuration conf,
int timeoutMs)
Returns a proxy to connect to the target HA service for health monitoring.
|
HAServiceProtocol |
HAServiceTarget.getHealthMonitorProxy(Configuration conf,
int timeoutMs,
int retries) |
HAServiceProtocol |
HAServiceTarget.getProxy(Configuration conf,
int timeoutMs) |
org.apache.hadoop.ha.ZKFCProtocol |
HAServiceTarget.getZKFCProxy(Configuration conf,
int timeoutMs) |
Modifier and Type | Method and Description |
---|---|
abstract ReplicaAccessorBuilder |
ReplicaAccessorBuilder.setConfiguration(Configuration conf)
Set the configuration to use.
|
Constructor and Description |
---|
HdfsAdmin(URI uri,
Configuration conf)
Create a new HdfsAdmin client.
|
Modifier and Type | Method and Description |
---|---|
Configuration |
InMemoryAliasMap.getConf() |
Modifier and Type | Method and Description |
---|---|
static InMemoryAliasMap |
InMemoryAliasMap.init(Configuration conf,
String blockPoolID) |
void |
InMemoryAliasMap.setConf(Configuration conf) |
static void |
InMemoryAliasMap.transferForBootstrap(javax.servlet.http.HttpServletResponse response,
Configuration conf,
InMemoryAliasMap aliasMap)
Transfer this aliasmap for bootstrapping standby Namenodes.
|
Modifier and Type | Method and Description |
---|---|
Configuration |
LevelDBFileRegionAliasMap.getConf() |
Configuration |
TextFileRegionAliasMap.getConf() |
Modifier and Type | Method and Description |
---|---|
void |
LevelDBFileRegionAliasMap.setConf(Configuration conf) |
void |
TextFileRegionAliasMap.setConf(Configuration conf) |
Modifier and Type | Method and Description |
---|---|
protected Configuration |
StateStoreDriver.getConf()
Get the State Store configuration.
|
Modifier and Type | Method and Description |
---|---|
boolean |
StateStoreDriver.init(Configuration config,
String id,
Collection<Class<? extends org.apache.hadoop.hdfs.server.federation.store.records.BaseRecord>> records,
org.apache.hadoop.hdfs.server.federation.metrics.StateStoreMetrics stateStoreMetrics)
Initialize the state store connection.
|
Modifier and Type | Method and Description |
---|---|
boolean |
StateStoreSerializableImpl.init(Configuration config,
String id,
Collection<Class<? extends org.apache.hadoop.hdfs.server.federation.store.records.BaseRecord>> records,
org.apache.hadoop.hdfs.server.federation.metrics.StateStoreMetrics metrics) |
Modifier and Type | Method and Description |
---|---|
Configuration |
FileSystemImage.getConf() |
Configuration |
FixedBlockResolver.getConf() |
Configuration |
SingleUGIResolver.getConf() |
Modifier and Type | Method and Description |
---|---|
abstract void |
DefaultAuditLogger.initialize(Configuration conf) |
void |
AuditLogger.initialize(Configuration conf)
Called during initialization of the logger.
|
void |
FixedBlockMultiReplicaResolver.setConf(Configuration conf) |
void |
FileSystemImage.setConf(Configuration conf) |
void |
FixedBlockResolver.setConf(Configuration conf) |
void |
SingleUGIResolver.setConf(Configuration conf) |
Constructor and Description |
---|
FSTreeWalk(Path root,
Configuration conf) |
Modifier and Type | Method and Description |
---|---|
Configuration |
AccessTokenProvider.getConf()
Return the conf.
|
Modifier and Type | Method and Description |
---|---|
void |
CredentialBasedAccessTokenProvider.setConf(Configuration conf) |
void |
AccessTokenProvider.setConf(Configuration configuration)
Set the conf.
|
void |
ConfCredentialBasedAccessTokenProvider.setConf(Configuration conf) |
void |
ConfRefreshTokenBasedAccessTokenProvider.setConf(Configuration conf) |
Constructor and Description |
---|
OAuth2ConnectionConfigurator(Configuration conf) |
OAuth2ConnectionConfigurator(Configuration conf,
org.apache.hadoop.security.authentication.client.ConnectionConfigurator sslConfigurator) |
Modifier and Type | Method and Description |
---|---|
Configuration |
EnumSetWritable.getConf() |
Configuration |
WritableComparator.getConf() |
Configuration |
ObjectWritable.getConf() |
Configuration |
GenericWritable.getConf() |
Configuration |
AbstractMapWritable.getConf() |
Modifier and Type | Method and Description |
---|---|
static <T extends Writable> |
WritableUtils.clone(T orig,
Configuration conf)
Make a copy of a writable object using serialization to a buffer.
|
static void |
IOUtils.copyBytes(InputStream in,
OutputStream out,
Configuration conf)
Copies from one stream to another.
|
static void |
IOUtils.copyBytes(InputStream in,
OutputStream out,
Configuration conf,
boolean close)
Copies from one stream to another.
|
static org.apache.hadoop.io.SequenceFile.Writer |
SequenceFile.createWriter(Configuration conf,
FSDataOutputStream out,
Class keyClass,
Class valClass,
org.apache.hadoop.io.SequenceFile.CompressionType compressionType,
CompressionCodec codec)
Deprecated.
|
static org.apache.hadoop.io.SequenceFile.Writer |
SequenceFile.createWriter(Configuration conf,
FSDataOutputStream out,
Class keyClass,
Class valClass,
org.apache.hadoop.io.SequenceFile.CompressionType compressionType,
CompressionCodec codec,
org.apache.hadoop.io.SequenceFile.Metadata metadata)
Deprecated.
|
static org.apache.hadoop.io.SequenceFile.Writer |
SequenceFile.createWriter(Configuration conf,
org.apache.hadoop.io.SequenceFile.Writer.Option... opts)
Create a new Writer with the given options.
|
static org.apache.hadoop.io.SequenceFile.Writer |
SequenceFile.createWriter(FileContext fc,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
org.apache.hadoop.io.SequenceFile.CompressionType compressionType,
CompressionCodec codec,
org.apache.hadoop.io.SequenceFile.Metadata metadata,
EnumSet<CreateFlag> createFlag,
org.apache.hadoop.fs.Options.CreateOpts... opts)
Construct the preferred type of SequenceFile Writer.
|
static org.apache.hadoop.io.SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass)
Deprecated.
|
static org.apache.hadoop.io.SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
int bufferSize,
short replication,
long blockSize,
boolean createParent,
org.apache.hadoop.io.SequenceFile.CompressionType compressionType,
CompressionCodec codec,
org.apache.hadoop.io.SequenceFile.Metadata metadata)
Deprecated.
|
static org.apache.hadoop.io.SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
int bufferSize,
short replication,
long blockSize,
org.apache.hadoop.io.SequenceFile.CompressionType compressionType,
CompressionCodec codec,
Progressable progress,
org.apache.hadoop.io.SequenceFile.Metadata metadata)
Deprecated.
|
static org.apache.hadoop.io.SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
org.apache.hadoop.io.SequenceFile.CompressionType compressionType)
Deprecated.
|
static org.apache.hadoop.io.SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
org.apache.hadoop.io.SequenceFile.CompressionType compressionType,
CompressionCodec codec)
Deprecated.
|
static org.apache.hadoop.io.SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
org.apache.hadoop.io.SequenceFile.CompressionType compressionType,
CompressionCodec codec,
Progressable progress)
Deprecated.
|
static org.apache.hadoop.io.SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
org.apache.hadoop.io.SequenceFile.CompressionType compressionType,
CompressionCodec codec,
Progressable progress,
org.apache.hadoop.io.SequenceFile.Metadata metadata)
Deprecated.
|
static org.apache.hadoop.io.SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
org.apache.hadoop.io.SequenceFile.CompressionType compressionType,
Progressable progress)
Deprecated.
|
static long |
MapFile.fix(FileSystem fs,
Path dir,
Class<? extends Writable> keyClass,
Class<? extends Writable> valueClass,
boolean dryrun,
Configuration conf)
This method attempts to fix a corrupt MapFile by re-creating its index.
|
static WritableComparator |
WritableComparator.get(Class<? extends WritableComparable> c,
Configuration conf)
Get a comparator for a
WritableComparable implementation. |
static org.apache.hadoop.io.SequenceFile.CompressionType |
SequenceFile.getDefaultCompressionType(Configuration job)
Get the compression type for the reduce outputs
|
static <K> K |
DefaultStringifier.load(Configuration conf,
String keyName,
Class<K> itemClass)
Restores the object from the configuration.
|
static <K> K[] |
DefaultStringifier.loadArray(Configuration conf,
String keyName,
Class<K> itemClass)
Restores the array of objects from the configuration.
|
static Class<?> |
ObjectWritable.loadClass(Configuration conf,
String className)
Find and load the class with given name className by first finding
it in the specified conf.
|
static Writable |
WritableFactories.newInstance(Class<? extends Writable> c,
Configuration conf)
Create a new instance of a class with a defined factory.
|
static Object |
ObjectWritable.readObject(DataInput in,
Configuration conf)
|
static Object |
ObjectWritable.readObject(DataInput in,
ObjectWritable objectWritable,
Configuration conf)
|
void |
EnumSetWritable.setConf(Configuration conf) |
void |
WritableComparator.setConf(Configuration conf) |
void |
ObjectWritable.setConf(Configuration conf) |
void |
GenericWritable.setConf(Configuration conf) |
void |
AbstractMapWritable.setConf(Configuration conf) |
static void |
SequenceFile.setDefaultCompressionType(Configuration job,
org.apache.hadoop.io.SequenceFile.CompressionType val)
Set the default compression type for sequence files.
|
static <K> void |
DefaultStringifier.store(Configuration conf,
K item,
String keyName)
Stores the item in the configuration with the given keyName.
|
static <K> void |
DefaultStringifier.storeArray(Configuration conf,
K[] items,
String keyName)
Stores the array of items in the configuration with the given keyName.
|
static void |
ObjectWritable.writeObject(DataOutput out,
Object instance,
Class declaredClass,
Configuration conf)
|
static void |
ObjectWritable.writeObject(DataOutput out,
Object instance,
Class declaredClass,
Configuration conf,
boolean allowCompactArrays)
|
Constructor and Description |
---|
DefaultStringifier(Configuration conf,
Class<T> c) |
WritableComparator(Class<? extends WritableComparable> keyClass,
Configuration conf,
boolean createInstances) |
Modifier and Type | Method and Description |
---|---|
Configuration |
PassthroughCodec.getConf() |
Configuration |
DefaultCodec.getConf() |
Configuration |
BZip2Codec.getConf()
Return the configuration used by this object.
|
Modifier and Type | Method and Description |
---|---|
static List<Class<? extends CompressionCodec>> |
CompressionCodecFactory.getCodecClasses(Configuration conf)
Get the list of codecs discovered via a Java ServiceLoader, or
listed in the configuration.
|
static Compressor |
CodecPool.getCompressor(CompressionCodec codec,
Configuration conf)
Get a
Compressor for the given CompressionCodec from the
pool or a new one. |
void |
Compressor.reinit(Configuration conf)
Prepare the compressor to be used in a new stream with settings defined in
the given Configuration
|
static void |
CompressionCodecFactory.setCodecClasses(Configuration conf,
List<Class> classes)
Sets a list of codec classes in the configuration.
|
void |
PassthroughCodec.setConf(Configuration conf) |
void |
DefaultCodec.setConf(Configuration conf) |
void |
BZip2Codec.setConf(Configuration conf)
Set the configuration to be used by this object.
|
Constructor and Description |
---|
CompressionCodecFactory(Configuration conf)
Find the codecs specified in the config value io.compression.codecs
and register them.
|
Modifier and Type | Method and Description |
---|---|
static void |
Server.bind(ServerSocket socket,
InetSocketAddress address,
int backlog,
Configuration conf,
String rangeConf) |
static int |
Client.getPingInterval(Configuration conf)
Get the ping interval from configuration;
If not set in the configuration, return the default value.
|
static int |
Client.getRpcTimeout(Configuration conf)
The time after which a RPC will timeout.
|
static int |
Client.getTimeout(Configuration conf)
Deprecated.
use
Client.getRpcTimeout(Configuration) instead |
void |
Server.refreshCallQueue(Configuration conf) |
void |
Server.refreshServiceAcl(Configuration conf,
org.apache.hadoop.security.authorize.PolicyProvider provider)
Refresh the service authorization ACL for the service handled by this server.
|
static void |
Client.setConnectTimeout(Configuration conf,
int timeout)
set the connection timeout value in configuration
|
static void |
Client.setPingInterval(Configuration conf,
int pingInterval)
set the ping interval value in configuration
|
Constructor and Description |
---|
Client(Class<? extends Writable> valueClass,
Configuration conf)
Construct an IPC client with the default SocketFactory.
|
Client(Class<? extends Writable> valueClass,
Configuration conf,
SocketFactory factory)
Construct an IPC client whose values are of the given
Writable
class. |
Server(String bindAddress,
int port,
Class<? extends Writable> paramClass,
int handlerCount,
Configuration conf) |
Server(String bindAddress,
int port,
Class<? extends Writable> rpcRequestClass,
int handlerCount,
int numReaders,
int queueSizePerHandler,
Configuration conf,
String serverName,
SecretManager<? extends TokenIdentifier> secretManager) |
Server(String bindAddress,
int port,
Class<? extends Writable> rpcRequestClass,
int handlerCount,
int numReaders,
int queueSizePerHandler,
Configuration conf,
String serverName,
SecretManager<? extends TokenIdentifier> secretManager,
String portRangeConfig)
Constructs a server listening on the named port and address.
|
Modifier and Type | Class and Description |
---|---|
class |
JobConf
A map/reduce job configuration.
|
Modifier and Type | Field and Description |
---|---|
protected Configuration |
SequenceFileRecordReader.conf |
Modifier and Type | Method and Description |
---|---|
Configuration |
RunningJob.getConfiguration()
Get the underlying job configuration
|
Modifier and Type | Method and Description |
---|---|
static int |
SkipBadRecords.getAttemptsToStartSkipping(Configuration conf)
Get the number of Task attempts AFTER which skip mode
will be kicked off.
|
static boolean |
SkipBadRecords.getAutoIncrMapperProcCount(Configuration conf)
Get the flag which if set to true,
SkipBadRecords.COUNTER_MAP_PROCESSED_RECORDS is incremented
by MapRunner after invoking the map function. |
static boolean |
SkipBadRecords.getAutoIncrReducerProcCount(Configuration conf)
Get the flag which if set to true,
SkipBadRecords.COUNTER_REDUCE_PROCESSED_GROUPS is incremented
by framework after invoking the reduce function. |
static long |
SkipBadRecords.getMapperMaxSkipRecords(Configuration conf)
Get the number of acceptable skip records surrounding the bad record PER
bad record in mapper.
|
static org.apache.hadoop.io.SequenceFile.Reader[] |
SequenceFileOutputFormat.getReaders(Configuration conf,
Path dir)
Open the output generated by this format.
|
static org.apache.hadoop.io.MapFile.Reader[] |
MapFileOutputFormat.getReaders(FileSystem ignored,
Path dir,
Configuration conf)
Open the output generated by this format.
|
static int |
FixedLengthInputFormat.getRecordLength(Configuration conf)
Get record length value
|
static long |
SkipBadRecords.getReducerMaxSkipGroups(Configuration conf)
Get the number of acceptable skip groups surrounding the bad group PER
bad group in reducer.
|
static Path |
SkipBadRecords.getSkipOutputPath(Configuration conf)
Get the directory to which skipped records are written.
|
static void |
SkipBadRecords.setAttemptsToStartSkipping(Configuration conf,
int attemptsToStartSkipping)
Set the number of Task attempts AFTER which skip mode
will be kicked off.
|
static void |
SkipBadRecords.setAutoIncrMapperProcCount(Configuration conf,
boolean autoIncr)
Set the flag which if set to true,
SkipBadRecords.COUNTER_MAP_PROCESSED_RECORDS is incremented
by MapRunner after invoking the map function. |
static void |
SkipBadRecords.setAutoIncrReducerProcCount(Configuration conf,
boolean autoIncr)
Set the flag which if set to true,
SkipBadRecords.COUNTER_REDUCE_PROCESSED_GROUPS is incremented
by framework after invoking the reduce function. |
static void |
SequenceFileInputFilter.setFilterClass(Configuration conf,
Class filterClass)
set the filter class
|
static void |
SkipBadRecords.setMapperMaxSkipRecords(Configuration conf,
long maxSkipRecs)
Set the number of acceptable skip records surrounding the bad record PER
bad record in mapper.
|
static void |
FixedLengthInputFormat.setRecordLength(Configuration conf,
int recordLength)
Set the length of each record
|
static void |
SkipBadRecords.setReducerMaxSkipGroups(Configuration conf,
long maxSkipGrps)
Set the number of acceptable skip groups surrounding the bad group PER
bad group in reducer.
|
Constructor and Description |
---|
JobClient(Configuration conf)
Build a job client with the given
Configuration ,
and connect to the default cluster |
JobClient(InetSocketAddress jobTrackAddr,
Configuration conf)
Build a job client, connect to the indicated job tracker.
|
JobConf(Configuration conf)
Construct a map/reduce job configuration.
|
JobConf(Configuration conf,
Class exampleClass)
Construct a map/reduce job configuration.
|
KeyValueLineRecordReader(Configuration job,
FileSplit split) |
SequenceFileAsTextRecordReader(Configuration conf,
FileSplit split) |
SequenceFileRecordReader(Configuration conf,
FileSplit split) |
Modifier and Type | Method and Description |
---|---|
Configuration |
CompositeRecordReader.getConf()
Return the configuration used by this object.
|
Configuration |
WrappedRecordReader.getConf() |
Modifier and Type | Method and Description |
---|---|
void |
CompositeRecordReader.setConf(Configuration conf)
Set the configuration to be used by this object.
|
void |
WrappedRecordReader.setConf(Configuration conf) |
Constructor and Description |
---|
CombineFileRecordReaderWrapper(FileInputFormat<K,V> inputFormat,
CombineFileSplit split,
Configuration conf,
Reporter reporter,
Integer idx) |
Constructor and Description |
---|
Submitter(Configuration conf) |
Modifier and Type | Method and Description |
---|---|
Configuration |
JobContext.getConfiguration()
Return the configuration for the job.
|
Modifier and Type | Method and Description |
---|---|
static void |
Job.addArchiveToClassPath(Path archive,
Configuration conf,
FileSystem fs)
Add an archive path to the current set of classpath entries.
|
static boolean |
Job.addArchiveToSharedCache(URI resource,
Configuration conf)
Add an archive to job config for shared cache processing.
|
static void |
Job.addCacheArchive(URI uri,
Configuration conf)
Add an archives to be localized to the conf.
|
static void |
Job.addCacheFile(URI uri,
Configuration conf)
Add a file to be localized to the conf.
|
static void |
Job.addFileToClassPath(Path file,
Configuration conf,
FileSystem fs)
Add a file path to the current set of classpath entries.
|
static void |
Job.addFileToClassPath(Path file,
Configuration conf,
FileSystem fs,
boolean addToCache)
Add a file path to the current set of classpath entries.
|
static boolean |
Job.addFileToSharedCache(URI resource,
Configuration conf)
Add a file to job config for shared cache processing.
|
static boolean |
Job.addFileToSharedCacheAndClasspath(URI resource,
Configuration conf)
Add a file to job config for shared cache processing.
|
static Map<String,Boolean> |
Job.getArchiveSharedCacheUploadPolicies(Configuration conf)
This is to get the shared cache upload policies for archives.
|
static int |
Job.getCompletionPollInterval(Configuration conf)
The interval at which waitForCompletion() should check.
|
static Map<String,Boolean> |
Job.getFileSharedCacheUploadPolicies(Configuration conf)
This is to get the shared cache upload policies for files.
|
static Job |
Job.getInstance(Cluster ignored,
Configuration conf)
Deprecated.
|
static Job |
Job.getInstance(Configuration conf)
|
static Job |
Job.getInstance(Configuration conf,
String jobName)
|
static Job |
Job.getInstance(JobStatus status,
Configuration conf)
|
static int |
Job.getProgressPollInterval(Configuration conf)
The interval at which monitorAndPrintJob() prints status
|
static org.apache.hadoop.mapreduce.Job.TaskStatusFilter |
Job.getTaskOutputFilter(Configuration conf)
Get the task output filter.
|
static void |
Job.setArchiveSharedCacheUploadPolicies(Configuration conf,
Map<String,Boolean> policies)
This is to set the shared cache upload policies for archives.
|
static void |
Job.setCacheArchives(URI[] archives,
Configuration conf)
Set the configuration with the given set of archives.
|
static void |
Job.setCacheFiles(URI[] files,
Configuration conf)
Set the configuration with the given set of files.
|
static void |
Job.setFileSharedCacheUploadPolicies(Configuration conf,
Map<String,Boolean> policies)
This is to set the shared cache upload policies for files.
|
static void |
Job.setTaskOutputFilter(Configuration conf,
org.apache.hadoop.mapreduce.Job.TaskStatusFilter newValue)
Modify the Configuration to set the task output filter.
|
Constructor and Description |
---|
Cluster(Configuration conf) |
Cluster(InetSocketAddress jobTrackAddr,
Configuration conf) |
Job(Configuration conf)
Deprecated.
|
Job(Configuration conf,
String jobName)
Deprecated.
|
Modifier and Type | Method and Description |
---|---|
static Configuration |
ValueAggregatorJob.setAggregatorDescriptors(Class<? extends ValueAggregatorDescriptor>[] descriptors) |
Modifier and Type | Method and Description |
---|---|
void |
UserDefinedValueAggregatorDescriptor.configure(Configuration conf)
Do nothing.
|
void |
ValueAggregatorDescriptor.configure(Configuration conf)
Configure the object
|
void |
ValueAggregatorBaseDescriptor.configure(Configuration conf)
get the input file name.
|
static Job |
ValueAggregatorJob.createValueAggregatorJob(Configuration conf,
String[] args)
Create an Aggregate based map/reduce job.
|
protected static ArrayList<ValueAggregatorDescriptor> |
ValueAggregatorJobBase.getAggregatorDescriptors(Configuration conf) |
protected static ValueAggregatorDescriptor |
ValueAggregatorJobBase.getValueAggregatorDescriptor(String spec,
Configuration conf) |
static void |
ValueAggregatorJobBase.setup(Configuration job) |
Constructor and Description |
---|
UserDefinedValueAggregatorDescriptor(String className,
Configuration conf) |
Modifier and Type | Method and Description |
---|---|
static void |
ChainReducer.addMapper(Job job,
Class<? extends Mapper> klass,
Class<?> inputKeyClass,
Class<?> inputValueClass,
Class<?> outputKeyClass,
Class<?> outputValueClass,
Configuration mapperConf)
Adds a
Mapper class to the chain reducer. |
static void |
ChainMapper.addMapper(Job job,
Class<? extends Mapper> klass,
Class<?> inputKeyClass,
Class<?> inputValueClass,
Class<?> outputKeyClass,
Class<?> outputValueClass,
Configuration mapperConf)
Adds a
Mapper class to the chain mapper. |
static void |
ChainReducer.setReducer(Job job,
Class<? extends Reducer> klass,
Class<?> inputKeyClass,
Class<?> inputValueClass,
Class<?> outputKeyClass,
Class<?> outputValueClass,
Configuration reducerConf)
Sets the
Reducer class to the chain job. |
Modifier and Type | Method and Description |
---|---|
Configuration |
DBConfiguration.getConf() |
Configuration |
DBInputFormat.getConf() |
Modifier and Type | Method and Description |
---|---|
static void |
DBConfiguration.configureDB(Configuration job,
String driverClass,
String dbUrl)
Sets the DB access related fields in the JobConf.
|
static void |
DBConfiguration.configureDB(Configuration conf,
String driverClass,
String dbUrl,
String userName,
String passwd)
Sets the DB access related fields in the
Configuration . |
protected RecordReader<LongWritable,T> |
OracleDataDrivenDBInputFormat.createDBRecordReader(org.apache.hadoop.mapreduce.lib.db.DBInputFormat.DBInputSplit split,
Configuration conf) |
protected RecordReader<LongWritable,T> |
DataDrivenDBInputFormat.createDBRecordReader(org.apache.hadoop.mapreduce.lib.db.DBInputFormat.DBInputSplit split,
Configuration conf) |
protected RecordReader<LongWritable,T> |
DBInputFormat.createDBRecordReader(org.apache.hadoop.mapreduce.lib.db.DBInputFormat.DBInputSplit split,
Configuration conf) |
static void |
DataDrivenDBInputFormat.setBoundingQuery(Configuration conf,
String query)
Set the user-defined bounding query to use with a user-defined query.
|
void |
DBInputFormat.setConf(Configuration conf)
Set the configuration to be used by this object.
|
static void |
OracleDBRecordReader.setSessionTimeZone(Configuration conf,
Connection conn)
Set session time zone
|
List<InputSplit> |
TextSplitter.split(Configuration conf,
ResultSet results,
String colName)
This method needs to determine the splits between two user-provided strings.
|
List<InputSplit> |
BooleanSplitter.split(Configuration conf,
ResultSet results,
String colName) |
List<InputSplit> |
FloatSplitter.split(Configuration conf,
ResultSet results,
String colName) |
List<InputSplit> |
BigDecimalSplitter.split(Configuration conf,
ResultSet results,
String colName) |
List<InputSplit> |
DateSplitter.split(Configuration conf,
ResultSet results,
String colName) |
List<InputSplit> |
DBSplitter.split(Configuration conf,
ResultSet results,
String colName)
Given a ResultSet containing one record (and already advanced to that record)
with two columns (a low value, and a high value, both of the same type), determine
a set of splits that span the given values.
|
List<InputSplit> |
IntegerSplitter.split(Configuration conf,
ResultSet results,
String colName) |
Constructor and Description |
---|
DataDrivenDBRecordReader(org.apache.hadoop.mapreduce.lib.db.DBInputFormat.DBInputSplit split,
Class<T> inputClass,
Configuration conf,
Connection conn,
DBConfiguration dbConfig,
String cond,
String[] fields,
String table,
String dbProduct) |
DBConfiguration(Configuration job) |
DBRecordReader(org.apache.hadoop.mapreduce.lib.db.DBInputFormat.DBInputSplit split,
Class<T> inputClass,
Configuration conf,
Connection conn,
DBConfiguration dbConfig,
String cond,
String[] fields,
String table) |
MySQLDataDrivenDBRecordReader(org.apache.hadoop.mapreduce.lib.db.DBInputFormat.DBInputSplit split,
Class<T> inputClass,
Configuration conf,
Connection conn,
DBConfiguration dbConfig,
String cond,
String[] fields,
String table) |
MySQLDBRecordReader(org.apache.hadoop.mapreduce.lib.db.DBInputFormat.DBInputSplit split,
Class<T> inputClass,
Configuration conf,
Connection conn,
DBConfiguration dbConfig,
String cond,
String[] fields,
String table) |
OracleDataDrivenDBRecordReader(org.apache.hadoop.mapreduce.lib.db.DBInputFormat.DBInputSplit split,
Class<T> inputClass,
Configuration conf,
Connection conn,
DBConfiguration dbConfig,
String cond,
String[] fields,
String table) |
OracleDBRecordReader(org.apache.hadoop.mapreduce.lib.db.DBInputFormat.DBInputSplit split,
Class<T> inputClass,
Configuration conf,
Connection conn,
DBConfiguration dbConfig,
String cond,
String[] fields,
String table) |
Modifier and Type | Field and Description |
---|---|
protected Configuration |
SequenceFileRecordReader.conf |
Modifier and Type | Method and Description |
---|---|
static int |
FixedLengthInputFormat.getRecordLength(Configuration conf)
Get record length value
|
static List<FileSplit> |
NLineInputFormat.getSplitsForFile(FileStatus status,
Configuration conf,
int numLinesPerSplit) |
static void |
FixedLengthInputFormat.setRecordLength(Configuration conf,
int recordLength)
Set the length of each record
|
Constructor and Description |
---|
KeyValueLineRecordReader(Configuration conf) |
Constructor and Description |
---|
ControlledJob(Configuration conf)
Construct a job.
|
Modifier and Type | Field and Description |
---|---|
protected Configuration |
CompositeRecordReader.conf |
Modifier and Type | Method and Description |
---|---|
Configuration |
CompositeRecordReader.getConf()
Return the configuration used by this object.
|
Modifier and Type | Method and Description |
---|---|
void |
CompositeRecordReader.setConf(Configuration conf)
Set the configuration to be used by this object.
|
void |
CompositeInputFormat.setFormat(Configuration conf)
Interpret a given string as a composite expression.
|
Constructor and Description |
---|
JoinRecordReader(int id,
Configuration conf,
int capacity,
Class<? extends WritableComparator> cmpcl) |
MultiFilterRecordReader(int id,
Configuration conf,
int capacity,
Class<? extends WritableComparator> cmpcl) |
Modifier and Type | Method and Description |
---|---|
static PathOutputCommitterFactory |
PathOutputCommitterFactory.getCommitterFactory(Path outputPath,
Configuration conf)
Get the committer factory for a configuration.
|
static org.apache.hadoop.io.MapFile.Reader[] |
MapFileOutputFormat.getReaders(Path dir,
Configuration conf)
Open the output generated by this format.
|
Modifier and Type | Method and Description |
---|---|
Configuration |
ManifestCommitter.getConf()
Get the config of the task attempt this instance was constructed
with.
|
Modifier and Type | Method and Description |
---|---|
Configuration |
BinaryPartitioner.getConf() |
Configuration |
KeyFieldBasedPartitioner.getConf() |
Configuration |
TotalOrderPartitioner.getConf() |
Configuration |
KeyFieldBasedComparator.getConf() |
Modifier and Type | Method and Description |
---|---|
static String |
TotalOrderPartitioner.getPartitionFile(Configuration conf)
Get the path to the SequenceFile storing the sorted partition keyset.
|
void |
BinaryPartitioner.setConf(Configuration conf) |
void |
KeyFieldBasedPartitioner.setConf(Configuration conf) |
void |
TotalOrderPartitioner.setConf(Configuration conf)
Read in the partition file and build indexing data structures.
|
void |
KeyFieldBasedComparator.setConf(Configuration conf) |
static void |
BinaryPartitioner.setLeftOffset(Configuration conf,
int offset)
Set the subarray to be used for partitioning to
bytes[offset:] in Python syntax. |
static void |
BinaryPartitioner.setOffsets(Configuration conf,
int left,
int right)
Set the subarray to be used for partitioning to
bytes[left:(right+1)] in Python syntax. |
static void |
TotalOrderPartitioner.setPartitionFile(Configuration conf,
Path p)
Set the path to the SequenceFile storing the sorted partition keyset.
|
static void |
BinaryPartitioner.setRightOffset(Configuration conf,
int offset)
Set the subarray to be used for partitioning to
bytes[:(offset+1)] in Python syntax. |
Constructor and Description |
---|
InputSampler(Configuration conf) |
Modifier and Type | Method and Description |
---|---|
static void |
TokenCache.cleanUpTokenReferral(Configuration conf)
Remove jobtoken referrals which don't make sense in the context
of the task execution.
|
static void |
TokenCache.obtainTokensForNamenodes(Credentials credentials,
Path[] ps,
Configuration conf)
Convenience method to obtain delegation tokens from namenodes
corresponding to the paths passed.
|
Constructor and Description |
---|
CLI(Configuration conf) |
Modifier and Type | Method and Description |
---|---|
protected void |
HistoryFileManager.serviceInit(Configuration conf) |
Modifier and Type | Field and Description |
---|---|
protected static Configuration |
RollingFileSystemSink.suppliedConf |
Modifier and Type | Method and Description |
---|---|
Configuration |
TableMapping.getConf() |
Configuration |
AbstractDNSToSwitchMapping.getConf() |
Configuration |
SocksSocketFactory.getConf() |
Configuration |
ScriptBasedMapping.getConf() |
Modifier and Type | Method and Description |
---|---|
void |
TableMapping.setConf(Configuration conf) |
void |
AbstractDNSToSwitchMapping.setConf(Configuration conf) |
void |
SocksSocketFactory.setConf(Configuration conf) |
void |
ScriptBasedMapping.setConf(Configuration conf)
Set the configuration to be used by this object.
|
Constructor and Description |
---|
AbstractDNSToSwitchMapping(Configuration conf)
Create an instance, caching the configuration file.
|
ScriptBasedMapping(Configuration conf)
Create an instance from the given configuration
|
Modifier and Type | Method and Description |
---|---|
static UserGroupInformation.AuthenticationMethod |
SecurityUtil.getAuthenticationMethod(Configuration conf) |
static String |
SecurityUtil.getClientPrincipal(Class<?> protocol,
Configuration conf)
Look up the client principal for a given protocol.
|
static org.apache.hadoop.security.KerberosInfo |
SecurityUtil.getKerberosInfo(Class<?> protocol,
Configuration conf)
Look up the KerberosInfo for a given protocol.
|
static TokenInfo |
SecurityUtil.getTokenInfo(Class<?> protocol,
Configuration conf)
Look up the TokenInfo for a given protocol.
|
static List<org.apache.hadoop.util.ZKUtil.ZKAuthInfo> |
SecurityUtil.getZKAuthInfos(Configuration conf,
String configKey)
Utility method to fetch ZK auth info from the configuration.
|
static void |
SecurityUtil.login(Configuration conf,
String keytabFileKey,
String userNameKey)
Login as a principal specified in config.
|
static void |
SecurityUtil.login(Configuration conf,
String keytabFileKey,
String userNameKey,
String hostname)
Login as a principal specified in config.
|
static Credentials |
Credentials.readTokenStorageFile(File filename,
Configuration conf)
Convenience method for reading a token storage file and loading its Tokens.
|
static Credentials |
Credentials.readTokenStorageFile(Path filename,
Configuration conf)
Convenience method for reading a token storage file and loading its Tokens.
|
static void |
SecurityUtil.setAuthenticationMethod(UserGroupInformation.AuthenticationMethod authenticationMethod,
Configuration conf) |
static void |
UserGroupInformation.setConfiguration(Configuration conf)
Set the static configuration for UGI.
|
static void |
SecurityUtil.setConfiguration(Configuration conf) |
void |
Credentials.writeTokenStorageFile(Path filename,
Configuration conf) |
void |
Credentials.writeTokenStorageFile(Path filename,
Configuration conf,
org.apache.hadoop.security.Credentials.SerializedFormat format) |
Modifier and Type | Method and Description |
---|---|
abstract CredentialProvider |
CredentialProviderFactory.createProvider(URI providerName,
Configuration conf) |
static List<CredentialProvider> |
CredentialProviderFactory.getProviders(Configuration conf) |
Modifier and Type | Method and Description |
---|---|
Configuration |
DefaultImpersonationProvider.getConf() |
Modifier and Type | Method and Description |
---|---|
void |
DefaultImpersonationProvider.setConf(Configuration conf) |
Modifier and Type | Method and Description |
---|---|
static Map<String,String> |
RestCsrfPreventionFilter.getFilterParams(Configuration conf,
String confPrefix)
Constructs a mapping of configuration properties to be used for filter
initialization.
|
static Map<String,String> |
XFrameOptionsFilter.getFilterParams(Configuration conf,
String confPrefix)
Constructs a mapping of configuration properties to be used for filter
initialization.
|
Modifier and Type | Method and Description |
---|---|
void |
Token.cancel(Configuration conf)
Cancel this delegation token.
|
void |
Token.TrivialRenewer.cancel(Token<?> token,
Configuration conf) |
abstract void |
TokenRenewer.cancel(Token<?> token,
Configuration conf)
Cancel the given token.
|
long |
Token.renew(Configuration conf)
Renew this delegation token.
|
long |
Token.TrivialRenewer.renew(Token<?> token,
Configuration conf) |
abstract long |
TokenRenewer.renew(Token<?> token,
Configuration conf)
Renew the given token.
|
Modifier and Type | Method and Description |
---|---|
Configuration |
Service.getConfig()
Get the configuration of this service.
|
Configuration |
AbstractService.getConfig() |
Modifier and Type | Method and Description |
---|---|
void |
Service.init(Configuration config)
Initialize the service.
|
void |
AbstractService.init(Configuration conf)
Initialize the service.
|
protected void |
CompositeService.serviceInit(Configuration conf) |
protected void |
AbstractService.serviceInit(Configuration conf)
All initialization code needed by a service.
|
protected void |
AbstractService.setConfig(Configuration conf)
Set the configuration for this service.
|
Modifier and Type | Method and Description |
---|---|
Configuration |
LaunchableService.bindArgs(Configuration config,
List<String> args)
Propagate the command line arguments.
|
Configuration |
AbstractLaunchableService.bindArgs(Configuration config,
List<String> args)
Propagate the command line arguments.
|
Modifier and Type | Method and Description |
---|---|
Configuration |
LaunchableService.bindArgs(Configuration config,
List<String> args)
Propagate the command line arguments.
|
Configuration |
AbstractLaunchableService.bindArgs(Configuration config,
List<String> args)
Propagate the command line arguments.
|
Modifier and Type | Method and Description |
---|---|
void |
DistCpOptions.appendToConf(Configuration conf)
Add options to configuration.
|
Constructor and Description |
---|
DistCp(Configuration configuration,
DistCpOptions inputOptions)
Public Constructor.
|
Modifier and Type | Method and Description |
---|---|
static <T> T |
ReflectionUtils.copy(Configuration conf,
T src,
T dst)
Make a copy of the writable object using serialization to a buffer.
|
static <T> T |
ReflectionUtils.newInstance(Class<T> theClass,
Configuration conf)
Create an object for the given class and initialize it from conf
|
static <T> T |
ReflectionUtils.newInstance(Class<T> theClass,
Configuration conf,
Class<?>[] argTypes,
Object... values)
Create an object for the given class and initialize it from conf
|
static int |
ToolRunner.run(Configuration conf,
Tool tool,
String[] args)
Runs the given
Tool by Tool.run(String[]) , after
parsing with the given generic arguments. |
static void |
ReflectionUtils.setConf(Object theObject,
Configuration conf)
Check and set 'configuration' if necessary.
|
Modifier and Type | Method and Description |
---|---|
static void |
FutureIO.propagateOptions(FSBuilder<?,?> builder,
Configuration conf,
String prefix,
boolean mandatory)
Propagate options to any builder, converting everything with the
prefix to an option where, if there were 2+ dot-separated elements,
it is converted to a schema.
|
static <T,U extends FSBuilder<T,U>> |
FutureIO.propagateOptions(FSBuilder<T,U> builder,
Configuration conf,
String optionalPrefix,
String mandatoryPrefix)
Propagate options to any builder, converting everything with the
prefix to an option where, if there were 2+ dot-separated elements,
it is converted to a schema.
|
Constructor and Description |
---|
Client(Configuration conf) |
Modifier and Type | Method and Description |
---|---|
static <T> T |
AHSProxy.createAHSProxy(Configuration conf,
Class<T> protocol,
InetSocketAddress ahsAddress) |
static <T> T |
NMProxy.createNMProxy(Configuration conf,
Class<T> protocol,
UserGroupInformation ugi,
org.apache.hadoop.yarn.ipc.YarnRPC rpc,
InetSocketAddress serverAddress) |
protected static <T> T |
ServerProxy.createRetriableProxy(Configuration conf,
Class<T> protocol,
UserGroupInformation user,
org.apache.hadoop.yarn.ipc.YarnRPC rpc,
InetSocketAddress serverAddress,
org.apache.hadoop.io.retry.RetryPolicy retryPolicy) |
protected static org.apache.hadoop.io.retry.RetryPolicy |
RMProxy.createRetryPolicy(Configuration conf,
long retryTime,
long retryInterval,
boolean isHAEnabled)
Fetch retry policy from Configuration and create the
retry policy with specified retryTime and retry interval.
|
protected static org.apache.hadoop.io.retry.RetryPolicy |
ServerProxy.createRetryPolicy(Configuration conf,
String maxWaitTimeStr,
long defMaxWaitTime,
String connectRetryIntervalStr,
long defRetryInterval) |
static <T> T |
ClientRMProxy.createRMProxy(Configuration configuration,
Class<T> protocol)
Create a proxy to the ResourceManager for the specified protocol.
|
static <T> T |
ClientRMProxy.createRMProxyFederation(Configuration configuration,
Class<T> protocol)
Create a proxy to the ResourceManager for the specified protocol.
|
protected static <T> T |
RMProxy.createRMProxyFederation(Configuration configuration,
Class<T> protocol,
RMProxy<T> instance)
This functionality is only used for NodeManager and only in non-HA mode.
|
static Text |
ClientRMProxy.getAMRMTokenService(Configuration conf) |
protected static <T> T |
AHSProxy.getProxy(Configuration conf,
Class<T> protocol,
InetSocketAddress rmAddress) |
static Text |
ClientRMProxy.getRMDelegationTokenService(Configuration conf)
Get the token service name to be used for RMDelegationToken.
|
static Text |
ClientRMProxy.getTokenService(Configuration conf,
String address,
String defaultAddr,
int defaultPort) |
Modifier and Type | Method and Description |
---|---|
static AppAdminClient |
AppAdminClient.createAppAdminClient(String appType,
Configuration conf)
Create a new instance of AppAdminClient.
|
Modifier and Type | Method and Description |
---|---|
protected void |
TimelineReaderClientImpl.serviceInit(Configuration conf) |
Modifier and Type | Method and Description |
---|---|
protected List<org.codehaus.jettison.json.JSONObject> |
LogsCLI.getAMContainerInfoForRMWebService(Configuration conf,
String appId) |
protected com.sun.jersey.api.client.ClientResponse |
LogsCLI.getClientResponseFromTimelineReader(Configuration conf,
String appId) |
com.sun.jersey.api.client.ClientResponse |
LogsCLI.getResponseFromNMWebService(Configuration conf,
com.sun.jersey.api.client.Client webServiceClient,
org.apache.hadoop.yarn.logaggregation.ContainerLogsRequest request,
String logFile) |
Modifier and Type | Class and Description |
---|---|
class |
YarnConfiguration |
Modifier and Type | Method and Description |
---|---|
static long |
YarnConfiguration.getSkipNodeInterval(Configuration conf)
Returns Timeout to skip node from scheduling if not heartbeated.
|
static float |
YarnConfiguration.getTimelineServiceVersion(Configuration conf)
Returns the timeline service version.
|
static boolean |
YarnConfiguration.isAclEnabled(Configuration conf) |
static boolean |
YarnConfiguration.isDistSchedulingEnabled(Configuration conf) |
static boolean |
YarnConfiguration.isOpportunisticContainerAllocationEnabled(Configuration conf) |
static boolean |
YarnConfiguration.numaAwarenessEnabled(Configuration conf)
Returns whether the NUMA awareness is enabled.
|
static boolean |
YarnConfiguration.shouldRMFailFast(Configuration conf) |
static boolean |
YarnConfiguration.systemMetricsPublisherEnabled(Configuration conf)
Returns whether the system publisher is enabled.
|
static boolean |
YarnConfiguration.timelineServiceEnabled(Configuration conf)
Returns whether the timeline service is enabled via configuration.
|
static boolean |
YarnConfiguration.timelineServiceV15Enabled(Configuration conf)
Returns whether the timeline service v.1,5 is enabled via configuration.
|
static boolean |
YarnConfiguration.timelineServiceV1Enabled(Configuration conf)
Returns whether the timeline service v.1 is enabled via configuration.
|
static boolean |
YarnConfiguration.timelineServiceV2Enabled(Configuration conf)
Returns whether the timeline service v.2 is enabled via configuration.
|
static boolean |
YarnConfiguration.useHttps(Configuration conf) |
Constructor and Description |
---|
YarnConfiguration(Configuration conf) |
Modifier and Type | Method and Description |
---|---|
protected void |
AsyncDispatcher.serviceInit(Configuration conf) |
Constructor and Description |
---|
LogReader(Configuration conf,
Path remoteAppLogFile) |
Modifier and Type | Field and Description |
---|---|
protected Configuration |
LogAggregationFileController.conf |
Modifier and Type | Method and Description |
---|---|
protected FileSystem |
LogAggregationFileController.getFileSystem(Configuration conf) |
void |
LogAggregationFileController.initialize(Configuration conf,
String controllerName)
Initialize the log file controller.
|
protected abstract void |
LogAggregationFileController.initInternal(Configuration conf)
Derived classes initialize themselves using this method.
|
Modifier and Type | Method and Description |
---|---|
org.apache.hadoop.security.KerberosInfo |
SchedulerSecurityInfo.getKerberosInfo(Class<?> protocol,
Configuration conf) |
org.apache.hadoop.security.KerberosInfo |
ContainerManagerSecurityInfo.getKerberosInfo(Class<?> protocol,
Configuration conf) |
TokenInfo |
SchedulerSecurityInfo.getTokenInfo(Class<?> protocol,
Configuration conf) |
TokenInfo |
ContainerManagerSecurityInfo.getTokenInfo(Class<?> protocol,
Configuration conf) |
Modifier and Type | Method and Description |
---|---|
org.apache.hadoop.security.KerberosInfo |
AdminSecurityInfo.getKerberosInfo(Class<?> protocol,
Configuration conf) |
TokenInfo |
AdminSecurityInfo.getTokenInfo(Class<?> protocol,
Configuration conf) |
Modifier and Type | Method and Description |
---|---|
org.apache.hadoop.security.KerberosInfo |
ClientTimelineSecurityInfo.getKerberosInfo(Class<?> protocol,
Configuration conf) |
org.apache.hadoop.security.KerberosInfo |
ClientRMSecurityInfo.getKerberosInfo(Class<?> protocol,
Configuration conf) |
TokenInfo |
ClientTimelineSecurityInfo.getTokenInfo(Class<?> protocol,
Configuration conf) |
TokenInfo |
ClientRMSecurityInfo.getTokenInfo(Class<?> protocol,
Configuration conf) |
Modifier and Type | Method and Description |
---|---|
protected void |
ServiceClient.serviceInit(Configuration configuration) |
Modifier and Type | Method and Description |
---|---|
static SharedCacheChecksum |
SharedCacheChecksumFactory.getChecksum(Configuration conf)
Get a new
SharedCacheChecksum object based on the configurable
algorithm implementation
(see yarn.sharedcache.checksum.algo.impl ). |
Modifier and Type | Method and Description |
---|---|
static ResourceCalculatorProcessTree |
ResourceCalculatorProcessTree.getResourceCalculatorProcessTree(String pid,
Class<? extends ResourceCalculatorProcessTree> clazz,
Configuration conf)
Create the ResourceCalculatorProcessTree rooted to specified process
from the class name and configure it.
|
Modifier and Type | Method and Description |
---|---|
static Text |
TimelineUtils.buildTimelineTokenService(Configuration conf) |
static float |
TimelineUtils.getTimelineServiceVersion(Configuration conf)
Returns the timeline service version.
|
static InetSocketAddress |
TimelineUtils.getTimelineTokenServiceAddress(Configuration conf) |
static String |
TimelineUtils.shortenFlowName(String flowName,
Configuration conf)
Shortens the flow name for the configured size by removing UUID if present.
|
static boolean |
TimelineUtils.timelineServiceEnabled(Configuration conf)
Returns whether the timeline service is enabled via configuration.
|
static boolean |
TimelineUtils.timelineServiceV1_5Enabled(Configuration conf)
Returns whether the timeline service v.1.5 is enabled by default via
configuration.
|
Copyright © 2023 Apache Software Foundation. All rights reserved.