org.apache.hadoop.mapreduce.v2.hs

Class HistoryFileManager

java.lang.Object

org.apache.hadoop.service.AbstractService

org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager

All Implemented Interfaces:

Closeable, AutoCloseable, Service

@InterfaceAudience.Public
 @InterfaceStability.Unstable
public class HistoryFileManager
extends AbstractService

This class provides a way to interact with history files in a thread safe manor.

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.JobListCache	jobListCache
protected ThreadPoolExecutor	moveToDoneExecutor

Constructor Summary

Constructors

Constructor and Description
HistoryFileManager()

Method Summary

Modifier and Type	Method and Description
protected org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.HistoryFileInfo	createHistoryFileInfo(Path historyFile, Path confFile, Path summaryFile, org.apache.hadoop.mapreduce.v2.jobhistory.JobIndexInfo jobIndexInfo, boolean isInDone)
protected org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.JobListCache	createJobListCache()
protected ThreadPoolExecutor	createMoveToDoneThreadPool(int numMoveThreads)
protected boolean	deleteDir(FileStatus serialDir)
protected List<FileStatus>	findTimestampedDirectories() Finds all history directories with a timestamp component by scanning the filesystem.
Collection<org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.HistoryFileInfo>	getAllFileInfo()
org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.HistoryFileInfo	getFileInfo(org.apache.hadoop.mapreduce.v2.api.records.JobId jobId)
protected static List<FileStatus>	scanDirectory(Path path, FileContext fc, PathFilter pathFilter)
protected List<FileStatus>	scanDirectoryForHistoryFiles(Path path, FileContext fc)
protected void	serviceInit(Configuration conf) All initialization code needed by a service.
void	serviceStop() Actions called during the transition to the STOPPED state.

Methods inherited from class org.apache.hadoop.service.AbstractService

close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, noteFailure, putBlocker, registerGlobalListener, registerServiceListener, removeBlocker, serviceStart, setConfig, start, stop, toString, unregisterGlobalListener, unregisterServiceListener, waitForServiceToStop

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

jobListCache

protected org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.JobListCache jobListCache

moveToDoneExecutor

@VisibleForTesting
protected ThreadPoolExecutor moveToDoneExecutor

Constructor Detail

HistoryFileManager

public HistoryFileManager()

Method Detail

serviceInit

protected void serviceInit(Configuration conf)
                    throws Exception

Description copied from class: AbstractService

All initialization code needed by a service. This method will only ever be called once during the lifecycle of a specific service instance. Implementations do not need to be synchronized as the logic in AbstractService.init(Configuration) prevents re-entrancy. The base implementation checks to see if the subclass has created a new configuration instance, and if so, updates the base class value

Overrides:

serviceInit in class AbstractService

Parameters:

conf - configuration

Throws:

Exception - on a failure -these will be caught, possibly wrapped, and will trigger a service stop

createMoveToDoneThreadPool

protected ThreadPoolExecutor createMoveToDoneThreadPool(int numMoveThreads)

serviceStop

public void serviceStop()
                 throws Exception

Description copied from class: AbstractService

Actions called during the transition to the STOPPED state. This method will only ever be called once during the lifecycle of a specific service instance. Implementations do not need to be synchronized as the logic in AbstractService.stop() prevents re-entrancy. Implementations MUST write this to be robust against failures, including checks for null references -and for the first failure to not stop other attempts to shut down parts of the service.

Overrides:

serviceStop in class AbstractService

Throws:

Exception - if needed -these will be caught and logged.

createJobListCache

protected org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.JobListCache createJobListCache()

createHistoryFileInfo

protected org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.HistoryFileInfo createHistoryFileInfo(Path historyFile,
                                                                                                     Path confFile,
                                                                                                     Path summaryFile,
                                                                                                     org.apache.hadoop.mapreduce.v2.jobhistory.JobIndexInfo jobIndexInfo,
                                                                                                     boolean isInDone)

scanDirectory

@VisibleForTesting
protected static List<FileStatus> scanDirectory(Path path,
                                                                   FileContext fc,
                                                                   PathFilter pathFilter)
                                                            throws IOException

Throws:

IOException

scanDirectoryForHistoryFiles

protected List<FileStatus> scanDirectoryForHistoryFiles(Path path,
                                                        FileContext fc)
                                                 throws IOException

Throws:

IOException

findTimestampedDirectories

protected List<FileStatus> findTimestampedDirectories()
                                               throws IOException

Finds all history directories with a timestamp component by scanning the filesystem. Used when the JobHistory server is started.

Returns:

list of history directories

Throws:

IOException

getAllFileInfo

public Collection<org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.HistoryFileInfo> getAllFileInfo()
                                                                                                throws IOException

Throws:

IOException

getFileInfo

public org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.HistoryFileInfo getFileInfo(org.apache.hadoop.mapreduce.v2.api.records.JobId jobId)
                                                                                 throws IOException

Throws:

IOException

deleteDir

protected boolean deleteDir(FileStatus serialDir)
                     throws AccessControlException,
                            FileNotFoundException,
                            UnsupportedFileSystemException,
                            IOException

Throws:

AccessControlException

FileNotFoundException

UnsupportedFileSystemException

IOException