org.apache.flume.sink.hdfs

Class AbstractHDFSWriter

java.lang.Object

org.apache.flume.sink.hdfs.AbstractHDFSWriter

All Implemented Interfaces:

Configurable, HDFSWriter

Direct Known Subclasses:

HDFSCompressedDataStream, HDFSDataStream, HDFSSequenceFile

@InterfaceAudience.Private
 @InterfaceStability.Evolving
public abstract class AbstractHDFSWriter
extends Object
implements HDFSWriter

Constructor Summary

Constructors

Constructor and Description
AbstractHDFSWriter()

Method Summary

Modifier and Type	Method and Description
void	configure(Context context) Request the implementing class to (re)configure itself.
int	getFsDesiredReplication()
int	getNumCurrentReplicas() This method gets the datanode replication count for the current open file.
protected void	hflushOrSync(org.apache.hadoop.fs.FSDataOutputStream os) If hflush is available in this version of HDFS, then this method calls hflush, else it calls sync.
protected boolean	isFileAppendable(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem hdfs, org.apache.hadoop.fs.Path dstPath)
boolean	isUnderReplicated() Contract for subclasses: Call registerCurrentStream() on open, unregisterCurrentStream() on close, and the base class takes care of the rest.
protected void	registerCurrentStream(org.apache.hadoop.fs.FSDataOutputStream outputStream, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path destPath)
protected void	unregisterCurrentStream()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flume.sink.hdfs.HDFSWriter

append, close, open, open, sync

Constructor Detail

AbstractHDFSWriter

public AbstractHDFSWriter()

Method Detail

configure

public void configure(Context context)

Description copied from interface: Configurable

Request the implementing class to (re)configure itself.

When configuration parameters are changed, they must be reflected by the component asap.

There are no thread safety guarantees on when configure might be called.

Specified by:

configure in interface Configurable

isUnderReplicated

public boolean isUnderReplicated()

Contract for subclasses: Call registerCurrentStream() on open, unregisterCurrentStream() on close, and the base class takes care of the rest.

Specified by:

isUnderReplicated in interface HDFSWriter

Returns:

registerCurrentStream

protected void registerCurrentStream(org.apache.hadoop.fs.FSDataOutputStream outputStream,
                                     org.apache.hadoop.fs.FileSystem fs,
                                     org.apache.hadoop.fs.Path destPath)

unregisterCurrentStream

protected void unregisterCurrentStream()

getFsDesiredReplication

public int getFsDesiredReplication()

getNumCurrentReplicas

public int getNumCurrentReplicas()
                          throws IllegalArgumentException,
                                 IllegalAccessException,
                                 InvocationTargetException

This method gets the datanode replication count for the current open file. If the pipeline isn't started yet or is empty, you will get the default replication factor.

If this function returns -1, it means you are not properly running with the HDFS-826 patch.

Throws:

InvocationTargetException

IllegalAccessException

IllegalArgumentException

hflushOrSync

protected void hflushOrSync(org.apache.hadoop.fs.FSDataOutputStream os)
                     throws IOException

If hflush is available in this version of HDFS, then this method calls hflush, else it calls sync.

Parameters:

os - - The stream to flush/sync

Throws:

IOException

isFileAppendable

protected boolean isFileAppendable(org.apache.hadoop.conf.Configuration conf,
                                   org.apache.hadoop.fs.FileSystem hdfs,
                                   org.apache.hadoop.fs.Path dstPath)
                            throws IOException

Throws:

IOException