org.apache.flume.sink.hdfs

Class AvroEventSerializer

java.lang.Object

org.apache.flume.sink.hdfs.AvroEventSerializer

All Implemented Interfaces:

Configurable, EventSerializer

public class AvroEventSerializer
extends Object
implements EventSerializer, Configurable

This class serializes Flume events into Avro data files. The Flume event body is read as an Avro datum, and is then written to the EventSerializer's output stream in Avro data file format.

The Avro schema is determined by reading a Flume event header. The schema may be specified either as a literal, by setting AVRO_SCHEMA_LITERAL_HEADER (not recommended, since the full schema must be transmitted in every event), or as a URL which the schema may be read from, by setting AVRO_SCHEMA_URL_HEADER. Schemas read from URLs are cached by instances of this class so that the overhead of retrieval is minimized.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	AvroEventSerializer.Builder

Field Summary

Fields

Modifier and Type	Field and Description
static String	AVRO_SCHEMA_LITERAL_HEADER
static String	AVRO_SCHEMA_URL_HEADER

Fields inherited from interface org.apache.flume.serialization.EventSerializer

CTX_PREFIX

Method Summary

Modifier and Type	Method and Description
void	afterCreate() Hook to write a header after file is opened for the first time.
void	afterReopen() Hook to handle any framing needed when file is re-opened (for write). Could have been named afterOpenForAppend().
void	beforeClose() Hook to write a trailer before the stream is closed.
void	configure(Context context) Request the implementing class to (re)configure itself.
void	flush() Hook to flush any internal write buffers to the underlying stream.
boolean	supportsReopen() Specify whether this output format supports reopening files for append.
void	write(Event event) Serialize and write the given event.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

AVRO_SCHEMA_LITERAL_HEADER

public static final String AVRO_SCHEMA_LITERAL_HEADER

See Also:

Constant Field Values

AVRO_SCHEMA_URL_HEADER

public static final String AVRO_SCHEMA_URL_HEADER

See Also:

Constant Field Values

Method Detail

configure

public void configure(Context context)

Description copied from interface: Configurable

Request the implementing class to (re)configure itself.

When configuration parameters are changed, they must be reflected by the component asap.

There are no thread safety guarantees on when configure might be called.

Specified by:

configure in interface Configurable

afterCreate

public void afterCreate()
                 throws IOException

Description copied from interface: EventSerializer

Hook to write a header after file is opened for the first time.

Specified by:

afterCreate in interface EventSerializer

Throws:

IOException

afterReopen

public void afterReopen()
                 throws IOException

Description copied from interface: EventSerializer

Hook to handle any framing needed when file is re-opened (for write).
Could have been named afterOpenForAppend().

Specified by:

afterReopen in interface EventSerializer

Throws:

IOException

write

public void write(Event event)
           throws IOException

Description copied from interface: EventSerializer

Serialize and write the given event.

Specified by:

write in interface EventSerializer

Parameters:

event - Event to write to the underlying stream.

Throws:

IOException

flush

public void flush()
           throws IOException

Description copied from interface: EventSerializer

Hook to flush any internal write buffers to the underlying stream. It is NOT necessary for an implementation to then call flush() / sync() on the underlying stream itself, since those semantics would be provided by the driver that calls this API.

Specified by:

flush in interface EventSerializer

Throws:

IOException

beforeClose

public void beforeClose()
                 throws IOException

Description copied from interface: EventSerializer

Hook to write a trailer before the stream is closed. Implementations must not buffer data in this call since EventSerializer.flush() is not guaranteed to be called after beforeClose().

Specified by:

beforeClose in interface EventSerializer

Throws:

IOException

supportsReopen

public boolean supportsReopen()

Description copied from interface: EventSerializer

Specify whether this output format supports reopening files for append. For example, this method should return false if EventSerializer.beforeClose() writes a trailer that "finalizes" the file (this type of behavior is file format-specific).
Could have been named supportsAppend().

Specified by:

supportsReopen in interface EventSerializer