org.apache.flume.sink

Class AbstractRpcSink

java.lang.Object

org.apache.flume.sink.AbstractSink

org.apache.flume.sink.AbstractRpcSink

All Implemented Interfaces:

BatchSizeSupported, Configurable, LifecycleAware, NamedComponent, Sink

Direct Known Subclasses:

AvroSink, ThriftSink

public abstract class AbstractRpcSink
extends AbstractSink
implements Configurable, BatchSizeSupported

This sink provides the basic RPC functionality for Flume. This sink takes several arguments which are used in RPC. This sink forms one half of Flume's tiered collection support. Events sent to this sink are transported over the network to the hostname / port pair using the RPC implementation encapsulated in RpcClient. The destination is an instance of Flume's .AvroSource or ThriftSource (based on which implementation of this class is used), which allows Flume agents to forward to other Flume agents, forming a tiered collection infrastructure. Of course, nothing prevents one from using this sink to speak to other custom built infrastructure that implements the same RPC protocol.

Events are taken from the configured Channel in batches of the configured batch-size. The batch size has no theoretical limits although all events in the batch must fit in memory. Generally, larger batches are far more efficient, but introduce a slight delay (measured in millis) in delivery. The batch behavior is such that underruns (i.e. batches smaller than the configured batch size) are possible. This is a compromise made to maintain low latency of event delivery. If the channel returns a null event, meaning it is empty, the batch is immediately sent, regardless of size. Batch underruns are tracked in the metrics. Empty batches do not incur an RPC roundtrip.

Configuration options

Parameter	Description	Unit (data type)	Default
hostname	The hostname to which events should be sent.	Hostname or IP (String)	none (required)
port	The port to which events should be sent on hostname.	TCP port (int)	none (required)
batch-size	The maximum number of events to send per RPC.	events (int)	100
connect-timeout	Maximum time to wait for the first Avro handshake and RPC request	milliseconds (long)	20000
request-timeout	Maximum time to wait RPC requests after the first	milliseconds (long)	20000
compression-type	Select compression type. Default is "none" and the only compression type available is "deflate"	compression type	none
compression-level	In the case compression type is "deflate" this value can be between 0-9. 0 being no compression and 1-9 is compression. The higher the number the better the compression. 6 is the default.	compression level	6

Metrics

TODO

Implementation Notes: Any implementation of this class must override the initializeRpcClient(Properties) method. This method will be called whenever this sink needs to create a new connection to the source.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flume.Sink

Sink.Status

Constructor Summary

Constructors

Constructor and Description
AbstractRpcSink()

Method Summary

Modifier and Type	Method and Description
void	configure(Context context) Request the implementing class to (re)configure itself.
long	getBatchSize() Returns the batch size
protected abstract RpcClient	initializeRpcClient(Properties props) Returns a new RpcClient instance configured using the given Properties object.
Sink.Status	process() Requests the sink to attempt to consume data from attached channel
void	start() The start() of RpcSink is more of an optimization that allows connection to be created before the process() loop is started.
void	stop() Stops a service or component.
String	toString()

Methods inherited from class org.apache.flume.sink.AbstractSink

getChannel, getLifecycleState, getName, setChannel, setName

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

AbstractRpcSink

public AbstractRpcSink()

Method Detail

configure

public void configure(Context context)

Description copied from interface: Configurable

Request the implementing class to (re)configure itself.

When configuration parameters are changed, they must be reflected by the component asap.

There are no thread safety guarantees on when configure might be called.

Specified by:

configure in interface Configurable

initializeRpcClient

protected abstract RpcClient initializeRpcClient(Properties props)

Returns a new RpcClient instance configured using the given Properties object. This method is called whenever a new connection needs to be created to the next hop.

Parameters:

props -

Returns:

start

public void start()

The start() of RpcSink is more of an optimization that allows connection to be created before the process() loop is started. In case it so happens that the start failed, the process() loop will itself attempt to reconnect as necessary. This is the expected behavior since it is possible that the downstream source becomes unavailable in the middle of the process loop and the sink will have to retry the connection again.

Specified by:

start in interface LifecycleAware

Overrides:

start in class AbstractSink

stop

public void stop()

Description copied from interface: LifecycleAware

Stops a service or component.

Implementations should determine the result of any stop logic and effect the return value of LifecycleAware.getLifecycleState() accordingly.

Specified by:

stop in interface LifecycleAware

Overrides:

stop in class AbstractSink

toString

public String toString()

Overrides:

toString in class AbstractSink

process

public Sink.Status process()
                    throws EventDeliveryException

Description copied from interface: Sink

Requests the sink to attempt to consume data from attached channel

Note: This method should be consuming from the channel within the bounds of a Transaction. On successful delivery, the transaction should be committed, and on failure it should be rolled back.

Specified by:

process in interface Sink

Returns:

READY if 1 or more Events were successfully delivered, BACKOFF if no data could be retrieved from the channel feeding this sink

Throws:

EventDeliveryException - In case of any kind of failure to deliver data to the next hop destination.

getBatchSize

public long getBatchSize()

Description copied from interface: BatchSizeSupported

Returns the batch size

Specified by:

getBatchSize in interface BatchSizeSupported