Apache Flume Sink Processors are used to facilitate the capabilities of load balancing and achieve the failover in case of sinks failure.
The following are the required properties of Apache Flume processors.
| Commands | Default | Description | sinks | – | The list of sinks in the group. | processor.type | default | The component type should be the default, failover, or load_balance. |
|---|
Example for Apache Flume Sink Processors.
agentone.sinkgroups = g1
agentone.sinkgroups.g1.sinks = k1 k2
agentone.sinkgroups.g1.processor.type = load_balance
Apache Flume provides different types of Flume Sink Processors which are mentioned below.
Let us see each Flume Sink Processor in detail.
1. Default Sink Processor
The Apache Flume Default Sink Processor accepts one sink for which a user can use the source - channel - sink pattern.
2. Failover Sink Processor
Apache Flume Failover Sink Processor is used to maintain a priority list which guarantees that even there is one event, it will be processed. It works by transferring failed sinks to a pool where they are assigned a cool-down period, increasing with sequential failures before they are retried. A sink is restored in the live pool after successfully sending the event. The sink has priority a priority associated with them, the larger the number, the higher the priority. While sending data if a Sink fails with no priority then the next Sink with the highest priority shall be tried next for sending Events.
For configuration of Failover Sink Processor, set a sink groups processor to failover and set priorities for all individual sinks. The specified priorities must be unique and the upper limit to failover time can be set (in milliseconds) using the max penalty property.
Example for Failover Sink Processors.
agentone.sinkgroups = g1
agentone.sinkgroups.g1.sinks = k1 k2
agentone.sinkgroups.g1.processor.type = failover
agentone.sinkgroups.g1.processor.priority.k1 = 5
agentone.sinkgroups.g1.processor.priority.k2 = 10
agentone.sinkgroups.g1.processor.maxpenalty = 10000
3. Load balancing Sink Processor
Load balancing sink processor is used to provide an ability to balance the load over multiple sinks and to do so it maintains an indexed list of active sinks on which the load must be distributed. It supports the distribution of load using round_robin or random selection mechanisms. The default selection mechanism is round_robin type but it can be overridden via configuration.
Example for Load balancing Sink Processors.
agentone.sinkgroups = g1
agentone.sinkgroups.g1.sinks = k1 k2
agentone.sinkgroups.g1.processor.type = load_balance
agentone.sinkgroups.g1.processor.backoff = true
agentone.sinkgroups.g1.processor.selector = random