Flink Component

Table of Contents

URI Format
Flink DataSet Callback
Flink DataStream Callback
Camel-Flink Producer call
See Also

Available as of Camel 2.18

This documentation page covers the Apache Flink component for the Apache Camel. The camel-flink component provides a bridge between Camel connectors and Flink tasks. This Camel Flink connector provides a way to route message from various transports, dynamically choosing a flink task to execute, use incoming message as input data for the task and finally deliver the results back to the Camel pipeline.

Maven users will need to add the following dependency to their pom.xml for this component:

<dependency>
    <groupId>org.apache.camel</groupId>
    <artifactId>camel-flink</artifactId>
    <version>x.x.x</version>
    <!-- use the same version as your Camel core version -->
</dependency>

URI Format

Currently, the Flink Component supports only Producers. One can create DataSet, DataStream jobs.

flink:dataset?dataset=#myDataSet&dataSetCallback=#dataSetCallback
flink:datastream?datastream=#myDataStream&dataStreamCallback=#dataStreamCallback

FlinkEndpoint Options

The Apache Flink component supports 7 endpoint options which are listed below:

{% raw %}

NameGroupDefaultJava TypeDescription

endpointType

producer

 

EndpointType

Required Type of the endpoint (dataset datastream).

collect

producer

true

boolean

Indicates if results should be collected or counted.

dataSet

producer

 

DataSet

DataSet to compute against.

dataSetCallback

producer

 

DataSetCallback

Function performing action against a DataSet.

dataStream

producer

 

DataStream

DataStream to compute against.

dataStreamCallback

producer

 

DataStreamCallback

Function performing action against a DataStream.

synchronous

advanced

false

boolean

Sets whether synchronous processing should be strictly used or Camel is allowed to use asynchronous processing (if supported).

{% endraw %}

FlinkComponent Options ^^^^^^^^

The Apache Flink component supports 4 options which are listed below.

{% raw %}

NameJava TypeDescription

dataSet

DataSet

DataSet to compute against.

dataStream

DataStream

DataStream to compute against.

dataSetCallback

DataSetCallback

Function performing action against a DataSet.

dataStreamCallback

DataStreamCallback

Function performing action against a DataStream.

{% endraw %}

Flink DataSet Callback

@Bean
public DataSetCallback<Long> dataSetCallback() {
    return new DataSetCallback<Long>() {
        public Long onDataSet(DataSet dataSet, Object... objects) {
            try {
                 dataSet.print();
                 return new Long(0);
            } catch (Exception e) {
                 return new Long(-1);
            }
        }
    };
}

Flink DataStream Callback

@Bean
public VoidDataStreamCallback dataStreamCallback() {
    return new VoidDataStreamCallback() {
        @Override
        public void doOnDataStream(DataStream dataStream, Object... objects) throws Exception {
            dataStream.flatMap(new Splitter()).print();

            environment.execute("data stream test");
        }
    };
}

Camel-Flink Producer call

CamelContext camelContext = new SpringCamelContext(context);

String pattern = "foo";

try {
    ProducerTemplate template = camelContext.createProducerTemplate();
    camelContext.start();
    Long count = template.requestBody("flink:dataSet?dataSet=#myDataSet&dataSetCallback=#countLinesContaining", pattern, Long.class);
    } finally {
        camelContext.stop();
    }

See Also