org.apache.paimon.flink.kafka

Class KafkaSinkFunction

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.streaming.api.functions.sink.RichSinkFunction<IN>

org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction<IN,org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.KafkaTransactionState,org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.KafkaTransactionContext>

org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer<SinkRecord>

org.apache.paimon.flink.kafka.KafkaSinkFunction

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.streaming.api.checkpoint.CheckpointedFunction, org.apache.flink.streaming.api.functions.sink.SinkFunction<SinkRecord>, LogSinkFunction

public class KafkaSinkFunction
extends org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer<SinkRecord>
implements LogSinkFunction

A FlinkKafkaProducer which implements LogSinkFunction to register WriteCallback.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer

org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.ContextStateSerializer, org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.KafkaTransactionContext, org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.KafkaTransactionState, org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.NextTransactionalIdHint, org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.NextTransactionalIdHintSerializer, org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.Semantic, org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.TransactionStateSerializer

Nested classes/interfaces inherited from class org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction

org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.State<TXN,CONTEXT>, org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.StateSerializer<TXN,CONTEXT>, org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.StateSerializerSnapshot<TXN,CONTEXT>, org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.TransactionHolder<TXN>

Nested classes/interfaces inherited from interface org.apache.paimon.flink.sink.LogSinkFunction

LogSinkFunction.WriteCallback

Nested classes/interfaces inherited from interface org.apache.flink.streaming.api.functions.sink.SinkFunction

org.apache.flink.streaming.api.functions.sink.SinkFunction.Context

Field Summary

Fields inherited from class org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer

asyncException, callback, DEFAULT_KAFKA_PRODUCERS_POOL_SIZE, DEFAULT_KAFKA_TRANSACTION_TIMEOUT, defaultTopicId, KEY_DISABLE_METRICS, pendingRecords, producerConfig, SAFE_SCALE_DOWN_FACTOR, semantic, topicPartitionsMap, writeTimestampToKafka

Fields inherited from class org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction

pendingCommitTransactions, state, userContext

Constructor Summary

Constructors

Constructor and Description
KafkaSinkFunction(String defaultTopic, org.apache.flink.streaming.connectors.kafka.KafkaSerializationSchema<SinkRecord> serializationSchema, Properties producerConfig, org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.Semantic semantic) Creates a KafkaSinkFunction for a given topic.

Method Summary

Modifier and Type	Method and Description
void	flush() Flush pending records.
void	open(org.apache.flink.configuration.Configuration configuration) Do not annotate with @override here to maintain compatibility with Flink 2.0+.
void	open(OpenContext openContext) Do not annotate with @override here to maintain compatibility with Flink 1.18-.
void	setWriteCallback(LogSinkFunction.WriteCallback writeCallback)

Methods inherited from class org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer

abort, acknowledgeMessage, beginTransaction, checkErroneous, close, commit, createProducer, finishProcessing, finishRecoveringContext, getPartitionsByTopic, getTransactionTimeout, ignoreFailuresAfterTransactionTimeout, initializeState, initializeUserContext, invoke, preCommit, recoverAndAbort, recoverAndCommit, setLogFailuresOnly, setTransactionalIdPrefix, setWriteTimestampToKafka, snapshotState

Methods inherited from class org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction

currentTransaction, enableTransactionTimeoutWarnings, finish, getUserContext, invoke, invoke, notifyCheckpointAborted, notifyCheckpointComplete, pendingTransactions, setTransactionTimeout

Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

getIterationRuntimeContext, getRuntimeContext, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.streaming.api.functions.sink.SinkFunction

finish, invoke, invoke, writeWatermark

Constructor Detail

KafkaSinkFunction

public KafkaSinkFunction(String defaultTopic,
                         org.apache.flink.streaming.connectors.kafka.KafkaSerializationSchema<SinkRecord> serializationSchema,
                         Properties producerConfig,
                         org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.Semantic semantic)

Creates a KafkaSinkFunction for a given topic. The sink produces its input to the topic. It accepts a KafkaSerializationSchema for serializing records to a ProducerRecord, including partitioning information.

Parameters:

defaultTopic - The default topic to write data to

serializationSchema - A serializable serialization schema for turning user objects into a kafka-consumable byte[] supporting key/value messages

producerConfig - Configuration properties for the KafkaProducer. 'bootstrap.servers.' is the only required argument.

semantic - Defines semantic that will be used by this producer (see KafkaSinkFunction.Semantic).

Method Detail

setWriteCallback

public void setWriteCallback(LogSinkFunction.WriteCallback writeCallback)

Specified by:

setWriteCallback in interface LogSinkFunction

open

public void open(OpenContext openContext)
          throws Exception

Do not annotate with @override here to maintain compatibility with Flink 1.18-.

Specified by:

open in interface org.apache.flink.api.common.functions.RichFunction

Throws:

Exception

open

public void open(org.apache.flink.configuration.Configuration configuration)
          throws Exception

Do not annotate with @override here to maintain compatibility with Flink 2.0+.

Specified by:

open in interface org.apache.flink.api.common.functions.RichFunction

Overrides:

open in class org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer<SinkRecord>

Throws:

Exception

flush

public void flush()
           throws org.apache.flink.streaming.connectors.kafka.FlinkKafkaException

Description copied from interface: LogSinkFunction

Flush pending records.

Specified by:

flush in interface LogSinkFunction

Throws:

org.apache.flink.streaming.connectors.kafka.FlinkKafkaException