org.apache.paimon.flink.sink.cdc

Class CdcDynamicTableParsingProcessFunction<T>

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.streaming.api.functions.ProcessFunction<T,Void>

org.apache.paimon.flink.sink.cdc.CdcDynamicTableParsingProcessFunction<T>

Type Parameters:

T - CDC change event type

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction

public class CdcDynamicTableParsingProcessFunction<T>
extends org.apache.flink.streaming.api.functions.ProcessFunction<T,Void>

A ProcessFunction to parse CDC change event to either a list of DataFields or CdcRecord and send them to different side outputs according to table name. This process function will capture newly added tables when syncing entire database and in cases where the newly added tables are including by attesting table filters.

This ProcessFunction can handle records for different tables at the same time.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.flink.streaming.api.functions.ProcessFunction

org.apache.flink.streaming.api.functions.ProcessFunction.Context, org.apache.flink.streaming.api.functions.ProcessFunction.OnTimerContext

Field Summary

Fields

Modifier and Type	Field and Description
static org.apache.flink.util.OutputTag<CdcMultiplexRecord>	DYNAMIC_OUTPUT_TAG
static org.apache.flink.util.OutputTag<org.apache.flink.api.java.tuple.Tuple2<Identifier,List<DataField>>>	DYNAMIC_SCHEMA_CHANGE_OUTPUT_TAG

Constructor Summary

Constructors

Constructor and Description
CdcDynamicTableParsingProcessFunction(String database, CatalogLoader catalogLoader, EventParser.Factory<T> parserFactory)

Method Summary

Modifier and Type	Method and Description
void	close()
void	open(org.apache.flink.configuration.Configuration parameters) Do not annotate with @override here to maintain compatibility with Flink 2.0+.
void	open(OpenContext openContext) Do not annotate with @override here to maintain compatibility with Flink 1.18-.
void	processElement(T raw, org.apache.flink.streaming.api.functions.ProcessFunction.Context context, org.apache.flink.util.Collector<Void> collector)

Methods inherited from class org.apache.flink.streaming.api.functions.ProcessFunction

onTimer

Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

getIterationRuntimeContext, getRuntimeContext, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DYNAMIC_OUTPUT_TAG

public static final org.apache.flink.util.OutputTag<CdcMultiplexRecord> DYNAMIC_OUTPUT_TAG

DYNAMIC_SCHEMA_CHANGE_OUTPUT_TAG

public static final org.apache.flink.util.OutputTag<org.apache.flink.api.java.tuple.Tuple2<Identifier,List<DataField>>> DYNAMIC_SCHEMA_CHANGE_OUTPUT_TAG

Constructor Detail

CdcDynamicTableParsingProcessFunction

public CdcDynamicTableParsingProcessFunction(String database,
                                             CatalogLoader catalogLoader,
                                             EventParser.Factory<T> parserFactory)

Method Detail

open

public void open(OpenContext openContext)
          throws Exception

Do not annotate with @override here to maintain compatibility with Flink 1.18-.

Throws:

Exception

open

public void open(org.apache.flink.configuration.Configuration parameters)
          throws Exception

Do not annotate with @override here to maintain compatibility with Flink 2.0+.

Specified by:

open in interface org.apache.flink.api.common.functions.RichFunction

Overrides:

open in class org.apache.flink.api.common.functions.AbstractRichFunction

Throws:

Exception

processElement

public void processElement(T raw,
                           org.apache.flink.streaming.api.functions.ProcessFunction.Context context,
                           org.apache.flink.util.Collector<Void> collector)
                    throws Exception

Specified by:

processElement in class org.apache.flink.streaming.api.functions.ProcessFunction<T,Void>

Throws:

Exception

close

public void close()
           throws Exception

Specified by:

close in interface org.apache.flink.api.common.functions.RichFunction

Overrides:

close in class org.apache.flink.api.common.functions.AbstractRichFunction

Throws:

Exception