org.apache.paimon.flink.source.operator

Class MonitorFunction

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.streaming.api.functions.source.RichSourceFunction<Split>

org.apache.paimon.flink.source.operator.MonitorFunction

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.streaming.api.checkpoint.CheckpointedFunction, org.apache.flink.streaming.api.functions.source.SourceFunction<Split>

public class MonitorFunction
extends org.apache.flink.streaming.api.functions.source.RichSourceFunction<Split>
implements org.apache.flink.streaming.api.checkpoint.CheckpointedFunction, org.apache.flink.api.common.state.CheckpointListener

This is the single (non-parallel) monitoring task, it is responsible for:

1. Monitoring snapshots of the Paimon table.
2. Creating the splits corresponding to the incremental files
3. Assigning them to downstream tasks for further processing.

The splits to be read are forwarded to the downstream ReadOperator which can have parallelism greater than one.

Currently, there are two features that rely on this monitor:

1. Consumer-id: rely on this function to do aligned snapshot consumption, and ensure that all data in a snapshot is consumed within each checkpoint.
2. Snapshot-watermark: when there is no watermark definition, the default Paimon table will pass the watermark recorded in the snapshot.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.streaming.api.functions.source.SourceFunction

org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T>

Constructor Summary

Constructors

Constructor and Description
MonitorFunction(ReadBuilder readBuilder, long monitorInterval, boolean emitSnapshotWatermark)

Method Summary

Modifier and Type	Method and Description
static org.apache.flink.streaming.api.datastream.DataStream<org.apache.flink.table.data.RowData>	buildSource(org.apache.flink.streaming.api.environment.StreamExecutionEnvironment env, String name, org.apache.flink.api.common.typeinfo.TypeInformation<org.apache.flink.table.data.RowData> typeInfo, ReadBuilder readBuilder, long monitorInterval, boolean emitSnapshotWatermark, boolean shuffleBucketWithPartition, BucketMode bucketMode)
void	cancel()
void	initializeState(org.apache.flink.runtime.state.FunctionInitializationContext context)
void	notifyCheckpointComplete(long checkpointId)
void	run(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<Split> ctx)
void	snapshotState(org.apache.flink.runtime.state.FunctionSnapshotContext ctx)

Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

close, getIterationRuntimeContext, getRuntimeContext, open, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

notifyCheckpointAborted

Methods inherited from interface org.apache.flink.api.common.functions.RichFunction

open

Constructor Detail

MonitorFunction

public MonitorFunction(ReadBuilder readBuilder,
                       long monitorInterval,
                       boolean emitSnapshotWatermark)

Method Detail

initializeState

public void initializeState(org.apache.flink.runtime.state.FunctionInitializationContext context)
                     throws Exception

Specified by:

initializeState in interface org.apache.flink.streaming.api.checkpoint.CheckpointedFunction

Throws:

Exception

snapshotState

public void snapshotState(org.apache.flink.runtime.state.FunctionSnapshotContext ctx)
                   throws Exception

Specified by:

snapshotState in interface org.apache.flink.streaming.api.checkpoint.CheckpointedFunction

Throws:

Exception

run

public void run(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<Split> ctx)
         throws Exception

Specified by:

run in interface org.apache.flink.streaming.api.functions.source.SourceFunction<Split>

Throws:

Exception

notifyCheckpointComplete

public void notifyCheckpointComplete(long checkpointId)

Specified by:

notifyCheckpointComplete in interface org.apache.flink.api.common.state.CheckpointListener

cancel

public void cancel()

Specified by:

cancel in interface org.apache.flink.streaming.api.functions.source.SourceFunction<Split>

buildSource

public static org.apache.flink.streaming.api.datastream.DataStream<org.apache.flink.table.data.RowData> buildSource(org.apache.flink.streaming.api.environment.StreamExecutionEnvironment env,
                                                                                                                    String name,
                                                                                                                    org.apache.flink.api.common.typeinfo.TypeInformation<org.apache.flink.table.data.RowData> typeInfo,
                                                                                                                    ReadBuilder readBuilder,
                                                                                                                    long monitorInterval,
                                                                                                                    boolean emitSnapshotWatermark,
                                                                                                                    boolean shuffleBucketWithPartition,
                                                                                                                    BucketMode bucketMode)