org.apache.paimon.flink.source.operator

Class MonitorSource

java.lang.Object

org.apache.paimon.flink.source.AbstractNonCoordinatedSource<Split>

org.apache.paimon.flink.source.operator.MonitorSource

All Implemented Interfaces:

Serializable, org.apache.flink.api.connector.source.Source<Split,SimpleSourceSplit,NoOpEnumState>, org.apache.flink.api.connector.source.SourceReaderFactory<Split,SimpleSourceSplit>

public class MonitorSource
extends AbstractNonCoordinatedSource<Split>

This is the single (non-parallel) monitoring task, it is responsible for:

1. Monitoring snapshots of the Paimon table.
2. Creating the splits corresponding to the incremental files
3. Assigning them to downstream tasks for further processing.

The splits to be read are forwarded to the downstream ReadOperator which can have parallelism greater than one.

Currently, there are two features that rely on this monitor:

1. Consumer-id: rely on this source to do aligned snapshot consumption, and ensure that all data in a snapshot is consumed within each checkpoint.
2. Snapshot-watermark: when there is no watermark definition, the default Paimon table will pass the watermark recorded in the snapshot.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
MonitorSource(ReadBuilder readBuilder, long monitorInterval, boolean emitSnapshotWatermark)

Method Summary

Modifier and Type	Method and Description
static org.apache.flink.streaming.api.datastream.DataStream<org.apache.flink.table.data.RowData>	buildSource(org.apache.flink.streaming.api.environment.StreamExecutionEnvironment env, String name, org.apache.flink.api.common.typeinfo.TypeInformation<org.apache.flink.table.data.RowData> typeInfo, ReadBuilder readBuilder, long monitorInterval, boolean emitSnapshotWatermark, boolean shuffleBucketWithPartition, boolean unawareBucket, NestedProjectedRowData nestedProjectedRowData)
org.apache.flink.api.connector.source.SourceReader<Split,SimpleSourceSplit>	createReader(org.apache.flink.api.connector.source.SourceReaderContext sourceReaderContext)
org.apache.flink.api.connector.source.Boundedness	getBoundedness()

Methods inherited from class org.apache.paimon.flink.source.AbstractNonCoordinatedSource

createEnumerator, getEnumeratorCheckpointSerializer, getSplitSerializer, restoreEnumerator

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

MonitorSource

public MonitorSource(ReadBuilder readBuilder,
                     long monitorInterval,
                     boolean emitSnapshotWatermark)

Method Detail

getBoundedness

public org.apache.flink.api.connector.source.Boundedness getBoundedness()

createReader

public org.apache.flink.api.connector.source.SourceReader<Split,SimpleSourceSplit> createReader(org.apache.flink.api.connector.source.SourceReaderContext sourceReaderContext)
                                                                                         throws Exception

Throws:

Exception

buildSource

public static org.apache.flink.streaming.api.datastream.DataStream<org.apache.flink.table.data.RowData> buildSource(org.apache.flink.streaming.api.environment.StreamExecutionEnvironment env,
                                                                                                                    String name,
                                                                                                                    org.apache.flink.api.common.typeinfo.TypeInformation<org.apache.flink.table.data.RowData> typeInfo,
                                                                                                                    ReadBuilder readBuilder,
                                                                                                                    long monitorInterval,
                                                                                                                    boolean emitSnapshotWatermark,
                                                                                                                    boolean shuffleBucketWithPartition,
                                                                                                                    boolean unawareBucket,
                                                                                                                    NestedProjectedRowData nestedProjectedRowData)