org.apache.paimon.flink.source.align

Class AlignedContinuousFileSplitEnumerator

java.lang.Object

org.apache.paimon.flink.source.ContinuousFileSplitEnumerator

org.apache.paimon.flink.source.align.AlignedContinuousFileSplitEnumerator

All Implemented Interfaces:

AutoCloseable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.api.connector.source.SplitEnumerator<FileStoreSourceSplit,PendingSplitsCheckpoint>

public class AlignedContinuousFileSplitEnumerator
extends ContinuousFileSplitEnumerator

An aligned continuously monitoring enumerator. It requires checkpoint to be aligned with snapshot.

There are two alignment cases here:

• The snapshot has been consumed, but the checkpoint has not yet been triggered: AlignedSourceReader will not request splits until checkpoint is triggered.
• The checkpoint has been triggered, but the snapshot has not yet been produced: The checkpoint will block until the snapshot of the upstream table is generated or times out.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.paimon.flink.source.ContinuousFileSplitEnumerator

ContinuousFileSplitEnumerator.PlanWithNextSnapshotId

Field Summary

Fields inherited from class org.apache.paimon.flink.source.ContinuousFileSplitEnumerator

consumerProgressCalculator, context, discoveryInterval, finished, nextSnapshotId, readersAwaitingSplit, scan, splitAssigner, splitGenerator

Constructor Summary

Constructors

Constructor and Description
AlignedContinuousFileSplitEnumerator(org.apache.flink.api.connector.source.SplitEnumeratorContext<FileStoreSourceSplit> context, Collection<FileStoreSourceSplit> remainSplits, Long nextSnapshotId, long discoveryInterval, StreamTableScan scan, BucketMode bucketMode, long alignTimeout, int splitPerTaskMax, boolean shuffleBucketWithPartition)

Method Summary

Modifier and Type	Method and Description
protected void	addSplits(Collection<FileStoreSourceSplit> splits)
void	addSplitsBack(List<FileStoreSourceSplit> splits, int subtaskId)
void	close()
protected SplitAssigner	createSplitAssigner(BucketMode bucketMode)
protected boolean	noMoreSplits()
void	notifyCheckpointAborted(long checkpointId)
void	notifyCheckpointComplete(long checkpointId)
protected void	processDiscoveredSplits(Optional<ContinuousFileSplitEnumerator.PlanWithNextSnapshotId> ignore, Throwable error)
protected Optional<ContinuousFileSplitEnumerator.PlanWithNextSnapshotId>	scanNextSnapshot()
PendingSplitsCheckpoint	snapshotState(long checkpointId)

Methods inherited from class org.apache.paimon.flink.source.ContinuousFileSplitEnumerator

addReader, assignSplits, assignSuggestedTask, handleSourceEvent, handleSplitRequest, start

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

AlignedContinuousFileSplitEnumerator

public AlignedContinuousFileSplitEnumerator(org.apache.flink.api.connector.source.SplitEnumeratorContext<FileStoreSourceSplit> context,
                                            Collection<FileStoreSourceSplit> remainSplits,
                                            @Nullable
                                            Long nextSnapshotId,
                                            long discoveryInterval,
                                            StreamTableScan scan,
                                            BucketMode bucketMode,
                                            long alignTimeout,
                                            int splitPerTaskMax,
                                            boolean shuffleBucketWithPartition)

Method Detail

addSplits

protected void addSplits(Collection<FileStoreSourceSplit> splits)

Overrides:

addSplits in class ContinuousFileSplitEnumerator

close

public void close()
           throws IOException

Specified by:

close in interface AutoCloseable

Specified by:

close in interface org.apache.flink.api.connector.source.SplitEnumerator<FileStoreSourceSplit,PendingSplitsCheckpoint>

Overrides:

close in class ContinuousFileSplitEnumerator

Throws:

IOException

addSplitsBack

public void addSplitsBack(List<FileStoreSourceSplit> splits,
                          int subtaskId)

Specified by:

addSplitsBack in interface org.apache.flink.api.connector.source.SplitEnumerator<FileStoreSourceSplit,PendingSplitsCheckpoint>

Overrides:

addSplitsBack in class ContinuousFileSplitEnumerator

snapshotState

public PendingSplitsCheckpoint snapshotState(long checkpointId)
                                      throws Exception

Specified by:

snapshotState in interface org.apache.flink.api.connector.source.SplitEnumerator<FileStoreSourceSplit,PendingSplitsCheckpoint>

Overrides:

snapshotState in class ContinuousFileSplitEnumerator

Throws:

Exception

notifyCheckpointAborted

public void notifyCheckpointAborted(long checkpointId)

notifyCheckpointComplete

public void notifyCheckpointComplete(long checkpointId)

Specified by:

notifyCheckpointComplete in interface org.apache.flink.api.common.state.CheckpointListener

Specified by:

notifyCheckpointComplete in interface org.apache.flink.api.connector.source.SplitEnumerator<FileStoreSourceSplit,PendingSplitsCheckpoint>

Overrides:

notifyCheckpointComplete in class ContinuousFileSplitEnumerator

scanNextSnapshot

protected Optional<ContinuousFileSplitEnumerator.PlanWithNextSnapshotId> scanNextSnapshot()

Overrides:

scanNextSnapshot in class ContinuousFileSplitEnumerator

processDiscoveredSplits

protected void processDiscoveredSplits(Optional<ContinuousFileSplitEnumerator.PlanWithNextSnapshotId> ignore,
                                       Throwable error)

Overrides:

processDiscoveredSplits in class ContinuousFileSplitEnumerator

noMoreSplits

protected boolean noMoreSplits()

Overrides:

noMoreSplits in class ContinuousFileSplitEnumerator

createSplitAssigner

protected SplitAssigner createSplitAssigner(BucketMode bucketMode)

Overrides:

createSplitAssigner in class ContinuousFileSplitEnumerator