org.apache.paimon.flink.source

Class ContinuousFileSplitEnumerator

java.lang.Object

org.apache.paimon.flink.source.ContinuousFileSplitEnumerator

All Implemented Interfaces:

AutoCloseable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.api.connector.source.SplitEnumerator<FileStoreSourceSplit,PendingSplitsCheckpoint>

Direct Known Subclasses:

AlignedContinuousFileSplitEnumerator

public class ContinuousFileSplitEnumerator
extends Object
implements org.apache.flink.api.connector.source.SplitEnumerator<FileStoreSourceSplit,PendingSplitsCheckpoint>

A continuously monitoring enumerator.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	ContinuousFileSplitEnumerator.PlanWithNextSnapshotId The result of scan.

Field Summary

Fields

Modifier and Type	Field and Description
protected ConsumerProgressCalculator	consumerProgressCalculator
protected org.apache.flink.api.connector.source.SplitEnumeratorContext<FileStoreSourceSplit>	context
protected long	discoveryInterval
protected boolean	finished
protected Long	nextSnapshotId
protected Set<Integer>	readersAwaitingSplit
protected StreamTableScan	scan
protected SplitAssigner	splitAssigner
protected FileStoreSourceSplitGenerator	splitGenerator

Constructor Summary

Constructors

Constructor and Description
ContinuousFileSplitEnumerator(org.apache.flink.api.connector.source.SplitEnumeratorContext<FileStoreSourceSplit> context, Collection<FileStoreSourceSplit> remainSplits, Long nextSnapshotId, long discoveryInterval, StreamTableScan scan, BucketMode bucketMode, int splitMaxPerTask, boolean shuffleBucketWithPartition)

Method Summary

Modifier and Type	Method and Description
void	addReader(int subtaskId)
protected void	addSplits(Collection<FileStoreSourceSplit> splits)
void	addSplitsBack(List<FileStoreSourceSplit> splits, int subtaskId)
protected void	assignSplits() Method should be synchronized because handleSplitRequest(int, java.lang.String) and processDiscoveredSplits(java.util.Optional<org.apache.paimon.flink.source.ContinuousFileSplitEnumerator.PlanWithNextSnapshotId>, java.lang.Throwable) have thread conflicts.
protected int	assignSuggestedTask(FileStoreSourceSplit split)
void	close()
protected SplitAssigner	createSplitAssigner(BucketMode bucketMode)
void	handleSourceEvent(int subtaskId, org.apache.flink.api.connector.source.SourceEvent sourceEvent)
void	handleSplitRequest(int subtaskId, String requesterHostname)
protected boolean	noMoreSplits()
void	notifyCheckpointComplete(long checkpointId)
protected void	processDiscoveredSplits(Optional<ContinuousFileSplitEnumerator.PlanWithNextSnapshotId> planWithNextSnapshotIdOptional, Throwable error)
protected Optional<ContinuousFileSplitEnumerator.PlanWithNextSnapshotId>	scanNextSnapshot()
PendingSplitsCheckpoint	snapshotState(long checkpointId)
void	start()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

notifyCheckpointAborted

Field Detail

context

protected final org.apache.flink.api.connector.source.SplitEnumeratorContext<FileStoreSourceSplit> context

discoveryInterval

protected final long discoveryInterval

readersAwaitingSplit

protected final Set<Integer> readersAwaitingSplit

splitGenerator

protected final FileStoreSourceSplitGenerator splitGenerator

scan

protected final StreamTableScan scan

splitAssigner

protected final SplitAssigner splitAssigner

consumerProgressCalculator

protected final ConsumerProgressCalculator consumerProgressCalculator

nextSnapshotId

@Nullable
protected Long nextSnapshotId

finished

protected boolean finished

Constructor Detail

ContinuousFileSplitEnumerator

public ContinuousFileSplitEnumerator(org.apache.flink.api.connector.source.SplitEnumeratorContext<FileStoreSourceSplit> context,
                                     Collection<FileStoreSourceSplit> remainSplits,
                                     @Nullable
                                     Long nextSnapshotId,
                                     long discoveryInterval,
                                     StreamTableScan scan,
                                     BucketMode bucketMode,
                                     int splitMaxPerTask,
                                     boolean shuffleBucketWithPartition)

Method Detail

addSplits

protected void addSplits(Collection<FileStoreSourceSplit> splits)

start

public void start()

Specified by:

start in interface org.apache.flink.api.connector.source.SplitEnumerator<FileStoreSourceSplit,PendingSplitsCheckpoint>

close

public void close()
           throws IOException

Specified by:

close in interface AutoCloseable

Specified by:

close in interface org.apache.flink.api.connector.source.SplitEnumerator<FileStoreSourceSplit,PendingSplitsCheckpoint>

Throws:

IOException

addReader

public void addReader(int subtaskId)

Specified by:

addReader in interface org.apache.flink.api.connector.source.SplitEnumerator<FileStoreSourceSplit,PendingSplitsCheckpoint>

handleSplitRequest

public void handleSplitRequest(int subtaskId,
                               @Nullable
                               String requesterHostname)

Specified by:

handleSplitRequest in interface org.apache.flink.api.connector.source.SplitEnumerator<FileStoreSourceSplit,PendingSplitsCheckpoint>

handleSourceEvent

public void handleSourceEvent(int subtaskId,
                              org.apache.flink.api.connector.source.SourceEvent sourceEvent)

Specified by:

handleSourceEvent in interface org.apache.flink.api.connector.source.SplitEnumerator<FileStoreSourceSplit,PendingSplitsCheckpoint>

addSplitsBack

public void addSplitsBack(List<FileStoreSourceSplit> splits,
                          int subtaskId)

Specified by:

addSplitsBack in interface org.apache.flink.api.connector.source.SplitEnumerator<FileStoreSourceSplit,PendingSplitsCheckpoint>

snapshotState

public PendingSplitsCheckpoint snapshotState(long checkpointId)
                                      throws Exception

Specified by:

snapshotState in interface org.apache.flink.api.connector.source.SplitEnumerator<FileStoreSourceSplit,PendingSplitsCheckpoint>

Throws:

Exception

notifyCheckpointComplete

public void notifyCheckpointComplete(long checkpointId)
                              throws Exception

Specified by:

notifyCheckpointComplete in interface org.apache.flink.api.common.state.CheckpointListener

Specified by:

notifyCheckpointComplete in interface org.apache.flink.api.connector.source.SplitEnumerator<FileStoreSourceSplit,PendingSplitsCheckpoint>

Throws:

Exception

scanNextSnapshot

protected Optional<ContinuousFileSplitEnumerator.PlanWithNextSnapshotId> scanNextSnapshot()

processDiscoveredSplits

protected void processDiscoveredSplits(Optional<ContinuousFileSplitEnumerator.PlanWithNextSnapshotId> planWithNextSnapshotIdOptional,
                                       Throwable error)

assignSplits

protected void assignSplits()

Method should be synchronized because handleSplitRequest(int, java.lang.String) and processDiscoveredSplits(java.util.Optional<org.apache.paimon.flink.source.ContinuousFileSplitEnumerator.PlanWithNextSnapshotId>, java.lang.Throwable) have thread conflicts.

assignSuggestedTask

protected int assignSuggestedTask(FileStoreSourceSplit split)

createSplitAssigner

protected SplitAssigner createSplitAssigner(BucketMode bucketMode)

noMoreSplits

protected boolean noMoreSplits()