org.apache.paimon.flink.source.assigners

Interface SplitAssigner

All Known Implementing Classes:

AlignedSplitAssigner, DynamicPartitionPruningAssigner, FIFOSplitAssigner, PreAssignSplitAssigner

public interface SplitAssigner

The SplitAssigner is responsible for deciding what splits should be processed next by which node. It determines split processing order and locality.

Method Summary

Modifier and Type	Method and Description
void	addSplit(int suggestedTask, FileStoreSourceSplit splits) Add one split of a specified subtask to the assigner.
void	addSplitsBack(int subtask, List<FileStoreSourceSplit> splits) Adds a set of splits to this assigner.
List<FileStoreSourceSplit>	getNext(int subtask, String hostname) Gets the next split.
Optional<Long>	getNextSnapshotId(int subtask) Gets the snapshot id of the next split.
int	numberOfRemainingSplits() Gets the current number of remaining splits.
Collection<FileStoreSourceSplit>	remainingSplits() Gets the remaining splits that this assigner has pending.

Method Detail

getNext

List<FileStoreSourceSplit> getNext(int subtask,
                                   @Nullable
                                   String hostname)

Gets the next split.

When this method returns an empty Optional, then the set of splits is assumed to be done and the source will finish once the readers finished their current splits.

addSplit

void addSplit(int suggestedTask,
              FileStoreSourceSplit splits)

Add one split of a specified subtask to the assigner.

addSplitsBack

void addSplitsBack(int subtask,
                   List<FileStoreSourceSplit> splits)

Adds a set of splits to this assigner. This happens for example when some split processing failed and the splits need to be re-added, or when new splits got discovered.

remainingSplits

Collection<FileStoreSourceSplit> remainingSplits()

Gets the remaining splits that this assigner has pending.

getNextSnapshotId

Optional<Long> getNextSnapshotId(int subtask)

Gets the snapshot id of the next split.

numberOfRemainingSplits

int numberOfRemainingSplits()

Gets the current number of remaining splits. This method should be guaranteed to be thread-safe.