org.apache.paimon.flink.source.assigners

Class PreAssignSplitAssigner

java.lang.Object

org.apache.paimon.flink.source.assigners.PreAssignSplitAssigner

All Implemented Interfaces:

SplitAssigner

public class PreAssignSplitAssigner
extends Object
implements SplitAssigner

Pre-calculate which splits each task should process according to the weight or given DynamicFilteringData, and then distribute the splits fairly.

Constructor Summary

Constructors

Constructor and Description
PreAssignSplitAssigner(int splitBatchSize, int parallelism, Collection<FileStoreSourceSplit> splits)
PreAssignSplitAssigner(int splitBatchSize, int parallelism, Collection<FileStoreSourceSplit> splits, Projection partitionRowProjection, org.apache.flink.table.connector.source.DynamicFilteringData dynamicFilteringData)
PreAssignSplitAssigner(int splitBatchSize, org.apache.flink.api.connector.source.SplitEnumeratorContext<FileStoreSourceSplit> context, Collection<FileStoreSourceSplit> splits)

Method Summary

Modifier and Type	Method and Description
void	addSplit(int suggestedTask, FileStoreSourceSplit split) Add one split of a specified subtask to the assigner.
void	addSplitsBack(int subtask, List<FileStoreSourceSplit> splits) Adds a set of splits to this assigner.
List<FileStoreSourceSplit>	getNext(int subtask, String hostname) Gets the next split.
Optional<Long>	getNextSnapshotId(int subtask) Gets the snapshot id of the next split.
int	numberOfRemainingSplits() Gets the current number of remaining splits.
SplitAssigner	ofDynamicPartitionPruning(Projection partitionRowProjection, org.apache.flink.table.connector.source.DynamicFilteringData dynamicFilteringData)
Collection<FileStoreSourceSplit>	remainingSplits() Gets the remaining splits that this assigner has pending.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

PreAssignSplitAssigner

public PreAssignSplitAssigner(int splitBatchSize,
                              org.apache.flink.api.connector.source.SplitEnumeratorContext<FileStoreSourceSplit> context,
                              Collection<FileStoreSourceSplit> splits)

PreAssignSplitAssigner

public PreAssignSplitAssigner(int splitBatchSize,
                              int parallelism,
                              Collection<FileStoreSourceSplit> splits,
                              Projection partitionRowProjection,
                              org.apache.flink.table.connector.source.DynamicFilteringData dynamicFilteringData)

PreAssignSplitAssigner

public PreAssignSplitAssigner(int splitBatchSize,
                              int parallelism,
                              Collection<FileStoreSourceSplit> splits)

Method Detail

getNext

public List<FileStoreSourceSplit> getNext(int subtask,
                                          @Nullable
                                          String hostname)

Description copied from interface: SplitAssigner

Gets the next split.

When this method returns an empty Optional, then the set of splits is assumed to be done and the source will finish once the readers finished their current splits.

Specified by:

getNext in interface SplitAssigner

addSplit

public void addSplit(int suggestedTask,
                     FileStoreSourceSplit split)

Description copied from interface: SplitAssigner

Add one split of a specified subtask to the assigner.

Specified by:

addSplit in interface SplitAssigner

addSplitsBack

public void addSplitsBack(int subtask,
                          List<FileStoreSourceSplit> splits)

Description copied from interface: SplitAssigner

Adds a set of splits to this assigner. This happens for example when some split processing failed and the splits need to be re-added, or when new splits got discovered.

Specified by:

addSplitsBack in interface SplitAssigner

remainingSplits

public Collection<FileStoreSourceSplit> remainingSplits()

Description copied from interface: SplitAssigner

Gets the remaining splits that this assigner has pending.

Specified by:

remainingSplits in interface SplitAssigner

getNextSnapshotId

public Optional<Long> getNextSnapshotId(int subtask)

Description copied from interface: SplitAssigner

Gets the snapshot id of the next split.

Specified by:

getNextSnapshotId in interface SplitAssigner

numberOfRemainingSplits

public int numberOfRemainingSplits()

Description copied from interface: SplitAssigner

Gets the current number of remaining splits. This method should be guaranteed to be thread-safe.

Specified by:

numberOfRemainingSplits in interface SplitAssigner

ofDynamicPartitionPruning

public SplitAssigner ofDynamicPartitionPruning(Projection partitionRowProjection,
                                               org.apache.flink.table.connector.source.DynamicFilteringData dynamicFilteringData)