org.apache.paimon.flink.lookup.partitioner

Class BucketShuffleStrategy

java.lang.Object

org.apache.paimon.flink.lookup.partitioner.BucketShuffleStrategy

All Implemented Interfaces:

Serializable, ShuffleStrategy

public class BucketShuffleStrategy
extends Object
implements ShuffleStrategy

BucketShuffleStrategy defines a strategy for determining the target subtask id for a given bucket id and join key hash. It also provides a method to retrieve the set of bucket ids that are required to be cached by a specific subtask.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
BucketShuffleStrategy(int numBuckets)

Method Summary

Modifier and Type	Method and Description
Set<Integer>	getRequiredCacheBucketIds(int subtaskId, int numSubtasks) Retrieves the set of bucket IDs that are required to be cached by a specific subtask.
int	getTargetSubtaskId(int bucketId, int joinKeyHash, int numSubtasks) Determines the target subtask ID for a given bucket ID and join key hash.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BucketShuffleStrategy

public BucketShuffleStrategy(int numBuckets)

Method Detail

getTargetSubtaskId

public int getTargetSubtaskId(int bucketId,
                              int joinKeyHash,
                              int numSubtasks)

Determines the target subtask ID for a given bucket ID and join key hash.

The method uses two different strategies based on the comparison between the number of buckets and the number of subtasks:

1. If the number of buckets is greater than or equal to the number of subtasks (numBuckets >= numSubtasks), then the target subtask ID is determined by assigning buckets to subtasks in a round-robin manner: e.g., numBuckets = 5, numSubtasks = 3 Bucket 0 -> Subtask 0 Bucket 1 -> Subtask 1 Bucket 2 -> Subtask 2 Bucket 3 -> Subtask 0 Bucket 4 -> Subtask 1

2. If the number of buckets is less than the number of subtasks (numBuckets < numSubtasks), then the target subtask ID is determined by distributing subtasks to buckets in a round-robin manner: e.g., numBuckets = 2, numSubtasks = 5 Bucket 0 -> Subtask 0,2,4 Bucket 1 -> Subtask 1,3

Specified by:

getTargetSubtaskId in interface ShuffleStrategy

Parameters:

bucketId - The ID of the bucket.

joinKeyHash - The hash of the join key.

numSubtasks - The total number of target subtasks.

Returns:

The target subtask ID for the given bucket ID and join key hash.

getRequiredCacheBucketIds

public Set<Integer> getRequiredCacheBucketIds(int subtaskId,
                                              int numSubtasks)

Retrieves the set of bucket IDs that are required to be cached by a specific subtask. This method uses the same strategy as getTargetSubtaskId(int, int, int) to ensure consistency in bucket to subtask mapping.

The method uses two different strategies based on the comparison between the number of buckets and the number of subtasks:

1. If the number of buckets is greater than or equal to the number of subtasks (numBuckets >= numSubtasks), then each subtask caches buckets in a round-robin manner. e.g., numBuckets = 5, numSubtasks = 3 Subtask 0 -> Bucket 0,3 Subtask 1 -> Bucket 1,4 Subtask 2 -> Bucket 2

2. If the number of buckets is less than the number of subtasks (numBuckets < numSubtasks), then each subtask caches buckets by polling from buckets in a round-robin manner: e.g., numBuckets = 2, numSubtasks = 5 Subtask 0 -> Bucket 0 Subtask 1 -> Bucket 1 Subtask 2 -> Bucket 0 Subtask 3 -> Bucket 1 Subtask 4 -> Bucket 0

Specified by:

getRequiredCacheBucketIds in interface ShuffleStrategy

Parameters:

subtaskId - The ID of the subtask.

numSubtasks - The total number of subtasks.

Returns:

The set of bucket IDs that are required to be cached by the specified subtask.