org.apache.paimon.flink.shuffle

Class RangeShuffle

java.lang.Object

org.apache.paimon.flink.shuffle.RangeShuffle

public class RangeShuffle
extends Object

RangeShuffle Util to shuffle the input stream by the sampling range. See `rangeShuffleBykey` method how to build the topo.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	RangeShuffle.KeyAndSizeExtractor<T> KeyAndSizeExtractor is responsible for extracting the sort key and row size.
static class	RangeShuffle.LocalSampleOperator<T> LocalSampleOperator wraps the sample logic on the partition side (the first phase of distributed sample algorithm).

Constructor Summary

Constructors

Constructor and Description
RangeShuffle()

Method Summary

Modifier and Type

Method and Description

static <T> org.apache.flink.streaming.api.datastream.DataStream<org.apache.flink.api.java.tuple.Tuple2<T,org.apache.flink.table.data.RowData>>

rangeShuffleByKey(org.apache.flink.streaming.api.datastream.DataStream<org.apache.flink.api.java.tuple.Tuple2<T,org.apache.flink.table.data.RowData>> inputDataStream, SerializableSupplier<Comparator<T>> keyComparator, org.apache.flink.api.common.typeinfo.TypeInformation<T> keyTypeInformation, int localSampleSize, int globalSampleSize, int rangeNum, int outParallelism, RowType valueRowType, boolean isSortBySize) The RelNode with range-partition distribution will create the following transformations.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

RangeShuffle

public RangeShuffle()

Method Detail

rangeShuffleByKey

public static <T> org.apache.flink.streaming.api.datastream.DataStream<org.apache.flink.api.java.tuple.Tuple2<T,org.apache.flink.table.data.RowData>> rangeShuffleByKey(org.apache.flink.streaming.api.datastream.DataStream<org.apache.flink.api.java.tuple.Tuple2<T,org.apache.flink.table.data.RowData>> inputDataStream,
                                                                                                                                                                        SerializableSupplier<Comparator<T>> keyComparator,
                                                                                                                                                                        org.apache.flink.api.common.typeinfo.TypeInformation<T> keyTypeInformation,
                                                                                                                                                                        int localSampleSize,
                                                                                                                                                                        int globalSampleSize,
                                                                                                                                                                        int rangeNum,
                                                                                                                                                                        int outParallelism,
                                                                                                                                                                        RowType valueRowType,
                                                                                                                                                                        boolean isSortBySize)

The RelNode with range-partition distribution will create the following transformations.

Explanation of the following figure: "[LSample, n]" means operator is LSample and parallelism is n, "LSample" means LocalSampleOperator, "GSample" means GlobalSampleOperator, "ARange" means AssignRangeId, "RRange" means RemoveRangeId.

 [IN,n]->[LSample,n]->[GSample,1]-BROADCAST
    \                                    \
     -----------------------------BATCH-[ARange,n]-PARTITION->[RRange,m]->
 

The streams except the sample and histogram process stream will be blocked, so the sample and histogram process stream does not care about requiredExchangeMode.