org.apache.paimon.flink.sorter

Class SortUtils

java.lang.Object

org.apache.paimon.flink.sorter.SortUtils

public class SortUtils
extends Object

This is a table sorter which will sort the records by the key of the shuffleKeyComparator generates. It is a global sort method, we will shuffle the input stream through shuffleKey. After sorted, we convert datastream from paimon RowData back to Flink RowData

                         toPaimonDataStream                         add key column                                  range shuffle by key                                    local sort                                              remove key
 DataStream[RowData] --------------------> DataStream[PaimonRowData] -------------------> DataStream[PaimonRowData] -------------------------> DataStream[PaimonRowData] -----------------------> DataStream[PaimonRowData sorted] ---------------------> DataStream[RowData sorted]
                                                                                                                                                                                                                                   back to flink RowData
 

Constructor Summary

Constructors

Constructor and Description
SortUtils()

Method Summary

Modifier and Type

Method and Description

static <KEY> org.apache.flink.streaming.api.datastream.DataStream<org.apache.flink.table.data.RowData>

sortStreamByKey(org.apache.flink.streaming.api.datastream.DataStream<org.apache.flink.table.data.RowData> inputStream, FileStoreTable table, RowType sortKeyType, org.apache.flink.api.common.typeinfo.TypeInformation<KEY> keyTypeInformation, SerializableSupplier<Comparator<KEY>> shuffleKeyComparator, org.apache.paimon.flink.sorter.SortUtils.KeyAbstract<KEY> shuffleKeyAbstract, org.apache.paimon.flink.sorter.SortUtils.ShuffleKeyConvertor<KEY> convertor, TableSortInfo tableSortInfo) Sort the input stream by the key specified.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SortUtils

public SortUtils()

Method Detail

sortStreamByKey

public static <KEY> org.apache.flink.streaming.api.datastream.DataStream<org.apache.flink.table.data.RowData> sortStreamByKey(org.apache.flink.streaming.api.datastream.DataStream<org.apache.flink.table.data.RowData> inputStream,
                                                                                                                              FileStoreTable table,
                                                                                                                              RowType sortKeyType,
                                                                                                                              org.apache.flink.api.common.typeinfo.TypeInformation<KEY> keyTypeInformation,
                                                                                                                              SerializableSupplier<Comparator<KEY>> shuffleKeyComparator,
                                                                                                                              org.apache.paimon.flink.sorter.SortUtils.KeyAbstract<KEY> shuffleKeyAbstract,
                                                                                                                              org.apache.paimon.flink.sorter.SortUtils.ShuffleKeyConvertor<KEY> convertor,
                                                                                                                              TableSortInfo tableSortInfo)

Sort the input stream by the key specified.

Type Parameters:

KEY - the KEY type in range shuffle

Parameters:

inputStream - the input data stream

table - the sorted file store table

sortKeyType - we will use paimon `BinaryExternalSortBuffer` to local sort, so we need to specify the key type.

keyTypeInformation - we will use range shuffle in global sort, so we need to range shuffle by the key first.

shuffleKeyComparator - comparator to compare the key when shuffle

shuffleKeyAbstract - abstract the key from the input `RowData`

convertor - convert the `KEY` to the sort key, then we can sort in `BinaryExternalSortBuffer`.

tableSortInfo - the necessary info of table sort.

Returns:

the global sorted data stream