org.apache.paimon.sort

Class AbstractBinaryExternalMerger<Entry>

java.lang.Object

org.apache.paimon.sort.AbstractBinaryExternalMerger<Entry>

Type Parameters:

Entry - Type of Entry to Merge sort.

All Implemented Interfaces:

Closeable, AutoCloseable

Direct Known Subclasses:

BinaryExternalMerger

public abstract class AbstractBinaryExternalMerger<Entry>
extends Object
implements Closeable

Spilled files Merger of BinaryExternalSortBuffer. It merges maxFanIn spilled files at most once.

Field Summary

Fields

Modifier and Type	Field and Description
protected IOManager	ioManager
protected int	pageSize

Constructor Summary

Constructors

Constructor and Description
AbstractBinaryExternalMerger(IOManager ioManager, int pageSize, int maxFanIn, SpillChannelManager channelManager, BlockCompressionFactory compressionCodecFactory, int compressionBlockSize)

Method Summary

Modifier and Type	Method and Description
protected abstract MutableObjectIterator<Entry>	channelReaderInputViewIterator(ChannelReaderInputView inView)
void	close()
BinaryMergeIterator<Entry>	getMergingIterator(List<ChannelWithMeta> channelIDs, List<FileIOChannel> openChannels) Returns an iterator that iterates over the merged result from all given channels.
List<ChannelWithMeta>	mergeChannelList(List<ChannelWithMeta> channelIDs) Merges the given sorted runs to a smaller number of sorted runs.
protected abstract Comparator<Entry>	mergeComparator()
protected abstract List<Entry>	mergeReusedEntries(int size)
protected abstract void	writeMergingOutput(MutableObjectIterator<Entry> mergeIterator, AbstractPagedOutputView output) read the merged stream and write the data back.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

pageSize

protected final int pageSize

ioManager

protected final IOManager ioManager

Constructor Detail

AbstractBinaryExternalMerger

public AbstractBinaryExternalMerger(IOManager ioManager,
                                    int pageSize,
                                    int maxFanIn,
                                    SpillChannelManager channelManager,
                                    BlockCompressionFactory compressionCodecFactory,
                                    int compressionBlockSize)

Method Detail

close

public void close()

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

getMergingIterator

public BinaryMergeIterator<Entry> getMergingIterator(List<ChannelWithMeta> channelIDs,
                                                     List<FileIOChannel> openChannels)
                                              throws IOException

Returns an iterator that iterates over the merged result from all given channels.

Parameters:

channelIDs - The channels that are to be merged and returned.

Returns:

An iterator over the merged records of the input channels.

Throws:

IOException - Thrown, if the readers encounter an I/O problem.

mergeChannelList

public List<ChannelWithMeta> mergeChannelList(List<ChannelWithMeta> channelIDs)
                                       throws IOException

Merges the given sorted runs to a smaller number of sorted runs.

Parameters:

channelIDs - The IDs of the sorted runs that need to be merged.

Returns:

A list of the IDs of the merged channels.

Throws:

IOException - Thrown, if the readers or writers encountered an I/O problem.

channelReaderInputViewIterator

protected abstract MutableObjectIterator<Entry> channelReaderInputViewIterator(ChannelReaderInputView inView)

Returns:

entry iterator reading from inView.

mergeComparator

protected abstract Comparator<Entry> mergeComparator()

Returns:

merging comparator used in merging.

mergeReusedEntries

protected abstract List<Entry> mergeReusedEntries(int size)

Returns:

reused entry object used in merging.

writeMergingOutput

protected abstract void writeMergingOutput(MutableObjectIterator<Entry> mergeIterator,
                                           AbstractPagedOutputView output)
                                    throws IOException

read the merged stream and write the data back.

Throws:

IOException