org.apache.paimon.mergetree.compact

Class ChangelogMergeTreeRewriter

java.lang.Object

org.apache.paimon.mergetree.compact.AbstractCompactRewriter

org.apache.paimon.mergetree.compact.MergeTreeCompactRewriter

org.apache.paimon.mergetree.compact.ChangelogMergeTreeRewriter

All Implemented Interfaces:

Closeable, AutoCloseable, CompactRewriter

Direct Known Subclasses:

FullChangelogMergeTreeCompactRewriter, LookupMergeTreeCompactRewriter

public abstract class ChangelogMergeTreeRewriter
extends MergeTreeCompactRewriter

A MergeTreeCompactRewriter which produces changelog files while performing compaction.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	ChangelogMergeTreeRewriter.UpgradeStrategy Strategy for upgrade.

Field Summary

Fields

Modifier and Type	Field and Description
protected int	maxLevel
protected CoreOptions.MergeEngine	mergeEngine

Fields inherited from class org.apache.paimon.mergetree.compact.MergeTreeCompactRewriter

keyComparator, mergeSorter, mfFactory, readerFactory, userDefinedSeqComparator, writerFactory

Constructor Summary

Constructors

Constructor and Description
ChangelogMergeTreeRewriter(int maxLevel, CoreOptions.MergeEngine mergeEngine, FileReaderFactory<KeyValue> readerFactory, KeyValueFileWriterFactory writerFactory, Comparator<InternalRow> keyComparator, FieldsComparator userDefinedSeqComparator, MergeFunctionFactory<KeyValue> mfFactory, MergeSorter mergeSorter, boolean produceChangelog, boolean forceDropDelete)

Method Summary

Modifier and Type	Method and Description
protected abstract MergeFunctionWrapper<ChangelogResult>	createMergeWrapper(int outputLevel)
CompactResult	rewrite(int outputLevel, boolean dropDelete, List<List<SortedRun>> sections) Rewrite sections to new level.
protected abstract boolean	rewriteChangelog(int outputLevel, boolean dropDelete, List<List<SortedRun>> sections)
protected boolean	rewriteLookupChangelog(int outputLevel, List<List<SortedRun>> sections)
CompactResult	upgrade(int outputLevel, DataFileMeta file) Upgrade file to new level, usually file data is not rewritten, only the metadata is updated.
protected abstract ChangelogMergeTreeRewriter.UpgradeStrategy	upgradeStrategy(int outputLevel, DataFileMeta file)

Methods inherited from class org.apache.paimon.mergetree.compact.MergeTreeCompactRewriter

notifyRewriteCompactBefore, readerForMergeTree, rewriteCompaction

Methods inherited from class org.apache.paimon.mergetree.compact.AbstractCompactRewriter

close, extractFilesFromSections

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

maxLevel

protected final int maxLevel

mergeEngine

protected final CoreOptions.MergeEngine mergeEngine

Constructor Detail

ChangelogMergeTreeRewriter

public ChangelogMergeTreeRewriter(int maxLevel,
                                  CoreOptions.MergeEngine mergeEngine,
                                  FileReaderFactory<KeyValue> readerFactory,
                                  KeyValueFileWriterFactory writerFactory,
                                  Comparator<InternalRow> keyComparator,
                                  @Nullable
                                  FieldsComparator userDefinedSeqComparator,
                                  MergeFunctionFactory<KeyValue> mfFactory,
                                  MergeSorter mergeSorter,
                                  boolean produceChangelog,
                                  boolean forceDropDelete)

Method Detail

rewriteChangelog

protected abstract boolean rewriteChangelog(int outputLevel,
                                            boolean dropDelete,
                                            List<List<SortedRun>> sections)

upgradeStrategy

protected abstract ChangelogMergeTreeRewriter.UpgradeStrategy upgradeStrategy(int outputLevel,
                                                                              DataFileMeta file)

createMergeWrapper

protected abstract MergeFunctionWrapper<ChangelogResult> createMergeWrapper(int outputLevel)

rewriteLookupChangelog

protected boolean rewriteLookupChangelog(int outputLevel,
                                         List<List<SortedRun>> sections)

rewrite

public CompactResult rewrite(int outputLevel,
                             boolean dropDelete,
                             List<List<SortedRun>> sections)
                      throws Exception

Description copied from interface: CompactRewriter

Rewrite sections to new level.

Specified by:

rewrite in interface CompactRewriter

Overrides:

rewrite in class MergeTreeCompactRewriter

Parameters:

outputLevel - new level

dropDelete - whether to drop the deletion, see MergeTreeCompactManager.triggerCompaction(boolean)

sections - list of sections (section is a list of SortedRuns, and key intervals between sections do not overlap)

Returns:

compaction result

Throws:

Exception - exception

upgrade

public CompactResult upgrade(int outputLevel,
                             DataFileMeta file)
                      throws Exception

Description copied from interface: CompactRewriter

Upgrade file to new level, usually file data is not rewritten, only the metadata is updated. But in some certain scenarios, we must rewrite file too, e.g. ChangelogMergeTreeRewriter

Specified by:

upgrade in interface CompactRewriter

Overrides:

upgrade in class AbstractCompactRewriter

Parameters:

outputLevel - new level

file - file to be updated

Returns:

compaction result

Throws:

Exception - exception