org.apache.paimon.mergetree

Class MergeTreeWriter

java.lang.Object

org.apache.paimon.mergetree.MergeTreeWriter

All Implemented Interfaces:

MemoryOwner, RecordWriter<KeyValue>

public class MergeTreeWriter
extends Object
implements RecordWriter<KeyValue>, MemoryOwner

A RecordWriter to write records and generate CompactIncrement.

Constructor Summary

Constructors

Constructor and Description

MergeTreeWriter(boolean writeBufferSpillable, MemorySize maxDiskSize, int sortMaxFan, CompressOptions sortCompression, IOManager ioManager, CompactManager compactManager, long maxSequenceNumber, Comparator<InternalRow> keyComparator, MergeFunction<KeyValue> mergeFunction, KeyValueFileWriterFactory writerFactory, boolean commitForceCompact, CoreOptions.ChangelogProducer changelogProducer, CommitIncrement increment, FieldsComparator userDefinedSeqComparator)

Method Summary

Modifier and Type	Method and Description
void	addNewFiles(List<DataFileMeta> files) Add files to the internal CompactManager.
void	close() Close this writer, the call will delete newly generated but not committed files.
void	compact(boolean fullCompaction) Compact files related to the writer.
Collection<DataFileMeta>	dataFiles() Get all data files maintained by this writer.
void	flushMemory() Flush memory of owner, release memory.
boolean	isCompacting() Check if a compaction is in progress, or if a compaction result remains to be fetched.
long	maxSequenceNumber() Get max sequence number of records written by this writer.
long	memoryOccupancy() Memory occupancy size of this owner.
CommitIncrement	prepareCommit(boolean waitCompaction) Prepare for a commit.
void	setMemoryPool(MemorySegmentPool memoryPool) Set MemorySegmentPool for the owner.
void	sync() Sync the writer.
void	withInsertOnly(boolean insertOnly) This method is called when the insert only status of the records changes.
void	write(KeyValue kv) Add a key-value element to the writer.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

MergeTreeWriter

public MergeTreeWriter(boolean writeBufferSpillable,
                       MemorySize maxDiskSize,
                       int sortMaxFan,
                       CompressOptions sortCompression,
                       IOManager ioManager,
                       CompactManager compactManager,
                       long maxSequenceNumber,
                       Comparator<InternalRow> keyComparator,
                       MergeFunction<KeyValue> mergeFunction,
                       KeyValueFileWriterFactory writerFactory,
                       boolean commitForceCompact,
                       CoreOptions.ChangelogProducer changelogProducer,
                       @Nullable
                       CommitIncrement increment,
                       @Nullable
                       FieldsComparator userDefinedSeqComparator)

Method Detail

setMemoryPool

public void setMemoryPool(MemorySegmentPool memoryPool)

Description copied from interface: MemoryOwner

Set MemorySegmentPool for the owner.

Specified by:

setMemoryPool in interface MemoryOwner

write

public void write(KeyValue kv)
           throws Exception

Description copied from interface: RecordWriter

Add a key-value element to the writer.

Specified by:

write in interface RecordWriter<KeyValue>

Throws:

Exception

compact

public void compact(boolean fullCompaction)
             throws Exception

Description copied from interface: RecordWriter

Compact files related to the writer. Note that compaction process is only submitted and may not be completed when the method returns.

Specified by:

compact in interface RecordWriter<KeyValue>

Parameters:

fullCompaction - whether to trigger full compaction or just normal compaction

Throws:

Exception

addNewFiles

public void addNewFiles(List<DataFileMeta> files)

Description copied from interface: RecordWriter

Add files to the internal CompactManager.

Specified by:

addNewFiles in interface RecordWriter<KeyValue>

Parameters:

files - files to add

dataFiles

public Collection<DataFileMeta> dataFiles()

Description copied from interface: RecordWriter

Get all data files maintained by this writer.

Specified by:

dataFiles in interface RecordWriter<KeyValue>

maxSequenceNumber

public long maxSequenceNumber()

Description copied from interface: RecordWriter

Get max sequence number of records written by this writer.

Specified by:

maxSequenceNumber in interface RecordWriter<KeyValue>

memoryOccupancy

public long memoryOccupancy()

Description copied from interface: MemoryOwner

Memory occupancy size of this owner.

Specified by:

memoryOccupancy in interface MemoryOwner

flushMemory

public void flushMemory()
                 throws Exception

Description copied from interface: MemoryOwner

Flush memory of owner, release memory.

Specified by:

flushMemory in interface MemoryOwner

Throws:

Exception

prepareCommit

public CommitIncrement prepareCommit(boolean waitCompaction)
                              throws Exception

Description copied from interface: RecordWriter

Prepare for a commit.

Specified by:

prepareCommit in interface RecordWriter<KeyValue>

Parameters:

waitCompaction - if this method need to wait for current compaction to complete

Returns:

Incremental files in this snapshot cycle

Throws:

Exception

isCompacting

public boolean isCompacting()

Description copied from interface: RecordWriter

Check if a compaction is in progress, or if a compaction result remains to be fetched.

Specified by:

isCompacting in interface RecordWriter<KeyValue>

sync

public void sync()
          throws Exception

Description copied from interface: RecordWriter

Sync the writer. The structure related to file reading and writing is thread unsafe, there are asynchronous threads inside the writer, which should be synced before reading data.

Specified by:

sync in interface RecordWriter<KeyValue>

Throws:

Exception

withInsertOnly

public void withInsertOnly(boolean insertOnly)

Description copied from interface: RecordWriter

This method is called when the insert only status of the records changes.

Specified by:

withInsertOnly in interface RecordWriter<KeyValue>

Parameters:

insertOnly - If true, all the following records would be of RowKind.INSERT, and no two records would have the same primary key.

close

public void close()
           throws Exception

Description copied from interface: RecordWriter

Close this writer, the call will delete newly generated but not committed files.

Specified by:

close in interface RecordWriter<KeyValue>

Throws:

Exception