org.apache.paimon.append

Class AppendOnlyWriter

java.lang.Object

org.apache.paimon.append.AppendOnlyWriter

All Implemented Interfaces:

MemoryOwner, BatchRecordWriter, RecordWriter<InternalRow>

public class AppendOnlyWriter
extends Object
implements BatchRecordWriter, MemoryOwner

A RecordWriter implementation that only accepts records which are always insert operations and don't have any unique keys or sort keys.

Constructor Summary

Constructors

Constructor and Description

AppendOnlyWriter(FileIO fileIO, IOManager ioManager, long schemaId, FileFormat fileFormat, long targetFileSize, RowType writeSchema, long maxSequenceNumber, CompactManager compactManager, IOFunction<List<DataFileMeta>,RecordReaderIterator<InternalRow>> bucketFileRead, boolean forceCompact, DataFilePathFactory pathFactory, CommitIncrement increment, boolean useWriteBuffer, boolean spillable, String fileCompression, CompressOptions spillCompression, SimpleColStatsCollector.Factory[] statsCollectors, MemorySize maxDiskSize, FileIndexOptions fileIndexOptions, boolean asyncFileWrite)

Method Summary

Modifier and Type	Method and Description
void	addNewFiles(List<DataFileMeta> files) Add files to the internal CompactManager.
void	close() Close this writer, the call will delete newly generated but not committed files.
void	compact(boolean fullCompaction) Compact files related to the writer.
Collection<DataFileMeta>	dataFiles() Get all data files maintained by this writer.
void	flushMemory() Flush memory of owner, release memory.
RowBuffer	getWriteBuffer()
boolean	isCompacting() Check if a compaction is in progress, or if a compaction result remains to be fetched.
long	maxSequenceNumber() Get max sequence number of records written by this writer.
long	memoryOccupancy() Memory occupancy size of this owner.
CommitIncrement	prepareCommit(boolean waitCompaction) Prepare for a commit.
void	setMemoryPool(MemorySegmentPool memoryPool) Set MemorySegmentPool for the owner.
void	sync() Sync the writer.
void	toBufferedWriter()
void	withInsertOnly(boolean insertOnly) This method is called when the insert only status of the records changes.
void	write(InternalRow rowData) Add a key-value element to the writer.
void	writeBundle(BundleRecords bundle) Add a batch elemens to the writer.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

AppendOnlyWriter

public AppendOnlyWriter(FileIO fileIO,
                        @Nullable
                        IOManager ioManager,
                        long schemaId,
                        FileFormat fileFormat,
                        long targetFileSize,
                        RowType writeSchema,
                        long maxSequenceNumber,
                        CompactManager compactManager,
                        IOFunction<List<DataFileMeta>,RecordReaderIterator<InternalRow>> bucketFileRead,
                        boolean forceCompact,
                        DataFilePathFactory pathFactory,
                        @Nullable
                        CommitIncrement increment,
                        boolean useWriteBuffer,
                        boolean spillable,
                        String fileCompression,
                        CompressOptions spillCompression,
                        SimpleColStatsCollector.Factory[] statsCollectors,
                        MemorySize maxDiskSize,
                        FileIndexOptions fileIndexOptions,
                        boolean asyncFileWrite)

Method Detail

write

public void write(InternalRow rowData)
           throws Exception

Description copied from interface: RecordWriter

Add a key-value element to the writer.

Specified by:

write in interface RecordWriter<InternalRow>

Throws:

Exception

writeBundle

public void writeBundle(BundleRecords bundle)
                 throws Exception

Description copied from interface: BatchRecordWriter

Add a batch elemens to the writer.

Specified by:

writeBundle in interface BatchRecordWriter

Throws:

Exception

compact

public void compact(boolean fullCompaction)
             throws Exception

Description copied from interface: RecordWriter

Compact files related to the writer. Note that compaction process is only submitted and may not be completed when the method returns.

Specified by:

compact in interface RecordWriter<InternalRow>

Parameters:

fullCompaction - whether to trigger full compaction or just normal compaction

Throws:

Exception

addNewFiles

public void addNewFiles(List<DataFileMeta> files)

Description copied from interface: RecordWriter

Add files to the internal CompactManager.

Specified by:

addNewFiles in interface RecordWriter<InternalRow>

Parameters:

files - files to add

dataFiles

public Collection<DataFileMeta> dataFiles()

Description copied from interface: RecordWriter

Get all data files maintained by this writer.

Specified by:

dataFiles in interface RecordWriter<InternalRow>

maxSequenceNumber

public long maxSequenceNumber()

Description copied from interface: RecordWriter

Get max sequence number of records written by this writer.

Specified by:

maxSequenceNumber in interface RecordWriter<InternalRow>

prepareCommit

public CommitIncrement prepareCommit(boolean waitCompaction)
                              throws Exception

Description copied from interface: RecordWriter

Prepare for a commit.

Specified by:

prepareCommit in interface RecordWriter<InternalRow>

Parameters:

waitCompaction - if this method need to wait for current compaction to complete

Returns:

Incremental files in this snapshot cycle

Throws:

Exception

isCompacting

public boolean isCompacting()

Description copied from interface: RecordWriter

Check if a compaction is in progress, or if a compaction result remains to be fetched.

Specified by:

isCompacting in interface RecordWriter<InternalRow>

sync

public void sync()
          throws Exception

Description copied from interface: RecordWriter

Sync the writer. The structure related to file reading and writing is thread unsafe, there are asynchronous threads inside the writer, which should be synced before reading data.

Specified by:

sync in interface RecordWriter<InternalRow>

Throws:

Exception

withInsertOnly

public void withInsertOnly(boolean insertOnly)

Description copied from interface: RecordWriter

This method is called when the insert only status of the records changes.

Specified by:

withInsertOnly in interface RecordWriter<InternalRow>

Parameters:

insertOnly - If true, all the following records would be of RowKind.INSERT, and no two records would have the same primary key.

close

public void close()
           throws Exception

Description copied from interface: RecordWriter

Close this writer, the call will delete newly generated but not committed files.

Specified by:

close in interface RecordWriter<InternalRow>

Throws:

Exception

toBufferedWriter

public void toBufferedWriter()
                      throws Exception

Throws:

Exception

setMemoryPool

public void setMemoryPool(MemorySegmentPool memoryPool)

Description copied from interface: MemoryOwner

Set MemorySegmentPool for the owner.

Specified by:

setMemoryPool in interface MemoryOwner

memoryOccupancy

public long memoryOccupancy()

Description copied from interface: MemoryOwner

Memory occupancy size of this owner.

Specified by:

memoryOccupancy in interface MemoryOwner

flushMemory

public void flushMemory()
                 throws Exception

Description copied from interface: MemoryOwner

Flush memory of owner, release memory.

Specified by:

flushMemory in interface MemoryOwner

Throws:

Exception

getWriteBuffer

@VisibleForTesting
public RowBuffer getWriteBuffer()