org.apache.paimon.utils

Interface RecordWriter<T>

Type Parameters:

T - type of record to write.

All Known Subinterfaces:

BatchRecordWriter

All Known Implementing Classes:

AppendOnlyWriter, MergeTreeWriter, PostponeBucketWriter

public interface RecordWriter<T>

The RecordWriter is responsible for writing data and handling in-progress files used to write yet un-staged data. The incremental files ready to commit is returned to the system by the prepareCommit(boolean).

Method Summary

Modifier and Type	Method and Description
void	addNewFiles(List<DataFileMeta> files) Add files to the internal CompactManager.
void	close() Close this writer, the call will delete newly generated but not committed files.
void	compact(boolean fullCompaction) Compact files related to the writer.
boolean	compactNotCompleted() Check if a compaction is in progress, or if a compaction result remains to be fetched, or if a compaction should be triggered later.
Collection<DataFileMeta>	dataFiles() Get all data files maintained by this writer.
long	maxSequenceNumber() Get max sequence number of records written by this writer.
CommitIncrement	prepareCommit(boolean waitCompaction) Prepare for a commit.
void	sync() Sync the writer.
void	withInsertOnly(boolean insertOnly) This method is called when the insert only status of the records changes.
void	write(T record) Add a key-value element to the writer.

Method Detail

write

void write(T record)
    throws Exception

Add a key-value element to the writer.

Throws:

Exception

compact

void compact(boolean fullCompaction)
      throws Exception

Compact files related to the writer. Note that compaction process is only submitted and may not be completed when the method returns.

Parameters:

fullCompaction - whether to trigger full compaction or just normal compaction

Throws:

Exception

addNewFiles

void addNewFiles(List<DataFileMeta> files)

Add files to the internal CompactManager.

Parameters:

files - files to add

dataFiles

Collection<DataFileMeta> dataFiles()

Get all data files maintained by this writer.

maxSequenceNumber

long maxSequenceNumber()

Get max sequence number of records written by this writer.

prepareCommit

CommitIncrement prepareCommit(boolean waitCompaction)
                       throws Exception

Prepare for a commit.

Parameters:

waitCompaction - if this method need to wait for current compaction to complete

Returns:

Incremental files in this snapshot cycle

Throws:

Exception

compactNotCompleted

boolean compactNotCompleted()

Check if a compaction is in progress, or if a compaction result remains to be fetched, or if a compaction should be triggered later.

sync

void sync()
   throws Exception

Sync the writer. The structure related to file reading and writing is thread unsafe, there are asynchronous threads inside the writer, which should be synced before reading data.

Throws:

Exception

withInsertOnly

void withInsertOnly(boolean insertOnly)

This method is called when the insert only status of the records changes.

Parameters:

insertOnly - If true, all the following records would be of RowKind.INSERT, and no two records would have the same primary key.

close

void close()
    throws Exception

Close this writer, the call will delete newly generated but not committed files.

Throws:

Exception