org.apache.paimon.operation

Interface FileStoreWrite<T>

Type Parameters:

T - type of record to write.

All Superinterfaces:

Restorable<List<FileStoreWrite.State<T>>>

All Known Subinterfaces:

BundleFileStoreWriter

All Known Implementing Classes:

AbstractFileStoreWrite, AppendOnlyFileStoreWrite, AppendOnlyFixedBucketFileStoreWrite, AppendOnlyUnawareBucketFileStoreWrite, KeyValueFileStoreWrite, MemoryFileStoreWrite

public interface FileStoreWrite<T>
extends Restorable<List<FileStoreWrite.State<T>>>

Write operation which provides RecordWriter creation and writes SinkRecord to FileStore.

Nested Class Summary

Nested Classes

Modifier and Type	Interface and Description
static class	FileStoreWrite.State<T> Recoverable state of FileStoreWrite.

Method Summary

Modifier and Type	Method and Description
void	close() Close the writer.
void	compact(BinaryRow partition, int bucket, boolean fullCompaction) Compact data stored in given partition and bucket.
void	notifyNewFiles(long snapshotId, BinaryRow partition, int bucket, List<DataFileMeta> files) Notify that some new files are created at given snapshot in given bucket.
List<CommitMessage>	prepareCommit(boolean waitCompaction, long commitIdentifier) Prepare commit in the write.
void	withCompactExecutor(ExecutorService compactExecutor)
void	withExecutionMode(boolean isStreamingMode) We detect whether it is in batch mode, if so, we do some optimization.
void	withIgnorePreviousFiles(boolean ignorePreviousFiles) Set whether the write operation should ignore previously stored files.
void	withInsertOnly(boolean insertOnly) This method is called when the insert only status of the records changes.
FileStoreWrite<T>	withIOManager(IOManager ioManager)
default FileStoreWrite<T>	withMemoryPool(MemorySegmentPool memoryPool) With memory pool for the current file store write.
FileStoreWrite<T>	withMemoryPoolFactory(MemoryPoolFactory memoryPoolFactory) With memory pool factory for the current file store write.
FileStoreWrite<T>	withMetricRegistry(MetricRegistry metricRegistry) With metrics to measure compaction.
void	write(BinaryRow partition, int bucket, T data) Write the data to the store according to the partition and bucket.

Methods inherited from interface org.apache.paimon.utils.Restorable

checkpoint, restore

Method Detail

withIOManager

FileStoreWrite<T> withIOManager(IOManager ioManager)

withMemoryPool

default FileStoreWrite<T> withMemoryPool(MemorySegmentPool memoryPool)

With memory pool for the current file store write.

Parameters:

memoryPool - the given memory pool.

withMemoryPoolFactory

FileStoreWrite<T> withMemoryPoolFactory(MemoryPoolFactory memoryPoolFactory)

With memory pool factory for the current file store write.

Parameters:

memoryPoolFactory - the given memory pool factory.

withIgnorePreviousFiles

void withIgnorePreviousFiles(boolean ignorePreviousFiles)

Set whether the write operation should ignore previously stored files.

Parameters:

ignorePreviousFiles - whether the write operation should ignore previously stored files.

withExecutionMode

void withExecutionMode(boolean isStreamingMode)

We detect whether it is in batch mode, if so, we do some optimization.

Parameters:

isStreamingMode - whether in streaming mode

withMetricRegistry

FileStoreWrite<T> withMetricRegistry(MetricRegistry metricRegistry)

With metrics to measure compaction.

withCompactExecutor

void withCompactExecutor(ExecutorService compactExecutor)

withInsertOnly

void withInsertOnly(boolean insertOnly)

This method is called when the insert only status of the records changes.

Parameters:

insertOnly - If true, all the following records would be of RowKind.INSERT, and no two records would have the same primary key.

write

void write(BinaryRow partition,
           int bucket,
           T data)
    throws Exception

Write the data to the store according to the partition and bucket.

Parameters:

partition - the partition of the data

bucket - the bucket id of the data

data - the given data

Throws:

Exception - the thrown exception when writing the record

compact

void compact(BinaryRow partition,
             int bucket,
             boolean fullCompaction)
      throws Exception

Compact data stored in given partition and bucket. Note that compaction process is only submitted and may not be completed when the method returns.

Parameters:

partition - the partition to compact

bucket - the bucket to compact

fullCompaction - whether to trigger full compaction or just normal compaction

Throws:

Exception - the thrown exception when compacting the records

notifyNewFiles

void notifyNewFiles(long snapshotId,
                    BinaryRow partition,
                    int bucket,
                    List<DataFileMeta> files)

Notify that some new files are created at given snapshot in given bucket.

Most probably, these files are created by another job. Currently this method is only used by the dedicated compact job to see files created by writer jobs.

Parameters:

snapshotId - the snapshot id where new files are created

partition - the partition where new files are created

bucket - the bucket where new files are created

files - the new files themselves

prepareCommit

List<CommitMessage> prepareCommit(boolean waitCompaction,
                                  long commitIdentifier)
                           throws Exception

Prepare commit in the write.

Parameters:

waitCompaction - if this method need to wait for current compaction to complete

commitIdentifier - identifier of the commit being prepared

Returns:

the file committable list

Throws:

Exception - the thrown exception

close

void close()
    throws Exception

Close the writer.

Throws:

Exception - the thrown exception