org.apache.paimon.table.sink

Class TableWriteImpl<T>

java.lang.Object

org.apache.paimon.table.sink.TableWriteImpl<T>

Type Parameters:

T - type of record to write into FileStore.

All Implemented Interfaces:

AutoCloseable, BatchTableWrite, InnerTableWrite, StreamTableWrite, TableWrite, Restorable<List<FileStoreWrite.State<T>>>

public class TableWriteImpl<T>
extends Object
implements InnerTableWrite, Restorable<List<FileStoreWrite.State<T>>>

TableWrite implementation.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	TableWriteImpl.RecordExtractor<T> Extractor to extract T from the SinkRecord.

Constructor Summary

Constructors

Constructor and Description
TableWriteImpl(RowType rowType, FileStoreWrite<T> write, KeyAndBucketExtractor<InternalRow> keyAndBucketExtractor, TableWriteImpl.RecordExtractor<T> recordExtractor, RowKindGenerator rowKindGenerator, boolean ignoreDelete)

Method Summary

Modifier and Type	Method and Description
List<FileStoreWrite.State<T>>	checkpoint() Extract state of the current operation instance.
void	close()
void	compact(BinaryRow partition, int bucket, boolean fullCompaction) Compact a bucket of a partition.
int	getBucket(InternalRow row) Calculate which bucket row belongs to.
BinaryRow	getPartition(InternalRow row) Calculate which partition row belongs to.
FileStoreWrite<T>	getWrite()
void	notifyNewFiles(long snapshotId, BinaryRow partition, int bucket, List<DataFileMeta> files) Notify that some new files are created at given snapshot in given bucket.
List<CommitMessage>	prepareCommit() Prepare commit for TableCommit.
List<CommitMessage>	prepareCommit(boolean waitCompaction, long commitIdentifier) Prepare commit for TableCommit.
void	restore(List<FileStoreWrite.State<T>> state) Restore state of a previous operation instance into the current operation instance.
SinkRecord	toLogRecord(SinkRecord record)
TableWriteImpl<T>	withBucketMode(BucketMode bucketMode)
TableWriteImpl<T>	withCompactExecutor(ExecutorService compactExecutor)
TableWriteImpl<T>	withExecutionMode(boolean isStreamingMode)
TableWriteImpl<T>	withIgnorePreviousFiles(boolean ignorePreviousFiles)
void	withInsertOnly(boolean insertOnly) This method is called when the insert only status of the records changes.
TableWriteImpl<T>	withIOManager(IOManager ioManager) With IOManager, this is needed if 'write-buffer-spillable' is set to true.
TableWriteImpl<T>	withMemoryPool(MemorySegmentPool memoryPool) With MemorySegmentPool for the current table write.
TableWriteImpl<T>	withMemoryPoolFactory(MemoryPoolFactory memoryPoolFactory)
TableWriteImpl<T>	withMetricRegistry(MetricRegistry metricRegistry) With metrics to measure compaction.
void	write(InternalRow row) Write a row to the writer.
void	write(InternalRow row, int bucket) Write a row with bucket.
SinkRecord	writeAndReturn(InternalRow row)
SinkRecord	writeAndReturn(InternalRow row, int bucket)
void	writeBundle(BinaryRow partition, int bucket, BundleRecords bundle) Write a bundle records directly, not per row.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TableWriteImpl

public TableWriteImpl(RowType rowType,
                      FileStoreWrite<T> write,
                      KeyAndBucketExtractor<InternalRow> keyAndBucketExtractor,
                      TableWriteImpl.RecordExtractor<T> recordExtractor,
                      @Nullable
                      RowKindGenerator rowKindGenerator,
                      boolean ignoreDelete)

Method Detail

withIgnorePreviousFiles

public TableWriteImpl<T> withIgnorePreviousFiles(boolean ignorePreviousFiles)

Specified by:

withIgnorePreviousFiles in interface InnerTableWrite

withExecutionMode

public TableWriteImpl<T> withExecutionMode(boolean isStreamingMode)

Specified by:

withExecutionMode in interface InnerTableWrite

withIOManager

public TableWriteImpl<T> withIOManager(IOManager ioManager)

Description copied from interface: TableWrite

With IOManager, this is needed if 'write-buffer-spillable' is set to true.

Specified by:

withIOManager in interface TableWrite

withMemoryPool

public TableWriteImpl<T> withMemoryPool(MemorySegmentPool memoryPool)

Description copied from interface: TableWrite

With MemorySegmentPool for the current table write.

Specified by:

withMemoryPool in interface TableWrite

withMemoryPoolFactory

public TableWriteImpl<T> withMemoryPoolFactory(MemoryPoolFactory memoryPoolFactory)

withCompactExecutor

public TableWriteImpl<T> withCompactExecutor(ExecutorService compactExecutor)

withBucketMode

public TableWriteImpl<T> withBucketMode(BucketMode bucketMode)

withInsertOnly

public void withInsertOnly(boolean insertOnly)

Description copied from interface: TableWrite

This method is called when the insert only status of the records changes.

Specified by:

withInsertOnly in interface TableWrite

Parameters:

insertOnly - If true, all the following records would be of RowKind.INSERT, and no two records would have the same primary key.

getPartition

public BinaryRow getPartition(InternalRow row)

Description copied from interface: TableWrite

Calculate which partition row belongs to.

Specified by:

getPartition in interface TableWrite

getBucket

public int getBucket(InternalRow row)

Description copied from interface: TableWrite

Calculate which bucket row belongs to.

Specified by:

getBucket in interface TableWrite

write

public void write(InternalRow row)
           throws Exception

Description copied from interface: TableWrite

Write a row to the writer.

Specified by:

write in interface TableWrite

Throws:

Exception

write

public void write(InternalRow row,
                  int bucket)
           throws Exception

Description copied from interface: TableWrite

Write a row with bucket.

Specified by:

write in interface TableWrite

Throws:

Exception

writeBundle

public void writeBundle(BinaryRow partition,
                        int bucket,
                        BundleRecords bundle)
                 throws Exception

Description copied from interface: TableWrite

Write a bundle records directly, not per row.

Specified by:

writeBundle in interface TableWrite

Throws:

Exception

writeAndReturn

@Nullable
public SinkRecord writeAndReturn(InternalRow row)
                                    throws Exception

Throws:

Exception

writeAndReturn

@Nullable
public SinkRecord writeAndReturn(InternalRow row,
                                           int bucket)
                                    throws Exception

Throws:

Exception

toLogRecord

public SinkRecord toLogRecord(SinkRecord record)

compact

public void compact(BinaryRow partition,
                    int bucket,
                    boolean fullCompaction)
             throws Exception

Description copied from interface: TableWrite

Compact a bucket of a partition. By default, it will determine whether to perform the compaction according to the 'num-sorted-run.compaction-trigger' option. If fullCompaction is true, it will force a full compaction, which is expensive.

NOTE: In Java API, full compaction is not automatically executed. If you set 'changelog-producer' to 'full-compaction', please execute this method regularly to produce changelog.

Specified by:

compact in interface TableWrite

Throws:

Exception

withMetricRegistry

public TableWriteImpl<T> withMetricRegistry(MetricRegistry metricRegistry)

Description copied from interface: TableWrite

With metrics to measure compaction.

Specified by:

withMetricRegistry in interface TableWrite

notifyNewFiles

public void notifyNewFiles(long snapshotId,
                           BinaryRow partition,
                           int bucket,
                           List<DataFileMeta> files)

Notify that some new files are created at given snapshot in given bucket.

Most probably, these files are created by another job. Currently this method is only used by the dedicated compact job to see files created by writer jobs.

prepareCommit

public List<CommitMessage> prepareCommit(boolean waitCompaction,
                                         long commitIdentifier)
                                  throws Exception

Description copied from interface: StreamTableWrite

Prepare commit for TableCommit. Collect incremental files for this write.

Specified by:

prepareCommit in interface StreamTableWrite

Parameters:

waitCompaction - whether to wait for the end of the background compaction.

commitIdentifier - Committed transaction ID, can start from 0. If there are multiple commits, please increment this ID.

Throws:

Exception

See Also:

StreamTableCommit.commit(long, java.util.List<org.apache.paimon.table.sink.CommitMessage>)

prepareCommit

public List<CommitMessage> prepareCommit()
                                  throws Exception

Description copied from interface: BatchTableWrite

Prepare commit for TableCommit. Collect incremental files for this write.

Specified by:

prepareCommit in interface BatchTableWrite

Throws:

Exception

See Also:

BatchTableCommit.commit(java.util.List<org.apache.paimon.table.sink.CommitMessage>)

close

public void close()
           throws Exception

Specified by:

close in interface AutoCloseable

Throws:

Exception

checkpoint

public List<FileStoreWrite.State<T>> checkpoint()

Description copied from interface: Restorable

Extract state of the current operation instance.

Specified by:

checkpoint in interface Restorable<List<FileStoreWrite.State<T>>>

restore

public void restore(List<FileStoreWrite.State<T>> state)

Description copied from interface: Restorable

Restore state of a previous operation instance into the current operation instance.

Specified by:

restore in interface Restorable<List<FileStoreWrite.State<T>>>

getWrite

@VisibleForTesting
public FileStoreWrite<T> getWrite()