org.apache.paimon.operation

Class AbstractFileStoreWrite<T>

java.lang.Object

org.apache.paimon.operation.AbstractFileStoreWrite<T>

Type Parameters:

T - type of record to write.

All Implemented Interfaces:

FileStoreWrite<T>, Restorable<List<FileStoreWrite.State<T>>>

Direct Known Subclasses:

MemoryFileStoreWrite

public abstract class AbstractFileStoreWrite<T>
extends Object
implements FileStoreWrite<T>

Base FileStoreWrite implementation.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	AbstractFileStoreWrite.WriterContainer<T> RecordWriter with the snapshot id it is created upon and the identifier of its last modified commit.

Nested classes/interfaces inherited from interface org.apache.paimon.operation.FileStoreWrite

FileStoreWrite.State<T>

Field Summary

Fields

Modifier and Type	Field and Description
protected CompactionMetrics	compactionMetrics
protected IOManager	ioManager
protected boolean	isStreamingMode
protected SnapshotManager	snapshotManager
protected String	tableName
protected Map<BinaryRow,Map<Integer,AbstractFileStoreWrite.WriterContainer<T>>>	writers

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	AbstractFileStoreWrite(SnapshotManager snapshotManager, FileStoreScan scan, IndexMaintainer.Factory<T> indexFactory, DeletionVectorsMaintainer.Factory dvMaintainerFactory, String tableName, int totalBuckets, RowType partitionType, int writerNumberMax, boolean legacyPartitionName)

Method Summary

Modifier and Type	Method and Description
List<FileStoreWrite.State<T>>	checkpoint() Extract state of the current operation instance.
void	close() Close the writer.
void	compact(BinaryRow partition, int bucket, boolean fullCompaction) Compact data stored in given partition and bucket.
protected static <T> java.util.function.Function<AbstractFileStoreWrite.WriterContainer<T>,Boolean>	createConflictAwareWriterCleanChecker(String commitUser, SnapshotManager snapshotManager)
protected static <T> java.util.function.Function<AbstractFileStoreWrite.WriterContainer<T>,Boolean>	createNoConflictAwareWriterCleanChecker()
protected abstract RecordWriter<T>	createWriter(Long snapshotId, BinaryRow partition, int bucket, List<DataFileMeta> restoreFiles, long restoredMaxSeqNumber, CommitIncrement restoreIncrement, ExecutorService compactExecutor, DeletionVectorsMaintainer deletionVectorsMaintainer)
protected abstract java.util.function.Function<AbstractFileStoreWrite.WriterContainer<T>,Boolean>	createWriterCleanChecker()
AbstractFileStoreWrite.WriterContainer<T>	createWriterContainer(BinaryRow partition, int bucket, boolean ignorePreviousFiles)
protected void	forceBufferSpill()
Map<BinaryRow,List<Integer>>	getActiveBuckets()
ExecutorService	getCompactExecutor()
protected AbstractFileStoreWrite.WriterContainer<T>	getWriterWrapper(BinaryRow partition, int bucket)
void	notifyNewFiles(long snapshotId, BinaryRow partition, int bucket, List<DataFileMeta> files) Notify that some new files are created at given snapshot in given bucket.
protected void	notifyNewWriter(RecordWriter<T> writer)
List<CommitMessage>	prepareCommit(boolean waitCompaction, long commitIdentifier) Prepare commit in the write.
void	restore(List<FileStoreWrite.State<T>> states) Restore state of a previous operation instance into the current operation instance.
void	withCompactExecutor(ExecutorService compactExecutor)
void	withExecutionMode(boolean isStreamingMode) We detect whether it is in batch mode, if so, we do some optimization.
void	withIgnorePreviousFiles(boolean ignorePreviousFiles) Set whether the write operation should ignore previously stored files.
void	withInsertOnly(boolean insertOnly) This method is called when the insert only status of the records changes.
FileStoreWrite<T>	withIOManager(IOManager ioManager)
FileStoreWrite<T>	withMemoryPoolFactory(MemoryPoolFactory memoryPoolFactory) With memory pool factory for the current file store write.
FileStoreWrite<T>	withMetricRegistry(MetricRegistry metricRegistry) With metrics to measure compaction.
void	write(BinaryRow partition, int bucket, T data) Write the data to the store according to the partition and bucket.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.paimon.operation.FileStoreWrite

withMemoryPool

Field Detail

snapshotManager

protected final SnapshotManager snapshotManager

ioManager

@Nullable
protected IOManager ioManager

writers

protected final Map<BinaryRow,Map<Integer,AbstractFileStoreWrite.WriterContainer<T>>> writers

isStreamingMode

protected boolean isStreamingMode

compactionMetrics

protected CompactionMetrics compactionMetrics

tableName

protected final String tableName

Constructor Detail

AbstractFileStoreWrite

protected AbstractFileStoreWrite(SnapshotManager snapshotManager,
                                 FileStoreScan scan,
                                 @Nullable
                                 IndexMaintainer.Factory<T> indexFactory,
                                 @Nullable
                                 DeletionVectorsMaintainer.Factory dvMaintainerFactory,
                                 String tableName,
                                 int totalBuckets,
                                 RowType partitionType,
                                 int writerNumberMax,
                                 boolean legacyPartitionName)

Method Detail

withIOManager

public FileStoreWrite<T> withIOManager(IOManager ioManager)

Specified by:

withIOManager in interface FileStoreWrite<T>

withMemoryPoolFactory

public FileStoreWrite<T> withMemoryPoolFactory(MemoryPoolFactory memoryPoolFactory)

Description copied from interface: FileStoreWrite

With memory pool factory for the current file store write.

Specified by:

withMemoryPoolFactory in interface FileStoreWrite<T>

Parameters:

memoryPoolFactory - the given memory pool factory.

withIgnorePreviousFiles

public void withIgnorePreviousFiles(boolean ignorePreviousFiles)

Description copied from interface: FileStoreWrite

Set whether the write operation should ignore previously stored files.

Specified by:

withIgnorePreviousFiles in interface FileStoreWrite<T>

Parameters:

ignorePreviousFiles - whether the write operation should ignore previously stored files.

withCompactExecutor

public void withCompactExecutor(ExecutorService compactExecutor)

Specified by:

withCompactExecutor in interface FileStoreWrite<T>

withInsertOnly

public void withInsertOnly(boolean insertOnly)

Description copied from interface: FileStoreWrite

This method is called when the insert only status of the records changes.

Specified by:

withInsertOnly in interface FileStoreWrite<T>

Parameters:

insertOnly - If true, all the following records would be of RowKind.INSERT, and no two records would have the same primary key.

write

public void write(BinaryRow partition,
                  int bucket,
                  T data)
           throws Exception

Description copied from interface: FileStoreWrite

Write the data to the store according to the partition and bucket.

Specified by:

write in interface FileStoreWrite<T>

Parameters:

partition - the partition of the data

bucket - the bucket id of the data

data - the given data

Throws:

Exception - the thrown exception when writing the record

compact

public void compact(BinaryRow partition,
                    int bucket,
                    boolean fullCompaction)
             throws Exception

Description copied from interface: FileStoreWrite

Compact data stored in given partition and bucket. Note that compaction process is only submitted and may not be completed when the method returns.

Specified by:

compact in interface FileStoreWrite<T>

Parameters:

partition - the partition to compact

bucket - the bucket to compact

fullCompaction - whether to trigger full compaction or just normal compaction

Throws:

Exception - the thrown exception when compacting the records

notifyNewFiles

public void notifyNewFiles(long snapshotId,
                           BinaryRow partition,
                           int bucket,
                           List<DataFileMeta> files)

Description copied from interface: FileStoreWrite

Notify that some new files are created at given snapshot in given bucket.

Most probably, these files are created by another job. Currently this method is only used by the dedicated compact job to see files created by writer jobs.

Specified by:

notifyNewFiles in interface FileStoreWrite<T>

Parameters:

snapshotId - the snapshot id where new files are created

partition - the partition where new files are created

bucket - the bucket where new files are created

files - the new files themselves

prepareCommit

public List<CommitMessage> prepareCommit(boolean waitCompaction,
                                         long commitIdentifier)
                                  throws Exception

Description copied from interface: FileStoreWrite

Prepare commit in the write.

Specified by:

prepareCommit in interface FileStoreWrite<T>

Parameters:

waitCompaction - if this method need to wait for current compaction to complete

commitIdentifier - identifier of the commit being prepared

Returns:

the file committable list

Throws:

Exception - the thrown exception

createWriterCleanChecker

protected abstract java.util.function.Function<AbstractFileStoreWrite.WriterContainer<T>,Boolean> createWriterCleanChecker()

createConflictAwareWriterCleanChecker

protected static <T> java.util.function.Function<AbstractFileStoreWrite.WriterContainer<T>,Boolean> createConflictAwareWriterCleanChecker(String commitUser,
                                                                                                                                          SnapshotManager snapshotManager)

createNoConflictAwareWriterCleanChecker

protected static <T> java.util.function.Function<AbstractFileStoreWrite.WriterContainer<T>,Boolean> createNoConflictAwareWriterCleanChecker()

close

public void close()
           throws Exception

Description copied from interface: FileStoreWrite

Close the writer.

Specified by:

close in interface FileStoreWrite<T>

Throws:

Exception - the thrown exception

checkpoint

public List<FileStoreWrite.State<T>> checkpoint()

Description copied from interface: Restorable

Extract state of the current operation instance.

Specified by:

checkpoint in interface Restorable<List<FileStoreWrite.State<T>>>

restore

public void restore(List<FileStoreWrite.State<T>> states)

Description copied from interface: Restorable

Restore state of a previous operation instance into the current operation instance.

Specified by:

restore in interface Restorable<List<FileStoreWrite.State<T>>>

getActiveBuckets

public Map<BinaryRow,List<Integer>> getActiveBuckets()

getWriterWrapper

protected AbstractFileStoreWrite.WriterContainer<T> getWriterWrapper(BinaryRow partition,
                                                                     int bucket)

createWriterContainer

@VisibleForTesting
public AbstractFileStoreWrite.WriterContainer<T> createWriterContainer(BinaryRow partition,
                                                                                          int bucket,
                                                                                          boolean ignorePreviousFiles)

withExecutionMode

public void withExecutionMode(boolean isStreamingMode)

Description copied from interface: FileStoreWrite

We detect whether it is in batch mode, if so, we do some optimization.

Specified by:

withExecutionMode in interface FileStoreWrite<T>

Parameters:

isStreamingMode - whether in streaming mode

withMetricRegistry

public FileStoreWrite<T> withMetricRegistry(MetricRegistry metricRegistry)

Description copied from interface: FileStoreWrite

With metrics to measure compaction.

Specified by:

withMetricRegistry in interface FileStoreWrite<T>

getCompactExecutor

@VisibleForTesting
public ExecutorService getCompactExecutor()

notifyNewWriter

protected void notifyNewWriter(RecordWriter<T> writer)

createWriter

protected abstract RecordWriter<T> createWriter(@Nullable
                                                Long snapshotId,
                                                BinaryRow partition,
                                                int bucket,
                                                List<DataFileMeta> restoreFiles,
                                                long restoredMaxSeqNumber,
                                                @Nullable
                                                CommitIncrement restoreIncrement,
                                                ExecutorService compactExecutor,
                                                @Nullable
                                                DeletionVectorsMaintainer deletionVectorsMaintainer)

forceBufferSpill

protected void forceBufferSpill()
                         throws Exception

Throws:

Exception