org.apache.paimon.operation

Class FileStoreCommitImpl

java.lang.Object

org.apache.paimon.operation.FileStoreCommitImpl

All Implemented Interfaces:

AutoCloseable, FileStoreCommit

public class FileStoreCommitImpl
extends Object
implements FileStoreCommit

Default implementation of FileStoreCommit.

This class provides an atomic commit method to the user.

1. Before calling commit(org.apache.paimon.manifest.ManifestCommittable, java.util.Map<java.lang.String, java.lang.String>), if user cannot determine if this commit is done before, user should first call filterCommitted(java.util.List<org.apache.paimon.manifest.ManifestCommittable>).
2. Before committing, it will first check for conflicts by checking if all files to be removed currently exists, and if modified files have overlapping key ranges with existing files.
3. After that it use the external lock (if provided) or the atomic rename of the file system to ensure atomicity.
4. If commit fails due to conflicts or exception it tries its best to clean up and aborts.
5. If atomic rename fails it tries again after reading the latest snapshot from step 2.

NOTE: If you want to modify this class, any exception during commit MUST NOT BE IGNORED. They must be thrown to restart the job. It is recommended to run FileStoreCommitTest thousands of times to make sure that your changes are correct.

Constructor Summary

Constructors

Constructor and Description

FileStoreCommitImpl(FileIO fileIO, SchemaManager schemaManager, String commitUser, RowType partitionType, String partitionDefaultName, FileStorePathFactory pathFactory, SnapshotManager snapshotManager, ManifestFile.Factory manifestFileFactory, ManifestList.Factory manifestListFactory, IndexManifestFile.Factory indexManifestFileFactory, FileStoreScan scan, int numBucket, MemorySize manifestTargetSize, MemorySize manifestFullCompactionSize, int manifestMergeMinCount, boolean dynamicPartitionOverwrite, Comparator<InternalRow> keyComparator, String branchName, StatsFileHandler statsFileHandler, BucketMode bucketMode, Integer manifestReadParallelism, List<CommitCallback> commitCallbacks, int commitMaxRetries)

Method Summary

Modifier and Type	Method and Description
void	abort(List<CommitMessage> commitMessages) Abort an unsuccessful commit.
void	close()
void	commit(ManifestCommittable committable, Map<String,String> properties) Commit from manifest committable.
void	commit(ManifestCommittable committable, Map<String,String> properties, boolean checkAppendFiles) Commit from manifest committable with checkAppendFiles.
void	commitStatistics(Statistics stats, long commitIdentifier) Commit new statistics.
void	dropPartitions(List<Map<String,String>> partitions, long commitIdentifier) Drop multiple partitions.
FileIO	fileIO()
List<ManifestCommittable>	filterCommitted(List<ManifestCommittable> committables) Find out which committables need to be retried when recovering from the failure.
FileStoreCommit	ignoreEmptyCommit(boolean ignoreEmptyCommit)
void	overwrite(Map<String,String> partition, ManifestCommittable committable, Map<String,String> properties) Overwrite from manifest committable and partition.
FileStorePathFactory	pathFactory()
void	truncateTable(long commitIdentifier)
FileStoreCommit	withLock(Lock lock) With global lock.
FileStoreCommit	withMetrics(CommitMetrics metrics) With metrics to measure commits.
FileStoreCommit	withPartitionExpire(PartitionExpire partitionExpire)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

FileStoreCommitImpl

public FileStoreCommitImpl(FileIO fileIO,
                           SchemaManager schemaManager,
                           String commitUser,
                           RowType partitionType,
                           String partitionDefaultName,
                           FileStorePathFactory pathFactory,
                           SnapshotManager snapshotManager,
                           ManifestFile.Factory manifestFileFactory,
                           ManifestList.Factory manifestListFactory,
                           IndexManifestFile.Factory indexManifestFileFactory,
                           FileStoreScan scan,
                           int numBucket,
                           MemorySize manifestTargetSize,
                           MemorySize manifestFullCompactionSize,
                           int manifestMergeMinCount,
                           boolean dynamicPartitionOverwrite,
                           @Nullable
                           Comparator<InternalRow> keyComparator,
                           String branchName,
                           StatsFileHandler statsFileHandler,
                           BucketMode bucketMode,
                           @Nullable
                           Integer manifestReadParallelism,
                           List<CommitCallback> commitCallbacks,
                           int commitMaxRetries)

Method Detail

withLock

public FileStoreCommit withLock(Lock lock)

Description copied from interface: FileStoreCommit

With global lock.

Specified by:

withLock in interface FileStoreCommit

ignoreEmptyCommit

public FileStoreCommit ignoreEmptyCommit(boolean ignoreEmptyCommit)

Specified by:

ignoreEmptyCommit in interface FileStoreCommit

withPartitionExpire

public FileStoreCommit withPartitionExpire(PartitionExpire partitionExpire)

Specified by:

withPartitionExpire in interface FileStoreCommit

filterCommitted

public List<ManifestCommittable> filterCommitted(List<ManifestCommittable> committables)

Description copied from interface: FileStoreCommit

Find out which committables need to be retried when recovering from the failure.

Specified by:

filterCommitted in interface FileStoreCommit

commit

public void commit(ManifestCommittable committable,
                   Map<String,String> properties)

Description copied from interface: FileStoreCommit

Commit from manifest committable.

Specified by:

commit in interface FileStoreCommit

commit

public void commit(ManifestCommittable committable,
                   Map<String,String> properties,
                   boolean checkAppendFiles)

Description copied from interface: FileStoreCommit

Commit from manifest committable with checkAppendFiles.

Specified by:

commit in interface FileStoreCommit

overwrite

public void overwrite(Map<String,String> partition,
                      ManifestCommittable committable,
                      Map<String,String> properties)

Description copied from interface: FileStoreCommit

Overwrite from manifest committable and partition.

Specified by:

overwrite in interface FileStoreCommit

Parameters:

partition - A single partition maps each partition key to a partition value. Depending on the user-defined statement, the partition might not include all partition keys. Also note that this partition does not necessarily equal to the partitions of the newly added key-values. This is just the partition to be cleaned up.

dropPartitions

public void dropPartitions(List<Map<String,String>> partitions,
                           long commitIdentifier)

Description copied from interface: FileStoreCommit

Drop multiple partitions. The Snapshot.CommitKind of generated snapshot is Snapshot.CommitKind#OVERWRITE.

Specified by:

dropPartitions in interface FileStoreCommit

Parameters:

partitions - A list of partition Maps. NOTE: cannot be empty!

truncateTable

public void truncateTable(long commitIdentifier)

Specified by:

truncateTable in interface FileStoreCommit

abort

public void abort(List<CommitMessage> commitMessages)

Description copied from interface: FileStoreCommit

Abort an unsuccessful commit. The data files will be deleted.

Specified by:

abort in interface FileStoreCommit

withMetrics

public FileStoreCommit withMetrics(CommitMetrics metrics)

Description copied from interface: FileStoreCommit

With metrics to measure commits.

Specified by:

withMetrics in interface FileStoreCommit

commitStatistics

public void commitStatistics(Statistics stats,
                             long commitIdentifier)

Description copied from interface: FileStoreCommit

Commit new statistics. The Snapshot.CommitKind of generated snapshot is Snapshot.CommitKind#ANALYZE.

Specified by:

commitStatistics in interface FileStoreCommit

pathFactory

public FileStorePathFactory pathFactory()

Specified by:

pathFactory in interface FileStoreCommit

fileIO

public FileIO fileIO()

Specified by:

fileIO in interface FileStoreCommit

close

public void close()

Specified by:

close in interface AutoCloseable

Specified by:

close in interface FileStoreCommit