org.apache.paimon.table.sink

Class TableCommitImpl

java.lang.Object

org.apache.paimon.table.sink.TableCommitImpl

All Implemented Interfaces:

AutoCloseable, BatchTableCommit, InnerTableCommit, StreamTableCommit, TableCommit

public class TableCommitImpl
extends Object
implements InnerTableCommit

An abstraction layer above FileStoreCommit to provide snapshot commit and expiration.

Constructor Summary

Constructors

Constructor and Description
TableCommitImpl(FileStoreCommit commit, Runnable expireSnapshots, PartitionExpire partitionExpire, TagAutoManager tagAutoManager, Lock lock, java.time.Duration consumerExpireTime, ConsumerManager consumerManager, CoreOptions.ExpireExecutionMode expireExecutionMode, String tableName, boolean forceCreatingSnapshot)

Method Summary

Modifier and Type	Method and Description
void	abort(List<CommitMessage> commitMessages) Abort an unsuccessful commit.
void	close()
void	commit(List<CommitMessage> commitMessages) Create a new commit.
void	commit(long identifier, List<CommitMessage> commitMessages) Create a new commit.
void	commit(ManifestCommittable committable)
void	commitMultiple(List<ManifestCommittable> committables, boolean checkAppendFiles)
void	expireSnapshots()
int	filterAndCommit(Map<Long,List<CommitMessage>> commitIdentifiersAndMessages) Filter out all List<CommitMessage> which have been committed and commit the remaining ones.
int	filterAndCommitMultiple(List<ManifestCommittable> committables)
int	filterAndCommitMultiple(List<ManifestCommittable> committables, boolean checkAppendFiles)
boolean	forceCreatingSnapshot()
ExecutorService	getExpireMainExecutor()
TableCommitImpl	ignoreEmptyCommit(boolean ignoreEmptyCommit) If this is set to true, when there is no new data, no snapshot will be generated.
void	truncateTable() Truncate table, like normal BatchTableCommit.commit(java.util.List<org.apache.paimon.table.sink.CommitMessage>), files are not immediately deleted, they are only logically deleted and will be deleted after the snapshot expires.
InnerTableCommit	withMetricRegistry(MetricRegistry registry)
TableCommitImpl	withOverwrite(Map<String,String> overwritePartitions) Overwrite writing, same as the 'INSERT OVERWRITE T PARTITION (...)' semantics of SQL.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TableCommitImpl

public TableCommitImpl(FileStoreCommit commit,
                       @Nullable
                       Runnable expireSnapshots,
                       @Nullable
                       PartitionExpire partitionExpire,
                       @Nullable
                       TagAutoManager tagAutoManager,
                       Lock lock,
                       @Nullable
                       java.time.Duration consumerExpireTime,
                       ConsumerManager consumerManager,
                       CoreOptions.ExpireExecutionMode expireExecutionMode,
                       String tableName,
                       boolean forceCreatingSnapshot)

Method Detail

forceCreatingSnapshot

public boolean forceCreatingSnapshot()

withOverwrite

public TableCommitImpl withOverwrite(@Nullable
                                     Map<String,String> overwritePartitions)

Description copied from interface: InnerTableCommit

Overwrite writing, same as the 'INSERT OVERWRITE T PARTITION (...)' semantics of SQL.

Specified by:

withOverwrite in interface InnerTableCommit

ignoreEmptyCommit

public TableCommitImpl ignoreEmptyCommit(boolean ignoreEmptyCommit)

Description copied from interface: InnerTableCommit

If this is set to true, when there is no new data, no snapshot will be generated. By default, empty commit is ignored.

• For Streaming: the default value of 'ignoreEmptyCommit' is false.
• For Batch: the default value of 'ignoreEmptyCommit' is true.

If there are no new files or compact files at the same time, no new commit will be generated regardless of the configuration (No one trigger commit interface).

Specified by:

ignoreEmptyCommit in interface InnerTableCommit

withMetricRegistry

public InnerTableCommit withMetricRegistry(MetricRegistry registry)

Specified by:

withMetricRegistry in interface InnerTableCommit

commit

public void commit(List<CommitMessage> commitMessages)

Description copied from interface: BatchTableCommit

Create a new commit. One commit may generate up to two snapshots, one for adding new files and the other for compaction. There will be some expiration policies after commit:

1. Snapshot expiration may occur according to three options:

• 'snapshot.time-retained': The maximum time of completed snapshots to retain.
• 'snapshot.num-retained.min': The minimum number of completed snapshots to retain.
• 'snapshot.num-retained.max': The maximum number of completed snapshots to retain.

2. Partition expiration may occur according to 'partition.expiration-time'. The partition check is expensive, so all partitions are not checked every time when invoking this method. The check frequency is controlled by 'partition.expiration-check-interval'. Partition expiration will create an 'OVERWRITE' snapshot.

Specified by:

commit in interface BatchTableCommit

Parameters:

commitMessages - commit messages from table write

truncateTable

public void truncateTable()

Description copied from interface: BatchTableCommit

Truncate table, like normal BatchTableCommit.commit(java.util.List<org.apache.paimon.table.sink.CommitMessage>), files are not immediately deleted, they are only logically deleted and will be deleted after the snapshot expires.

Specified by:

truncateTable in interface BatchTableCommit

commit

public void commit(long identifier,
                   List<CommitMessage> commitMessages)

Description copied from interface: StreamTableCommit

Create a new commit. One commit may generate up to two snapshots, one for adding new files and the other for compaction. There will be some expiration policies after commit:

1. Snapshot expiration may occur according to three options:

• 'snapshot.time-retained': The maximum time of completed snapshots to retain.
• 'snapshot.num-retained.min': The minimum number of completed snapshots to retain.
• 'snapshot.num-retained.max': The maximum number of completed snapshots to retain.

2. Partition expiration may occur according to 'partition.expiration-time'. The partition check is expensive, so all partitions are not checked every time when invoking this method. The check frequency is controlled by 'partition.expiration-check-interval'. Partition expiration will create an 'OVERWRITE' snapshot.

Compared to StreamTableCommit.filterAndCommit(java.util.Map<java.lang.Long, java.util.List<org.apache.paimon.table.sink.CommitMessage>>), this method does not check if commitIdentifier has been committed, so this method might be faster. Please only use this method if you can make sure that the commitIdentifier hasn't been committed before.

Specified by:

commit in interface StreamTableCommit

Parameters:

identifier - Committed transaction ID, can start from 0. If there are multiple commits, please increment this ID.

commitMessages - commit messages from table write

See Also:

StreamTableWrite.prepareCommit(boolean, long)

filterAndCommit

public int filterAndCommit(Map<Long,List<CommitMessage>> commitIdentifiersAndMessages)

Description copied from interface: StreamTableCommit

Filter out all List<CommitMessage> which have been committed and commit the remaining ones.

Compared to StreamTableCommit.commit(long, java.util.List<org.apache.paimon.table.sink.CommitMessage>), this method will first check if a commitIdentifier has been committed, so this method might be slower. A common usage of this method is to retry the commit process after a failure.

Specified by:

filterAndCommit in interface StreamTableCommit

Parameters:

commitIdentifiersAndMessages - a map containing all CommitMessages in question. The key is the commitIdentifier.

Returns:

number of List<CommitMessage> committed.

commit

public void commit(ManifestCommittable committable)

commitMultiple

public void commitMultiple(List<ManifestCommittable> committables,
                           boolean checkAppendFiles)

filterAndCommitMultiple

public int filterAndCommitMultiple(List<ManifestCommittable> committables)

filterAndCommitMultiple

public int filterAndCommitMultiple(List<ManifestCommittable> committables,
                                   boolean checkAppendFiles)

expireSnapshots

public void expireSnapshots()

close

public void close()
           throws Exception

Specified by:

close in interface AutoCloseable

Throws:

Exception

abort

public void abort(List<CommitMessage> commitMessages)

Description copied from interface: TableCommit

Abort an unsuccessful commit. The data files will be deleted.

Specified by:

abort in interface TableCommit

getExpireMainExecutor

@VisibleForTesting
public ExecutorService getExpireMainExecutor()