public class TableCommitImpl extends Object implements InnerTableCommit
FileStoreCommit
to provide snapshot commit and expiration.Constructor and Description |
---|
TableCommitImpl(FileStoreCommit commit,
Runnable expireSnapshots,
PartitionExpire partitionExpire,
TagAutoManager tagAutoManager,
Lock lock,
java.time.Duration consumerExpireTime,
ConsumerManager consumerManager,
CoreOptions.ExpireExecutionMode expireExecutionMode,
String tableName,
boolean forceCreatingSnapshot) |
Modifier and Type | Method and Description |
---|---|
void |
abort(List<CommitMessage> commitMessages)
Abort an unsuccessful commit.
|
void |
close() |
void |
commit(List<CommitMessage> commitMessages)
Create a new commit.
|
void |
commit(long identifier,
List<CommitMessage> commitMessages)
Create a new commit.
|
void |
commit(ManifestCommittable committable) |
void |
commitMultiple(List<ManifestCommittable> committables,
boolean checkAppendFiles) |
void |
expireSnapshots() |
int |
filterAndCommit(Map<Long,List<CommitMessage>> commitIdentifiersAndMessages)
Filter out all
List<CommitMessage> which have been committed and commit the remaining
ones. |
int |
filterAndCommitMultiple(List<ManifestCommittable> committables) |
int |
filterAndCommitMultiple(List<ManifestCommittable> committables,
boolean checkAppendFiles) |
boolean |
forceCreatingSnapshot() |
ExecutorService |
getExpireMainExecutor() |
TableCommitImpl |
ignoreEmptyCommit(boolean ignoreEmptyCommit)
If this is set to true, when there is no new data, no snapshot will be generated.
|
void |
truncateTable()
Truncate table, like normal
BatchTableCommit.commit(java.util.List<org.apache.paimon.table.sink.CommitMessage>) , files are not immediately deleted, they are only
logically deleted and will be deleted after the snapshot expires. |
InnerTableCommit |
withMetricRegistry(MetricRegistry registry) |
TableCommitImpl |
withOverwrite(Map<String,String> overwritePartitions)
Overwrite writing, same as the 'INSERT OVERWRITE T PARTITION (...)' semantics of SQL.
|
public TableCommitImpl(FileStoreCommit commit, @Nullable Runnable expireSnapshots, @Nullable PartitionExpire partitionExpire, @Nullable TagAutoManager tagAutoManager, Lock lock, @Nullable java.time.Duration consumerExpireTime, ConsumerManager consumerManager, CoreOptions.ExpireExecutionMode expireExecutionMode, String tableName, boolean forceCreatingSnapshot)
public boolean forceCreatingSnapshot()
public TableCommitImpl withOverwrite(@Nullable Map<String,String> overwritePartitions)
InnerTableCommit
withOverwrite
in interface InnerTableCommit
public TableCommitImpl ignoreEmptyCommit(boolean ignoreEmptyCommit)
InnerTableCommit
If there are no new files or compact files at the same time, no new commit will be generated regardless of the configuration (No one trigger commit interface).
ignoreEmptyCommit
in interface InnerTableCommit
public InnerTableCommit withMetricRegistry(MetricRegistry registry)
withMetricRegistry
in interface InnerTableCommit
public void commit(List<CommitMessage> commitMessages)
BatchTableCommit
1. Snapshot expiration may occur according to three options:
2. Partition expiration may occur according to 'partition.expiration-time'. The partition check is expensive, so all partitions are not checked every time when invoking this method. The check frequency is controlled by 'partition.expiration-check-interval'. Partition expiration will create an 'OVERWRITE' snapshot.
commit
in interface BatchTableCommit
commitMessages
- commit messages from table writepublic void truncateTable()
BatchTableCommit
BatchTableCommit.commit(java.util.List<org.apache.paimon.table.sink.CommitMessage>)
, files are not immediately deleted, they are only
logically deleted and will be deleted after the snapshot expires.truncateTable
in interface BatchTableCommit
public void commit(long identifier, List<CommitMessage> commitMessages)
StreamTableCommit
1. Snapshot expiration may occur according to three options:
2. Partition expiration may occur according to 'partition.expiration-time'. The partition check is expensive, so all partitions are not checked every time when invoking this method. The check frequency is controlled by 'partition.expiration-check-interval'. Partition expiration will create an 'OVERWRITE' snapshot.
Compared to StreamTableCommit.filterAndCommit(java.util.Map<java.lang.Long, java.util.List<org.apache.paimon.table.sink.CommitMessage>>)
, this method does not check if
commitIdentifier
has been committed, so this method might be faster. Please only use
this method if you can make sure that the commitIdentifier
hasn't been committed
before.
commit
in interface StreamTableCommit
identifier
- Committed transaction ID, can start from 0. If there are multiple
commits, please increment this ID.commitMessages
- commit messages from table writeStreamTableWrite.prepareCommit(boolean, long)
public int filterAndCommit(Map<Long,List<CommitMessage>> commitIdentifiersAndMessages)
StreamTableCommit
List<CommitMessage>
which have been committed and commit the remaining
ones.
Compared to StreamTableCommit.commit(long, java.util.List<org.apache.paimon.table.sink.CommitMessage>)
, this method will first check if a commitIdentifier
has been committed, so this method might be slower. A common usage of this
method is to retry the commit process after a failure.
filterAndCommit
in interface StreamTableCommit
commitIdentifiersAndMessages
- a map containing all CommitMessage
s in question.
The key is the commitIdentifier
.List<CommitMessage>
committed.public void commit(ManifestCommittable committable)
public void commitMultiple(List<ManifestCommittable> committables, boolean checkAppendFiles)
public int filterAndCommitMultiple(List<ManifestCommittable> committables)
public int filterAndCommitMultiple(List<ManifestCommittable> committables, boolean checkAppendFiles)
public void expireSnapshots()
public void close() throws Exception
close
in interface AutoCloseable
Exception
public void abort(List<CommitMessage> commitMessages)
TableCommit
abort
in interface TableCommit
@VisibleForTesting public ExecutorService getExpireMainExecutor()
Copyright © 2023–2024 The Apache Software Foundation. All rights reserved.