org.apache.paimon.table.sink

Interface StreamTableCommit

All Superinterfaces:

AutoCloseable, TableCommit

All Known Subinterfaces:

InnerTableCommit

All Known Implementing Classes:

TableCommitImpl

@Public
public interface StreamTableCommit
extends TableCommit

A TableCommit for stream processing. You can use this class to commit multiple times.

Since:

0.4.0

See Also:

StreamWriteBuilder

Method Summary

Modifier and Type	Method and Description
void	commit(long commitIdentifier, List<CommitMessage> commitMessages) Create a new commit.
int	filterAndCommit(Map<Long,List<CommitMessage>> commitIdentifiersAndMessages) Filter out all List<CommitMessage> which have been committed and commit the remaining ones.

Methods inherited from interface org.apache.paimon.table.sink.TableCommit

abort

Methods inherited from interface java.lang.AutoCloseable

close

Method Detail

commit

void commit(long commitIdentifier,
            List<CommitMessage> commitMessages)

Create a new commit. One commit may generate up to two snapshots, one for adding new files and the other for compaction. There will be some expiration policies after commit:

1. Snapshot expiration may occur according to three options:

• 'snapshot.time-retained': The maximum time of completed snapshots to retain.
• 'snapshot.num-retained.min': The minimum number of completed snapshots to retain.
• 'snapshot.num-retained.max': The maximum number of completed snapshots to retain.

2. Partition expiration may occur according to 'partition.expiration-time'. The partition check is expensive, so all partitions are not checked every time when invoking this method. The check frequency is controlled by 'partition.expiration-check-interval'. Partition expiration will create an 'OVERWRITE' snapshot.

Compared to filterAndCommit(java.util.Map<java.lang.Long, java.util.List<org.apache.paimon.table.sink.CommitMessage>>), this method does not check if commitIdentifier has been committed, so this method might be faster. Please only use this method if you can make sure that the commitIdentifier hasn't been committed before.

Parameters:

commitIdentifier - Committed transaction ID, can start from 0. If there are multiple commits, please increment this ID.

commitMessages - commit messages from table write

See Also:

StreamTableWrite.prepareCommit(boolean, long)

filterAndCommit

int filterAndCommit(Map<Long,List<CommitMessage>> commitIdentifiersAndMessages)

Filter out all List<CommitMessage> which have been committed and commit the remaining ones.

Compared to commit(long, java.util.List<org.apache.paimon.table.sink.CommitMessage>), this method will first check if a commitIdentifier has been committed, so this method might be slower. A common usage of this method is to retry the commit process after a failure.

Parameters:

commitIdentifiersAndMessages - a map containing all CommitMessages in question. The key is the commitIdentifier.

Returns:

number of List<CommitMessage> committed.