org.apache.paimon.flink.sink

Interface Committer<CommitT,GlobalCommitT>

All Superinterfaces:

AutoCloseable

All Known Implementing Classes:

StoreCommitter, StoreMultiCommitter

public interface Committer<CommitT,GlobalCommitT>
extends AutoCloseable

The Committer is responsible for creating and committing an aggregated committable, which we call committable (see combine(long, long, java.util.List<CommitT>)).

The Committer runs with parallelism equal to 1.

Nested Class Summary

Nested Classes

Modifier and Type	Interface and Description
static interface	Committer.Context Context to create Committer.
static interface	Committer.Factory<CommitT,GlobalCommitT> Factory to create Committer.

Method Summary

Modifier and Type	Method and Description
GlobalCommitT	combine(long checkpointId, long watermark, GlobalCommitT t, List<CommitT> committables)
GlobalCommitT	combine(long checkpointId, long watermark, List<CommitT> committables) Compute an aggregated committable from a list of committables.
void	commit(List<GlobalCommitT> globalCommittables) Commits the given GlobalCommitT.
static Committer.Context	createContext(String commitUser, org.apache.flink.metrics.groups.OperatorMetricGroup metricGroup, boolean streamingCheckpointEnabled, boolean isRestored, org.apache.flink.api.common.state.OperatorStateStore stateStore)
default int	filterAndCommit(List<GlobalCommitT> globalCommittables)
int	filterAndCommit(List<GlobalCommitT> globalCommittables, boolean checkAppendFiles) Filter out all GlobalCommitT which have committed, and commit the remaining GlobalCommitT.
boolean	forceCreatingSnapshot()
Map<Long,List<CommitT>>	groupByCheckpoint(Collection<CommitT> committables)

Methods inherited from interface java.lang.AutoCloseable

close

Method Detail

forceCreatingSnapshot

boolean forceCreatingSnapshot()

combine

GlobalCommitT combine(long checkpointId,
                      long watermark,
                      List<CommitT> committables)
               throws IOException

Compute an aggregated committable from a list of committables.

Throws:

IOException

combine

GlobalCommitT combine(long checkpointId,
                      long watermark,
                      GlobalCommitT t,
                      List<CommitT> committables)

commit

void commit(List<GlobalCommitT> globalCommittables)
     throws IOException,
            InterruptedException

Commits the given GlobalCommitT.

Throws:

IOException

InterruptedException

filterAndCommit

int filterAndCommit(List<GlobalCommitT> globalCommittables,
                    boolean checkAppendFiles)
             throws IOException

Filter out all GlobalCommitT which have committed, and commit the remaining GlobalCommitT.

Throws:

IOException

filterAndCommit

default int filterAndCommit(List<GlobalCommitT> globalCommittables)
                     throws IOException

Throws:

IOException

groupByCheckpoint

Map<Long,List<CommitT>> groupByCheckpoint(Collection<CommitT> committables)

createContext

static Committer.Context createContext(String commitUser,
                                       @Nullable
                                       org.apache.flink.metrics.groups.OperatorMetricGroup metricGroup,
                                       boolean streamingCheckpointEnabled,
                                       boolean isRestored,
                                       org.apache.flink.api.common.state.OperatorStateStore stateStore)