org.apache.paimon.flink.sink

Class StoreMultiCommitter

java.lang.Object

org.apache.paimon.flink.sink.StoreMultiCommitter

All Implemented Interfaces:

AutoCloseable, Committer<MultiTableCommittable,WrappedManifestCommittable>

public class StoreMultiCommitter
extends Object
implements Committer<MultiTableCommittable,WrappedManifestCommittable>

StoreMultiCommitter for multiple dynamic store. During the commit process, it will group the WrappedManifestCommittables by their table identifier and use different committers to commit to different tables.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.paimon.flink.sink.Committer

Committer.Context, Committer.Factory<CommitT,GlobalCommitT>

Constructor Summary

Constructors

Constructor and Description
StoreMultiCommitter(Catalog.Loader catalogLoader, Committer.Context context)
StoreMultiCommitter(Catalog.Loader catalogLoader, Committer.Context context, boolean ignoreEmptyCommit, Map<String,String> dynamicOptions)

Method Summary

Modifier and Type	Method and Description
void	close()
WrappedManifestCommittable	combine(long checkpointId, long watermark, List<MultiTableCommittable> committables) Compute an aggregated committable from a list of committables.
WrappedManifestCommittable	combine(long checkpointId, long watermark, WrappedManifestCommittable wrappedManifestCommittable, List<MultiTableCommittable> committables)
void	commit(List<WrappedManifestCommittable> committables) Commits the given GlobalCommitT.
int	filterAndCommit(List<WrappedManifestCommittable> globalCommittables, boolean checkAppendFiles) Filter out all GlobalCommitT which have committed, and commit the remaining GlobalCommitT.
boolean	forceCreatingSnapshot()
Map<Long,List<MultiTableCommittable>>	groupByCheckpoint(Collection<MultiTableCommittable> committables)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.paimon.flink.sink.Committer

createContext, filterAndCommit

Constructor Detail

StoreMultiCommitter

public StoreMultiCommitter(Catalog.Loader catalogLoader,
                           Committer.Context context)

StoreMultiCommitter

public StoreMultiCommitter(Catalog.Loader catalogLoader,
                           Committer.Context context,
                           boolean ignoreEmptyCommit,
                           Map<String,String> dynamicOptions)

Method Detail

forceCreatingSnapshot

public boolean forceCreatingSnapshot()

Specified by:

forceCreatingSnapshot in interface Committer<MultiTableCommittable,WrappedManifestCommittable>

combine

public WrappedManifestCommittable combine(long checkpointId,
                                          long watermark,
                                          List<MultiTableCommittable> committables)

Description copied from interface: Committer

Compute an aggregated committable from a list of committables.

Specified by:

combine in interface Committer<MultiTableCommittable,WrappedManifestCommittable>

combine

public WrappedManifestCommittable combine(long checkpointId,
                                          long watermark,
                                          WrappedManifestCommittable wrappedManifestCommittable,
                                          List<MultiTableCommittable> committables)

Specified by:

combine in interface Committer<MultiTableCommittable,WrappedManifestCommittable>

commit

public void commit(List<WrappedManifestCommittable> committables)
            throws IOException,
                   InterruptedException

Description copied from interface: Committer

Commits the given GlobalCommitT.

Specified by:

commit in interface Committer<MultiTableCommittable,WrappedManifestCommittable>

Throws:

IOException

InterruptedException

filterAndCommit

public int filterAndCommit(List<WrappedManifestCommittable> globalCommittables,
                           boolean checkAppendFiles)
                    throws IOException

Description copied from interface: Committer

Filter out all GlobalCommitT which have committed, and commit the remaining GlobalCommitT.

Specified by:

filterAndCommit in interface Committer<MultiTableCommittable,WrappedManifestCommittable>

Throws:

IOException

groupByCheckpoint

public Map<Long,List<MultiTableCommittable>> groupByCheckpoint(Collection<MultiTableCommittable> committables)

Specified by:

groupByCheckpoint in interface Committer<MultiTableCommittable,WrappedManifestCommittable>

close

public void close()
           throws Exception

Specified by:

close in interface AutoCloseable

Throws:

Exception