org.apache.paimon.flink.source.operator

Class CombinedCompactorSource<T>

java.lang.Object

org.apache.paimon.flink.source.AbstractNonCoordinatedSource<T>

org.apache.paimon.flink.source.operator.CombinedCompactorSource<T>

All Implemented Interfaces:

Serializable, org.apache.flink.api.connector.source.Source<T,SimpleSourceSplit,NoOpEnumState>, org.apache.flink.api.connector.source.SourceReaderFactory<T,SimpleSourceSplit>

Direct Known Subclasses:

CombinedAwareBatchSource, CombinedAwareStreamingSource, CombinedUnawareBatchSource, CombinedUnawareStreamingSource

public abstract class CombinedCompactorSource<T>
extends AbstractNonCoordinatedSource<T>

This is the single (non-parallel) monitoring task, it is responsible for:

1. Monitoring snapshots of the Paimon table.
2. Creating the splits Split or compaction task UnawareAppendCompactionTask corresponding to the incremental files
3. Assigning them to downstream tasks for further processing.

The splits to be read are forwarded to the downstream MultiTablesReadOperator which can have parallelism greater than one.

Currently, only dedicated compaction job for multi-tables rely on this monitor. This is the single (non-parallel) monitoring task, it is responsible for the new Paimon table.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected CatalogLoader	catalogLoader
protected Pattern	databasePattern
protected Pattern	excludingPattern
protected Pattern	includingPattern
protected boolean	isStreaming

Constructor Summary

Constructors

Constructor and Description
CombinedCompactorSource(CatalogLoader catalogLoader, Pattern includingPattern, Pattern excludingPattern, Pattern databasePattern, boolean isStreaming)

Method Summary

Modifier and Type	Method and Description
org.apache.flink.api.connector.source.Boundedness	getBoundedness()

Methods inherited from class org.apache.paimon.flink.source.AbstractNonCoordinatedSource

createEnumerator, getEnumeratorCheckpointSerializer, getSplitSerializer, restoreEnumerator

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.api.connector.source.SourceReaderFactory

createReader

Field Detail

catalogLoader

protected final CatalogLoader catalogLoader

includingPattern

protected final Pattern includingPattern

excludingPattern

protected final Pattern excludingPattern

databasePattern

protected final Pattern databasePattern

isStreaming

protected final boolean isStreaming

Constructor Detail

CombinedCompactorSource

public CombinedCompactorSource(CatalogLoader catalogLoader,
                               Pattern includingPattern,
                               Pattern excludingPattern,
                               Pattern databasePattern,
                               boolean isStreaming)

Method Detail

getBoundedness

public org.apache.flink.api.connector.source.Boundedness getBoundedness()