org.apache.paimon.flink.source.operator

Class CombinedCompactorSourceFunction<T>

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.streaming.api.functions.source.RichSourceFunction<T>

org.apache.paimon.flink.source.operator.CombinedCompactorSourceFunction<T>

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, org.apache.flink.streaming.api.functions.source.SourceFunction<T>

Direct Known Subclasses:

CombinedAwareBatchSourceFunction, CombinedAwareStreamingSourceFunction, CombinedUnawareBatchSourceFunction, CombinedUnawareStreamingSourceFunction

public abstract class CombinedCompactorSourceFunction<T>
extends org.apache.flink.streaming.api.functions.source.RichSourceFunction<T>

This is the single (non-parallel) monitoring task, it is responsible for:

1. Monitoring snapshots of the Paimon table.
2. Creating the splits Split or compaction task UnawareAppendCompactionTask corresponding to the incremental files
3. Assigning them to downstream tasks for further processing.

The splits to be read are forwarded to the downstream MultiTablesReadOperator which can have parallelism greater than one.

Currently, only dedicated compaction job for multi-tables rely on this monitor. This is the single (non-parallel) monitoring task, it is responsible for the new Paimon table.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.streaming.api.functions.source.SourceFunction

org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T>

Field Summary

Fields

Modifier and Type	Field and Description
protected Catalog.Loader	catalogLoader
protected org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T>	ctx
protected Pattern	databasePattern
protected Pattern	excludingPattern
protected Pattern	includingPattern
protected AtomicBoolean	isRunning
protected boolean	isStreaming

Constructor Summary

Constructors

Constructor and Description
CombinedCompactorSourceFunction(Catalog.Loader catalogLoader, Pattern includingPattern, Pattern excludingPattern, Pattern databasePattern, boolean isStreaming)

Method Summary

Modifier and Type	Method and Description
void	cancel()
void	open(org.apache.flink.configuration.Configuration parameters)
void	run(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T> sourceContext)

Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

close, getIterationRuntimeContext, getRuntimeContext, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.api.common.functions.RichFunction

open

Field Detail

catalogLoader

protected final Catalog.Loader catalogLoader

includingPattern

protected final Pattern includingPattern

excludingPattern

protected final Pattern excludingPattern

databasePattern

protected final Pattern databasePattern

isStreaming

protected final boolean isStreaming

isRunning

protected transient AtomicBoolean isRunning

ctx

protected transient org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T> ctx

Constructor Detail

CombinedCompactorSourceFunction

public CombinedCompactorSourceFunction(Catalog.Loader catalogLoader,
                                       Pattern includingPattern,
                                       Pattern excludingPattern,
                                       Pattern databasePattern,
                                       boolean isStreaming)

Method Detail

open

public void open(org.apache.flink.configuration.Configuration parameters)
          throws Exception

Specified by:

open in interface org.apache.flink.api.common.functions.RichFunction

Overrides:

open in class org.apache.flink.api.common.functions.AbstractRichFunction

Throws:

Exception

run

public void run(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T> sourceContext)
         throws Exception

Throws:

Exception

cancel

public void cancel()