org.apache.paimon.flink.source

Class BucketUnawareCompactSource

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.streaming.api.functions.source.RichSourceFunction<UnawareAppendCompactionTask>

org.apache.paimon.flink.source.BucketUnawareCompactSource

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, org.apache.flink.streaming.api.functions.source.SourceFunction<UnawareAppendCompactionTask>

public class BucketUnawareCompactSource
extends org.apache.flink.streaming.api.functions.source.RichSourceFunction<UnawareAppendCompactionTask>

Source Function for unaware-bucket Compaction.

Note: The function is the source function of unaware-bucket compactor coordinator. It will read the latest snapshot continuously by compactionCoordinator, and generate new compaction tasks. The source function is used in unaware-bucket compaction job (both stand-alone and write-combined). Besides, we don't need to save state in this function, it will invoke a full scan when starting up, and scan continuously for the following snapshot.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.streaming.api.functions.source.SourceFunction

org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T>

Constructor Summary

Constructors

Constructor and Description
BucketUnawareCompactSource(FileStoreTable table, boolean isStreaming, long scanInterval, Predicate filter)

Method Summary

Modifier and Type	Method and Description
static org.apache.flink.streaming.api.datastream.DataStreamSource<UnawareAppendCompactionTask>	buildSource(org.apache.flink.streaming.api.environment.StreamExecutionEnvironment env, BucketUnawareCompactSource source, boolean streaming, String tableIdentifier)
void	cancel()
void	open(org.apache.flink.configuration.Configuration parameters) Do not annotate with @override here to maintain compatibility with Flink 2.0+.
void	open(OpenContext openContext) Do not annotate with @override here to maintain compatibility with Flink 1.18-.
void	run(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<UnawareAppendCompactionTask> sourceContext)

Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

close, getIterationRuntimeContext, getRuntimeContext, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BucketUnawareCompactSource

public BucketUnawareCompactSource(FileStoreTable table,
                                  boolean isStreaming,
                                  long scanInterval,
                                  @Nullable
                                  Predicate filter)

Method Detail

open

public void open(OpenContext openContext)
          throws Exception

Do not annotate with @override here to maintain compatibility with Flink 1.18-.

Throws:

Exception

open

public void open(org.apache.flink.configuration.Configuration parameters)
          throws Exception

Do not annotate with @override here to maintain compatibility with Flink 2.0+.

Specified by:

open in interface org.apache.flink.api.common.functions.RichFunction

Overrides:

open in class org.apache.flink.api.common.functions.AbstractRichFunction

Throws:

Exception

run

public void run(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<UnawareAppendCompactionTask> sourceContext)
         throws Exception

Throws:

Exception

cancel

public void cancel()

buildSource

public static org.apache.flink.streaming.api.datastream.DataStreamSource<UnawareAppendCompactionTask> buildSource(org.apache.flink.streaming.api.environment.StreamExecutionEnvironment env,
                                                                                                                  BucketUnawareCompactSource source,
                                                                                                                  boolean streaming,
                                                                                                                  String tableIdentifier)