org.apache.paimon.flink.source

Class BaseDataTableSource

java.lang.Object

org.apache.paimon.flink.source.FlinkTableSource

org.apache.paimon.flink.source.BaseDataTableSource

All Implemented Interfaces:

org.apache.flink.table.connector.source.abilities.SupportsAggregatePushDown, org.apache.flink.table.connector.source.abilities.SupportsFilterPushDown, org.apache.flink.table.connector.source.abilities.SupportsLimitPushDown, SupportsLookupCustomShuffle, org.apache.flink.table.connector.source.abilities.SupportsProjectionPushDown, org.apache.flink.table.connector.source.abilities.SupportsWatermarkPushDown, org.apache.flink.table.connector.source.DynamicTableSource, org.apache.flink.table.connector.source.LookupTableSource, org.apache.flink.table.connector.source.ScanTableSource

Direct Known Subclasses:

DataTableSource

public abstract class BaseDataTableSource
extends FlinkTableSource
implements org.apache.flink.table.connector.source.LookupTableSource, org.apache.flink.table.connector.source.abilities.SupportsWatermarkPushDown, org.apache.flink.table.connector.source.abilities.SupportsAggregatePushDown, SupportsLookupCustomShuffle

Table source to create StaticFileStoreSource or ContinuousFileStoreSource under batch mode or streaming mode.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.paimon.flink.source.FlinkTableSource

FlinkTableSource.SplitStatistics

Nested classes/interfaces inherited from interface org.apache.flink.table.connector.source.LookupTableSource

org.apache.flink.table.connector.source.LookupTableSource.LookupContext, org.apache.flink.table.connector.source.LookupTableSource.LookupRuntimeProvider

Nested classes/interfaces inherited from interface org.apache.flink.table.connector.source.DynamicTableSource

org.apache.flink.table.connector.source.DynamicTableSource.Context, org.apache.flink.table.connector.source.DynamicTableSource.DataStructureConverter

Nested classes/interfaces inherited from interface org.apache.flink.table.connector.source.abilities.SupportsLookupCustomShuffle

SupportsLookupCustomShuffle.InputDataPartitioner

Nested classes/interfaces inherited from interface org.apache.flink.table.connector.source.ScanTableSource

org.apache.flink.table.connector.source.ScanTableSource.ScanContext, org.apache.flink.table.connector.source.ScanTableSource.ScanRuntimeProvider

Nested classes/interfaces inherited from interface org.apache.flink.table.connector.source.abilities.SupportsFilterPushDown

org.apache.flink.table.connector.source.abilities.SupportsFilterPushDown.Result

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.flink.table.factories.DynamicTableFactory.Context	context
protected Long	countPushed
protected LogStoreTableFactory	logStoreTableFactory
protected org.apache.flink.table.catalog.ObjectIdentifier	tableIdentifier
protected boolean	unbounded
protected org.apache.flink.api.common.eventtime.WatermarkStrategy<org.apache.flink.table.data.RowData>	watermarkStrategy

Fields inherited from class org.apache.paimon.flink.source.FlinkTableSource

FLINK_INFER_SCAN_PARALLELISM, limit, predicate, projectFields, splitStatistics, table

Constructor Summary

Constructors

Constructor and Description

BaseDataTableSource(org.apache.flink.table.catalog.ObjectIdentifier tableIdentifier, Table table, boolean unbounded, org.apache.flink.table.factories.DynamicTableFactory.Context context, LogStoreTableFactory logStoreTableFactory, Predicate predicate, int[][] projectFields, Long limit, org.apache.flink.api.common.eventtime.WatermarkStrategy<org.apache.flink.table.data.RowData> watermarkStrategy, Long countPushed)

Method Summary

Modifier and Type	Method and Description
boolean	applyAggregates(List<int[]> groupingSets, List<org.apache.flink.table.expressions.AggregateExpression> aggregateExpressions, org.apache.flink.table.types.DataType producedDataType)
void	applyWatermark(org.apache.flink.api.common.eventtime.WatermarkStrategy<org.apache.flink.table.data.RowData> watermarkStrategy)
String	asSummaryString()
protected abstract List<String>	dynamicPartitionFilteringFields()
org.apache.flink.table.connector.ChangelogMode	getChangelogMode()
protected FileStoreLookupFunction	getFileStoreLookupFunction(org.apache.flink.table.connector.source.LookupTableSource.LookupContext context, FileStoreTable table, int[] projection, int[] joinKey)
org.apache.flink.table.connector.source.LookupTableSource.LookupRuntimeProvider	getLookupRuntimeProvider(org.apache.flink.table.connector.source.LookupTableSource.LookupContext context)
Optional<SupportsLookupCustomShuffle.InputDataPartitioner>	getPartitioner() This method is used to retrieve a custom partitioner that will be applied to the input stream of lookup-join node.
org.apache.flink.table.connector.source.ScanTableSource.ScanRuntimeProvider	getScanRuntimeProvider(org.apache.flink.table.connector.source.ScanTableSource.ScanContext scanContext)
boolean	isUnbounded()

Methods inherited from class org.apache.paimon.flink.source.FlinkTableSource

applyFilters, applyLimit, applyProjection, getPredicateWithScanPartitions, getTable, inferSourceParallelism, scanSplitsForInference, supportsNestedProjection

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.table.connector.source.DynamicTableSource

copy

Methods inherited from interface org.apache.flink.table.connector.source.abilities.SupportsProjectionPushDown

applyProjection

Field Detail

tableIdentifier

protected final org.apache.flink.table.catalog.ObjectIdentifier tableIdentifier

unbounded

protected final boolean unbounded

context

protected final org.apache.flink.table.factories.DynamicTableFactory.Context context

logStoreTableFactory

@Nullable
protected final LogStoreTableFactory logStoreTableFactory

watermarkStrategy

@Nullable
protected org.apache.flink.api.common.eventtime.WatermarkStrategy<org.apache.flink.table.data.RowData> watermarkStrategy

countPushed

@Nullable
protected Long countPushed

Constructor Detail

BaseDataTableSource

public BaseDataTableSource(org.apache.flink.table.catalog.ObjectIdentifier tableIdentifier,
                           Table table,
                           boolean unbounded,
                           org.apache.flink.table.factories.DynamicTableFactory.Context context,
                           @Nullable
                           LogStoreTableFactory logStoreTableFactory,
                           @Nullable
                           Predicate predicate,
                           @Nullable
                           int[][] projectFields,
                           @Nullable
                           Long limit,
                           @Nullable
                           org.apache.flink.api.common.eventtime.WatermarkStrategy<org.apache.flink.table.data.RowData> watermarkStrategy,
                           @Nullable
                           Long countPushed)

Method Detail

getChangelogMode

public org.apache.flink.table.connector.ChangelogMode getChangelogMode()

Specified by:

getChangelogMode in interface org.apache.flink.table.connector.source.ScanTableSource

getScanRuntimeProvider

public org.apache.flink.table.connector.source.ScanTableSource.ScanRuntimeProvider getScanRuntimeProvider(org.apache.flink.table.connector.source.ScanTableSource.ScanContext scanContext)

Specified by:

getScanRuntimeProvider in interface org.apache.flink.table.connector.source.ScanTableSource

dynamicPartitionFilteringFields

protected abstract List<String> dynamicPartitionFilteringFields()

applyWatermark

public void applyWatermark(org.apache.flink.api.common.eventtime.WatermarkStrategy<org.apache.flink.table.data.RowData> watermarkStrategy)

Specified by:

applyWatermark in interface org.apache.flink.table.connector.source.abilities.SupportsWatermarkPushDown

getLookupRuntimeProvider

public org.apache.flink.table.connector.source.LookupTableSource.LookupRuntimeProvider getLookupRuntimeProvider(org.apache.flink.table.connector.source.LookupTableSource.LookupContext context)

Specified by:

getLookupRuntimeProvider in interface org.apache.flink.table.connector.source.LookupTableSource

getFileStoreLookupFunction

protected FileStoreLookupFunction getFileStoreLookupFunction(org.apache.flink.table.connector.source.LookupTableSource.LookupContext context,
                                                             FileStoreTable table,
                                                             int[] projection,
                                                             int[] joinKey)

applyAggregates

public boolean applyAggregates(List<int[]> groupingSets,
                               List<org.apache.flink.table.expressions.AggregateExpression> aggregateExpressions,
                               org.apache.flink.table.types.DataType producedDataType)

Specified by:

applyAggregates in interface org.apache.flink.table.connector.source.abilities.SupportsAggregatePushDown

asSummaryString

public String asSummaryString()

Specified by:

asSummaryString in interface org.apache.flink.table.connector.source.DynamicTableSource

isUnbounded

public boolean isUnbounded()

Specified by:

isUnbounded in class FlinkTableSource

getPartitioner

public Optional<SupportsLookupCustomShuffle.InputDataPartitioner> getPartitioner()

Description copied from interface: SupportsLookupCustomShuffle

This method is used to retrieve a custom partitioner that will be applied to the input stream of lookup-join node.

Specified by:

getPartitioner in interface SupportsLookupCustomShuffle

Returns:

An SupportsLookupCustomShuffle.InputDataPartitioner that defines how records should be distributed across the different subtasks. If the connector expects the input data to remain in its original distribution, an Optional.empty() should be returned.