org.apache.paimon.table.source

Class DataTableStreamScan

java.lang.Object

org.apache.paimon.table.source.DataTableStreamScan

All Implemented Interfaces:

DataTableScan, InnerTableScan, StreamDataTableScan, StreamTableScan, TableScan, Restorable<Long>

Direct Known Subclasses:

LookupDataTableScan

public class DataTableStreamScan
extends Object
implements StreamDataTableScan

StreamTableScan implementation for streaming planning.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.paimon.table.source.TableScan

TableScan.Plan

Field Summary

Fields

Modifier and Type	Field and Description
protected SnapshotReader	snapshotReader

Constructor Summary

Constructors

Constructor and Description
DataTableStreamScan(CoreOptions options, SnapshotReader snapshotReader, SnapshotManager snapshotManager, ChangelogManager changelogManager, boolean supportStreamingReadOverwrite, DefaultValueAssigner defaultValueAssigner)

Method Summary

Modifier and Type	Method and Description
Long	checkpoint() Checkpoint to return next snapshot id.
protected BoundedChecker	createBoundedChecker()
protected FollowUpScanner	createFollowUpScanner()
protected StartingScanner	createStartingScanner(boolean isStreaming)
org.apache.paimon.table.source.AbstractDataTableScan	dropStats()
protected SnapshotReader.Plan	handleOverwriteSnapshot(Snapshot snapshot)
List<PartitionEntry>	listPartitionEntries()
void	notifyCheckpointComplete(Long nextSnapshot) Notifies the checkpoint complete with next snapshot id.
CoreOptions	options()
TableScan.Plan	plan() Plan splits, throws EndOfScanException if the scan is ended.
void	restore(Long nextSnapshotId) Restore from checkpoint next snapshot id.
void	restore(Long nextSnapshotId, boolean scanAllSnapshot) Restore from checkpoint next snapshot id with scan kind.
StartingContext	startingContext()
Long	watermark() Current watermark for consumed snapshot.
org.apache.paimon.table.source.AbstractDataTableScan	withBucket(int bucket)
org.apache.paimon.table.source.AbstractDataTableScan	withBucketFilter(Filter<Integer> bucketFilter)
DataTableStreamScan	withFilter(Predicate predicate)
org.apache.paimon.table.source.AbstractDataTableScan	withLevelFilter(Filter<Integer> levelFilter)
org.apache.paimon.table.source.AbstractDataTableScan	withMetricsRegistry(MetricRegistry metricsRegistry)
org.apache.paimon.table.source.AbstractDataTableScan	withPartitionFilter(List<BinaryRow> partitions)
org.apache.paimon.table.source.AbstractDataTableScan	withPartitionFilter(Map<String,String> partitionSpec)
org.apache.paimon.table.source.AbstractDataTableScan	withPartitionsFilter(List<Map<String,String>> partitions)
DataTableScan	withShard(int indexOfThisSubtask, int numberOfParallelSubtasks) Specify the shard to be read, and allocate sharded files to read records.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.paimon.table.source.InnerTableScan

dropStats, withBucketFilter, withLevelFilter, withLimit, withMetricsRegistry, withPartitionFilter, withPartitionFilter, withPartitionsFilter

Methods inherited from interface org.apache.paimon.table.source.TableScan

listPartitions

Field Detail

snapshotReader

protected final SnapshotReader snapshotReader

Constructor Detail

DataTableStreamScan

public DataTableStreamScan(CoreOptions options,
                           SnapshotReader snapshotReader,
                           SnapshotManager snapshotManager,
                           ChangelogManager changelogManager,
                           boolean supportStreamingReadOverwrite,
                           DefaultValueAssigner defaultValueAssigner)

Method Detail

withFilter

public DataTableStreamScan withFilter(Predicate predicate)

Specified by:

withFilter in interface InnerTableScan

startingContext

public StartingContext startingContext()

Specified by:

startingContext in interface StreamDataTableScan

plan

public TableScan.Plan plan()

Description copied from interface: TableScan

Plan splits, throws EndOfScanException if the scan is ended.

Specified by:

plan in interface TableScan

listPartitionEntries

public List<PartitionEntry> listPartitionEntries()

Specified by:

listPartitionEntries in interface TableScan

handleOverwriteSnapshot

@Nullable
protected SnapshotReader.Plan handleOverwriteSnapshot(Snapshot snapshot)

createFollowUpScanner

protected FollowUpScanner createFollowUpScanner()

createBoundedChecker

protected BoundedChecker createBoundedChecker()

checkpoint

@Nullable
public Long checkpoint()

Description copied from interface: StreamTableScan

Checkpoint to return next snapshot id.

Specified by:

checkpoint in interface StreamTableScan

Specified by:

checkpoint in interface Restorable<Long>

watermark

@Nullable
public Long watermark()

Description copied from interface: StreamTableScan

Current watermark for consumed snapshot.

Specified by:

watermark in interface StreamTableScan

restore

public void restore(@Nullable
                    Long nextSnapshotId)

Description copied from interface: StreamTableScan

Restore from checkpoint next snapshot id.

Specified by:

restore in interface StreamTableScan

Specified by:

restore in interface Restorable<Long>

restore

public void restore(@Nullable
                    Long nextSnapshotId,
                    boolean scanAllSnapshot)

Description copied from interface: StreamDataTableScan

Restore from checkpoint next snapshot id with scan kind.

Specified by:

restore in interface StreamDataTableScan

notifyCheckpointComplete

public void notifyCheckpointComplete(@Nullable
                                     Long nextSnapshot)

Description copied from interface: StreamTableScan

Notifies the checkpoint complete with next snapshot id.

Specified by:

notifyCheckpointComplete in interface StreamTableScan

withShard

public DataTableScan withShard(int indexOfThisSubtask,
                               int numberOfParallelSubtasks)

Description copied from interface: DataTableScan

Specify the shard to be read, and allocate sharded files to read records.

Specified by:

withShard in interface DataTableScan

withBucket

@VisibleForTesting
public org.apache.paimon.table.source.AbstractDataTableScan withBucket(int bucket)

withBucketFilter

public org.apache.paimon.table.source.AbstractDataTableScan withBucketFilter(Filter<Integer> bucketFilter)

Specified by:

withBucketFilter in interface InnerTableScan

withPartitionFilter

public org.apache.paimon.table.source.AbstractDataTableScan withPartitionFilter(Map<String,String> partitionSpec)

Specified by:

withPartitionFilter in interface InnerTableScan

withPartitionFilter

public org.apache.paimon.table.source.AbstractDataTableScan withPartitionFilter(List<BinaryRow> partitions)

Specified by:

withPartitionFilter in interface InnerTableScan

withPartitionsFilter

public org.apache.paimon.table.source.AbstractDataTableScan withPartitionsFilter(List<Map<String,String>> partitions)

Specified by:

withPartitionsFilter in interface InnerTableScan

withLevelFilter

public org.apache.paimon.table.source.AbstractDataTableScan withLevelFilter(Filter<Integer> levelFilter)

Specified by:

withLevelFilter in interface InnerTableScan

withMetricsRegistry

public org.apache.paimon.table.source.AbstractDataTableScan withMetricsRegistry(MetricRegistry metricsRegistry)

Specified by:

withMetricsRegistry in interface InnerTableScan

dropStats

public org.apache.paimon.table.source.AbstractDataTableScan dropStats()

Specified by:

dropStats in interface InnerTableScan

options

public CoreOptions options()

createStartingScanner

protected StartingScanner createStartingScanner(boolean isStreaming)