org.apache.paimon.table.source

Interface ReadBuilder

All Superinterfaces:

Serializable

All Known Implementing Classes:

ReadBuilderImpl

@Public
public interface ReadBuilder
extends Serializable

An interface for building the TableScan and TableRead.

Example of distributed reading:

 // 1. Create a ReadBuilder (Serializable)
 Table table = catalog.getTable(...);
 ReadBuilder builder = table.newReadBuilder()
     .withFilter(...)
     .withReadType(...);

 // 2. Plan splits in 'Coordinator' (or named 'Driver'):
 List<Split> splits = builder.newScan().plan().splits();

 // 3. Distribute these splits to different tasks

 // 4. Read a split in task
 TableRead read = builder.newRead();
 RecordReader<InternalRow> reader = read.createReader(split);
 reader.forEachRemaining(...);

 

newStreamScan() will create a stream scan, which can perform continuously planning:

 TableScan scan = builder.newStreamScan();
 while (true) {
     List<Split> splits = scan.plan().splits();
     ...
 }
 

NOTE: InternalRow cannot be saved in memory. It may be reused internally, so you need to convert it into your own data structure or copy it.

Since:

0.4.0

Method Summary

Modifier and Type	Method and Description
TableRead	newRead() Create a TableRead to read Splits.
TableScan	newScan() Create a TableScan to perform batch planning.
StreamTableScan	newStreamScan() Create a TableScan to perform streaming planning.
RowType	readType() Returns read row type.
String	tableName() A name to identify the table.
ReadBuilder	withBucketFilter(Filter<Integer> bucketFilter) Push bucket filter.
default ReadBuilder	withFilter(List<Predicate> predicates) Apply filters to the readers to decrease the number of produced records.
ReadBuilder	withFilter(Predicate predicate) Push filters, will filter the data as much as possible, but it is not guaranteed that it is a complete filter.
ReadBuilder	withLimit(int limit) the row number pushed down.
ReadBuilder	withPartitionFilter(Map<String,String> partitionSpec) Push partition filter.
ReadBuilder	withProjection(int[] projection) Apply projection to the reader, if you need nested row pruning, use withReadType(RowType) instead.
default ReadBuilder	withProjection(int[][] projection) Deprecated.
ReadBuilder	withReadType(RowType readType) Push read row type to the reader, support nested row pruning.
ReadBuilder	withShard(int indexOfThisSubtask, int numberOfParallelSubtasks) Specify the shard to be read, and allocate sharded files to read records.

Method Detail

tableName

String tableName()

A name to identify the table.

readType

RowType readType()

Returns read row type.

withFilter

default ReadBuilder withFilter(List<Predicate> predicates)

Apply filters to the readers to decrease the number of produced records.

This interface filters records as much as possible, however some produced records may not satisfy all predicates. Users need to recheck all records.

withFilter

ReadBuilder withFilter(Predicate predicate)

Push filters, will filter the data as much as possible, but it is not guaranteed that it is a complete filter.

withPartitionFilter

ReadBuilder withPartitionFilter(Map<String,String> partitionSpec)

Push partition filter.

withBucketFilter

ReadBuilder withBucketFilter(Filter<Integer> bucketFilter)

Push bucket filter. Note that this method cannot be used simultaneously with withShard(int, int).

Reason: Bucket filtering and sharding are different logical mechanisms for selecting subsets of table data. Applying both methods simultaneously introduces conflicting selection criteria.

withReadType

ReadBuilder withReadType(RowType readType)

Push read row type to the reader, support nested row pruning.

Parameters:

readType - read row type, can be a pruned type from Table.rowType()

Since:

1.0.0

withProjection

ReadBuilder withProjection(int[] projection)

Apply projection to the reader, if you need nested row pruning, use withReadType(RowType) instead.

withProjection

@Deprecated
default ReadBuilder withProjection(int[][] projection)

Deprecated.

Apply projection to the reader, only support top level projection.

withLimit

ReadBuilder withLimit(int limit)

the row number pushed down.

withShard

ReadBuilder withShard(int indexOfThisSubtask,
                      int numberOfParallelSubtasks)

Specify the shard to be read, and allocate sharded files to read records. Note that this method cannot be used simultaneously with withBucketFilter(Filter).

Reason: Sharding and bucket filtering are different logical mechanisms for selecting subsets of table data. Applying both methods simultaneously introduces conflicting selection criteria.

newScan

TableScan newScan()

Create a TableScan to perform batch planning.

newStreamScan

StreamTableScan newStreamScan()

Create a TableScan to perform streaming planning.

newRead

TableRead newRead()

Create a TableRead to read Splits.