org.apache.paimon.table.source

Interface ReadBuilder

All Superinterfaces:

Serializable

All Known Implementing Classes:

ReadBuilderImpl

@Public
public interface ReadBuilder
extends Serializable

An interface for building the TableScan and TableRead.

Example of distributed reading:

 // 1. Create a ReadBuilder (Serializable)
 Table table = catalog.getTable(...);
 ReadBuilder builder = table.newReadBuilder()
     .withFilter(...)
     .withProjection(...);

 // 2. Plan splits in 'Coordinator' (or named 'Driver'):
 List<Split> splits = builder.newScan().plan().splits();

 // 3. Distribute these splits to different tasks

 // 4. Read a split in task
 TableRead read = builder.newRead();
 RecordReader<InternalRow> reader = read.createReader(split);
 reader.forEachRemaining(...);

 

newStreamScan() will create a stream scan, which can perform continuously planning:

 TableScan scan = builder.newStreamScan();
 while (true) {
     List<Split> splits = scan.plan().splits();
     ...
 }
 

NOTE: InternalRow cannot be saved in memory. It may be reused internally, so you need to convert it into your own data structure or copy it.

Since:

0.4.0

Method Summary

Modifier and Type	Method and Description
TableRead	newRead() Create a TableRead to read Splits.
TableScan	newScan() Create a TableScan to perform batch planning.
StreamTableScan	newStreamScan() Create a TableScan to perform streaming planning.
RowType	readType() Returns read row type, projected by withProjection(int[]).
String	tableName() A name to identify the table.
ReadBuilder	withBucketFilter(Filter<Integer> bucketFilter) Push bucket filter.
default ReadBuilder	withFilter(List<Predicate> predicates) Apply filters to the readers to decrease the number of produced records.
ReadBuilder	withFilter(Predicate predicate) Push filters, will filter the data as much as possible, but it is not guaranteed that it is a complete filter.
ReadBuilder	withLimit(int limit) the row number pushed down.
ReadBuilder	withPartitionFilter(Map<String,String> partitionSpec) Push partition filter.
default ReadBuilder	withProjection(int[] projection) Apply projection to the reader.
ReadBuilder	withProjection(int[][] projection) Push nested projection.
ReadBuilder	withShard(int indexOfThisSubtask, int numberOfParallelSubtasks) Specify the shard to be read, and allocate sharded files to read records.

Method Detail

tableName

String tableName()

A name to identify the table.

readType

RowType readType()

Returns read row type, projected by withProjection(int[]).

withFilter

default ReadBuilder withFilter(List<Predicate> predicates)

Apply filters to the readers to decrease the number of produced records.

This interface filters records as much as possible, however some produced records may not satisfy all predicates. Users need to recheck all records.

withFilter

ReadBuilder withFilter(Predicate predicate)

Push filters, will filter the data as much as possible, but it is not guaranteed that it is a complete filter.

withPartitionFilter

ReadBuilder withPartitionFilter(Map<String,String> partitionSpec)

Push partition filter.

withBucketFilter

ReadBuilder withBucketFilter(Filter<Integer> bucketFilter)

Push bucket filter. Note that this method cannot be used simultaneously with withShard(int, int).

Reason: Bucket filtering and sharding are different logical mechanisms for selecting subsets of table data. Applying both methods simultaneously introduces conflicting selection criteria.

withProjection

default ReadBuilder withProjection(int[] projection)

Apply projection to the reader.

NOTE: Nested row projection is currently not supported.

withProjection

ReadBuilder withProjection(int[][] projection)

Push nested projection. For example, [[0, 2, 1], ...] specifies to include the 2nd field of the 3rd field of the 1st field in the top-level row.

withLimit

ReadBuilder withLimit(int limit)

the row number pushed down.

withShard

ReadBuilder withShard(int indexOfThisSubtask,
                      int numberOfParallelSubtasks)

Specify the shard to be read, and allocate sharded files to read records. Note that this method cannot be used simultaneously with withBucketFilter(Filter).

Reason: Sharding and bucket filtering are different logical mechanisms for selecting subsets of table data. Applying both methods simultaneously introduces conflicting selection criteria.

newScan

TableScan newScan()

Create a TableScan to perform batch planning.

newStreamScan

StreamTableScan newStreamScan()

Create a TableScan to perform streaming planning.

newRead

TableRead newRead()

Create a TableRead to read Splits.