Read#

Interface#

class TableRead#

Given a Split or a list of Split, generate a reader for batch reading.

A real-time split may create at most one successful reader. Calls using the same real-time split must not run concurrently, and the split must not be reused by multiple queries. A failed reader creation may be retried sequentially while its read-view ticket remains valid.

Public Functions

virtual ~TableRead() = default#

virtual Result<std::unique_ptr<BatchReader>> CreateReader(const std::vector<std::shared_ptr<Split>> &splits) = 0#

Creates a BatchReader instance for reading data.

This method creates a BatchReader that will be responsible for reading data from the provided splits.

Note

BatchReaders created by the same TableRead are not thread-safe for concurrent reading.

Parameters:: splits – A vector of shared pointers to Split instances representing the data to be read.
Returns:: A Result containing a unique pointer to the BatchReader instance.

virtual Result<std::unique_ptr<BatchReader>> CreateReader(const std::shared_ptr<Split> &split) = 0#

Creates a BatchReader instance for a single split.

Parameters:: split – A shared pointer to the Split instance that defines the data to be read.
Returns:: A Result containing a unique pointer to the BatchReader instance.

virtual Result<std::unique_ptr<CountReader>> CreateCountReader(const std::vector<std::shared_ptr<Split>> &splits)#

Creates a CountReader for count queries on the specified splits.

Implementations may override this to provide a more efficient count path.

Public Static Functions

static Result<std::unique_ptr<TableRead>> Create(std::unique_ptr<ReadContext> context)#

Create an instance of TableRead.

Parameters:: context – A unique pointer to the ReadContext used for read operations.
Returns:: A Result containing a unique pointer to the TableRead instance.

class ReadContextBuilder#

ReadContextBuilder used to build a ReadContext, has input validation.

Public Functions

explicit ReadContextBuilder(const std::string &path)#

Constructs a ReadContextBuilder with required parameters.

Parameters:: path – The root path of the table.

explicit ReadContextBuilder(const std::shared_ptr<FormatTable> &table)#

Constructs a ReadContextBuilder for a format table that is already loaded: the only way to read one whose schema lives in a metastore rather than under its location, such as a table a REST catalog serves.

The table carries what such a location does not say, so SetTableSchema(), WithFileSystem(), WithFileSystemSchemeToIdentifierMap() and a branch are refused here rather than ignored.

Parameters:: table – The format table to read, as Catalog::GetFormatTable() hands it back.

~ReadContextBuilder()#

ReadContextBuilder(ReadContextBuilder&&) noexcept#

ReadContextBuilder &operator=(ReadContextBuilder&&) noexcept#

ReadContextBuilder &SetReadFieldNames(const std::vector<std::string> &read_field_names)#

Set the schema fields to read from the table.

If not set, all fields from the table schema will be read. This is useful for projection pushdown to reduce I/O and improve performance by reading only the required columns.

Note

Currently supports top-level field selection. For nested field selection use SetReadSchema(std::unique_ptr<ArrowSchema>) instead.

Parameters:: read_field_names – Vector of field names to read from the table.
Returns:: Reference to this builder for method chaining.

ReadContextBuilder &SetReadFieldIds(const std::vector<int32_t> &read_field_ids)#

Set the schema fields to read from the table.

If not set, all fields from the table schema will be read. This is useful for projection pushdown to reduce I/O and improve performance by reading only the required columns.

Note

Currently supports top-level field selection.

Note

SetReadFieldIds() and SetReadFieldNames() are mutually exclusive. Calling both will ignore the read schema set by SetReadFieldNames().

Parameters:: read_field_ids – Vector of field ids to read from the table.
Returns:: Reference to this builder for method chaining.

ReadContextBuilder &SetReadSchema(std::unique_ptr<ArrowSchema> read_schema)#

Set the read Arrow Schema for nested column pruning.

The read schema is an Arrow C Data Interface schema where STRUCT types may contain only a subset of the original sub-fields, enabling nested column pruning to reduce I/O. Field matching is based on field name: the system looks up each field by name in the table schema and rebuilds the aligned schema using the table schema’s type and metadata. Metadata propagation from the user-provided schema is whitelist-based: currently only “paimon.map.selected-keys” is preserved and merged into the final aligned schema.

To prune map entries by key, attach metadata “paimon.map.selected-keys” to the target map field in read schema. The value is a comma-separated key list, for example: “k1,k2”. Only map fields with string key type (Arrow utf8) are supported.

Attaching this metadata to a MAP field only filters the returned MAP after reading. To push down selected keys from a shared-shredding MAP and return them as STRUCT children, build the field with MapSharedShreddingAccessBuilder. To read selected paths from a VARIANT field, build the field with VariantAccessBuilder.

Example:

auto map_field = arrow::field("m", arrow::map(arrow::utf8(), arrow::int32()));
auto map_meta = arrow::KeyValueMetadata::Make(
    {"paimon.map.selected-keys"}, {"k1,k2"});
auto projected_schema = arrow::schema({
    arrow::field("id", arrow::int64()),
map_field->WithMetadata(map_meta),
});

auto c_schema = std::make_unique<ArrowSchema>();
arrow::ExportSchema(*projected_schema, c_schema.get());

ReadContextBuilder builder("/path/to/table");
builder.SetReadSchema(std::move(c_schema));

Note

Priority: read_schema > read_field_ids > read_field_names. When set, read_field_ids and read_field_names are ignored.

Parameters:: read_schema – Arrow C Schema. Ownership of schema resources is transferred to the built ReadContext.
Returns:: Reference to this builder for method chaining.

ReadContextBuilder &SetOptions(const std::map<std::string, std::string> &options)#

Set a configuration options map to set some option entries which are not defined in the table schema or whose values you want to overwrite.

Note

The options map will clear the options added by AddOption() before.

Parameters:: options – The configuration options map.
Returns:: Reference to this builder for method chaining.

ReadContextBuilder &AddOption(const std::string &key, const std::string &value)#

Add a single configuration option which is not defined in the table schema or whose value you want to overwrite.

If you want to add multiple options, call AddOption() multiple times or use SetOptions() instead.

Parameters:

key – The option key.
value – The option value.

Returns:

Reference to this builder for method chaining.

ReadContextBuilder &SetPredicate(const std::shared_ptr<Predicate> &predicate)#

Set a predicate for filtering data during reading.

The predicate is used for both partition pruning and data filtering. It can significantly improve performance by reducing the amount of data that needs to be read and processed.

Parameters:: predicate – Shared pointer to the predicate for data filtering.
Returns:: Reference to this builder for method chaining.

ReadContextBuilder &EnablePredicateFilter(bool enabled)#

Whether to perform precise filtering according to predicates for data read from format reader.

Parameters:: enabled – Whether to enable precise filtering (default: false)
Returns:: Reference to this builder for method chaining.

ReadContextBuilder &EnablePrefetch(bool enabled)#

Enable or disable prefetching of data batches from individual files.

When enabled, the reader will prefetch multiple batches in parallel to improve throughput by overlapping I/O with computation. This is particularly beneficial for high-latency storage systems.

Parameters:: enabled – Whether to enable prefetching (default: false)
Returns:: Reference to this builder for method chaining.

ReadContextBuilder &EnableLateMaterializing(bool enabled)#

Enable or disable late materialization (probe/payload two-phase reads).

When enabled, each parallel reader under the prefetch layer performs a probe read of predicate columns first and only materializes payload columns for matched rows.

Note

Without a pushed-down predicate the late-materializing reader degrades to a plain passthrough.

Parameters:: enabled – Whether to enable late materialization (default: false)
Returns:: Reference to this builder for method chaining.

ReadContextBuilder &SetReadAheadCacheEnabled(bool enabled)#

Enable or disable the read-ahead cache for read operations.

A read-ahead cache is used to prebuffer data ranges before they are needed, which can improve read performance by reducing redundant I/O operations.

Parameters:: enabled – Whether to enable the read-ahead cache (default: true)
Returns:: Reference to this builder for method chaining.

ReadContextBuilder &WithCacheConfig(const CacheConfig &config)#

Set the cache configuration for prefetch read operations.

Parameters:: config – The cache configuration to use.
Returns:: Reference to this builder for method chaining.

ReadContextBuilder &SetWarmupLevel(WarmupLevel level)#

Set how far the reader prepares the next file before it is read.

Warmup overlaps remote-storage latency with the read of the current file. Higher levels hide more latency but use more memory, and may warm files that a query never reads (for example when a LIMIT stops the scan early).