Scan#

Bucket pruning#

For fixed-bucket append and primary-key tables, an equality predicate on every bucket key lets the scan derive the target bucket using the table’s bucket function. Other buckets are excluded from the scan plan without requiring an explicit bucket ID from the caller. An explicit bucket filter takes precedence. Queries that do not constrain all bucket keys with equality, and bucket-unaware tables, keep the existing scan behavior. Both scan types use a shared selector that computes the bucket with each manifest entry’s total bucket count, so rescaled files are not filtered using the current table’s bucket count. Historical schemas remain eligible when their ordered bucket-key field IDs, types and bucket function match. Changes to unrelated columns do not disable pruning. Incompatible bucket schemas and nonpositive total bucket counts retain the existing filtering behavior.

This inference prunes data files at the manifest-entry level. Manifest min/max-bucket skipping requires an explicit bucket filter. When the snapshot live-manifest-entry cache is enabled, inferred scans cache candidates by bucket, current bucket count and current schema ID. Files with other bucket counts or schema IDs remain in the cached candidates and are filtered after lookup. This permits cache reuse without discarding files that require a different bucket calculation or schema fallback.

Decimal literals are rescaled to the bucket field’s type only when the conversion is exact. NaN literals and decimals that cannot be represented exactly disable inferred bucket pruning.

Interface#

class TableScan#

A scanner interface for reading table’s meta and create a plan.

Public Functions

virtual ~TableScan() = default#

virtual Result<std::shared_ptr<Plan>> CreatePlan() = 0#

Create a scan plan.

Returns:: A Result containing a shared pointer to the created Plan or an error status.

virtual std::shared_ptr<Metrics> GetMetrics() const#

Retrieve metrics related to scan planning operations.

Returns:: A point-in-time snapshot of scan metrics. Mutating the returned object does not affect metrics collected by this scan.

virtual Result<std::vector<std::map<std::string, std::string>>> ListPartitions() const#

Lists the partitions the scan can see, each as its partition values keyed by field name, in a stable order.

A table with no partition keys has none.

Only a format table answers it today: its partitions are the directories a scan descends, so listing them costs no more than planning. Every other table type returns NotImplemented.

Returns:: A Result containing the partitions, or an error status.

Public Static Functions

static Result<std::unique_ptr<TableScan>> Create(std::unique_ptr<ScanContext> context)#

Create an instance of TableScan.

Parameters:: context – A unique pointer to the ScanContext used for scan operations.
Returns:: A Result containing a unique pointer to the TableScan instance.

class ScanContextBuilder#

ScanContextBuilder used to build a ScanContext, has input validation.

Public Functions

explicit ScanContextBuilder(const std::string &path)#

Constructs a ScanContextBuilder with required parameters.

Parameters:: path – The root path of the table.

explicit ScanContextBuilder(const std::shared_ptr<FormatTable> &table)#

Constructs a ScanContextBuilder for a format table that is already loaded: the only way to scan one whose schema lives in a metastore rather than under its location, such as a table a REST catalog serves.

The table carries what such a location does not say, so SetTableSchema() and WithFileSystem() are refused here rather than ignored.

Parameters:: table – The format table to scan, as Catalog::GetFormatTable() hands it back.

~ScanContextBuilder()#

ScanContextBuilder &SetLimit(int32_t limit)#: If limit is not set, it defaults to unlimited.

ScanContextBuilder &SetBucketFilter(int32_t bucket_filter)#: Set a bucket filter to scan only specific bucket.

ScanContextBuilder &SetPartitionFilter(const std::vector<std::map<std::string, std::string>> &partition_filters)#: partition_filters in vector is supposed to be OR, filter in map is supposed to be AND, e.g., partition_filters is {{k1=1,k2=10}, {k1=2,k2=20}} => OR(AND(k1=1,k2=10), AND(k1=2,k2=20))

ScanContextBuilder &SetPredicate(const std::shared_ptr<Predicate> &predicate)#: Set a predicate for filtering data.

ScanContextBuilder &SetGlobalIndexResult(const std::shared_ptr<GlobalIndexResult> &global_index_result)#

Sets the result of a global index search (e.g., row ids (may with scores) from a distributed index lookup).

This is used to push down index-filtered row ids into the scan for efficient data retrieval.

ScanContextBuilder &WithRealtimeContext(const std::shared_ptr<RealtimeContext> &realtime_context)#: Enables process-local union reads with the real-time stores owned by realtime_context.

ScanContextBuilder &AddOption(const std::string &key, const std::string &value)#: The options added or set in ScanContextBuilder have high priority and will be merged with the options in table schema.

ScanContextBuilder &SetOptions(const std::map<std::string, std::string> &options)#

Set a configuration options map to set some option entries which are not defined in the table schema or whose values you want to overwrite.

Note

The options map will clear the options added by AddOption() before.

Parameters:: options – The configuration options map.
Returns:: Reference to this builder for method chaining.

ScanContextBuilder &WithStreamingMode(bool is_streaming_mode)#: Set whether the scan is in streaming mode.

Note

if not set, is_streaming_mode = false

ScanContextBuilder &WithMemoryPool(const std::shared_ptr<MemoryPool> &memory_pool)#

Set custom memory pool for memory management.

Note

if not set, memory_pool is default pool

Parameters:: memory_pool – The memory pool to use.
Returns:: Reference to this builder for method chaining.

ScanContextBuilder &WithExecutor(const std::shared_ptr<Executor> &executor)#

Set custom executor for task execution.

Note

if not set, executor is default executor

Parameters:: executor – The executor to use.
Returns:: Reference to this builder for method chaining.

ScanContextBuilder &WithFileSystem(const std::shared_ptr<FileSystem> &file_system)#

Sets a custom file system instance to be used for all file operations in this scan context.

This bypasses the global file system registry and uses the provided implementation directly.

Note

If not set, use default file system (configured in Options::FILE_SYSTEM)

Parameters:: file_system – The file system to use.
Returns:: Reference to this builder for method chaining.

ScanContextBuilder &WithCatalog(const std::shared_ptr<Catalog> &catalog, const Identifier &identifier)#

Plans a native table through its own file system - including the per-table temporary credentials a catalog that issues them hands out through Catalog::GetTableFileSystem.

This is a shorthand for WithFileSystem(catalog->GetTableFileSystem(identifier)); an explicit WithFileSystem() takes precedence, so the catalog is not asked.

Parameters:

catalog – Non-null catalog, read when Finish() builds the context.
identifier – The native table to scan.

Returns:

Reference to this builder for method chaining.

ScanContextBuilder &SetTableSchema(const std::string &table_schema)#

Set the table schema as a string to avoid schema loading I/O operations.

This optimization allows the scanner to use a pre-loaded schema instead of reading it from the table metadata, which can improve performance especially in scenarios with many small scan operations.

Note

The user must ensure that the schema string is valid and matches the table.

Note

If not set, the schema will be loaded from the table path.

Parameters:: table_schema – String representation of the table schema.
Returns:: Reference to this builder for method chaining.

ScanContextBuilder &WithCache(const std::shared_ptr<Cache> &cache)#

Inject a cache for scan operations.

Passing nullptr disables cache.

Returns:: Reference to this builder for method chaining.

Result<std::unique_ptr<ScanContext>> Finish()#

Build and return a ScanContext instance with input validation.

Returns:: Result containing the constructed ScanContext or an error status.

class ScanContext#

ScanContext is some configuration for table scan operations.

Please do not use this class directly, use ScanContextBuilder to build a ScanContext which has input validation.