Skip to main content

Global Index

Overview​

Global Index is a powerful indexing mechanism for Data Evolution (append) tables. It enables efficient row-level lookups and filtering without full-table scans. Paimon supports multiple global index types:

  • BTree Index: A B-tree based index for scalar column lookups. Supports equality, IN, range predicates, and can be combined across multiple columns with AND/OR logic.
  • Bitmap Index: A bitmap based index for enum-like scalar dimensions and tag columns. Supports equality, IN, prefix match on string columns, complement predicates, and null checks with compressed row-id bitmaps.
  • Vector Index: An approximate nearest neighbor (ANN) index powered by Paimon's vector index library for vector similarity search.
  • Full-Text Index: A full-text search index backed by the native full-text engine for text retrieval. Supports term matching and relevance scoring.
  • Hybrid Search: A multi-route search API that combines results from multiple vector routes, multiple full-text routes, or both before reading table rows.
Index TypeBest ForNotes
BTreeScalar filters on numeric, string, date, and timestamp columnsBest when predicates are selective, such as equality, IN, range, and null checks.
BitmapEnum-like dimensions and tag columnsBest for equality, IN, string prefix match, complement predicates, and null checks over compressed row-id bitmaps.
VectorTop-K similarity search on embeddingsUses ANN algorithms. Tune build-time and search-time options to balance recall, latency, and index size.
Full-TextKeyword search over text columnsUses full-text scoring and tokenizer configuration stored with each index file.
Hybrid SearchCombining multiple vector routes, multiple full-text routes, or vector and full-text retrieval togetherRuns multiple scored routes and merges them with a ranker before reading rows.

Global indexes work on top of Data Evolution tables. To use global indexes, your table must have:

  • 'bucket' = '-1' (unaware-bucket mode)
  • 'row-tracking.enabled' = 'true'
  • 'data-evolution.enabled' = 'true'

Global index queries may not be exact when the index only covers part of the table data. If a query predicate matches the index, Paimon returns only the results from the indexed portion. Matching records in data that has not been indexed yet will not be returned.

Prerequisites​

Create a table with the required properties:

CREATE TABLE my_table (
    id INT,
    name STRING,
    embedding ARRAY<FLOAT>,
    content STRING
) TBLPROPERTIES (
    'bucket' = '-1',
    'row-tracking.enabled' = 'true',
    'data-evolution.enabled' = 'true',
    'global-index.enabled' = 'true'
);

Lifecycle​

Create global indexes for all partitions or only selected partitions:

CALL sys.create_global_index(
    table => 'db.my_table',
    index_column => 'name',
    index_type => 'btree'
);

CALL sys.create_global_index(
    table => 'db.my_table',
    index_column => 'name',
    index_type => 'btree',
    partitions => 'dt=2026-06-18;dt=2026-06-19'
);

PyPaimon global index build currently supports single-column BTree indexes, single-column Bitmap indexes, single-column paimon-vindex IVF vector indexes, and single-column full-text indexes on tables with row tracking enabled.

Drop index files:

CALL sys.drop_global_index(
    table => 'db.my_table',
    index_column => 'name',
    index_type => 'btree'
);

Global indexes are stored in index files and recorded in table metadata. To inspect index files and their row-id coverage, query the table_indexes system table:

SELECT index_type, index_field_name, row_range_start, row_range_end
FROM my_table$table_indexes
WHERE index_field_name IS NOT NULL;

You can also query file_key_ranges to inspect data file row-id ranges and diagnose coverage:

SELECT file_path, first_row_id, record_count
FROM my_table$file_key_ranges;

For workloads that need newly appended rows to become visible only after existing global indexes cover them, enable the visibility callback:

ALTER TABLE my_table SET (
    'visibility-callback.enabled' = 'true',
    'visibility-callback.timeout' = '30 min'
);

Coverage and Freshness​

Global index files cover row-id ranges. If more rows are appended after an index is built, those new rows are not automatically covered by the existing index files. Build the global index again to create index files for newly uncovered data. By default, queries use fast search and only read indexed row ranges; rows in uncovered ranges are not returned for that indexed query.

To improve freshness for query types that support raw-data search, set:

ALTER TABLE my_table SET ('global-index.search-mode' = 'full');

With full search, supported global-index queries first use the snapshot nextRowId and global index row-id coverage to detect whether any row range is missing from the index. Raw data is scanned only when such a gap exists. Use detail search when data files may have been rewritten or updated after index creation; it scans data file metadata to find the exact unindexed row ranges and can handle index invalidation caused by updates or rewrites.

To temporarily disable global-index scan acceleration while keeping the index files, set:

ALTER TABLE my_table SET ('global-index.enabled' = 'false');

Set it back to true to use global indexes during scans again.

Build Options​

These table options affect global index build and read behavior:

OptionDefaultDescription
global-index.enabledtrueWhether scans can use global indexes.
global-index.search-modefastSearch mode for global-index queries. fast searches indexed data only. full checks snapshot nextRowId against global index row-id coverage and scans raw data only if a gap exists. detail scans data file metadata to find exact unindexed rows and can handle index invalidation caused by updates or rewrites.
global-index.external-pathNot setRoot directory for global index files. If not set, files are stored under the table index directory.
sorted-index.records-per-range10000000Expected number of records per sorted global index file for BTree and Bitmap builds.
sorted-index.build.max-parallelism4096Maximum Flink or Spark parallelism for building sorted global indexes.
global-index.row-count-per-shard100000Target row count per shard for non-sorted global index builds such as vector and full-text indexes.
global-index.build.max-shard32Preferred maximum shard count for global index builds.
global-index.build.max-parallelism4096Maximum Flink or Spark parallelism for building non-sorted global indexes.
global-index.thread-num32Maximum number of concurrent threads for global index I/O.
visibility-callback.enabledfalseWhether batch or bounded-stream commits wait until existing global indexes cover newly added files.
visibility-callback.timeout30 minMaximum wait time for visibility callback.

BTree Index​

Use BTree indexes for scalar column lookups and range predicates. See BTree Index for build and query examples.

Bitmap Index​

Use Bitmap indexes for enum-like dimensions and tag columns queried by equality, IN, string prefix match, complement, or null predicates. See Bitmap Index for build examples, options, and file format details.

Vector Index​

Use Vector indexes for approximate nearest neighbor (ANN) search. See Vector Index for supported index types, options, and search examples.

Full-Text Index​

Use Full-Text indexes for text retrieval backed by the native full-text engine. See Full-Text Index for tokenizer options and search examples.

Use Hybrid Search to query multiple vector routes, multiple full-text routes, or a mix of both in one request. See Hybrid Search for route configuration, rankers, and API examples.