Global Index

Overview

Global Index is a powerful indexing mechanism for Data Evolution (append) tables. It enables efficient row-level lookups and filtering without full-table scans. Paimon supports multiple global index types:

BTree Index: A B-tree based index for scalar column lookups. Supports equality, IN, range predicates, and can be combined across multiple columns with AND/OR logic.
Bitmap Index: A bitmap based index for enum-like scalar dimensions and tag columns. Supports equality, IN, prefix match on string columns, complement predicates, and null checks with compressed row-id bitmaps.
Vector Index: An approximate nearest neighbor (ANN) index powered by Paimon's vector index library for vector similarity search.
Full-Text Index: A full-text search index backed by the native full-text engine for text retrieval. Supports term matching and relevance scoring.
Hybrid Search: A multi-route search API that combines results from multiple vector routes, multiple full-text routes, or both before reading table rows.

Index Type	Best For	Notes
BTree	Scalar filters on numeric, string, date, and timestamp columns	Best when predicates are selective, such as equality, IN, range, and null checks.
Bitmap	Enum-like dimensions and tag columns	Best for equality, IN, string prefix match, complement predicates, and null checks over compressed row-id bitmaps.
Vector	Top-K similarity search on embeddings	Uses ANN algorithms. Tune build-time and search-time options to balance recall, latency, and index size.
Full-Text	Keyword search over text columns	Uses full-text scoring and tokenizer configuration stored with each index file.
Hybrid Search	Combining multiple vector routes, multiple full-text routes, or vector and full-text retrieval together	Runs multiple scored routes and merges them with a ranker before reading rows.

Global indexes work on top of Data Evolution tables. To use global indexes, your table must have:

'bucket' = '-1' (unaware-bucket mode)
'row-tracking.enabled' = 'true'
'data-evolution.enabled' = 'true'

Global index queries may not be exact when the index only covers part of the table data. If a query predicate matches the index, Paimon returns only the results from the indexed portion. Matching records in data that has not been indexed yet will not be returned.

Prerequisites

Create a table with the required properties:

SQL
Python SDK

CREATE TABLE my_table (
    id INT,
    name STRING,
    embedding ARRAY<FLOAT>,
    content STRING
) TBLPROPERTIES (
    'bucket' = '-1',
    'row-tracking.enabled' = 'true',
    'data-evolution.enabled' = 'true',
    'global-index.enabled' = 'true'
);

import pyarrow as pa

from pypaimon import Schema

schema = Schema.from_pyarrow_schema(
    pa.schema([
        pa.field("id", pa.int32()),
        pa.field("name", pa.string()),
        pa.field("embedding", pa.list_(pa.float32())),
        pa.field("content", pa.string()),
    ]),
    options={
        "bucket": "-1",
        "row-tracking.enabled": "true",
        "data-evolution.enabled": "true",
        "global-index.enabled": "true",
    },
)

catalog.create_table("db.my_table", schema, ignore_if_exists=False)

Lifecycle

Create global indexes for all partitions or only selected partitions:

SQL
Python SDK

CALL sys.create_global_index(
    table => 'db.my_table',
    index_column => 'name',
    index_type => 'btree'
);

CALL sys.create_global_index(
    table => 'db.my_table',
    index_column => 'name',
    index_type => 'btree',
    partitions => 'dt=2026-06-18;dt=2026-06-19'
);

table = catalog.get_table("db.my_table")

added_files = table.create_global_index("name")
print(added_files)

The API returns the number of committed index files. You can pass build options and restrict the build to selected partitions:

added_files = table.create_global_index(
    "name",
    index_type="btree",
    partitions=[{"dt": "2026-06-18"}, {"dt": "2026-06-19"}],
    options={"sorted-index.records-per-range": "10000000"},
)

PyPaimon global index build currently supports single-column BTree indexes, single-column Bitmap indexes, single-column paimon-vindex vector indexes, and single-column full-text indexes on tables with row tracking enabled.

Drop index files:

SQL
Python SDK

CALL sys.drop_global_index(
    table => 'db.my_table',
    index_column => 'name',
    index_type => 'btree'
);

table = catalog.get_table("db.my_table")

dropped_files = table.drop_global_index("name", index_type="btree")
print(dropped_files)

You can also restrict the drop to selected partitions, or count matched files without committing:

matched_files = table.drop_global_index(
    "name",
    index_type="btree",
    partitions=[{"dt": "2026-06-18"}, {"dt": "2026-06-19"}],
    dry_run=True,
)

Global indexes are stored in index files and recorded in table metadata. To inspect index files and their row-id coverage, query the table_indexes system table:

SQL
Python SDK

SELECT index_type, index_field_name, row_range_start, row_range_end
FROM my_table$table_indexes
WHERE index_field_name IS NOT NULL;

import pyarrow.compute as pc

table_indexes = catalog.get_table("db.my_table$table_indexes")
read_builder = table_indexes.new_read_builder().with_projection([
    "index_type",
    "index_field_name",
    "row_range_start",
    "row_range_end",
])

pa_table = read_builder.new_read().to_arrow(
    read_builder.new_scan().plan().splits()
)
pa_table = pa_table.filter(pc.is_valid(pa_table["index_field_name"]))
print(pa_table)

You can also query file_key_ranges to inspect data file row-id ranges and diagnose coverage:

SQL
Python SDK

SELECT file_path, first_row_id, record_count
FROM my_table$file_key_ranges;

file_key_ranges = catalog.get_table("db.my_table$file_key_ranges")
read_builder = file_key_ranges.new_read_builder().with_projection([
    "file_path",
    "first_row_id",
    "record_count",
])

pa_table = read_builder.new_read().to_arrow(
    read_builder.new_scan().plan().splits()
)
print(pa_table)

For workloads that need newly appended rows to become visible only after existing global indexes cover them, enable the visibility callback:

SQL
Python SDK

ALTER TABLE my_table SET (
    'visibility-callback.enabled' = 'true',
    'visibility-callback.timeout' = '30 min'
);

from pypaimon.schema.schema_change import SchemaChange

catalog.alter_table(
    "db.my_table",
    [
        SchemaChange.set_option("visibility-callback.enabled", "true"),
        SchemaChange.set_option("visibility-callback.timeout", "30 min"),
    ],
)

Coverage and Freshness

Global index files cover row-id ranges. If more rows are appended after an index is built, those new rows are not automatically covered by the existing index files. Build the global index again to create index files for newly uncovered data. Scalar queries default to full search, while vector and full-text queries default to fast and only search indexed row ranges.

To improve freshness for query types that support raw-data search, set:

SQL
Python SDK

ALTER TABLE my_table SET ('vector-index.search-mode' = 'full');

from pypaimon.schema.schema_change import SchemaChange

catalog.alter_table(
    "db.my_table",
    [SchemaChange.set_option("vector-index.search-mode", "full")],
)

For a read-only override on an existing Table instance:

full_table = table.copy({"vector-index.search-mode": "full"})

With full search, supported global-index queries first use the snapshot nextRowId and global index row-id coverage to detect whether any row range is missing from the index. Raw data is scanned only when such a gap exists. Use detail search when data files may have been rewritten or updated after index creation; it scans data file metadata to find the exact unindexed row ranges and can handle index invalidation caused by updates or rewrites.

Use scalar-index.search-mode, vector-index.search-mode, or full-text-index.search-mode for one index family. The legacy global-index.search-mode has no default and is used as a fallback when the corresponding family-specific option is not explicitly configured.

To temporarily disable global-index scan acceleration while keeping the index files, set:

SQL
Python SDK

ALTER TABLE my_table SET ('global-index.enabled' = 'false');

from pypaimon.schema.schema_change import SchemaChange

catalog.alter_table(
    "db.my_table",
    [SchemaChange.set_option("global-index.enabled", "false")],
)

Set it back to true to use global indexes during scans again.

Build Options

These table options affect global index build and read behavior:

Option	Default	Description
`global-index.enabled`	`true`	Whether scans can use global indexes.
`global-index.search-mode`	Not set	Legacy fallback search mode for global-index queries. Family-specific options take precedence.
`scalar-index.search-mode`	`fast`	Search mode for BTree and Bitmap queries.
`vector-index.search-mode`	`fast`	Search mode for vector queries.
`full-text-index.search-mode`	`fast`	Search mode for full-text queries.
`global-index.external-path`	Not set	Root directory for global index files. If not set, files are stored under the table index directory.
`global-index.column-update-action`	`THROW_ERROR`	Action for updates to indexed columns. `THROW_ERROR` rejects the update, `DROP_PARTITION_INDEX` drops affected partition indexes, and `IGNORE` keeps existing index files unchanged. Use `IGNORE` only when a stale index is acceptable; for example, a vector updated from `NULL` to a value remains invisible until the index is rebuilt.
`sorted-index.records-per-range`	`10000000`	Expected number of records per sorted global index file for BTree and Bitmap builds.
`sorted-index.build.max-parallelism`	`4096`	Maximum Flink or Spark parallelism for building sorted global indexes.
`global-index.row-count-per-shard`	`100000`	Target row count per shard for non-sorted global index builds such as vector and full-text indexes.
`global-index.build.max-shard`	`32`	Preferred maximum shard count for global index builds.
`global-index.build.max-parallelism`	`4096`	Maximum Flink or Spark parallelism for building non-sorted global indexes.
`global-index.thread-num`	`32`	Maximum number of concurrent threads for global index I/O.
`visibility-callback.enabled`	`false`	Whether batch or bounded-stream commits wait until existing global indexes cover newly added files.
`visibility-callback.timeout`	`30 min`	Maximum wait time for visibility callback.

BTree Index

Use BTree indexes for scalar column lookups and range predicates. See BTree Index for build and query examples.

Bitmap Index

Use Bitmap indexes for enum-like dimensions and tag columns queried by equality, IN, string prefix match, complement, or null predicates. See Bitmap Index for build examples, options, and file format details.

Vector Index

Use Vector indexes for approximate nearest neighbor (ANN) search. See Vector Index for supported index types, options, and search examples.

Full-Text Index

Use Full-Text indexes for text retrieval backed by the native full-text engine. See Full-Text Index for tokenizer options and search examples.

Hybrid Search

Use Hybrid Search to query multiple vector routes, multiple full-text routes, or a mix of both in one request. See Hybrid Search for route configuration, rankers, and API examples.

Overview​

Prerequisites​

Lifecycle​

Coverage and Freshness​

Build Options​

BTree Index​

Bitmap Index​

Vector Index​

Full-Text Index​

Hybrid Search​