Vector Index

Vector Index provides approximate nearest neighbor (ANN) search for vector similarity search scenarios such as recommendation systems, image retrieval, and RAG (Retrieval Augmented Generation) applications.

Supported vector index types:

Index Type	Description
ivf-flat	IVF index with flat vector storage.
ivf-pq	IVF index with product quantization.
ivf-sq	IVF index with 8-bit scalar-quantized residuals.
ivf-rq	IVF index with rotated residual quantization.
diskann	paimon-vindex DiskANN index with graph traversal and persisted rerank vectors.

Choose the index type based on the trade-off you want:

Index Type	Best For
ivf-flat	Highest recall among IVF variants when storage and memory are acceptable.
ivf-pq	The smallest IVF files when stronger quantization loss is acceptable.
ivf-sq	High compact-index throughput with one byte per vector dimension.
ivf-rq	Higher compact-IVF recall when additional scan work is acceptable.
diskann	High-recall immutable collections served from local SSD or a complete local cache.

See the paimon-vindex documentation for its index selection guidance, storage architecture, and native API details.

Upgrading paimon-vindex indexes to 0.3.0

paimon-vindex 0.3.0 does not read the experimental ivf-hnsw-flat and ivf-hnsw-sq files written by 0.2.x. Rebuild those indexes as ivf-flat, ivf-pq, ivf-sq, ivf-rq, or diskann before removing the old runtime. Keep the source vectors or a 0.2-compatible index copy until the upgrade is accepted.

For table prerequisites, see Global Index. For refresh, coverage modes, and shared build options, see Manage Global Indexes.

Build Vector Index

Create and populate the shared example table first. The build and search examples below use its three-dimensional embedding column. For this small dataset, use ivf-flat with one cluster; production workloads should choose an index type and cluster count for their own data.

For Python, install pypaimon[vindex] (or its matching paimon-vindex==0.5.0 dependency) before building or querying native vector indexes.

Spark / Flink SQL

CALL sys.create_global_index(
    table => 'db.my_table',
    index_column => 'embedding',
    index_type => 'ivf-flat',
    options => 'ivf-flat.dimension=3,ivf-flat.distance.metric=cosine,ivf-flat.nlist=1'
);

Python SDK

table = catalog.get_table("db.my_table")

added_files = table.create_global_index(
    "embedding",
    index_type="ivf-flat",
    options={
        "ivf-flat.dimension": "3",
        "ivf-flat.distance.metric": "cosine",
        "ivf-flat.nlist": "1",
    },
)
print(added_files)

To limit a build to selected partitions, pass partitions with the same build options. The shared example table is partitioned by dt:

added_files = table.create_global_index(
    "embedding",
    index_type="ivf-flat",
    partitions=[{"dt": "2026-06-18"}, {"dt": "2026-06-19"}],
    options={
        "ivf-flat.dimension": "3",
        "ivf-flat.distance.metric": "cosine",
        "ivf-flat.nlist": "1",
        "global-index.row-count-per-shard": "100000",
    },
)
print(added_files)

For ARRAY<FLOAT> columns, specify the dimension with <index-type>.dimension. For VECTOR<FLOAT, n> columns, Paimon uses the dimension from the column type. Every query vector must have that same dimension.

Python Tables with Deletion Vectors

Python can build the native vector indexes listed above on data-evolution tables with deletion vectors enabled, including tables created with the multimodal API's default options:

import pyarrow as pa
import pypaimon.multimodal as pm

connection = pm.connect(options={"warehouse": "/tmp/paimon-warehouse"})
table = connection.create_table(
    "vectors",
    schema=pa.schema([("id", pa.int64()), ("embedding", pa.list_(pa.float32(), 2))]),
    options={"file.format": "parquet"},
)
table.add([{"id": 1, "embedding": [0.0, 1.0]}, {"id": 2, "embedding": [1.0, 1.0]}])
table.create_index("embedding", "ivf-flat", options={
    "ivf-flat.nlist": "1", "ivf-flat.distance.metric": "l2",
})
table.delete("id = 1")
print(table.search([0.0, 1.0]).select(["id"]).limit(1).to_list())  # [{"id": 2}]

Index builds preserve physical row IDs, including rows already marked as deleted. Search applies the deletion vectors from its query snapshot before selecting top-K results, so logical deletes do not require rebuilding the index. Logical deletion does not physically remove vectors from existing index files.

Vector Search

Python indexed vector searches use the global-index.thread-num table option, the same as Java. It must be a positive integer and defaults to 32. Single and batch queries open and search at most that many index shards concurrently, bounded by the number of splits. Set it to 1 for serial shard searches. It applies to both paimon-vindex and Lumina indexes. Results retain the same score and row-ID tie ordering. Each shard can also use native search and I/O threads, so choose the thread count together with those settings and the available memory. In Python, raw-vector fallback continues to use the table read parallelism.

The examples search the three-dimensional IVF-flat index built above. A limit of 5 returns at most five matches; the three-row sample table returns fewer. ivf.nprobe=1 matches the single cluster in this example.

Spark SQL

-- Search for top-5 nearest neighbors
SELECT id, name, __paimon_search_score
FROM vector_search('my_table', 'embedding', array(1.0f, 2.0f, 3.0f), 5)
ORDER BY __paimon_search_score DESC, id;

Flink SQL (Procedure)

Unlike Spark's table-valued function, Flink uses a CALL procedure to perform vector search. The procedure returns JSON-serialized rows as strings.

-- Search for top-5 nearest neighbors
CALL sys.vector_search(
    `table` => 'db.my_table',
    vector_column => 'embedding',
    query_vector => '1.0,2.0,3.0',
    top_k => 5
);

-- With projection (only return specific columns)
CALL sys.vector_search(
    `table` => 'db.my_table',
    vector_column => 'embedding',
    query_vector => '1.0,2.0,3.0',
    top_k => 5,
    projection => 'id,name,__paimon_search_score',
    options => 'ivf.nprobe=1',
    `where` => 'id > 100'
);

Java API

Table table = catalog.getTable(identifier);

// Step 1: Build vector search
float[] queryVector = {1.0f, 2.0f, 3.0f};
GlobalIndexResult result = table.newVectorSearchBuilder()
        .withVector(queryVector)
        .withLimit(5)
        .withVectorColumn("embedding")
        .withOption("ivf.nprobe", "1")
        .executeLocal();

// Step 2: Read matching rows using the search result
ReadBuilder readBuilder = table.newReadBuilder();
TableScan.Plan plan = readBuilder.newScan().withGlobalIndexResult(result).plan();
try (RecordReader<InternalRow> reader = readBuilder.newRead().createReader(plan)) {
    reader.forEachRemaining(row -> {
        System.out.println("id=" + row.getInt(0) + ", name=" + row.getString(1));
    });
}

Batch results keep the same order as input vectors.

float[][] queryVectors = {
    {1.0f, 2.0f, 3.0f},
    {3.0f, 2.0f, 1.0f}
};
List<GlobalIndexResult> batchResults = table.newBatchVectorSearchBuilder()
        .withVectors(queryVectors)
        .withLimit(5)
        .withVectorColumn("embedding")
        .executeBatchLocal();
// batchResults.get(i) corresponds to queryVectors[i].

For Java, use Table.newVectorSearchBuilder() to produce a global index result, then pass the result to TableScan.withGlobalIndexResult.

Python SDK

table = catalog.get_table("db.my_table")

# Step 1: Build vector search
result = (
    table.new_vector_search_builder()
    .with_vector_column("embedding")
    .with_query_vector([1.0, 2.0, 3.0])
    .with_limit(5)
    .with_option("ivf.nprobe", "1")
    .execute_local()
)

# Step 2: Read matching rows using the search result
read_builder = table.new_read_builder()
scan = read_builder.new_scan().with_global_index_result(result)
plan = scan.plan()
table_read = read_builder.new_read()
pa_table = table_read.to_arrow(plan.splits())
print(pa_table)

You can also add a scalar filter to pre-filter rows before vector search:

from pypaimon.common.predicate_builder import PredicateBuilder

table.create_global_index("category", index_type="btree")

predicate = (
    PredicateBuilder(table.fields)
    .equal("category", "electronics")
)

result = (
    table.new_vector_search_builder()
    .with_vector_column("embedding")
    .with_query_vector([1.0, 2.0, 3.0])
    .with_limit(5)
    .with_filter(predicate)
    .execute_local()
)

The scalar filter is evaluated with matching scalar global indexes before vector search. Build a BTree index for frequently used metadata filters, such as category, tenant_id, or event_time, so vector search can restrict the candidate row ids before running ANN search.

An index answer that is only a candidate set — contains, ends-with or LIKE on a BTree index, or a conjunction with a member no index can evaluate — is never ranked as if it matched, because a non-matching but closer row would take a top-k slot from a matching one. By default such candidates are excluded and a warning is logged, so the result may hold fewer than limit rows; set global-index.filter.refine-from-data=true to verify them by reading their filter columns instead. That read runs on the caller and can cover every candidate row.

A search selects top-K row IDs; reading those rows through a regular table scan does not imply score order. See Read Scored Results for SQL ordering and Java/Python score access.

Search Options

Search-time options are passed with each vector search request:

Option	Default	Description
ivf.nprobe	Automatic	Explicit number of IVF clusters to probe. When omitted, paimon-vindex derives the width from the index, top_k, and filter selectivity.
ivf.max_initial_filter_expansion_factor	Disabled	Positive integer limiting filter-driven expansion of the initial automatic IVF probe width. A factor of 1 disables initial filter expansion. Progressive retries may still probe more clusters when fewer than top_k filtered results are found.
ivf.refine_factor	Disabled	Retrieves top_k * refine_factor IVF candidates and reranks them with the original vectors stored in the Paimon table. It is most useful for compressed indexes such as ivf-pq, ivf-sq, and ivf-rq when recall is more important than latency.
ivf_pq.batch_table_reuse	auto	IVF-PQ batch search distance-table reuse mode: auto, on, or off. Other index types and scalar searches ignore it.
ivf_pq.batch_table_reuse.max_bytes	512 MiB	Positive long integer limiting the memory used by IVF-PQ batch distance-table reuse. Search falls back to direct table construction when the reusable tables exceed the budget.
diskann.l_search	Automatic	paimon-vindex DiskANN graph candidate width. The automatic value uses calibration when available, otherwise max(100, 2 * top_k).

Use the same distance metric at build time and query time. Search options can be passed per query, so you can use a larger ivf.nprobe or diskann.l_search for higher recall queries and a smaller value for latency-sensitive queries. Do not set both in one query.

ivf.max_initial_filter_expansion_factor applies only to automatic IVF search and cannot be combined with ivf.nprobe or diskann.l_search. Lower factors reduce initial filtered-search work but may reduce Recall@K compared with uncapped automatic search. Progressive expansion occurs only when fewer than top_k valid results are returned; if the capped initial search already fills top_k, probing stops.

ivf.refine_factor can also be configured with refine_factor, rerank_factor, and hyphenated spellings such as ivf.refine-factor. Setting ivf.refine_factor=1 still performs the raw-vector rerank for the indexed candidates; leaving it unset skips the rerank stage.

Build Options

Supported paimon-vindex options:

Option	Default	Description
<index-type>.dimension	128	Vector dimension for ARRAY<FLOAT> columns. Ignored for VECTOR<FLOAT, n> columns.
<index-type>.distance.metric	inner_product	Distance metric. Supported values: l2, cosine, inner_product.
<index-type>.ivf.coarse-assignment	auto	Build-time list assignment for all IVF types. auto uses Vamana when dimension × nlist ≥ 1,000,000 and exact assignment otherwise; exact always uses exact assignment.
<index-type>.ivf.train.max-points-per-centroid	256	Positive coarse K-means training limit for all IVF types: at most nlist × value vectors.
<index-type>.train.sample-ratio	1.0	Ratio of vectors sampled for native index training. Must be greater than 0 and less than or equal to 1. Lower values reduce training memory and build cost, but may reduce index quality.
<index-type>.nlist	Automatic	Number of clusters for the four IVF types. When omitted, paimon-vindex resolves it from the shard's non-null vector count.
<index-type>.pq.code-ratio	0.0625	Relative PQ-code budget for ivf-pq and diskann.
<index-type>.pq.m	Automatic	Expert override for the PQ sub-vector count used by ivf-pq and diskann.
ivf-pq.ivf.pq-encoding	auto	IVF-PQ build-time encoding. auto selects an accelerated backend when supported; canonical uses the canonical encoder.
<index-type>.pq.train.max-points-per-centroid	256	Positive PQ training limit for ivf-pq and diskann: at most 2^pq.bits × value vectors per subquantizer.
ivf-pq.pq.use-opq	Automatic	Explicitly enables or disables OPQ. Without an explicit value, a target-recall of at least 0.9 enables it.
ivf-rq.rq.bits	4	Persisted IVF-RQ residual width. Supported values are 1 through 8; changing it requires rebuilding the index.
<index-type>.target-recall	Not set	Build-policy hint used by ivf-pq and diskann. Validate the resulting recall on held-out queries.
<index-type>.max-bytes-per-vector	Not set	Storage objective and conservative preflight bound for ivf-pq, ivf-rq, and diskann.

Use the index-prefixed keys in table properties, SQL procedure options, and Python build options. For example, the following IVF-PQ build selects the non-default exact and canonical paths and lowers both training limits from 256 to 128:

CALL sys.create_global_index(
    table => 'db.model_embeddings',
    index_column => 'embedding',
    index_type => 'ivf-pq',
    options => 'ivf-pq.ivf.coarse-assignment=exact,ivf-pq.ivf.pq-encoding=canonical,ivf-pq.ivf.train.max-points-per-centroid=128,ivf-pq.pq.train.max-points-per-centroid=128'
);

Omit these options to retain the defaults in the table above. The native names ivf.coarse-assignment, ivf.pq-encoding, ivf.train.max-points-per-centroid, and pq.train.max-points-per-centroid are also accepted directly; prefer index-prefixed names in table properties to avoid sharing a setting across index types. Use fields.<field-name>.<option> to override a table property for one vector column. Explicit procedure or Python options override table properties; within either source, field-prefixed options override index-prefixed options, which override bare native names. An inapplicable stored table property is ignored, while an explicitly supplied option that does not apply to the selected index type is rejected.

Additional paimon-vindex DiskANN build options:

Option	Default	Description
diskann.build-preset	balanced	Coherent fast_build, balanced, or high_recall build policy.
diskann.deployment-profile	Not set	Deployment objective used to choose a storage layout.
diskann.pq.bits	8	Resident PQ-code width. Supported values are 4 and 8.
diskann.max-degree	64	Maximum graph out-degree.
diskann.build-search-list-size	max(100, max-degree)	Candidate width during graph construction.
diskann.alpha	1.2	Robust-prune threshold.
diskann.seed	42	Reproducible initialization and build-order seed.
diskann.memory-budget-bytes	8 GiB	Internal graph-build memory estimate used to select normal or sharded construction.
diskann.storage-layout	auto	Explicit compact or interleaved layout override.
diskann.raw-vector-encoding	auto	Explicit f32 or f16 persisted rerank-vector encoding.
diskann.build-distance	auto	Explicit product-quantized or full-precision build traversal override.

Quantized and DiskANN Builds

The following are alternative builds for a separate, populated table db.model_embeddings whose embedding ARRAY<FLOAT> values have dimension 768. They do not apply to the three-dimensional sample table. Choose one alternative; train quantized indexes on a representative dataset large enough for the chosen clustering and quantization parameters.

CALL sys.create_global_index(
    table => 'db.model_embeddings',
    index_column => 'embedding',
    index_type => 'ivf-pq',
    options => 'ivf-pq.dimension=768,ivf-pq.distance.metric=cosine,ivf-pq.nlist=256,ivf-pq.pq.code-ratio=0.0625'
);

Alternatively, build a DiskANN index:

CALL sys.create_global_index(
    table => 'db.model_embeddings',
    index_column => 'embedding',
    index_type => 'diskann',
    options => 'diskann.dimension=768,diskann.distance.metric=l2,diskann.build-preset=balanced'
);

Pass the same option keys to Python's create_global_index(..., options={...}). Use full 768-dimensional query vectors when searching either of these indexes.

Per-Field Options

The options above can also be set at the table level (in TBLPROPERTIES), where they are shared by every vector column of the same index type. When a table has multiple vector columns, you can scope an option to a single column with fields.<field-name>.<option>. The field-level form takes precedence over the column-agnostic option for that column. Use the stored table column name exactly as <field-name>. Field-level vector options do not include the index-type prefix; for example, use fields.image_embedding.nlist to override the shared ivf-pq.nlist option for image_embedding:

Spark SQL

CREATE TABLE multi_embedding_table (
    id INT,
    title_embedding ARRAY<FLOAT>,
    image_embedding ARRAY<FLOAT>
) TBLPROPERTIES (
    'row-tracking.enabled' = 'true',
    'data-evolution.enabled' = 'true',
    'global-index.enabled' = 'true',
    -- per-column dimensions
    'fields.title_embedding.dimension' = '768',
    'fields.image_embedding.dimension' = '512',
    -- shared by every ivf-pq column, overridden only for 'image_embedding'
    'ivf-pq.nlist' = '256',
    'fields.image_embedding.nlist' = '512',
    -- per-column training sample ratio
    'fields.image_embedding.train.sample-ratio' = '0.5'
);

Python SDK

import pyarrow as pa

from pypaimon import Schema

schema = Schema.from_pyarrow_schema(
    pa.schema([
        pa.field("id", pa.int32()),
        pa.field("title_embedding", pa.list_(pa.float32())),
        pa.field("image_embedding", pa.list_(pa.float32())),
    ]),
    options={
        "row-tracking.enabled": "true",
        "data-evolution.enabled": "true",
        "global-index.enabled": "true",
        "fields.title_embedding.dimension": "768",
        "fields.image_embedding.dimension": "512",
        "ivf-pq.nlist": "256",
        "fields.image_embedding.nlist": "512",
        "fields.image_embedding.train.sample-ratio": "0.5",
    },
)

catalog.create_table("db.multi_embedding_table", schema, ignore_if_exists=False)

These properties configure subsequent IVF-PQ builds; creating the table does not build an index. After loading data, a build on title_embedding uses nlist=256, while one on image_embedding uses nlist=512 and trains with half of its non-null vectors.

Drop Vector Index

SQL

CALL sys.drop_global_index(
    table => 'db.my_table',
    index_column => 'embedding',
    index_type => 'ivf-flat'
);

Python SDK

table = catalog.get_table("db.my_table")

dropped_files = table.drop_global_index("embedding", index_type="ivf-flat")
print(dropped_files)

You can also restrict the drop to selected partitions, or count matched files without committing:

matched_files = table.drop_global_index(
    "embedding",
    index_type="ivf-flat",
    partitions=[{"dt": "2026-06-18"}, {"dt": "2026-06-19"}],
    dry_run=True,
)
print(matched_files)