Skip to main content

Vector Index

Vector Index provides approximate nearest neighbor (ANN) search for vector similarity search scenarios such as recommendation systems, image retrieval, and RAG (Retrieval Augmented Generation) applications.

Supported vector index types:

Index TypeDescription
ivf-flatIVF index with flat vector storage.
ivf-pqIVF index with product quantization.
ivf-sqIVF index with 8-bit scalar-quantized residuals.
ivf-rqIVF index with rotated residual quantization.
diskannpaimon-vindex DiskANN index with graph traversal and persisted rerank vectors.

Choose the index type based on the trade-off you want:

Index TypeBest For
ivf-flatHighest recall among IVF variants when storage and memory are acceptable.
ivf-pqThe smallest IVF files when stronger quantization loss is acceptable.
ivf-sqHigh compact-index throughput with one byte per vector dimension.
ivf-rqHigher compact-IVF recall when additional scan work is acceptable.
diskannHigh-recall immutable collections served from local SSD or a complete local cache.

See the paimon-vindex documentation for its index selection guidance, storage architecture, and native API details.

Upgrading paimon-vindex indexes to 0.3.0

paimon-vindex 0.3.0 does not read the experimental ivf-hnsw-flat and ivf-hnsw-sq files written by 0.2.x. Rebuild those indexes as ivf-flat, ivf-pq, ivf-sq, ivf-rq, or diskann before removing the old runtime. Keep the source vectors or a 0.2-compatible index copy until the upgrade is accepted.

For table prerequisites, see Global Index. For refresh, coverage modes, and shared build options, see Manage Global Indexes.

Build Vector Index​

Create and populate the shared example table first. The build and search examples below use its three-dimensional embedding column. For this small dataset, use ivf-flat with one cluster; production workloads should choose an index type and cluster count for their own data.

For Python, install pypaimon[vindex] (or its matching paimon-vindex==0.4.0 dependency) before building or querying native vector indexes.

CALL sys.create_global_index(
    table => 'db.my_table',
    index_column => 'embedding',
    index_type => 'ivf-flat',
    options => 'ivf-flat.dimension=3,ivf-flat.distance.metric=cosine,ivf-flat.nlist=1'
);

For ARRAY<FLOAT> columns, specify the dimension with <index-type>.dimension. For VECTOR<FLOAT, n> columns, Paimon uses the dimension from the column type. Every query vector must have that same dimension.

The examples search the three-dimensional IVF-flat index built above. A limit of 5 returns at most five matches; the three-row sample table returns fewer. ivf.nprobe=1 matches the single cluster in this example.

-- Search for top-5 nearest neighbors
SELECT id, name, __paimon_search_score
FROM vector_search('my_table', 'embedding', array(1.0f, 2.0f, 3.0f), 5)
ORDER BY __paimon_search_score DESC, id;

A search selects top-K row IDs; reading those rows through a regular table scan does not imply score order. See Read Scored Results for SQL ordering and Java/Python score access.

Search Options​

Search-time options are passed with each vector search request:

OptionDefaultDescription
ivf.nprobeAutomaticExplicit number of IVF clusters to probe. When omitted, paimon-vindex derives the width from the index, top_k, and filter selectivity.
ivf.max_initial_filter_expansion_factorDisabledPositive integer limiting filter-driven expansion of the initial automatic IVF probe width. A factor of 1 disables initial filter expansion. Progressive retries may still probe more clusters when fewer than top_k filtered results are found.
ivf.refine_factorDisabledRetrieves top_k * refine_factor IVF candidates and reranks them with the original vectors stored in the Paimon table. It is most useful for compressed indexes such as ivf-pq, ivf-sq, and ivf-rq when recall is more important than latency.
ivf_pq.batch_table_reuseautoIVF-PQ batch search distance-table reuse mode: auto, on, or off. Other index types and scalar searches ignore it.
ivf_pq.batch_table_reuse.max_bytes512 MiBPositive long integer limiting the memory used by IVF-PQ batch distance-table reuse. Search falls back to direct table construction when the reusable tables exceed the budget.
diskann.l_searchAutomaticpaimon-vindex DiskANN graph candidate width. The automatic value uses calibration when available, otherwise max(100, 2 * top_k).

Use the same distance metric at build time and query time. Search options can be passed per query, so you can use a larger ivf.nprobe or diskann.l_search for higher recall queries and a smaller value for latency-sensitive queries. Do not set both in one query.

ivf.max_initial_filter_expansion_factor applies only to automatic IVF search and cannot be combined with ivf.nprobe or diskann.l_search. Lower factors reduce initial filtered-search work but may reduce Recall@K compared with uncapped automatic search. Progressive expansion occurs only when fewer than top_k valid results are returned; if the capped initial search already fills top_k, probing stops.

ivf.refine_factor can also be configured with refine_factor, rerank_factor, and hyphenated spellings such as ivf.refine-factor. Setting ivf.refine_factor=1 still performs the raw-vector rerank for the indexed candidates; leaving it unset skips the rerank stage.

Build Options​

Supported paimon-vindex options:

OptionDefaultDescription
<index-type>.dimension128Vector dimension for ARRAY<FLOAT> columns. Ignored for VECTOR<FLOAT, n> columns.
<index-type>.distance.metricinner_productDistance metric. Supported values: l2, cosine, inner_product.
<index-type>.train.sample-ratio1.0Ratio of vectors sampled for native index training. Must be greater than 0 and less than or equal to 1. Lower values reduce training memory and build cost, but may reduce index quality.
<index-type>.nlistAutomaticNumber of clusters for the four IVF types. When omitted, paimon-vindex resolves it from the shard's non-null vector count.
<index-type>.pq.code-ratio0.0625Relative PQ-code budget for ivf-pq and diskann.
<index-type>.pq.mAutomaticExpert override for the PQ sub-vector count used by ivf-pq and diskann.
ivf-pq.pq.use-opqAutomaticExplicitly enables or disables OPQ. Without an explicit value, a target-recall of at least 0.9 enables it.
ivf-rq.rq.bits4Persisted IVF-RQ residual width. Supported values are 1 through 8; changing it requires rebuilding the index.
<index-type>.target-recallNot setBuild-policy hint used by ivf-pq and diskann. Validate the resulting recall on held-out queries.
<index-type>.max-bytes-per-vectorNot setStorage objective and conservative preflight bound for ivf-pq, ivf-rq, and diskann.

Additional paimon-vindex DiskANN build options:

OptionDefaultDescription
diskann.build-presetbalancedCoherent fast_build, balanced, or high_recall build policy.
diskann.deployment-profileNot setDeployment objective used to choose a storage layout.
diskann.pq.bits8Resident PQ-code width. Supported values are 4 and 8.
diskann.max-degree64Maximum graph out-degree.
diskann.build-search-list-sizemax(100, max-degree)Candidate width during graph construction.
diskann.alpha1.2Robust-prune threshold.
diskann.seed42Reproducible initialization and build-order seed.
diskann.memory-budget-bytes8 GiBInternal graph-build memory estimate used to select normal or sharded construction.
diskann.storage-layoutautoExplicit compact or interleaved layout override.
diskann.raw-vector-encodingautoExplicit f32 or f16 persisted rerank-vector encoding.
diskann.build-distanceautoExplicit product-quantized or full-precision build traversal override.

Quantized and DiskANN Builds​

The following are alternative builds for a separate, populated table db.model_embeddings whose embedding ARRAY<FLOAT> values have dimension 768. They do not apply to the three-dimensional sample table. Choose one alternative; train quantized indexes on a representative dataset large enough for the chosen clustering and quantization parameters.

CALL sys.create_global_index(
    table => 'db.model_embeddings',
    index_column => 'embedding',
    index_type => 'ivf-pq',
    options => 'ivf-pq.dimension=768,ivf-pq.distance.metric=cosine,ivf-pq.nlist=256,ivf-pq.pq.code-ratio=0.0625'
);

Alternatively, build a DiskANN index:

CALL sys.create_global_index(
    table => 'db.model_embeddings',
    index_column => 'embedding',
    index_type => 'diskann',
    options => 'diskann.dimension=768,diskann.distance.metric=l2,diskann.build-preset=balanced'
);

Pass the same option keys to Python's create_global_index(..., options={...}). Use full 768-dimensional query vectors when searching either of these indexes.

Per-Field Options​

The options above can also be set at the table level (in TBLPROPERTIES), where they are shared by every vector column of the same index type. When a table has multiple vector columns, you can scope an option to a single column with fields.<field-name>.<option>. The field-level form takes precedence over the column-agnostic option for that column. Use the stored table column name exactly as <field-name>. Field-level vector options do not include the index-type prefix; for example, use fields.image_embedding.nlist to override the shared ivf-pq.nlist option for image_embedding:

CREATE TABLE multi_embedding_table (
    id INT,
    title_embedding ARRAY<FLOAT>,
    image_embedding ARRAY<FLOAT>
) TBLPROPERTIES (
    'row-tracking.enabled' = 'true',
    'data-evolution.enabled' = 'true',
    'global-index.enabled' = 'true',
    -- per-column dimensions
    'fields.title_embedding.dimension' = '768',
    'fields.image_embedding.dimension' = '512',
    -- shared by every ivf-pq column, overridden only for 'image_embedding'
    'ivf-pq.nlist' = '256',
    'fields.image_embedding.nlist' = '512',
    -- per-column training sample ratio
    'fields.image_embedding.train.sample-ratio' = '0.5'
);

These properties configure subsequent IVF-PQ builds; creating the table does not build an index. After loading data, a build on title_embedding uses nlist=256, while one on image_embedding uses nlist=512 and trains with half of its non-null vectors.

Drop Vector Index​

CALL sys.drop_global_index(
    table => 'db.my_table',
    index_column => 'embedding',
    index_type => 'ivf-flat'
);