Hybrid Search

Hybrid search queries multiple scored routes in one request and ranks the results before reading table rows. A route can be a vector search route or a full-text search route, so hybrid search can combine multiple vector indexes, multiple full-text indexes, or a mix of vector and full-text indexes. This is useful when one table stores several searchable representations for the same record, such as title embeddings, body embeddings, and content text.

Create the Example Table

This example uses a separate table with two three-dimensional vector columns. In a configured Spark Paimon catalog, select the existing db database and run:

CREATE TABLE hybrid_documents (
    id INT,
    title_embedding ARRAY<FLOAT>,
    body_embedding ARRAY<FLOAT>,
    content STRING
) TBLPROPERTIES (
    'row-tracking.enabled' = 'true',
    'data-evolution.enabled' = 'true'
);

INSERT INTO hybrid_documents VALUES
    (1, array(1.0f, 0.0f, 0.0f), array(0.0f, 1.0f, 0.0f), 'paimon hybrid search'),
    (2, array(0.9f, 0.1f, 0.0f), array(0.1f, 0.9f, 0.0f), 'paimon lake format'),
    (3, array(0.0f, 1.0f, 0.0f), array(1.0f, 0.0f, 0.0f), 'vector search tutorial');

The Java and Python examples query this same table through their catalog. Use matching dimensions and model preprocessing for stored and query vectors in a real pipeline. The sample vectors here only illustrate route configuration.

Build the Route Indexes

Before running hybrid search, create a global index for every vector or text column used by the query:

CALL sys.create_global_index(
    table => 'db.hybrid_documents',
    index_column => 'title_embedding',
    index_type => 'ivf-flat',
    options => 'ivf-flat.dimension=3,ivf-flat.distance.metric=cosine,ivf-flat.nlist=1'
);

CALL sys.create_global_index(
    table => 'db.hybrid_documents',
    index_column => 'body_embedding',
    index_type => 'ivf-flat',
    options => 'ivf-flat.dimension=3,ivf-flat.distance.metric=cosine,ivf-flat.nlist=1'
);

CALL sys.create_global_index(
    table => 'db.hybrid_documents',
    index_column => 'content',
    index_type => 'full-text'
);

Alternatively, after creating and populating the table, build the same indexes with Python. Install pypaimon[vindex,full-text] for the native index dependencies:

table = catalog.get_table("db.hybrid_documents")
for column in ("title_embedding", "body_embedding"):
    table.create_global_index(column, index_type="ivf-flat", options={
        "ivf-flat.dimension": "3",
        "ivf-flat.distance.metric": "cosine",
        "ivf-flat.nlist": "1",
    })
table.create_global_index("content", index_type="full-text")

The sample uses one IVF cluster and queries it with ivf.nprobe=1. Larger indexes can use different build and search options.

Choose a Ranker

For Spark SQL, use the hybrid_search(table_name, vector_routes, full_text_routes, limit[, ranker]) table-valued function. The fourth argument is the final number of ranked results to return. The optional fifth argument selects the ranker. Supported rankers are rrf, weighted_score, and mrr; the default is rrf.

Rankers combine route scores differently:

Ranker	Best For	Description
rrf	Combining routes with different score scales	Reciprocal rank fusion uses each route's rank order, so vector and full-text scores do not need to be normalized.
weighted_score	Weighting routes by normalized score, not just rank	Min-max normalizes each route's scores to [0, 1], then sums them weighted by route weight, so weights (not raw score magnitude) control each route's influence. The exposed __paimon_search_score is therefore a per-query relative value in [0, sum of weights], not a raw similarity or BM25 score.
mrr	Emphasizing top-ranked hits from each route	Weighted reciprocal-rank fusion sums weight / rank for each row returned by a route, where rank starts at 1.

Configure Routes

The second argument is an array of vector route configs created by named_struct:

Field	Required	Default	Description
field or vector_column	Yes	N/A	Vector column to search.
query_vector	Yes	N/A	Query vector for this route.
limit	No	Final limit	Top K results to retrieve from this vector column before ranking.
weight	No	1.0	Weight for this route when ranking results.
options	No	Empty map	Route-specific vector search options such as ivf.nprobe and diskann.l_search.

The third argument is an array of full-text route configs created by named_struct:

Field	Required	Default	Description
column	Yes	N/A	Text column to search.
query	Yes	N/A	Full-text query string for this route.
limit	No	Final limit	Top K results to retrieve from this text column before ranking.
weight	No	1.0	Weight for this route when ranking results.
options	No	Empty map	Reserved for future full-text search options. Only an empty map is accepted.

Within each route array, every named_struct should use the same fields because Spark requires array elements to have the same struct type. Full-text route options is currently a reserved field; pass map() when the field is needed for struct-type consistency.

Use route limit values larger than the final limit when each route should contribute enough candidates for ranking. For example, with a final limit of 10, route limits such as 50 or 100 give the ranker more candidates to merge.

Run Hybrid Search

Route limits and the final limit are upper bounds. The three-row sample table returns at most three rows even when a route asks for 50 and the final limit is 10.

Spark SQL

Vector and full-text routes can be searched together:

SELECT id, __paimon_search_score
FROM hybrid_search(
  'hybrid_documents',
  array(
    named_struct(
      'field', 'title_embedding',
      'query_vector', array(1.0f, 0.0f, 0.0f),
      'limit', 50,
      'weight', 2.0f,
      'options', map('ivf.nprobe', '1'))),
  array(
    named_struct(
      'column', 'content',
      'query', '{"match":{"query":"paimon search"}}',
      'limit', 50,
      'weight', 1.0f,
      'options', map())),
  10,
  'rrf')
ORDER BY __paimon_search_score DESC, id;

To search two vector indexes without a full-text route, pass array() as the third argument:

SELECT id, __paimon_search_score
FROM hybrid_search(
  'hybrid_documents',
  array(
    named_struct(
      'field', 'title_embedding',
      'query_vector', array(1.0f, 0.0f, 0.0f),
      'limit', 50,
      'weight', 2.0f,
      'options', map('ivf.nprobe', '1')),
    named_struct(
      'field', 'body_embedding',
      'query_vector', array(0.0f, 1.0f, 0.0f),
      'limit', 50,
      'weight', 1.0f,
      'options', map('ivf.nprobe', '1'))),
  array(),
  10,
  'weighted_score')
ORDER BY __paimon_search_score DESC, id;

Use mrr when you want weighted reciprocal-rank fusion:

SELECT id, __paimon_search_score
FROM hybrid_search(
  'hybrid_documents',
  array(
    named_struct(
      'field', 'title_embedding',
      'query_vector', array(1.0f, 0.0f, 0.0f),
      'limit', 50,
      'weight', 2.0f,
      'options', map('ivf.nprobe', '1'))),
  array(),
  10,
  'mrr')
ORDER BY __paimon_search_score DESC, id;

Spark SQL adds __paimon_search_score to expose the ranked score. Use ORDER BY when the displayed row order must follow that score.

Java API

Table table = catalog.getTable(identifier);

HybridSearchBuilder searchBuilder =
        table.newHybridSearchBuilder()
                .addVectorRoute(
                        "title_embedding",
                        new float[] {1.0f, 0.0f, 0.0f},
                        50,
                        2.0f,
                        java.util.Collections.singletonMap("ivf.nprobe", "1"))
                .addFullTextRoute(
                        "content",
                        "{\"match\":{\"query\":\"paimon search\"}}",
                        50,
                        1.0f,
                        java.util.Collections.emptyMap())
                .withLimit(10)
                .withRrfRanker();
GlobalIndexResult result = searchBuilder.executeLocal();

ReadBuilder readBuilder = table.newReadBuilder();
TableScan.Plan plan = readBuilder.newScan().withGlobalIndexResult(result).plan();
try (RecordReader<InternalRow> reader = readBuilder.newRead().createReader(plan)) {
    reader.forEachRemaining(row -> {
        System.out.println("id=" + row.getInt(0));
    });
}

To search two vector indexes without a full-text route, add two vector routes:

GlobalIndexResult vectorOnlyResult =
        table.newHybridSearchBuilder()
                .addVectorRoute(
                        "title_embedding",
                        new float[] {1.0f, 0.0f, 0.0f},
                        50,
                        2.0f,
                        java.util.Collections.singletonMap("ivf.nprobe", "1"))
                .addVectorRoute(
                        "body_embedding",
                        new float[] {0.0f, 1.0f, 0.0f},
                        50,
                        1.0f,
                        java.util.Collections.singletonMap("ivf.nprobe", "1"))
                .withLimit(10)
                .withWeightedScoreRanker()
                .executeLocal();

Use withRanker("mrr") for weighted reciprocal-rank fusion:

GlobalIndexResult mrrResult =
        table.newHybridSearchBuilder()
                .addVectorRoute(
                        "title_embedding",
                        new float[] {1.0f, 0.0f, 0.0f},
                        50,
                        1.0f,
                        java.util.Collections.singletonMap("ivf.nprobe", "1"))
                .withLimit(10)
                .withRanker("mrr")
                .executeLocal();

For Java, use Table.newHybridSearchBuilder() to configure routes, final limit, and ranker directly. VectorSearch and HybridSearch are internal pushdown representations.

Python SDK

table = catalog.get_table("db.hybrid_documents")

result = (
    table.new_hybrid_search_builder()
    .add_vector_route(
        "title_embedding",
        [1.0, 0.0, 0.0],
        limit=50,
        weight=2.0,
        options={"ivf.nprobe": "1"},
    )
    .add_full_text_route(
        "content",
        '{"match":{"query":"paimon search"}}',
        limit=50,
        weight=1.0,
    )
    .with_limit(10)
    .with_rrf_ranker()
    .execute_local()
)

read_builder = table.new_read_builder()
scan = read_builder.new_scan().with_global_index_result(result)
plan = scan.plan()
table_read = read_builder.new_read()
pa_table = table_read.to_arrow(plan.splits())
print(pa_table)

To search two vector indexes without a full-text route, add two vector routes:

result = (
    table.new_hybrid_search_builder()
    .add_vector_route(
        "title_embedding",
        [1.0, 0.0, 0.0],
        limit=50,
        weight=2.0,
        options={"ivf.nprobe": "1"},
    )
    .add_vector_route(
        "body_embedding",
        [0.0, 1.0, 0.0],
        limit=50,
        weight=1.0,
        options={"ivf.nprobe": "1"},
    )
    .with_limit(10)
    .with_weighted_score_ranker()
    .execute_local()
)

Use with_ranker("mrr") for weighted reciprocal-rank fusion:

result = (
    table.new_hybrid_search_builder()
    .add_vector_route(
        "title_embedding",
        [1.0, 0.0, 0.0],
        limit=50,
        weight=1.0,
        options={"ivf.nprobe": "1"},
    )
    .with_limit(10)
    .with_ranker("mrr")
    .execute_local()
)

Coverage and Candidate Limits

Each route depends on its own index coverage and search mode. Review Coverage and Freshness when ingesting or updating data. The ranker can only combine candidates returned by the routes; increase route limits when a small candidate set restricts final recall.

For Java and Python, the subsequent table scan does not preserve ranking order. See Read Scored Results to access scores by row ID.

Row Filters

A row filter given to a hybrid search (WHERE on the Spark hybrid_search result, or withFilter on the Java builder) is applied to every route before its own top-k, so vector and full-text candidates are drawn from the same filtered row set before the ranker merges them. Full-text routes follow the same rules as a standalone full-text search; see Full-Text Row Filters. In particular, under scalar-index.search-mode=fast a full-text route whose filter columns have no scalar index contributes no candidates, so the fused result comes from the other routes only.