Full-Text Index

Full-text index lets Paimon run top-k text retrieval on a string column through the Global Index framework. The index type is full-text. The Paimon integration module is paimon-full-text, backed by the standalone paimon-full-text-index native library.

The search API passes the table column name separately from the query. The query itself is a JSON string accepted by paimon-full-text-index.

Full-text indexes are scored by the native engine and can be used directly, or combined with vector routes in Hybrid Search.

For table prerequisites, see Global Index. For refresh, coverage modes, and shared build options, see Manage Global Indexes.

Build Full-Text Index

Build full-text indexes on STRING, CHAR, or VARCHAR columns. Null values are skipped by the native text index, while row-id coverage is still tracked by Paimon.

Start with the populated shared example table. Choose one tokenizer before the initial build: the SQL and Python calls below show alternative configurations. Repeating a build with different options does not retokenize rows already covered by the index.

SQL

CALL sys.create_global_index(
    table => 'db.my_table',
    index_column => 'content',
    index_type => 'full-text'
);

Use the ngram tokenizer for short character fragments or substring-like lookup:

CALL sys.create_global_index(
    table => 'db.my_table',
    index_column => 'content',
    index_type => 'full-text',
    options => 'full-text.tokenizer=ngram,full-text.ngram.min-gram=2,full-text.ngram.max-gram=3'
);

Use jieba for Chinese word segmentation:

CALL sys.create_global_index(
    table => 'db.my_table',
    index_column => 'content',
    index_type => 'full-text',
    options => 'full-text.tokenizer=jieba'
);

Python SDK

table = catalog.get_table("db.my_table")

added_files = table.create_global_index(
    "content",
    index_type="full-text",
)
print(added_files)

Use the ngram tokenizer for short character fragments or substring-like lookup:

added_files = table.create_global_index(
    "content",
    index_type="full-text",
    options={
        "full-text.tokenizer": "ngram",
        "full-text.ngram.min-gram": "2",
        "full-text.ngram.max-gram": "3",
    },
)
print(added_files)

Install the PyPaimon full-text extra before building or querying full-text indexes:

pip install 'pypaimon[full-text]'

All native full-text options must use the public full-text. prefix in Paimon. Paimon removes this prefix once and passes the remaining keys to the native library. For example, set full-text.tokenizer=ngram, not tokenizer=ngram.

Supported full-text index options:

Option	Default	Description
full-text.tokenizer	default	Tokenizer used by the native full-text index. Supported values are default, simple, whitespace, raw, ngram, and jieba.
full-text.ngram.min-gram	3	Minimum gram length for the ngram tokenizer.
full-text.ngram.max-gram	3	Maximum gram length for the ngram tokenizer.
full-text.ngram.prefix-only	false	Whether the ngram tokenizer only emits prefix ngrams.
full-text.jieba.search-mode	true	Whether the jieba tokenizer uses search mode.
full-text.jieba.ordinal-position	true	Whether the jieba tokenizer uses ordinal positions.
full-text.lower-case	true	Whether configurable tokenizers lowercase emitted tokens.
full-text.max-token-length	40	Maximum token length kept by configurable tokenizers.
full-text.ascii-folding	true	Whether to normalize non-ASCII Latin characters to ASCII.
full-text.stem	true	Whether to apply stemming.
full-text.language	english	Language used by stemming and built-in stop-word filters.
full-text.remove-stop-words	true	Whether to remove built-in stop words for the configured language.
full-text.stop-words	empty	Semicolon-separated custom stop words. Requires full-text.remove-stop-words=true.
full-text.with-position	true	Whether to store term positions. Keep this enabled for phrase queries.

The default tokenizer uses English full-text defaults: lower-case, stemming, stop-word removal, ASCII folding, maximum token length 40, and positions. Set full-text.with-position=false only when phrase search is not needed.

Tokenizer settings are stored in the global index file metadata. Existing index files keep the tokenizer options they were built with, even if later index builds use different options.

To change tokenization for already indexed rows, drop the index and build it again with the new options. Use the same partition scope for both operations if changing only selected partitions. Coordinate the rebuild with queries that rely on complete coverage; fast searches only indexed ranges.

Full-Text Search

Spark SQL uses the full_text_search(table_name, column, query, limit) table-valued function. The column argument is the Paimon table column to search. The query argument is passed to the native full-text reader as a JSON string.

Spark SQL

-- Search for the top 10 rows matching any query term.
SELECT id, content, __paimon_search_score
FROM full_text_search(
    'my_table',
    'content',
    '{"match":{"query":"paimon lake format"}}',
    10
)
ORDER BY __paimon_search_score DESC, id;

-- Require all query terms.
SELECT id, content, __paimon_search_score
FROM full_text_search(
    'my_table',
    'content',
    '{"match":{"query":"paimon lake format","operator":"And"}}',
    10
)
ORDER BY __paimon_search_score DESC, id;

-- Phrase query. The index must be built with full-text.with-position=true.
SELECT id, content, __paimon_search_score
FROM full_text_search(
    'my_table',
    'content',
    '{"match_phrase":{"query":"paimon lake"}}',
    10
)
ORDER BY __paimon_search_score DESC, id;

-- Filter rows before ranking: the top 10 is taken among rows with category = 'lake'.
SELECT id, content, __paimon_search_score
FROM full_text_search(
    'my_table',
    'content',
    '{"match":{"query":"paimon lake format"}}',
    10
)
WHERE category = 'lake' AND dt >= '2026-09-01'
ORDER BY __paimon_search_score DESC, id;

Spark exposes the score as the __paimon_search_score metadata column.

Java API

Table table = catalog.getTable(identifier);

GlobalIndexResult result =
        table.newFullTextSearchBuilder()
                .withQuery("content", "{\"match\":{\"query\":\"paimon lake\"}}")
                .withLimit(10)
                .executeLocal();

ReadBuilder readBuilder = table.newReadBuilder();
TableScan.Plan plan = readBuilder.newScan().withGlobalIndexResult(result).plan();
try (RecordReader<InternalRow> reader = readBuilder.newRead().createReader(plan)) {
    reader.forEachRemaining(row -> System.out.println(row));
}

Add a row filter with withFilter to rank only matching rows:

Predicate filter = new PredicateBuilder(table.rowType()).equal(1, BinaryString.fromString("lake"));

GlobalIndexResult result =
        table.newFullTextSearchBuilder()
                .withQuery("content", "{\"match\":{\"query\":\"paimon lake\"}}")
                .withFilter(filter)
                .withLimit(10)
                .executeLocal();

Python SDK

table = catalog.get_table("db.my_table")

result = (
    table.new_full_text_search_builder()
    .with_query("content", '{"match":{"query":"paimon lake"}}')
    .with_limit(10)
    .execute_local()
)

read_builder = table.new_read_builder()
plan = read_builder.new_scan().with_global_index_result(result).plan()
pa_table = read_builder.new_read().to_arrow(plan.splits())
print(pa_table)

By default, full-text-index.search-mode=fast searches indexed row ranges only. When full-text-index.search-mode is full or detail, Paimon also covers unindexed row ranges by reading raw rows and building a temporary native full-text index for the searched column.

Row Filters

A full-text search can carry a row filter: a Spark WHERE clause on the full_text_search result, withFilter on the Java builder, or withFilter on a hybrid search. Partition predicates in the filter prune partitions; the remaining predicates are evaluated before top-k ranking, so the result is the top-k among matching rows rather than a filtered subset of the unfiltered top-k.

Non-partition predicates are resolved into an exact set of matching rows before ranking, and that set is handed to the existing full-text index, so BM25 statistics are always those of the full corpus:

• Rows whose filter columns are covered by a scalar global index (BTree, Bitmap, Multivalue, FM) are decided by the index, the same way vector search pre-filters rows. When the index can only produce candidates (a conjunct no index could evaluate, or contains, ends-with and LIKE on a BTree index), those candidates are never ranked as if they matched: with global-index.filter.refine-from-data=true they are verified by reading their filter columns, otherwise (the default) they are excluded and a warning is logged, so the result may hold fewer than limit rows. The read runs on the caller and can cover every candidate row, which is why it is opt-in; prefer an index that answers the predicate exactly, such as Bitmap or FM for contains.
• Rows whose filter columns are not covered by a scalar index follow scalar-index.search-mode:

scalar-index.search-mode	Rows without an index on the filter columns
fast (default)	Excluded from the search; a warning is logged
full / detail	Decided by reading their filter columns

Build a scalar index on the columns you filter by to avoid the data read. With full-text-index.search-mode=fast, only rows covered by the full-text index are considered in either case; rows outside the coverage are searched through a temporary index in full mode, where the filter is applied the same way. Predicates the engine cannot push down (for example, UDFs) are applied by the engine after the search, so such queries may return fewer than limit rows.

Primary-key full-text indexes do not support row filters yet; see Primary-Key Indexes.

Query DSL

The query DSL is a JSON object with one top-level query type. Use the examples below as JSON strings in Spark SQL, Java, Python, or hybrid search routes.

Match

Use match for normal term search. The default operator is Or.

{
  "match": {
    "query": "paimon lake format",
    "operator": "And"
  }
}

Field	Required	Default	Description
query	Yes	N/A	Query text.
operator	No	Or	How query terms are combined. Supported values are Or and And.
boost	No	1.0	Score multiplier for this query.
fuzziness	No	0	Edit distance for fuzzy matching. Use an integer or auto.
max_expansions	No	50	Maximum fuzzy expansions. maxExpansions is also accepted.
prefix_length	No	0	Number of leading characters that must match exactly. prefixLength is also accepted.

Phrase

Use match_phrase for ordered terms. Phrase search requires positions, so build the index with full-text.with-position=true.

{
  "match_phrase": {
    "query": "paimon lake",
    "slop": 1
  }
}

Field	Required	Default	Description
query	Yes	N/A	Phrase text.
slop	No	0	Number of positional moves allowed when matching the phrase.

Boolean

Use boolean to combine nested queries.

{
  "boolean": {
    "must": [
      {"match": {"query": "paimon"}}
    ],
    "should": [
      {"match_phrase": {"query": "lake format"}}
    ],
    "must_not": [
      {"match": {"query": "vector"}}
    ]
  }
}

Field	Required	Default	Description
must	No	empty	Queries that must match.
should	No	empty	Queries that may match and contribute score.
must_not	No	empty	Queries that exclude matching rows.

The native reader also accepts a queries array of occurrence/query pairs, such as ["Must", {"match": {"query": "paimon"}}].

Multi Match

multi_match searches several native text fields in one query.

{
  "multi_match": {
    "query": "paimon search",
    "columns": ["title", "body"],
    "boosts": [2.0, 1.0]
  }
}

The current Paimon global index build path creates one full-text index for one table column. To search multiple Paimon table columns, create one full-text index per column and combine them with Hybrid Search. multi_match is useful for native multi-field indexes provided by adapters that write multiple native text fields.

Boost Demotion

Use boost to down-rank rows that also match a negative query.

{
  "boost": {
    "positive": {
      "match": {
        "query": "paimon"
      }
    },
    "negative": {
      "match": {
        "query": "vector"
      }
    },
    "negative_boost": 0.3
  }
}

Field	Required	Default	Description
positive	Yes	N/A	Query that contributes the main score.
negative	Yes	N/A	Query used to down-rank matching rows.
negative_boost	No	0.5	Positive multiplier applied to rows that also match negative.

Drop Full-Text Index

SQL

CALL sys.drop_global_index(
    table => 'db.my_table',
    index_column => 'content',
    index_type => 'full-text'
);

Python SDK

table = catalog.get_table("db.my_table")

dropped_files = table.drop_global_index(
    "content",
    index_type="full-text",
)
print(dropped_files)

You can also restrict the drop to selected partitions, or count matched files without committing:

matched_files = table.drop_global_index(
    "content",
    index_type="full-text",
    partitions=[{"dt": "2026-06-18"}, {"dt": "2026-06-19"}],
    dry_run=True,
)
print(matched_files)

Notes

• The table column is supplied by full_text_search(..., column, query, limit), withQuery(column, query), or with_query(column, query).
• The optional column field inside the JSON query belongs to the native paimon-full-text-index DSL. Most Paimon users should omit it and pass the Paimon table column through the Paimon API instead.
• The query string must be valid JSON understood by the native full-text reader. Invalid query syntax fails during search.
• limit must be positive. Full-text search selects top-K scored matches, which can be combined with vector results by hybrid search rankers. A subsequent table scan does not guarantee score order. Use explicit SQL ordering or access scores by row ID; see Read Scored Results.