Full-Text Index

Full-text index lets Paimon run top-k text retrieval on a string column through the Global Index framework. The index type is full-text. The Paimon integration module is paimon-full-text, backed by the standalone paimon-full-text-index native library.

The search API passes the table column name separately from the query. The query itself is a JSON string accepted by paimon-full-text-index.

Full-text indexes are scored by the native engine and can be used directly, or combined with vector routes in Hybrid Search.

Build Full-Text Index

Build full-text indexes on STRING, CHAR, or VARCHAR columns. Null values are skipped by the native text index, while row-id coverage is still tracked by Paimon.

SQL

CALL sys.create_global_index(
    table => 'db.my_table',
    index_column => 'content',
    index_type => 'full-text'
);

Use the ngram tokenizer for short character fragments or substring-like lookup:

CALL sys.create_global_index(
    table => 'db.my_table',
    index_column => 'content',
    index_type => 'full-text',
    options => 'full-text.tokenizer=ngram,full-text.ngram.min-gram=2,full-text.ngram.max-gram=3'
);

Use jieba for Chinese word segmentation:

CALL sys.create_global_index(
    table => 'db.my_table',
    index_column => 'content',
    index_type => 'full-text',
    options => 'full-text.tokenizer=jieba'
);

Python SDK

table = catalog.get_table("db.my_table")

added_files = table.create_global_index(
    "content",
    index_type="full-text",
)
print(added_files)

Use the ngram tokenizer for short character fragments or substring-like lookup:

added_files = table.create_global_index(
    "content",
    index_type="full-text",
    options={
        "full-text.tokenizer": "ngram",
        "full-text.ngram.min-gram": "2",
        "full-text.ngram.max-gram": "3",
    },
)
print(added_files)

Install the PyPaimon full-text extra before building or querying full-text indexes:

pip install 'pypaimon[full-text]'

All native full-text options must use the public full-text. prefix in Paimon. Paimon removes this prefix once and passes the remaining keys to the native library. For example, set full-text.tokenizer=ngram, not tokenizer=ngram.

Supported full-text index options:

Option	Default	Description
full-text.tokenizer	default	Tokenizer used by the native full-text index. Supported values are default, simple, whitespace, raw, ngram, and jieba.
full-text.ngram.min-gram	3	Minimum gram length for the ngram tokenizer.
full-text.ngram.max-gram	3	Maximum gram length for the ngram tokenizer.
full-text.ngram.prefix-only	false	Whether the ngram tokenizer only emits prefix ngrams.
full-text.jieba.search-mode	true	Whether the jieba tokenizer uses search mode.
full-text.jieba.ordinal-position	true	Whether the jieba tokenizer uses ordinal positions.
full-text.lower-case	true	Whether configurable tokenizers lowercase emitted tokens.
full-text.max-token-length	40	Maximum token length kept by configurable tokenizers.
full-text.ascii-folding	true	Whether to normalize non-ASCII Latin characters to ASCII.
full-text.stem	true	Whether to apply stemming.
full-text.language	english	Language used by stemming and built-in stop-word filters.
full-text.remove-stop-words	true	Whether to remove built-in stop words for the configured language.
full-text.stop-words	empty	Semicolon-separated custom stop words. Requires full-text.remove-stop-words=true.
full-text.with-position	true	Whether to store term positions. Keep this enabled for phrase queries.

The default tokenizer uses English full-text defaults: lower-case, stemming, stop-word removal, ASCII folding, maximum token length 40, and positions. Set full-text.with-position=false only when phrase search is not needed.

Tokenizer settings are stored in the global index file metadata. Existing index files keep the tokenizer options they were built with, even if later index builds use different options.

Drop Full-Text Index

SQL

CALL sys.drop_global_index(
    table => 'db.my_table',
    index_column => 'content',
    index_type => 'full-text'
);

Python SDK

table = catalog.get_table("db.my_table")

dropped_files = table.drop_global_index(
    "content",
    index_type="full-text",
)
print(dropped_files)

You can also restrict the drop to selected partitions, or count matched files without committing:

matched_files = table.drop_global_index(
    "content",
    index_type="full-text",
    partitions=[{"dt": "2026-06-18"}, {"dt": "2026-06-19"}],
    dry_run=True,
)
print(matched_files)

Full-Text Search

Spark SQL uses the full_text_search(table_name, column, query, limit) table-valued function. The column argument is the Paimon table column to search. The query argument is passed to the native full-text reader as a JSON string.

Spark SQL

-- Search for the top 10 rows matching any query term.
SELECT id, content, __paimon_search_score
FROM full_text_search(
    'my_table',
    'content',
    '{"match":{"query":"paimon lake format"}}',
    10
);

-- Require all query terms.
SELECT id, content, __paimon_search_score
FROM full_text_search(
    'my_table',
    'content',
    '{"match":{"query":"paimon lake format","operator":"And"}}',
    10
);

-- Phrase query. The index must be built with full-text.with-position=true.
SELECT id, content, __paimon_search_score
FROM full_text_search(
    'my_table',
    'content',
    '{"match_phrase":{"query":"paimon lake"}}',
    10
);

Spark exposes the score as the __paimon_search_score metadata column.

Java API

Table table = catalog.getTable(identifier);

GlobalIndexResult result =
        table.newFullTextSearchBuilder()
                .withQuery("content", "{\"match\":{\"query\":\"paimon lake\"}}")
                .withLimit(10)
                .executeLocal();

ReadBuilder readBuilder = table.newReadBuilder();
TableScan.Plan plan = readBuilder.newScan().withGlobalIndexResult(result).plan();
try (RecordReader<InternalRow> reader = readBuilder.newRead().createReader(plan)) {
    reader.forEachRemaining(row -> System.out.println(row));
}

Python SDK

table = catalog.get_table("db.my_table")

result = (
    table.new_full_text_search_builder()
    .with_query("content", '{"match":{"query":"paimon lake"}}')
    .with_limit(10)
    .execute_local()
)

read_builder = table.new_read_builder()
plan = read_builder.new_scan().with_global_index_result(result).plan()
pa_table = read_builder.new_read().to_arrow(plan.splits())
print(pa_table)

By default, full-text-index.search-mode=fast searches indexed row ranges only. When full-text-index.search-mode is full or detail, Paimon also covers unindexed row ranges by reading raw rows and building a temporary native full-text index for the searched column.

Query DSL

The query DSL is a JSON object with one top-level query type. Use the examples below as JSON strings in Spark SQL, Java, Python, or hybrid search routes.

Match

Use match for normal term search. The default operator is Or.

{
  "match": {
    "query": "paimon lake format",
    "operator": "And"
  }
}

Field	Required	Default	Description
query	Yes	N/A	Query text.
operator	No	Or	How query terms are combined. Supported values are Or and And.
boost	No	1.0	Score multiplier for this query.
fuzziness	No	0	Edit distance for fuzzy matching. Use an integer or auto.
max_expansions	No	50	Maximum fuzzy expansions. maxExpansions is also accepted.
prefix_length	No	0	Number of leading characters that must match exactly. prefixLength is also accepted.

Phrase

Use match_phrase for ordered terms. Phrase search requires positions, so build the index with full-text.with-position=true.

{
  "match_phrase": {
    "query": "paimon lake",
    "slop": 1
  }
}

Field	Required	Default	Description
query	Yes	N/A	Phrase text.
slop	No	0	Number of positional moves allowed when matching the phrase.

Boolean

Use boolean to combine nested queries.

{
  "boolean": {
    "must": [
      {"match": {"query": "paimon"}}
    ],
    "should": [
      {"match_phrase": {"query": "lake format"}}
    ],
    "must_not": [
      {"match": {"query": "vector"}}
    ]
  }
}

Field	Required	Default	Description
must	No	empty	Queries that must match.
should	No	empty	Queries that may match and contribute score.
must_not	No	empty	Queries that exclude matching rows.

The native reader also accepts a queries array of occurrence/query pairs, such as ["Must", {"match": {"query": "paimon"}}].

Multi Match

multi_match searches several native text fields in one query.

{
  "multi_match": {
    "query": "paimon search",
    "columns": ["title", "body"],
    "boosts": [2.0, 1.0]
  }
}

The current Paimon global index build path creates one full-text index for one table column. To search multiple Paimon table columns, create one full-text index per column and combine them with Hybrid Search. multi_match is useful for native multi-field indexes provided by adapters that write multiple native text fields.

Boost Demotion

Use boost to down-rank rows that also match a negative query.

{
  "boost": {
    "positive": {
      "match": {
        "query": "paimon"
      }
    },
    "negative": {
      "match": {
        "query": "vector"
      }
    },
    "negative_boost": 0.3
  }
}

Field	Required	Default	Description
positive	Yes	N/A	Query that contributes the main score.
negative	Yes	N/A	Query used to down-rank matching rows.
negative_boost	No	0.5	Positive multiplier applied to rows that also match negative.

Notes

• The table column is supplied by full_text_search(..., column, query, limit), withQuery(column, query), or with_query(column, query).
• The optional column field inside the JSON query belongs to the native paimon-full-text-index DSL. Most Paimon users should omit it and pass the Paimon table column through the Paimon API instead.
• The query string must be valid JSON understood by the native full-text reader. Invalid query syntax fails during search.
• limit must be positive. Full-text results are returned in score order and can be merged with vector results by hybrid search rankers.