FM Index

The FM index is an exact byte-oriented substring index for CHAR, VARCHAR, and STRING columns. It supports CONTAINS needles of any byte length without a configured gram size. Null values do not match; empty needles follow the normal Paimon predicate semantics.

The writer divides source rows into independent partitions and appends them to one checksummed container file. Each partition stores a compressed wavelet matrix, sampled suffix-array values, row boundaries, and null rows. It does not duplicate the source values. Reads demand-load bounded blocks instead of downloading the complete container. If locating matches would cost too much, the index declines the predicate so the normal data path can scan the source values and preserve exact results.

Create a Global FM Index

Create the index on a Data Evolution table with row tracking enabled:

CALL sys.create_global_index(
    table => 'db.documents',
    index_column => 'content',
    index_type => 'fmindex',
    options => 'fm-index.partition-row-count=100000'
);

Drop it with the same index type:

CALL sys.drop_global_index(
    table => 'db.documents',
    index_column => 'content',
    index_type => 'fmindex'
);

See Global Index for row-tracking requirements, lifecycle, and partial-coverage semantics.

Query

After a connector converts a literal substring expression to Paimon's CONTAINS predicate, the normal batch scan uses the FM index automatically. For example, Spark pushes down:

SELECT id, content
FROM documents
WHERE content LIKE '%needle%';

Results are exact inside every indexed row range. A table-wide global-index query can still omit matches in row ranges that have never been indexed; rebuild the index for newly appended ranges or use the visibility callback described on the Global Index page.

For a primary-key table, configure pk-fm.index.columns instead. Primary-key FM indexes follow data compaction and scan uncovered files through the ordinary data path, so partial coverage does not make CONTAINS results incomplete. See Primary-Key Indexes.

Options

Global options can be passed to create_global_index. For a primary-key index, place the same keys in fields.<column>.pk-fm.index.options; the fm-index. prefix may be omitted inside that JSON object.

Option	Default	Description
fm-index.partition-size	16 mb	Maximum encoded text buffered by one independently readable partition.
fm-index.partition-row-count	100000	Maximum source rows in one partition.
fm-index.sa-sample-rate	32	Suffix-array sample rate. A smaller power of two speeds locate at the cost of a larger index.
fm-index.compression	lz4	Compression codec for independently checksummed FM blocks.
fm-index.compression-level	1	Compression level for codecs which support levels.
fm-index.read-cache-size	64 mb	Maximum decoded rank and sample block cache per indexer.
fm-index.demand-page-size	512 kb	Target contiguous range size when demand-loading blocks.
fm-index.locate-cost-ratio	0.001	Maximum estimated suffix-array locate work relative to source text bytes before declining index evaluation.

fm-index.partition-size and fm-index.partition-row-count bound build memory and the unit of independent reads. Smaller partitions reduce peak construction memory but increase the number of partitions searched per query. All partitions produced by one writer are stored in one physical index file and represented by one index manifest entry. A lower fm-index.sa-sample-rate accelerates locating matched rows but stores more suffix-array samples.

Limitations

• The index is single-column and accepts only character string types.
• Matching is byte-oriented and case-sensitive; it does not apply a tokenizer, collation, or Unicode normalization.
• Global FM indexes inherit the partial-coverage behavior of table-wide global indexes.
• Primary-key FM indexes are built from eligible compact output, not directly from Level-0 appends.