Skip to main content

Overview

A multimodal table keeps metadata, binary payloads, embeddings, and text in one logical table. An ingestion pipeline can append records, enrich selected columns, and build indexes for filtering or retrieval.

This chapter focuses on append tables with Data Evolution: column files share row IDs, so adding captions or replacing embeddings can leave large media payloads in their existing files. It is an append table with additional storage and update capabilities, not a primary-key table that deduplicates records by a business key.

One logical record combines normal columns, dedicated BLOB payloads, and optional dedicated vectors by row ID; indexes select matching row IDs for reads.

Start Here​

Follow the Quick Start to create a table, write sample data, update selected columns, and build a scalar index. It uses Spark SQL in a configured Paimon catalog. The topic pages also provide Flink, Java, and Python examples where supported.

If you need to…Read
Store images, audio, documents, or collections of binary objectsBLOB Storage
Reuse payloads across tables or export a temporary URLBLOB References and Sharing
Address video frames while storing complete encoded videosVideo Frame Storage
Store fixed-dimension embeddingsVector Storage
Query metadata whose structure varies between rowsVariant Storage
Backfill or replace selected columns; update and delete rowsData Evolution
Understand file ranges, column versions, and row reconstructionFile Layout and Reads
Compact column files or materialize logical deletionsData Evolution Maintenance
Choose an index and retrieve matching recordsGlobal Index

Choose Storage and Indexing Separately​

Storage determines how values are represented and read. Indexes determine how queries find matching row IDs. A stored vector does not automatically have an ANN index, and a text column does not automatically have a full-text index.

CapabilitySetup for this append-table workflow
Partial-column updates and managed BLOB storageEnable row-tracking.enabled and data-evolution.enabled at table creation.
Dedicated vector filesEnable both properties above and set vector.file.format = vortex.
Global indexesUse a Data Evolution table, load data, then explicitly build indexes.
VariantUse Parquet and a supported engine; Data Evolution is optional.

Data Evolution requires no primary key and bucket = -1, the default for append tables. The basic properties are:

'row-tracking.enabled' = 'true',
'data-evolution.enabled' = 'true'

For JVM primary-key workloads, use the separate guides for BLOB storage and indexes. Their requirements differ from this chapter.

Plan for Updates and Freshness​

  • A normal-column update writes the selected columns over complete affected file ranges. It saves I/O on untouched columns; it can still process more rows than the update predicate matches.
  • Appending data does not refresh existing indexes. The default fast search mode can omit uncovered rows; choose a coverage policy.
  • Ordinary updates and compaction preserve row IDs. Some maintenance operations reassign them; coordinate row-ID lifetime with BLOB views or applications that persist row references.