Overview
A multimodal table keeps metadata, binary payloads, embeddings, and text in one logical table. An ingestion pipeline can append records, enrich selected columns, and build indexes for filtering or retrieval.
This chapter focuses on append tables with Data Evolution: column files share row IDs, so adding captions or replacing embeddings can leave large media payloads in their existing files. It is an append table with additional storage and update capabilities, not a primary-key table that deduplicates records by a business key.
Start Here
Follow the Quick Start to create a table, write sample data, update selected columns, and build a scalar index. It uses Spark SQL in a configured Paimon catalog. The topic pages also provide Flink, Java, and Python examples where supported.
| If you need to… | Read |
|---|---|
| Store images, audio, documents, or collections of binary objects | BLOB Storage |
| Reuse payloads across tables or export a temporary URL | BLOB References and Sharing |
| Address video frames while storing complete encoded videos | Video Frame Storage |
| Store fixed-dimension embeddings | Vector Storage |
| Query metadata whose structure varies between rows | Variant Storage |
| Backfill or replace selected columns; update and delete rows | Data Evolution |
| Understand file ranges, column versions, and row reconstruction | File Layout and Reads |
| Compact column files or materialize logical deletions | Data Evolution Maintenance |
| Choose an index and retrieve matching records | Global Index |
Choose Storage and Indexing Separately
Storage determines how values are represented and read. Indexes determine how queries find matching row IDs. A stored vector does not automatically have an ANN index, and a text column does not automatically have a full-text index.
| Capability | Setup for this append-table workflow |
|---|---|
| Partial-column updates and managed BLOB storage | Enable row-tracking.enabled and data-evolution.enabled at table creation. |
| Dedicated vector files | Enable both properties above and set vector.file.format = vortex. |
| Global indexes | Use a Data Evolution table, load data, then explicitly build indexes. |
| Variant | Use Parquet and a supported engine; Data Evolution is optional. |
Data Evolution requires no primary key and bucket = -1, the default for append
tables. The basic properties are:
'row-tracking.enabled' = 'true',
'data-evolution.enabled' = 'true'
For JVM primary-key workloads, use the separate guides for BLOB storage and indexes. Their requirements differ from this chapter.
Plan for Updates and Freshness
- A normal-column update writes the selected columns over complete affected file ranges. It saves I/O on untouched columns; it can still process more rows than the update predicate matches.
- Appending data does not refresh existing indexes. The default
fastsearch mode can omit uncovered rows; choose a coverage policy. - Ordinary updates and compaction preserve row IDs. Some maintenance operations reassign them; coordinate row-ID lifetime with BLOB views or applications that persist row references.