BLOB Storage

Primary-key tables can store top-level BLOB, ARRAY<BLOB>, and MAP<K, BLOB> payloads in table-managed files. Unlike the positional BLOB files used by append tables, managed BLOB payloads have stable descriptors. MergeTree sorting, deduplication, and compaction can therefore reorder or remove rows without rewriting the surviving payload bytes.

Three pieces of storage work together:

Piece	Contents	Lifecycle
Data file	A BlobDescriptor for each non-null scalar, array element, or map value	Rows and descriptors are merged by the table's merge engine
.managed.blob pack	Immutable payload bytes, potentially shared by several rows and files	Surviving descriptors continue to reference the same bytes
.blobref sidecar	Exact set of managed packs referenced by one data file	Owned by that data file and replaced with compacted output

For general BLOB concepts and read options, see BLOB Storage. Before adopting managed storage, check Requirements and Limitations and Garbage Collection. The sections below cover table creation, merge-engine behavior, and the payload lifecycle.

The append-only video-frame-field mode is deliberately separate from primary-key managed BLOB storage. It writes self-contained .video packs containing complete encoded videos and embedded frame-run indexes, and is not supported on primary-key tables. Enabling it does not change .managed.blob packs, .blobref ownership, or their garbage-collection behavior.

Create a Table

Use blob-field to mark scalar, array, or map fields whose payloads should be stored in managed BLOB files. blob-descriptor-field and blob-view-field are inline forms: their serialized descriptor or view metadata stays in the normal data file and is not materialized into a managed BLOB file.

The following Flink SQL example accepts scalar, array, and map values:

CREATE TABLE media (
    id BIGINT,
    name STRING,
    content BYTES COMMENT '__BLOB_FIELD; media content',
    attachments ARRAY<BYTES> COMMENT '__BLOB_FIELD; related files',
    renditions MAP<STRING, BYTES> COMMENT '__BLOB_FIELD; named renditions',
    PRIMARY KEY (id) NOT ENFORCED
) WITH (
    'merge-engine' = 'deduplicate',
    'changelog-producer' = 'none',
    'blob.target-file-size' = '128 mb'
);

INSERT INTO media VALUES
    (1, 'logo', X'89504E470D0A1A0A', ARRAY[X'25504446', NULL], MAP['thumbnail', X'89504E47']);

For a primary-key table, every non-null Blob value in a blob-field is externalized before it enters the MergeTree sort buffer. Its payload is copied into a new table-managed BLOB pack regardless of the value's backing representation. Reads return the payload bytes by default; the existing blob-as-descriptor read option can expose descriptors instead. A blob-descriptor-field is written inline to the normal data file and does not participate in managed storage or its reference sidecars.

Copy Payloads from Another Table

When descriptor-backed BLOBs are copied to another table, the target normally rebuilds a FileIO from its catalog context. For a managed blob-field, this is a copy flow: the target writes the payload into its own BLOB storage and does not retain the source descriptor. If the source table uses table-scoped credentials, configure blob-descriptor.source-table on the target so that the source table's FileIO is used to materialize the payload:

ALTER TABLE media_copy SET TBLPROPERTIES (
    'blob-descriptor.source-table' = 'db.media$branch_rt'
);

Use blob-descriptor-field to retain literal descriptors, or blob-view-field to retain a logical, no-copy reference to an upstream row. Other blob-descriptor.* filesystem options remain sufficient when the source storage can be accessed with static configuration; source-table is for table-scoped FileIO credentials.

The source table must belong to the same catalog. A branch suffix is supported. Target tables without a catalog loader, including external tables in REST catalogs, are not supported. When this option is set, it takes precedence over other blob-descriptor.* options; remove it before switching back to descriptor-specific filesystem configuration.

Arrays, Maps, and Pack Size

ARRAY<BLOB> is externalized element by element. Every non-null Blob element is copied into managed storage, while array order, a null array, and null elements are preserved. An empty array writes no payload. ARRAY<BLOB> uses blob-field; blob-descriptor-field and blob-view-field remain scalar-only declarations.

MAP<K, BLOB> is externalized value by value. Keys remain in the normal data file and every non-null value is replaced with a descriptor to managed storage. A null map, an empty map, and null values are preserved. Supported key types are the integer family, BOOLEAN, DECIMAL, DATE, TIME, BINARY, VARBINARY (BYTES), CHAR, and VARCHAR; blob-descriptor-field and blob-view-field remain scalar-only declarations.

blob.target-file-size controls when a writer rolls to a new managed payload pack. A pack can contain payloads from multiple rows, and a row descriptor records its URI, offset, and length.

note

On append tables, blob-descriptor-field is descriptor-only storage and non-null values must provide a descriptor. Append-table blob-field storage still requires row tracking and data evolution. The managed raw-byte externalization described here is specific to supported primary-key tables.

Requirements and Limitations

Primary-key managed BLOB storage has the following requirements:

Item	Requirement
Managed BLOB declaration	BLOB, ARRAY<BLOB>, and MAP<K, BLOB> use blob-field; blob-descriptor-field remains inline
Merge engine	deduplicate, partial-update, or first-row
Changelog producer	none only
Key usage	A managed BLOB column cannot be a primary, partition, bucket, or sequence key
External data paths	data-file.external-paths is not supported
PK clustering override	pk-clustering-override is not supported

row-tracking.enabled and data-evolution.enabled are not required for this primary-key mode.

Partial-update with BLOB fields

Primary-key tables may use merge-engine=partial-update with scalar blob-descriptor-field, managed blob-field (BLOB, ARRAY<BLOB>, or MAP<K, BLOB>), and scalar blob-view-field:

Mode	Partial-update	Changelog	Notes
Scalar blob-descriptor-field	Supported	Same as other PU tables	Inline descriptor bytes
blob-field (managed scalar, array, or map)	Supported	changelog-producer=none only	Payload in .managed.blob
Scalar blob-view-field	Supported	Same as other PU tables	Resolve on read via catalog
ARRAY / MAP descriptor or view	Not supported	—	Scalar only

The ordering fields on the left-hand side of fields.<ordering-field[,ordering-field...]>.sequence-group cannot contain BLOB values. This includes BLOB, ARRAY<BLOB>, and MAP<K, BLOB> fields in any storage mode. BLOB fields are supported on the right-hand side as fields protected by the sequence group.

CREATE TABLE media_meta (
    id BIGINT,
    name STRING,
    content BYTES COMMENT '__BLOB_DESCRIPTOR_FIELD; external video',
    ts INT,
    PRIMARY KEY (id) NOT ENFORCED
) WITH (
    'merge-engine' = 'partial-update',
    'fields.ts.sequence-group' = 'name,content'
);

For managed BLOB columns, set changelog-producer=none (same as deduplicate managed BLOB tables). Managed BLOB retract records do not retain payload values. When a managed BLOB field is protected by a sequence group and retract records are processed, it therefore cannot use an aggregate function that depends on the original retract payload. last_value is supported because it always clears the field on retract; other aggregate functions require fields.<field-name>.ignore-retract=true. This restriction does not apply when ignore-delete=true or when the managed BLOB field is not protected by a sequence group.

CREATE TABLE training_chunks (
    id BIGINT,
    name STRING,
    chunk BYTES COMMENT '__BLOB_FIELD; raw training block',
    PRIMARY KEY (id) NOT ENFORCED
) WITH (
    'merge-engine' = 'partial-update',
    'changelog-producer' = 'none',
    'blob-field' = 'chunk'
);

Non-null values update the corresponding column; null values do not update the field (standard partial-update semantics). Within a sequence group, a null sequence value skips the entire group. Without aggregate functions, an incoming sequence value that is newer or equal replaces every field in the group, including replacing a BLOB value with null; an older record is ignored. With aggregate functions, every record with a non-null sequence value participates in aggregation or retraction, even when its sequence value is older. For example, last_value clears the field for both newer and older retract records.

Managed BLOB partial updates externalize each non-null scalar BLOB, array element, or map value into a .managed.blob pack. Empty collections and collections containing only null values write no payload. Unreachable packs from repeated updates are reclaimed by remove_orphan_blobs after they are older than older_than.

blob-view-field columns store serialized view structs inline. Reads resolve upstream blob bytes through the catalog when blob-view.resolve.enabled is true (default). Append upstream tables used by sys.blob_view(...) must enable row-tracking.enabled and data-evolution.enabled.

First-row with BLOB fields

Primary-key tables may use merge-engine=first-row with managed blob-field (BLOB, ARRAY<BLOB>, or MAP<K, BLOB>):

Mode	First-row	Changelog	Notes
blob-field (managed scalar, array, or map)	Supported	changelog-producer=none only	Keeps the first non-retract row and its managed payload

For managed BLOB columns, set changelog-producer=none. first-row normally also supports the lookup changelog producer, but managed BLOB storage requires none.

first-row keeps the earliest surviving row for the same primary key. Later updates to the same key therefore do not replace an existing managed BLOB payload. This differs from deduplicate, which keeps the latest row.

By default, first-row does not accept DELETE or UPDATE_BEFORE records. Configure ignore-delete=true if your pipeline may emit them.

CREATE TABLE training_chunks (
    id BIGINT,
    name STRING,
    chunk BYTES COMMENT '__BLOB_FIELD; raw training block',
    PRIMARY KEY (id) NOT ENFORCED
) WITH (
    'merge-engine' = 'first-row',
    'changelog-producer' = 'none',
    'blob-field' = 'chunk'
);

Managed BLOB Update, Delete, and Compaction

Each incoming non-null scalar BLOB, array element, or map value is written as a new descriptor and payload before merge rules are applied. An update later discarded by partial-update or sequence rules can therefore leave an orphaned payload pack. A delete record does not write a new payload. The merge engine determines the logical final row during reads and compaction. Each data file's .blobref sidecar records managed packs referenced by non-retract key-values in that file, which may include intermediate partial-update payloads before compaction.

The example below uses deduplicate. A newer descriptor for key 7 replaces its old descriptor; the payload for unchanged key 11 remains shared with the input files.

Compaction preserves descriptors for surviving values and creates new .blobref sidecars from the compacted output. It does not copy the referenced payload bytes into new .managed.blob packs. This keeps ordinary compaction cost proportional to row metadata instead of BLOB size.

The .blobref file is owned by its data file through DataFileMeta.extraFiles, so it follows the data file through snapshot, tag, branch, rollback, expiration, and deletion lifecycles. Shared .managed.blob packs are deliberately not extra files because more than one retained data file can reference the same pack.

Garbage Collection

Unreferenced .managed.blob packs are reclaimed by managed blob orphan cleanup. Local cleanup is LocalManagedBlobOrphanFilesClean. Spark exposes the same cleanup as remove_orphan_blobs; Flink exposes it as remove_orphan_blobs. The cleaner reads every retained data file's .blobref sidecar across snapshots, tags, and branches, then deletes packs that are not referenced and whose modification time is earlier than the absolute older_than cutoff (1 day before the run starts by default). remove_orphan_files never deletes .managed.blob packs.

This cleanup is best-effort. It lists snapshots, collects used packs twice, and aborts the run (deletes nothing) if the snapshot topology or used-pack set changed between those collections. That shrinks the window in which a concurrent commit can change reachability. Standard Paimon compaction does not make a pack that was unreachable at the final collection reachable afterward: it only reuses packs referenced by its input data files, and deletion-conflict detection rejects a stale compact whose inputs have already been removed. Under these standard compaction invariants, no separate commit lease is required for that compaction path.

older_than provides a grace period for packs created by a writer but not yet referenced by a committed snapshot. Standard writers create new UUID-named packs; choose a cutoff far enough behind the current time for writes, commits, and retries to finish. The one-day default assumes those operations complete within one day. Writers that publish references to pre-existing old packs, or commit implementations that bypass normal deletion-conflict detection, are outside this safety model.

A missing, corrupt, or unsupported .blobref sidecar on a data file that still exists is unsafe: that run skips deleting every .managed.blob file. ADD entries left in unmerged manifests after snapshot expire, whose data files are already gone, are ignored. An empty, valid sidecar is different from a missing sidecar: it explicitly states that the data file references no managed payload pack.

Snapshot expiration still deletes only the data file and its .blobref extra file. Pack bytes are reclaimed on the next managed blob orphan cleanup run after they become unreachable.

Reference Metadata

The BLOB reference set is recorded in a data-file-owned .blobref sidecar rather than IndexManifest. BLOB reachability is an exact dependency of one data file, and DataFileMeta.extraFiles already provides the required lifecycle for such metadata. Using IndexManifest would require a new snapshot-level index lifecycle and would be a larger compatibility change.

Each sidecar is immutable, versioned, checksummed, and deterministic. It stores only managed payload identities; descriptor and view fields remain inline and are not added to the managed reference set. Managed pack identity is derived from the descriptor URI and its reserved .managed.blob suffix, so a valid reference is retained even when the pack and the data file are in different directories.