BLOB Storage

Primary-key tables can store top-level BLOB, ARRAY<BLOB>, and MAP<K, BLOB> payloads in table-managed files. Unlike the positional BLOB files used by append tables, managed BLOB payloads have stable descriptors. MergeTree sorting, deduplication, and compaction can therefore reorder or remove rows without rewriting the surviving payload bytes.

This mode stores:

• a serialized BlobDescriptor for each scalar value, non-null array element, or non-null map value;
• the payload in an immutable .managed.blob pack; and
• one .blobref sidecar for every data file, containing the exact managed packs referenced by that file.

For general BLOB concepts and read options, see BLOB Storage.

Create a Table

Use blob-field to mark scalar, array, or map fields whose payloads should be stored in managed BLOB files. blob-descriptor-field and blob-view-field are inline forms: their serialized descriptor or view metadata stays in the normal data file and is not materialized into a managed BLOB file.

The following example accepts both a scalar value and an ordered array of values:

CREATE TABLE media (
    id BIGINT,
    name STRING,
    content BYTES COMMENT '__BLOB_FIELD; media content',
    attachments ARRAY<BYTES> COMMENT '__BLOB_FIELD; related files',
    renditions MAP<STRING, BYTES> COMMENT '__BLOB_FIELD; named renditions',
    PRIMARY KEY (id) NOT ENFORCED
) WITH (
    'merge-engine' = 'deduplicate',
    'changelog-producer' = 'none',
    'blob.target-file-size' = '128 mb'
);

INSERT INTO media VALUES
    (1, 'logo', X'89504E470D0A1A0A', ARRAY[X'25504446', NULL], MAP['thumbnail', X'89504E47']);

For a primary-key table, every non-null Blob value in a blob-field is externalized before it enters the MergeTree sort buffer. Its payload is copied into a new table-managed BLOB pack regardless of the value's backing representation. Reads return the payload bytes by default; the existing blob-as-descriptor read option can expose descriptors instead. A blob-descriptor-field is written inline to the normal data file and does not participate in managed storage or its reference sidecars.

When descriptor-backed BLOBs are copied to another table, the target normally rebuilds a FileIO from its catalog context. For a managed blob-field, this is a copy flow: the target writes the payload into its own BLOB storage and does not retain the source descriptor. If the source table uses table-scoped credentials, configure blob-descriptor.source-table on the target so that the source table's FileIO is used to materialize the payload:

ALTER TABLE media_copy SET TBLPROPERTIES (
    'blob-descriptor.source-table' = 'db.media$branch_rt'
);

Use blob-descriptor-field to retain literal descriptors, or blob-view-field to retain a logical, no-copy reference to an upstream row. Other blob-descriptor.* filesystem options remain sufficient when the source storage can be accessed with static configuration; source-table is for table-scoped FileIO credentials.

The source table must belong to the same catalog. A branch suffix is supported. Target tables without a catalog loader, including external tables in REST catalogs, are not supported. When this option is set, it takes precedence over other blob-descriptor.* options; remove it before switching back to descriptor-specific filesystem configuration.

ARRAY<BLOB> is externalized element by element. Every non-null Blob element is copied into managed storage, while array order, a null array, and null elements are preserved. An empty array writes no payload. ARRAY<BLOB> uses blob-field; blob-descriptor-field and blob-view-field remain scalar-only declarations.

MAP<K, BLOB> is externalized value by value. Keys remain in the normal data file and every non-null value is replaced with a descriptor to managed storage. A null map, an empty map, and null values are preserved. Supported key types are the integer family, CHAR, and VARCHAR; blob-descriptor-field and blob-view-field remain scalar-only declarations.

blob.target-file-size controls when a writer rolls to a new managed payload pack. A pack can contain payloads from multiple rows, and a row descriptor records its URI, offset, and length.

note

On append tables, blob-descriptor-field is descriptor-only storage and writes must provide a descriptor. Append-table blob-field storage still requires row tracking and data evolution. The managed raw-byte externalization described here is specific to supported primary-key tables.

Requirements and Limitations

Primary-key managed BLOB storage has the following requirements:

Item	Requirement
Managed BLOB declaration	BLOB, ARRAY<BLOB>, and MAP<K, BLOB> use blob-field; blob-descriptor-field remains inline
Merge engine	deduplicate only
Changelog producer	none only
Key usage	A managed BLOB column cannot be a primary, partition, bucket, or sequence key
External data paths	data-file.external-paths is not supported
PK clustering override	pk-clustering-override is not supported

row-tracking.enabled and data-evolution.enabled are not required for this primary-key mode.

Update, Delete, and Compaction

An update writes a new descriptor and managed payload when a scalar value, array element, or map value changes. A delete record does not write a new payload. Deduplication determines the final rows of each data file, and its .blobref sidecar contains only the managed packs referenced by those rows.

Compaction preserves descriptors for surviving values and creates new .blobref sidecars from the compacted output. It does not copy the referenced payload bytes into new .managed.blob packs. This keeps ordinary compaction cost proportional to row metadata instead of BLOB size.

The .blobref file is owned by its data file through DataFileMeta.extraFiles, so it follows the data file through snapshot, tag, branch, rollback, expiration, and deletion lifecycles. Shared .managed.blob packs are deliberately not extra files because more than one retained data file can reference the same pack.

Garbage Collection

Garbage collection of unreferenced .managed.blob packs is not implemented yet. Updates, deletes, compaction, or an ambiguous writer failure can therefore leave payload packs that are no longer reachable from current rows.

The ordinary orphan-file cleaner intentionally preserves all .managed.blob files. This fail-safe behavior prevents it from deleting a payload that is still reachable from a snapshot, tag, branch, or another retained root, but it also means unused BLOB storage can grow until a root-aware BLOB garbage collector is available.

A future collector must compute reachability across all retained roots and treat a missing, corrupt, or unsupported .blobref sidecar as unsafe to delete. An empty, valid sidecar is different from a missing sidecar: it explicitly states that the data file references no managed payload pack.

Reference Metadata

The BLOB reference set is recorded in a data-file-owned .blobref sidecar rather than IndexManifest. BLOB reachability is an exact dependency of one data file, and DataFileMeta.extraFiles already provides the required lifecycle for such metadata. Using IndexManifest would require a new snapshot-level index lifecycle and would be a larger compatibility change.

Each sidecar is immutable, versioned, checksummed, and deterministic. It stores only managed payload identities; descriptor and view fields remain inline and are not added to the managed reference set. Managed pack identity is derived from the descriptor URI and its reserved .managed.blob suffix, so a valid reference is retained even when the pack and the data file are in different directories.