Skip to main content

Blob Storage

Overview​

The BLOB (Binary Large Object) type is a data type designed for storing multimodal data such as images, videos, audio files, and other large binary objects in Paimon tables. Unlike traditional BYTES type which stores binary data inline with other columns, BLOB type stores large binary data in separate files and maintains references to them, providing better performance for large objects. A field can also use ARRAY<BLOB> for an ordered collection or MAP<K, BLOB> for keyed blobs.

The Blob Storage is based on Data Evolution mode.

The Blob type is ideal for:

  • Image Storage: Store product images, user avatars, medical imaging data
  • Video Content: Store video clips, surveillance footage, multimedia content
  • Audio Files: Store voice recordings, music files, podcast episodes
  • Document Storage: Store PDF documents, office files, large text files
  • Machine Learning: Store embeddings, model weights, feature vectors
  • Any Large Binary Data: Any data that is too large to store efficiently inline

Storage Layout​

When you define a table with a Blob column, Paimon automatically separates the storage:

  1. Normal Data Files (e.g., .parquet, .orc): Store regular columns (INT, STRING, etc.)
  2. Blob Data Files (.blob): Store the actual blob data

For append-only video-frame workloads, one or more scalar BLOB fields can instead use the video pack format (.video). A video pack concatenates several complete encoded videos and embeds a compact frame-run index. The frame ordinal lives in VideoFrameDescriptor; it does not need a frame_index or frame_timestamp column in the normal data file. When multiple video fields are configured, their physical boundaries may be nested, but every payload change must occur at a logical episode boundary so the normal and dedicated files remain one aligned file group.

For example, given a table with schema (id INT, name STRING, picture BLOB):

table/
├── bucket-0/
│   ├── data-uuid-0.parquet      # Contains id, name columns
│   ├── data-uuid-1.blob         # Contains picture blob data
│   ├── data-uuid-2.blob         # Contains more picture blob data
│   └── ...
├── manifest/
├── schema/
└── snapshot/

This separation provides several benefits:

  • Efficient column projection (reading non-blob columns doesn't load blob data)
  • Optimized file rolling based on blob size
  • Better compression for regular columnar data

For details about the blob file format structure, see File Format - BLOB.

Storage Modes​

Paimon supports three storage modes for BLOB fields, selected via comment directives on the column:

  1. Default blob storage (__BLOB_FIELD) Blob bytes are written to Paimon-managed .blob files under the table path. This mode supports BLOB, ARRAY<BLOB>, and MAP<K, BLOB> fields.

  2. Descriptor-only storage (__BLOB_DESCRIPTOR_FIELD) Only serialized BlobDescriptor bytes are stored inline in data files. Paimon does not write .blob files for these fields, and non-null values must provide descriptor-based input.

  3. Blob view storage (__BLOB_VIEW_FIELD) Serialized BlobViewStruct bytes are stored inline. The struct points to a BLOB value in an upstream table by table identifier, BLOB field, and row id. The actual blob bytes are resolved from the upstream table at read time.

This allows one table to mix different storage modes for different BLOB columns. ARRAY<BLOB> and MAP<K, BLOB> are supported only by __BLOB_FIELD; descriptor-only and blob-view comment directives accept scalar BLOB fields only. Map keys support the integer family, BOOLEAN, DECIMAL, DATE, TIME, BINARY, VARBINARY (BYTES), CHAR, and VARCHAR. Use non-null keys for compatibility across Flink, Spark, and Python.

Table Options​

OptionRequiredDefaultTypeDescription
video-frame-field
No-StringNames one or more comma-separated scalar BLOB fields whose logical values are frames in complete encoded videos packed into .video files. Configured fields must share logical episode boundaries. This version supports append-only tables and exact VideoFrameDescriptor-backed input only.
blob-as-descriptor
NofalseBooleanControls read output format for blob fields. When set to true, queries return serialized BlobDescriptor bytes; when false, queries return actual blob bytes. This option is dynamic and can be changed with ALTER TABLE ... SET.
blob-write-null-on-missing-file
NofalseBoolean

When enabled for Flink writes, if a descriptor BLOB value references a file that does not exist, Paimon writes NULL for that value and logs a warning instead of failing when reading the descriptor.

blob-view.resolve.enabled
NotrueBoolean

Controls whether blob view fields are resolved to the upstream BLOB content at read time. Set to false when forwarding blob view references from one view table to another.

blob.target-file-size
No(same as target-file-size)MemorySizeTarget size for blob files. When a blob file reaches this size, a new file is created. If not specified, uses the same value as target-file-size.
row-tracking.enabled
Yes*falseBooleanMust be enabled for blob tables to support row-level operations.
data-evolution.enabled
Yes*falseBooleanMust be enabled for blob tables to support schema evolution.

*Required for blob functionality to work correctly.

Specifically, if the storage system of the input BlobDescriptor differs from that used by Paimon, you can specify the storage configuration for the input blob descriptor using the prefix blob-descriptor.. For example, if the source data is stored in a different OSS endpoint, you can configure it as below (using flink sql as an example):

CREATE TABLE image_table (
    id INT,
    name STRING,
    image BYTES COMMENT '__BLOB_FIELD'
) WITH (
    'row-tracking.enabled' = 'true',
    'data-evolution.enabled' = 'true',
    'fs.oss.endpoint' = 'aaa',                   -- This is for Paimon's own config
    'blob-descriptor.fs.oss.endpoint' = 'bbb'    -- This is for input blob descriptors' config
);

Creating a Table​

The recommended way to create a blob table in SQL is to use the comment directive __BLOB_FIELD, __BLOB_DESCRIPTOR_FIELD, or __BLOB_VIEW_FIELD on the column. Paimon automatically converts the column to the corresponding BLOB type and registers it in the corresponding option.

CREATE TABLE image_table (
    id INT,
    name STRING,
    image BYTES COMMENT '__BLOB_FIELD; product image',
    gallery ARRAY<BYTES> COMMENT '__BLOB_FIELD; product gallery'
) WITH (
    'row-tracking.enabled' = 'true',
    'data-evolution.enabled' = 'true'
);

-- Multiple blob columns with different storage modes
CREATE TABLE media_table (
    id INT,
    photo BYTES COMMENT '__BLOB_FIELD; original photo',
    thumbnail BYTES COMMENT '__BLOB_DESCRIPTOR_FIELD; thumbnail descriptor',
    preview BYTES COMMENT '__BLOB_VIEW_FIELD; preview from upstream'
) WITH (
    'row-tracking.enabled' = 'true',
    'data-evolution.enabled' = 'true'
);

Video Frame Storage​

Use video-frame-field when the logical table has one row per frame while the physical storage and decoding unit is a complete encoded video. The comma-separated option also marks each named SQL BYTES or BINARY column as a BLOB column. It selects the dedicated .video format; the ordinary .blob version and layout are not changed.

CREATE TABLE video_frames (
    episode_id BIGINT,
    camera_front BYTES,
    camera_wrist BYTES
) WITH (
    'row-tracking.enabled' = 'true',
    'data-evolution.enabled' = 'true',
    'video-frame-field' = 'camera_front,camera_wrist'
);

Each logical value is a VideoFrameDescriptor: its URI range identifies one complete encoded video and its frame ordinal selects a frame inside that video. On write, a .video data region concatenates raw video payloads without ordinary BLOB entry wrappers. Four embedded delta-varint indexes record physical video lengths, logical run lengths, run-to-video references, and the first frame ordinal of each run. Consecutive frames therefore need one run entry rather than one index entry per row. NULL and data-evolution placeholders use negative run references.

Physical reuse uses exact payload descriptor identity (URI, offset, and length), not a content hash. It is local to each .video file: every pack is self-contained and never points at payloads owned by another Paimon data file. A pack can contain multiple MP4 payloads. After a size or row target is reached, rolling waits for the current physical video group to end, making the target soft for large videos. A later commit or an earlier roll can store another physical copy of the same source video. For multi-camera tables, payload boundaries may be nested across fields as long as every transition is an episode boundary. Once a target is reached within an episode, normal, BLOB, and vector rolling is deferred until the next payload boundary and all active writers close together. The resulting file group remains row-aligned and a single large episode may exceed the configured target.

Compaction byte-copies the complete encoded-video ranges into a new self-contained .video pack and rebuilds the embedded indexes; it does not decode or re-encode frames. The aligned normal data file contains only application columns such as episode_id, state, and action. Paimon stores the frame mapping in the .video descriptor/index path.

Current restrictions:

  • Append-only tables only; primary-key tables continue to use managed BLOB storage.
  • Every configured field must be a scalar BLOB; ARRAY<BLOB> and MAP<K, BLOB> continue to use .blob.
  • Non-null writes must be exact descriptor-backed BlobRef values containing a VideoFrameDescriptor. Inline bytes and ordinary BlobDescriptor values are rejected.
  • Version 1 addresses frames by zero-based presentation-order ordinal with stride one. It does not store PTS values or parse codec/container metadata.
  • A BlobConsumer callback is not supported for the video field.

Reads of a .video field return serialized VideoFrameDescriptor values even when blob-as-descriptor is false. Materializing the complete MP4 once per logical frame would defeat the format's purpose.

For Python ingestion and PyTorch DataLoader usage, including decoder-session reuse in workers, see PyPaimon Multimodal API: Video Frame Storage.

The comment directive format is __DIRECTIVE; optional user comment. Paimon converts BYTES/BINARY to BLOB, ARRAY<BYTES>/ARRAY<BINARY> to ARRAY<BLOB>, and MAP<K, BYTES>/MAP<K, BINARY> to MAP<K, BLOB>. It registers the field in the corresponding option and stores the text after ; as the column's real comment.

Supported directives:

DirectiveStorage modeOption
__BLOB_FIELDRaw scalar, array elements, or map values in .blob filesblob-field
__BLOB_DESCRIPTOR_FIELDDescriptor bytes inlineblob-descriptor-field
__BLOB_VIEW_FIELDView reference inlineblob-view-field

Adding a Blob Column​

The same comment directive works with ALTER TABLE ADD COLUMN:

ALTER TABLE image_table ADD picture BYTES COMMENT '__BLOB_FIELD';

ALTER TABLE image_table
    ADD video BYTES COMMENT '__BLOB_DESCRIPTOR_FIELD; promotional video';

Inserting Blob Data​

INSERT INTO image_table VALUES (1, 'sample', X'89504E470D0A1A0A', NULL);

INSERT INTO image_table
SELECT id, name, content, CAST(NULL AS ARRAY<BYTES>) FROM source_table;

For an ARRAY<BLOB> field, write an array of binary values. The array itself and individual elements may be null when the corresponding array and element types are nullable:

INSERT INTO image_table
VALUES (1, 'sample', X'89504E470D0A1A0A', ARRAY[X'01', NULL, X'02']);

Querying Blob Data​

-- Select all columns including blob
SELECT * FROM image_table;

-- Select only non-blob columns (efficient - doesn't load blob data)
SELECT id, name FROM image_table;

-- Return descriptor bytes instead of actual blob bytes
ALTER TABLE image_table SET ('blob-as-descriptor' = 'true');
SELECT image FROM image_table;

Blob Construct Sources (Java API)​

Blob blob = Blob.fromData(imageBytes);                                         // byte array
Blob blob = Blob.fromLocal("/path/to/image.png");                              // local file
Blob blob = Blob.fromFile(fileIO, "s3://bucket/path/to/image.png");            // any FileIO
Blob blob = Blob.fromFile(fileIO, "s3://bucket/large-file.bin", 1024, 2048);   // partial file
Blob blob = Blob.fromHttp("https://example.com/image.png");                    // HTTP URL
Blob blob = Blob.fromInputStream(() -> new FileInputStream("..."));            // InputStream
Blob blob = Blob.fromDescriptor(uriReader, descriptor);                        // BlobDescriptor

Descriptor-Only Storage​

If you want downstream tables to reuse upstream blob files (no copying and no new .blob files), use __BLOB_DESCRIPTOR_FIELD:

CREATE TABLE descriptor_table (
    id INT,
    image BYTES COMMENT '__BLOB_DESCRIPTOR_FIELD; reused image'
) WITH (
    'row-tracking.enabled' = 'true',
    'data-evolution.enabled' = 'true'
);

Paimon stores only serialized BlobDescriptor bytes in normal data files. Reading the blob follows the descriptor URI to access bytes, and non-null values require descriptor input for those fields.

Presigned URLs for OSS Blobs​

Paimon can create a temporary HTTPS URL for an OSS-backed BlobDescriptor. The descriptor may refer to a byte range inside a larger .blob file. Paimon first materializes exactly that range as a separate OSS object without a file extension, sets its content type to application/octet-stream, and then returns a presigned GET URL.

Configure the catalog with the standard public HTTPS OSS endpoint. Internal endpoints and endpoints without the https scheme are rejected because external consumers must be able to fetch the returned URL:

fs.oss.endpoint=https://oss-cn-hangzhou.aliyuncs.com

To obtain descriptors from a regular BLOB column, read the source table with blob-as-descriptor=true. The SQL functions have strict and error-tolerant forms:

-- Flink SQL
SELECT sys.descriptor_to_presigned_url(
    'default.image_table',
    image,
    INTERVAL '5' MINUTE)
FROM image_table /*+ OPTIONS('blob-as-descriptor'='true') */;

SELECT sys.try_descriptor_to_presigned_url(
    'default.image_table',
    image,
    INTERVAL '5' MINUTE)
FROM image_table /*+ OPTIONS('blob-as-descriptor'='true') */;
-- Spark SQL
ALTER TABLE image_table SET TBLPROPERTIES ('blob-as-descriptor' = 'true');

SELECT sys.descriptor_to_presigned_url(
    'default.image_table',
    image,
    INTERVAL '5' MINUTE)
FROM image_table;

In both Flink and Spark, source_table must be a non-null string literal in database.table form. catalog.database.table is also accepted when the catalog matches the catalog that owns the function. Dynamic columns and CASE expressions are rejected during planning. The strict function propagates row errors; try_descriptor_to_presigned_url returns NULL for row-level errors. The validity interval must be positive whole seconds.

The Java API uses the same table-root safety check:

DataTable table = ...;
Blob blob = ...;
String url =
        blob.toPresignedUrl(
                table.fileIO(), table.location(), Duration.ofMinutes(5));

The materialized object's bytes are unchanged. Consumers must inspect the content instead of relying on a URL suffix or a format-specific content type. For direct model image_url inputs, use only formats verified with the target model; PDF is not covered by this entry point.

Materialized objects are keyed by the complete descriptor fingerprint. Repeated short-term calls for the same descriptor reuse an object whose length, content type, and fingerprint still match, while generating a fresh URL. The first version does not resolve races with stale-cache cleanup or orphan-file cleaning.

Treat the returned URL as a bearer credential: submit it immediately to the external service, do not persist it, and never write the URL or its signature/security-token query parameters to logs.

Blob View​

Blob view is useful when a downstream table should reference BLOB values already stored in an upstream table, without copying the bytes or creating new .blob files. A blob view field stores only a small BlobViewStruct inline. When the field is read, Paimon resolves the referenced BLOB from the upstream table.

Blob view requires:

  • the upstream table to have row tracking enabled, so each row has a stable _ROW_ID
  • the downstream field to be declared with __BLOB_VIEW_FIELD comment directive
  • writes to provide a serialized BlobViewStruct; in Flink SQL, use the built-in sys.blob_view function

The Flink SQL function signature is:

sys.blob_view(table_name, field_name, row_id)

Arguments:

  • table_name: the upstream table name. It must be fully qualified as database.table or catalog.database.table. Unqualified table names are rejected.
  • field_name: the upstream BLOB field name.
  • row_id: the _ROW_ID value from the upstream row-tracking table.

The following example writes a downstream table whose image_ref field views the image field in image_table:

CREATE TABLE image_table (
    id INT,
    name STRING,
    image BYTES COMMENT '__BLOB_FIELD'
) WITH (
    'row-tracking.enabled' = 'true',
    'data-evolution.enabled' = 'true'
);

CREATE TABLE image_view_table (
    id INT,
    label STRING,
    image_ref BYTES COMMENT '__BLOB_VIEW_FIELD'
) WITH (
    'row-tracking.enabled' = 'true',
    'data-evolution.enabled' = 'true'
);

INSERT INTO image_view_table
SELECT
    id,
    name AS label,
    sys.blob_view('default.image_table', 'image', _ROW_ID)
FROM `image_table$row_tracking`;

If the current Paimon catalog name is included in the table name, the function also accepts catalog.database.table:

SELECT sys.blob_view('my_catalog.default.image_table', 'image', _ROW_ID)
FROM `image_table$row_tracking`;

Reads from image_view_table.image_ref return the referenced BLOB bytes in the same way as normal blob fields. The referenced upstream table and row must remain available for the view to be resolved.

Forward Blob View References

By default, reading a blob view field resolves the BlobViewStruct and returns the upstream BLOB content. If you want to import data from one blob view table into another blob view table without copying the BLOB bytes, read the source table with blob-view.resolve.enabled=false and write the result into a target field declared with __BLOB_VIEW_FIELD.

With this option disabled, Paimon preserves the serialized BlobViewStruct during reads. When the preserved value is written to another blob view field, the target table stores the same upstream reference instead of creating a chained view reference.

For example, if table T1 contains blob view references to BLOBs in table T0, importing T1 into T2 with blob-view.resolve.enabled=false makes T2 keep referencing T0 directly.

CREATE TABLE t2 (
    id INT,
    image_ref BYTES COMMENT '__BLOB_VIEW_FIELD'
) WITH (
    'row-tracking.enabled' = 'true',
    'data-evolution.enabled' = 'true'
);

-- Flink SQL example: the source table is read with blob view resolution disabled.
INSERT INTO t2
SELECT id, image_ref
FROM t1 /*+ OPTIONS('blob-view.resolve.enabled'='false') */;

MERGE INTO Support​

For Data Evolution writes in Flink and Spark:

  • raw-data BLOB columns are still rejected in partial-column MERGE INTO updates
  • descriptor-based BLOB columns are allowed

For the Python equivalent, see Blob Storage in pypaimon.

Limitations​

  1. Primary Key Tables: Managed BLOB storage in primary-key tables has additional requirements; see Primary-Key BLOB Storage. Primary-key tables support merge-engine=partial-update with scalar blob-descriptor-field, managed blob-field (changelog-producer=none), and scalar blob-view-field; ARRAY / MAP descriptor or view fields are not supported.
  2. No Predicate Pushdown: Blob columns cannot be used in filter predicates.
  3. No Statistics: Statistics collection is not supported for blob columns.
  4. Append Table Options: For append tables, row-tracking.enabled and data-evolution.enabled must be set to true.
  5. Blob View Dependency: Blob view fields depend on the referenced upstream table and row. If the upstream data is removed or no longer readable, the view cannot be resolved.
  6. Collection Storage Mode: ARRAY<BLOB> and MAP<K, BLOB> are supported only for raw managed blob storage (__BLOB_FIELD), not for __BLOB_DESCRIPTOR_FIELD or __BLOB_VIEW_FIELD.

Best Practices​

  1. Use Column Projection: Always select only the columns you need. Avoid SELECT * if you don't need blob data.

  2. Set Appropriate Target File Size: Configure blob.target-file-size based on your blob sizes. Larger values mean fewer files but larger individual files.

  3. Use Descriptor Fields When Reusing External Blob Files: Use __BLOB_DESCRIPTOR_FIELD for fields that should keep descriptor references instead of writing new .blob files.

  4. Use Blob View to Avoid Copying BLOB Data: Use __BLOB_VIEW_FIELD when a downstream table only needs to reference BLOB values from an upstream table.

  5. Use Partitioning: Partition your blob tables by date or other dimensions to improve query performance and data management.