Video Frame Storage

Store one logical row per frame while keeping complete encoded videos as the physical payload. Use ordinary BLOB storage when each row contains an independent image, audio clip, or binary object.

Create a Video Table

Use video-frame-field when the logical table has one row per frame while the physical storage and decoding unit is a complete encoded video. The comma-separated option also marks each named SQL BYTES or BINARY column as a BLOB column. It selects the dedicated .video format; the ordinary .blob version and layout are not changed.

Flink SQL

CREATE TABLE video_frames (
    episode_id BIGINT,
    camera_front BYTES,
    camera_wrist BYTES
) WITH (
    'row-tracking.enabled' = 'true',
    'data-evolution.enabled' = 'true',
    'video-frame-field' = 'camera_front,camera_wrist'
);

Spark SQL

CREATE TABLE video_frames (
    episode_id BIGINT,
    camera_front BINARY,
    camera_wrist BINARY
) TBLPROPERTIES (
    'row-tracking.enabled' = 'true',
    'data-evolution.enabled' = 'true',
    'video-frame-field' = 'camera_front,camera_wrist'
);

Creating the table does not ingest or decode video files. Use the PyPaimon video ingestion API to write complete videos and their logical frame rows.

Descriptors and Physical Layout

Each VideoFrameDescriptor identifies an encoded video range and a frame ordinal. A .video file packs raw videos, optional keyframe indexes, and delta-varint row mappings. Consecutive frames share one run; negative references represent NULL or data-evolution placeholders.

.video (version 1, in file order)
├─ Complete encoded videos A, B, ...
├─ Keyframe index blocks A, B, ...
│  ├─ Video metadata ranges: offset and length within the encoded video
│  └─ Compressed keyframe entries: frame ordinal, PTS, and packet byte position
├─ Metadata indexes
│  ├─ Video payload lengths
│  ├─ Keyframe index block lengths
│  └─ Run lengths, video references, and starting frame indices
└─ Footer: five 4-byte index sizes, 4-byte magic, and 1-byte version

Each video index stores initialization ranges such as MP4 moov and keyframe positions. A cold reader prefetches these ranges and a bounded GOP window, then fetches uncached decoder reads on demand. Zero length selects the scan fallback. PyPaimon generates indexes for supported ISO BMFF videos and preserves them during rewrites. See the file format specification.

Reuse, Rolling, and Compaction

Physical reuse uses exact payload descriptor identity (URI, offset, and length), not a content hash. It is local to each .video file: every pack is self-contained and never points at payloads owned by another Paimon data file. A pack can contain multiple MP4 payloads. After a size or row target is reached, rolling waits for the current physical video group to end, making the target soft for large videos. A later commit or an earlier roll can store another physical copy of the same source video. For multi-camera tables, payload boundaries may be nested across fields as long as every transition is an episode boundary. Once a target is reached within an episode, normal, BLOB, and vector rolling is deferred until the next payload boundary and all active writers close together. The resulting file group remains row-aligned and a single large episode may exceed the configured target.

When BLOB compaction is enabled, it copies videos and keyframe indexes into a new self-contained .video pack without decoding frames. The aligned normal data file contains only application columns such as episode_id, state, and action. blob-compaction.enabled defaults to false; ordinary normal-file compaction can leave the video packs unchanged. See Data Evolution Maintenance.

Requirements and Limitations

• Append-only tables only; primary-key tables continue to use managed BLOB storage.
• Every configured field must be a scalar BLOB; ARRAY<BLOB> and MAP<K, BLOB> continue to use .blob.
• Non-null writes must be exact descriptor-backed BlobRef values containing a VideoFrameDescriptor. Inline bytes and ordinary BlobDescriptor values are rejected.
• A BlobConsumer callback is not supported for the video field.

Read Frames

A frame descriptor identifies the encoded video and the frame to decode. It does not contain decoded image pixels. Read behavior differs by API:

API	How to obtain frame descriptors
Flink and Spark SQL	Set blob-as-descriptor=true. With false, the SQL adapter materializes the complete encoded video payload for each selected frame row.
PyPaimon reads of .video fields	Return serialized VideoFrameDescriptor values even when blob-as-descriptor=false.
Java internal-row API	row.getBlob(position) returns a descriptor-backed BlobRef; call toDescriptor() to access the frame descriptor. toData() reads the complete video payload.

After ingesting data, query descriptors without loading a complete video for every row:

Flink SQL

SELECT episode_id, camera_front, camera_wrist
FROM video_frames /*+ OPTIONS('blob-as-descriptor'='true') */;

Spark SQL

ALTER TABLE video_frames SET TBLPROPERTIES ('blob-as-descriptor' = 'true');
SELECT episode_id, camera_front, camera_wrist FROM video_frames;

This table property remains enabled for subsequent SQL reads.

For Python ingestion and PyTorch DataLoader usage, including decoder-session reuse in workers, see PyPaimon Multimodal API: Video Frame Storage.