Variant Storage

Overview

The VARIANT type stores semi-structured data whose shape can differ from row to row. A value can be an object, an array, or a scalar such as a string, number, or boolean. This makes VARIANT suitable for event attributes, model output, tool parameters, and other data that evolves too frequently to model every field as a table column.

Paimon stores VARIANT with the Parquet Variant binary encoding, instead of storing its JSON text. The data is parsed when it is converted to VARIANT, and queries can extract a path as a requested SQL type.

VARIANT does not require Data Evolution mode. SQL support requires Spark 4.0 or later, or Flink 2.1 or later. Variant data files must use Parquet; ORC and Avro do not support this type.

Variant vs. JSON String

A JSON string and a VARIANT value can represent the same logical document, but they have different storage and query behavior.

	JSON stored as STRING	VARIANT
Logical type	An opaque string	Semi-structured, typed data
Validation	Any text can be written; JSON validity is checked only when a JSON function parses it	Input is parsed when it is converted to VARIANT
Physical storage	UTF-8 text in one string column	Binary value and metadata fields
Value types	Numbers, booleans, and nulls are text until parsed	Types are encoded in the value
Field access	A JSON function interprets the text during the query	A Variant function reads a path and converts it to the requested SQL type
Physical optimization	The whole string is read and interpreted	Frequently queried fields can be shredded into typed sub-columns
Best fit	Preserving the original JSON text, or rarely inspecting the content	Repeatedly querying evolving semi-structured data

Converting JSON to VARIANT preserves its data semantics, but not its textual representation. Do not rely on the original whitespace, object field order, or number formatting after conversion. If the exact source text is required for auditing, keep it in a separate STRING column.

Write and Query Variant

The following Spark SQL example creates a Variant column and converts JSON strings with parse_json:

CREATE TABLE events (
    id BIGINT,
    payload VARIANT
) USING paimon
TBLPROPERTIES (
    'file.format' = 'parquet'
);

INSERT INTO events VALUES
    (1, parse_json('{"user":{"id":1001},"city":"Hangzhou","active":true}')),
    (2, parse_json('{"user":{"id":1002},"city":"Beijing","score":9.5}'));

Use variant_get to extract a path and specify its result type:

SELECT
    id,
    variant_get(payload, '$.user.id', 'bigint') AS user_id,
    variant_get(payload, '$.city', 'string') AS city
FROM events
WHERE variant_get(payload, '$.active', 'boolean') = true;

Paths can address nested objects and arrays, for example $.user.id, $.items[0].sku, or $[0]. A missing path returns NULL. Conversion behavior for incompatible values is defined by the query engine and the Variant function being used.

With a JSON string, an equivalent query uses string-oriented JSON functions and normally parses the text while evaluating the query:

SELECT
    id,
    CAST(get_json_object(payload_json, '$.user.id') AS BIGINT) AS user_id
FROM json_events;

Storage Layout

Plain Variant

By default, a Variant column is stored as two binary fields inside the Parquet file:

payload (GROUP)
├── value      BYTE_ARRAY  -- encoded values and structure
└── metadata   BYTE_ARRAY  -- object-key dictionary and encoding metadata

The binary representation avoids storing and reparsing JSON syntax. It also retains the type of each scalar value. Extracting a sub-field from a plain Variant still requires reading the binary value field for that Variant column.

Shredded Variant

Shredding materializes selected Variant paths as typed Parquet sub-columns while keeping the logical table column as VARIANT. For example, shredding age and city produces a layout similar to:

payload (GROUP)
├── metadata   BYTE_ARRAY
├── value      BYTE_ARRAY OPTIONAL       -- fields that were not shredded
└── typed_value (GROUP) OPTIONAL
    ├── age (GROUP)
    │   ├── value        BYTE_ARRAY OPTIONAL
    │   └── typed_value  INT32 OPTIONAL
    └── city (GROUP)
        ├── value        BYTE_ARRAY OPTIONAL
        └── typed_value  BYTE_ARRAY OPTIONAL (STRING)

The typed_value fields are normal typed Parquet leaves. A reader that pushes Variant path extractions into the scan can read the required leaves instead of decoding the complete Variant value. This is most useful when a document is wide but queries repeatedly access a small set of paths.

Shredding is lossless:

• Fields absent from the shredding schema remain in the overflow value bytes.
• A value that does not match the configured shredded type also remains in value.
• Reading the complete Variant transparently reconstructs it from typed fields and overflow bytes.
• A table can contain both plain and shredded files. Changing the shredding configuration affects new files and does not rewrite existing files.

Shredding trades additional write work and physical columns for faster projected reads. It may not help workloads that usually select the entire Variant, access unpredictable paths, or write very sparse documents with few repeated fields.

Configure Shredding

Explicit Schema

Set variant.shreddingSchema to a JSON-encoded Paimon ROW type. Top-level fields identify Variant columns in the table; their nested types describe the paths to materialize.

CREATE TABLE user_events (
    id BIGINT,
    payload VARIANT
) USING paimon
TBLPROPERTIES (
    'file.format' = 'parquet',
    'variant.shreddingSchema' =
    '{"type":"ROW","fields":[{"name":"payload","type":{"type":"ROW","fields":[{"name":"user_id","type":"BIGINT"},{"name":"city","type":"STRING"}]}}]}'
);

This schema shreds $.user_id as BIGINT and $.city as STRING. Other fields remain available through the overflow bytes. The legacy key parquet.variant.shreddingSchema is accepted as an alias.

Shredded typed values support character and binary types, booleans, decimal and numeric types, and nested ROW and ARRAY types. A VARIANT branch in the shredding schema leaves that branch in its binary representation.

Use an explicit schema when the important query paths and their types are known. It gives files a predictable physical layout and avoids inference buffering.

Automatic Schema Inference

Paimon can infer a shredding schema independently for each output file:

CREATE TABLE inferred_events (
    id BIGINT,
    payload VARIANT
) USING paimon
TBLPROPERTIES (
    'file.format' = 'parquet',
    'variant.inferShreddingSchema' = 'true',
    'variant.shredding.maxInferBufferRow' = '4096',
    'variant.shredding.maxSchemaWidth' = '300',
    'variant.shredding.maxSchemaDepth' = '50',
    'variant.shredding.minFieldCardinalityRatio' = '0.1'
);

The writer buffers the initial rows of a file, infers their common shape, creates the physical schema, and then flushes those rows. Rare fields, fields beyond the configured width or depth, and fields with incompatible observed types remain unshredded.

Option	Default	Description
variant.inferShreddingSchema	false	Enables per-file shredding schema inference when no explicit schema is configured.
variant.shredding.maxInferBufferRow	4096	Maximum number of initial rows buffered for inference.
variant.shredding.maxSchemaWidth	300	Maximum number of fields in an inferred shredding schema.
variant.shredding.maxSchemaDepth	50	Maximum Variant traversal depth during inference.
variant.shredding.minFieldCardinalityRatio	0.1	Minimum fraction of sampled non-null Variant values that must contain a field before it is shredded.

Automatic inference is convenient for exploratory or rapidly evolving data. Because inference is per file, different files can have different shredded paths; readers use each file's physical schema transparently.

Query Pushdown

Shredding and query pushdown are separate capabilities. Shredding creates typed physical columns; the query engine must also push the requested Variant paths into the Paimon scan to avoid reading the full Variant.

Spark 4.1 and later can push variant_get extractions into Paimon when spark.sql.variant.pushVariantIntoScan is enabled:

SET spark.sql.variant.pushVariantIntoScan = true;

SELECT
    variant_get(payload, '$.user_id', 'bigint'),
    variant_get(payload, '$.city', 'string')
FROM user_events;

The SQL and result are the same for plain, shredded, and mixed-layout files. Shredded files provide the largest I/O benefit because the requested paths are available as independent Parquet leaves. Selecting the complete Variant, such as SELECT payload, requires reading and reconstructing the full value.

Limitations

• Variant data files must use Parquet. ORC and Avro are not supported.
• VARIANT cannot be used as a primary key or partition key.
• Variant extraction does not by itself create an index or guarantee predicate pushdown. Use shredding to reduce sub-column I/O; use an appropriate index or modeled table column when a path needs indexed filtering.