Skip to main content

Schema

A schema file defines the fields, keys, and options of a table at a particular schema ID. Readers use schema IDs in snapshots and data file metadata to interpret records written before or after a schema change.

Schema ID and Format Version​

These two numbers serve different purposes:

  • id identifies a table schema. IDs start at 0; an update creates a new schema ID.
  • version identifies the schema JSON format. The current format version is 3.

In the default layout, schema ID 0 is stored in schema/schema-0. A file named schema-3 does not imply format version 3.

JSON Fields​

FieldTypeMeaning
versionIntegerSchema JSON format version. See Compatibility.
idLongSchema ID, used in the file name and metadata references.
fieldsArray of DataFieldOrdered table fields, including stable field IDs.
highestFieldIdIntegerHighest field ID allocated, including nested fields; used when allocating IDs for new fields.
partitionKeysArray of stringsNames of the partition fields.
primaryKeysArray of stringsNames of the primary-key fields; empty for a table without a primary key.
optionsMap of strings to stringsTable options. Map entry order has no meaning.
commentString, optionalTable comment.
timeMillisLongSchema creation time in milliseconds since the Unix epoch.

Example​

This example is schema ID 0, serialized with format version 3:

{
"version" : 3,
"id" : 0,
"fields" : [ {
"id" : 0,
"name" : "order_id",
"type" : "BIGINT NOT NULL"
}, {
"id" : 1,
"name" : "order_name",
"type" : "STRING"
}, {
"id" : 2,
"name" : "order_user_id",
"type" : "BIGINT"
}, {
"id" : 3,
"name" : "order_shop_id",
"type" : "BIGINT"
} ],
"highestFieldId" : 3,
"partitionKeys" : [ ],
"primaryKeys" : [ "order_id" ],
"options" : {
"bucket" : "5"
},
"comment" : "",
"timeMillis" : 1720496663041
}

Compatibility​

Older schema files can omit options whose defaults have since changed. Readers preserve their original behavior when decoding those schemas:

Schema formatMissing field or optionReader behavior
No version fieldversionTreat as format version 1.
Version 1bucketSupply bucket = 1.
Versions 1 and 2file.formatSupply file.format = orc.
Older files without a timestamptimeMillisUse 0.

These compatibility defaults do not replace options explicitly stored in the schema. New tables use current defaults, including Parquet as the default data file format.

DataField​

A DataField describes one column, including fields nested inside a row type.

FieldTypeMeaning
idIntegerStable field identifier used for schema evolution.
nameStringField name.
typeString or JSON type objectLogical type, including nullability and nested type information.
descriptionString, optionalField comment.
defaultValueString, optionalStored default value definition. Engine support determines how it is used.
generationExpressionString, optionalSQL scalar expression generating a physical top-level column. Mutually exclusive with defaultValue.

Primitive types commonly use strings such as BIGINT NOT NULL. Nested types need their structured type representation. See Data Types for the logical type reference.

Generated Columns​

A top-level field, including a partition field, may define generationExpression, for example:

{
"id": 2,
"name": "dt",
"type": "DATE",
"generationExpression": "CAST(event_time AS DATE)"
}

Writers compile expressions for omitted columns into Paimon Transform operations once at initialization. These operations define the automatically generated value, including CAST formatting and invalid-input behavior; the write engine's ANSI mode or implicit CAST rules do not change that definition. Spark and Flink 1.18+ SQL inserts with an explicit target column list fill omitted generated columns through the same evaluator, including source-column defaults, before writing. Flink automatically fills only nullable generated columns outside the primary key. See Flink SQL Write for details. Expressions may reference ordinary top-level fields and compose supported scalar calls, such as YEAR(event_time), UPPER(name), CONCAT(UPPER(name), '-', name), and DATE_ADD(CAST(event_time AS DATE), days). The expression result type must match the declared column type. Nested field references, references to other generated columns, arbitrary SQL, UDFs, and nondeterministic functions are not supported. TIMESTAMP WITH LOCAL TIME ZONE inputs and cast targets are rejected because their results can depend on a writer's default time zone.

Generated columns are physical fields; readers use the stored values without evaluating the expression. A cast from TIMESTAMP WITHOUT TIME ZONE to DATE uses the timestamp's calendar date, independently of the writer's default time zone.

This is a write-time column-filling feature, not a strict computed-column constraint. Adapted writers evaluate an expression only when the input schema omits its column. If the input schema includes the column, its supplied value is preserved without evaluating or checking the expression, including an explicit NULL. Normal type, NOT NULL, primary-key, and partition constraints still apply. Stored values may therefore differ from the expression result; readers and optimizers must not assume that the expression is an invariant or infer partition filters from it. Updating a source column does not recompute a generated column that is already supplied in the write row.

Definitions are supported at table creation. Changing a definition or renaming, dropping, or changing the type of any source is not supported. A generated column cannot be used as rowkind.field.

All primary-key merge engines are supported. Column filling happens before merging; the merge engine applies its normal rules to generated columns without recomputing expressions after a merge. For example, a partial-update merge can retain an existing generated value when its source changes. Data Evolution partial-column updates write only the selected columns. Unselected generated columns keep their existing values, and selected generated columns accept the supplied values without requiring their expression's source columns.

Use clients supporting this attribute for writes; older clients may ignore it. Its presence does not establish compatibility with older writers. See Spark DDL for SQL examples.

Update Schema​

A schema update writes a new schema file rather than replacing the old definition:

my_table/
└── schema/
├── schema-0
├── schema-1
└── schema-2

The latest schema defines the current table, while existing snapshots and data files can still reference earlier schema IDs. Retain schema files that are needed to interpret existing data. Use the supported Flink schema changes or Spark schema changes instead of editing schema JSON directly.