Primary Key Tables

Paimon merges changes to primary key tables using its LSM storage. Iceberg readers use the files and deletion information published in Iceberg metadata, so the publication mode determines how soon they see an update.

Choose a Read Mode

Mode	Required options	Files eligible for incremental publication
Full compaction	Enable metadata.iceberg.storage	Highest-level LSM files
Deletion vectors	Enable storage, format v3, deletion vectors, and 64-bit bitmaps	Files above L0 plus deletion vectors

Initial publication and metadata rebuilds use snapshot splits that can be read directly without Paimon's merge logic. These may include files outside the incremental rules above. Use compaction to establish a predictable visibility boundary.

Use the full-compaction mode when the reader does not support Iceberg v3 deletion vectors. Use the deletion-vector mode when the reader supports it and you need to expose changes without waiting for full compaction to the highest level.

Full-Compaction Example

First prepare the Paimon and Iceberg catalogs for Flink or Spark. Create the following table through the Paimon catalog.

Flink SQL

CREATE TABLE paimon_catalog.`default`.orders (
    order_id BIGINT,
    status STRING,
    payment DOUBLE,
    PRIMARY KEY (order_id) NOT ENFORCED
) WITH (
    'metadata.iceberg.storage' = 'hadoop-catalog'
);

Spark SQL

CREATE TABLE paimon_catalog.`default`.orders (
    order_id BIGINT,
    status STRING,
    payment DOUBLE
) TBLPROPERTIES (
    'primary-key' = 'order_id',
    'metadata.iceberg.storage' = 'hadoop-catalog'
);

Run these statements in either engine. Explicit full compaction makes the visibility boundary clear without configuring a very short production compaction interval. Wait for the compaction job to finish before querying.

INSERT INTO paimon_catalog.`default`.orders VALUES
    (1, 'SUBMITTED', CAST(NULL AS DOUBLE)),
    (2, 'COMPLETED', 200.0),
    (3, 'SUBMITTED', CAST(NULL AS DOUBLE));

CALL paimon_catalog.sys.compact(`table` => 'default.orders', compact_strategy => 'full');

SELECT order_id, status, payment
FROM iceberg_catalog.`default`.orders
WHERE status = 'COMPLETED'
ORDER BY order_id;

order_id  status     payment
2         COMPLETED  200.0

Update order 1 through Paimon, compact again, and read the result:

INSERT INTO paimon_catalog.`default`.orders VALUES (1, 'COMPLETED', 100.0);

CALL paimon_catalog.sys.compact(`table` => 'default.orders', compact_strategy => 'full');

SELECT order_id, status, payment
FROM iceberg_catalog.`default`.orders
WHERE status = 'COMPLETED'
ORDER BY order_id;

order_id  status     payment
1         COMPLETED  100.0
2         COMPLETED  200.0

Timeliness

Without Iceberg deletion vectors, incremental publication selects only the highest LSM level. A completed Paimon write alone does not guarantee that Iceberg can see the change. Full compaction moves the merged result to that level.

For continuous workloads, choose a compaction schedule based on the required freshness and the cost of rewriting data:

Option	Purpose
compaction.optimization-interval	Trigger optimization compaction based on an elapsed interval
full-compaction.delta-commits	Trigger full compaction after a number of delta commits in Flink streaming writes

Neither option has a default interval/count. See dedicated compaction for scheduling and resource configuration. Reader caching and catalog publication add to the end-to-end delay.

Deletion Vector Support

Paimon can publish deletion vectors in the Iceberg v3 format. A deletion vector marks obsolete row positions so a compatible reader can filter them without merging all versions of a key.

Set all three options when creating the table:

'metadata.iceberg.format-version' = '3',
'deletion-vectors.enabled' = 'true',
'deletion-vectors.bitmap64' = 'true'

Use an Iceberg reader with v3 deletion-vector support. Iceberg introduced this support in 1.8.0; verify support in the actual engine connector as well. Iceberg 1.8.x requires JDK 11 or later.

Visibility still depends on publication

During incremental publication in this mode, Paimon selects primary key data files with a level greater than zero. L0 files are not included in those incremental updates. Changes must reach eligible files and their deletion vectors must be committed and published before a refreshed Iceberg reader sees them. This mode removes the requirement to compact all the way to the highest level; it does not guarantee immediate visibility of every write.

Example: Read an Updated Key

Use the Flink catalogs and batch settings from the catalog setup, with an Iceberg runtime that supports deletion vectors.

CREATE TABLE paimon_catalog.`default`.orders_dv (
    order_id BIGINT,
    status STRING,
    PRIMARY KEY (order_id) NOT ENFORCED
) WITH (
    'metadata.iceberg.storage' = 'hadoop-catalog',
    'metadata.iceberg.format-version' = '3',
    'deletion-vectors.enabled' = 'true',
    'deletion-vectors.bitmap64' = 'true'
);

INSERT INTO paimon_catalog.`default`.orders_dv VALUES
    (1, 'SUBMITTED'), (2, 'COMPLETED');

INSERT INTO paimon_catalog.`default`.orders_dv VALUES (1, 'COMPLETED');

SELECT order_id, status
FROM iceberg_catalog.`default`.orders_dv
ORDER BY order_id;

Once the writes and metadata publication complete, the expected rows are:

order_id  status
1         COMPLETED
2         COMPLETED

To inspect the deletion-vector indexes on the Paimon side:

SELECT * FROM paimon_catalog.`default`.`orders_dv$table_indexes`
WHERE index_type = 'DELETION_VECTORS';

Existing Tables with 32-Bit Deletion Vectors

Changing deletion-vectors.bitmap64 does not convert existing 32-bit index files. Before enabling Iceberg deletion-vector publication:

1. Stop all writers to the table.
2. Run a full compaction and wait for it to finish.
3. Set deletion-vectors.bitmap64 to true and metadata.iceberg.format-version to 3.
4. Restart the writers with the updated options and verify reads through the Iceberg connector.

For Flink SQL, step 3 is:

ALTER TABLE paimon_catalog.`default`.orders_dv SET (
    'deletion-vectors.bitmap64' = 'true',
    'metadata.iceberg.format-version' = '3'
);

Switching from format v2 to v3 rebuilds the Iceberg metadata on the next publication instead of reusing the v2 base. Verify the resulting snapshots and tags, and check every reader's v3 support before upgrading.