Bucketed Append

A bucketed append table distributes rows using a fixed number of buckets and a bucket key. Rows with the same bucket key in the same partition are routed to the same bucket. This supports bucket pruning, compatible bucketed joins, and ordered streaming reads within a bucket. It does not deduplicate rows or create a primary key.

Create a Bucketed Table

Use a positive bucket count and choose bucket-key columns that match your query or ordering requirements. The following examples assume a configured Paimon catalog and create a separate table named bucketed_table.

Flink

CREATE TABLE bucketed_table (
    product_id BIGINT,
    price DOUBLE,
    sales BIGINT
) WITH (
    'bucket' = '8',
    'bucket-key' = 'product_id'
);

Spark

CREATE TABLE bucketed_table (
    product_id BIGINT,
    price DOUBLE,
    sales BIGINT
) USING paimon
TBLPROPERTIES (
    'bucket' = '8',
    'bucket-key' = 'product_id'
);

Bucket count controls the distribution, while the bucket key determines which rows are colocated. A skewed key can concentrate work in a few buckets. More buckets can improve pruning for selective lookups, but can also create more small files for a small write workload.

Data Skipping

When a query contains equality (=) or IN predicates on the complete bucket-key, Paimon can compute the candidate buckets and skip files in the other buckets:

SELECT * FROM bucketed_table WHERE product_id = 12345;
SELECT * FROM bucketed_table WHERE product_id IN (1, 2, 3);

An equality lookup reads the matching bucket. An IN lookup may read several buckets, and different key values can map to the same bucket. Rows inside the selected buckets still need to satisfy the query predicate.

For a composite key such as bucket-key = product_id,region, the predicate must constrain both columns to finite equality or IN values. Filtering on only product_id, or using only a range predicate, does not identify a fixed set of buckets through this optimization.

Bucketed Join

Spark can use compatible bucket distributions to avoid a shuffle in a batch join. Enable V2 bucketing and join on the distribution keys:

SET spark.sql.sources.v2.bucketing.enabled = true;

CREATE TABLE fact_table (order_id INT, f1 STRING) USING paimon
TBLPROPERTIES ('bucket' = '10', 'bucket-key' = 'order_id');

CREATE TABLE dim_table (order_id INT, f2 STRING) USING paimon
TBLPROPERTIES ('bucket' = '10', 'primary-key' = 'order_id');

SELECT * FROM fact_table AS fact
JOIN dim_table AS dim ON fact.order_id = dim.order_id;

In this example, the fact table is an append table and the dimension table is a primary key table. They use the same bucket count and compatible distribution keys. Spark uses the partitioning reported by the Paimon source when planning the join; verify the resulting plan with EXPLAIN because the chosen join strategy also depends on Spark's optimizer.

Bucketed Streaming

With bucket-append-ordered = true (the default), a streaming reader preserves append order within the same partition and bucket. It provides neither an event-time sort nor a global order across buckets.

To use one queue within each partition, explicitly set bucket = 1. Omitting bucket selects the unaware-bucket layout, which has no such ordering guarantee.

Streaming Read Order

• Within one partition and bucket, earlier appended records are read before later appended records.
• Different buckets can be read by different tasks, so records from those buckets can interleave downstream.
• scan.plan-sort-partition = true sorts planned files by partition fields. This is useful when reading the initial snapshot of a partitioned table; it does not sort individual records by event time or make all downstream tasks emit in one global order.

Keep bucket-append-ordered = true when consumers rely on append order. To enable incremental clustering on a bucketed table, set it to false; clustering rewrites rows in clustering order and gives up that append-order guarantee.

For scan startup modes, watermarks, alignment, and bounded streaming reads, see Streaming.