Batch Writes

Write Arrow or pandas data with an explicit write-and-commit lifecycle. These examples assume you have a table from Catalogs and Tables.

Batch Write

A batch write has two steps: write data files, then publish them in a snapshot. You can write multiple Arrow or pandas batches with one writer. Call prepare_commit() once, pass its messages to the commit object, and close both objects. Create a new writer for another commit.

The following example shows alternative input formats. Replace the placeholder values with data matching your table schema; the quick start contains a complete executable example.

table = catalog.get_table('database_name.table_name')

# 1. Create table write and commit
write_builder = table.new_batch_write_builder()
table_write = write_builder.new_write()
table_commit = write_builder.new_commit()

# 2. Choose the method for your input format:
# 2.1 Write pandas.DataFrame
dataframe = ...
table_write.write_pandas(dataframe)

# 2.2 Write pyarrow.Table
pa_table = ...
table_write.write_arrow(pa_table)

# 2.3 Write pyarrow.RecordBatch
record_batch = ...
table_write.write_arrow_batch(record_batch)

# 3. Publish the written batches in one snapshot
commit_messages = table_write.prepare_commit()
table_commit.commit(commit_messages)

# 4. Close resources
table_write.close()
table_commit.close()

new_batch_write_builder() keeps postpone-bucket writes in bucket-postpone. Use new_postpone_fixed_bucket_write_builder() to write real buckets. New partitions are buffered until prepare_commit(); existing ones are incremental. This builder currently supports bucket-function.type=default only.

Commit Identity

Write builders generate a commit user by default. Use with_commit_user() before creating writers, updates or committers to share an application identity across distributed tasks or restore a stream application:

write_builder = table.new_stream_write_builder().with_commit_user('application-1')
table_write = write_builder.new_write()
table_commit = write_builder.new_commit()

Writers and their committer must use the same identity. Use different identities for independent applications. Batch and postpone fixed-bucket builders expose the same method. Changing a builder affects only operations created afterward.

Native Write and Commit

With the pypaimon-rust extension installed, set write.native.enabled=true to use Rust for Parquet/BLOB writes and commit.native.enabled=true to use Rust for REST snapshot publication. Batch and stream writers retain the same prepare-and-commit lifecycle. Native commits also support overwrite, truncate_table(), truncate_partitions() and commit callbacks. Overwrite and truncate record an OVERWRITE snapshot even when no files match, and remove index metadata for the replaced partitions. Physical files remain available to historical snapshots. Changelog-only commits also publish a snapshot. For Parquet tables, both paths support manifest.compression=null, deflate, snappy, zstd, bzip2 and xz, including manifests retained from earlier snapshots. Overwrite and truncate partition specs convert textual values to the declared partition type. None and the configured default partition name select NULL partitions; malformed values are rejected before publication.

Standard REST tables with query-auth.enabled=true can write and commit without requesting SELECT permission. The REST server still authorizes snapshot publication. Row filters and column masks continue to apply to explicit reads; internal sequence-number and HASH-index restoration reads manifest metadata. For query-authorized targets, Native row-ID updates, predicate updates/deletes, upserts, shard updates and MERGE require an unrestricted whole-table SELECT grant before reading target rows or evaluating assignments. A row filter, column mask or denied grant rejects the operation; ordinary writes still require no SELECT grant. MERGE table sources obtain their own grants independently from the target, including historical snapshots and branches. Query-authorized cross-partition index bootstrap remains unsupported.

Whole-partition predicate deletes can use Native on primary-key tables configured with input changelogs. Native upserts also accept fully NULL source key columns with a different physical type, including selected NULL dictionary values.

Native selection checks the formats used by the operation. An unused changelog-file.format does not prevent Native Parquet writes when no INPUT changelog is produced, including batch overwrite. Likewise, vector.file.format does not prevent Native updates on tables without VECTOR fields. These operations continue to write Parquet/BLOB files; an active dedicated format must still be supported by the Native path.

The same Native selection applies to direct construction of TableWrite, BatchTableWrite, StreamTableWrite and PostponeFixedBucketBatchTableWrite. Their public classes, constructor signatures and Arrow/Pandas batch overrides are preserved. Python execution is initialized only when that backend is selected.

write_ray() uses the shared Python coordinator and lets workers select Native writers. For postpone fixed-bucket writes, overwrite plans use the new input unless an explicit bucket plan was supplied. A complete primary-key with_write_type() leaves the writer unchanged; primary-key column subsets remain unsupported.

Append with_write_type() preserves the selected column order. Both the full table schema and the selected schema are accepted as Arrow input; only selected columns are written. Selected Arrow inputs can also carry partition, hash and configured row-kind columns used for routing and event handling. The selected columns must retain their declared order. Named Rows bind fields by name. Reads restore omitted partition columns from split metadata. Selected-schema inputs must include the routing columns they need; write_arrow(data, bucket=...) also accepts a bucket assigned upstream.

Configure with_write_type() before the first effective write. Empty batches and ignored events do not lock the selection. Row-to-Arrow encoding failures before execution starts also leave it configurable. After writing, the same ordered selection can be set again, but changing its columns or their order is rejected, including after prepare_commit() or idle writer cleanup. Create a new writer to use a different write type.

Changing with_blob_consumer() affects newly created partition/bucket writers. Active bucket writers retain their consumer across stream prepare_commit(identifier) calls. Unaware bucket writers retire on an idle prepare; fixed bucket writers also require a newer committed identifier from the same commit user. A reopened bucket uses the current consumer.

Batch prepare_commit() accepts no arguments and can be called once. Stream prepare_commit(identifier) requires an integer-like value in the signed 64-bit range. These call boundaries are the same for Python and Native writers.

Native commit selection requires a standard REST catalog and a reconstructible built-in FileIO. Other catalogs and custom Python FileIO implementations retain the Python path. Binary partition keys require partition.legacy-name=false for portable names across implementations.

To replace data, configure overwrite on the write builder before creating the writer and commit object:

# overwrite whole table
write_builder = table.new_batch_write_builder().overwrite()

# overwrite partition 'dt=2024-01-01'
write_builder = table.new_batch_write_builder().overwrite({'dt': '2024-01-01'})

Parquet Page Indexes

Python Parquet writers write page indexes by default on PyArrow >= 13; older versions omit them. Set parquet.write-page-index.enabled=false to disable them, or true to require them. Explicit true on PyArrow < 13 fails before creating files.

This affects new files only, independently of read-side filtering. When enabled, page-level statistics move from page headers into the index. Check reader compatibility when upgrading; PyArrow reads do not currently use these indexes.

Manifest Merging

Python commits retain existing manifest files without merging or rewriting them, including during commit retries. Run manifest compaction through a centralized maintenance service, for example using the Java engines' compact_manifest procedure.

The table options manifest.merge-min-count, manifest.merge.skip-on-write-only, and write-only do not enable manifest merging in Python. manifest.target-file-size still controls the size of newly written manifest files.

Commit Callback

You can register CommitCallback instances on a TableCommit to be notified after each successful snapshot commit. This is useful for post-commit actions such as syncing metadata to external systems.

Implementations must be idempotent — a callback may be invoked more than once for the same commit if a failure occurs right after the commit succeeds.

from pypaimon.write.commit_callback import CommitCallback, CommitCallbackContext

class MyCallback(CommitCallback):
    def call(self, context: CommitCallbackContext) -> None:
        print(f"Committed snapshot {context.snapshot.id}, "
              f"{len(context.commit_entries)} entries, "
              f"identifier {context.identifier}")

    def close(self) -> None:
        pass  # release resources if needed

write_builder = table.new_batch_write_builder()
table_write = write_builder.new_write()
table_commit = write_builder.new_commit()
table_commit.add_commit_callback(MyCallback())

table_write.write_arrow(data)
table_commit.commit(table_write.prepare_commit())
table_write.close()
table_commit.close()

CommitCallbackContext provides:

Field	Type	Description
snapshot	Snapshot	The committed snapshot (id, commit_kind, time_millis, next_row_id, …)
commit_entries	List[ManifestEntry]	Delta manifest entries in this commit (each carries file.first_row_id when row-tracking is enabled)
identifier	int	Commit identifier