Runtime Configuration

Use SQL hints for options that apply to one statement, session settings for a job or session, and ALTER TABLE ... SET for persisted table options. See FlinkConnectorOptions for the full option reference.

Checkpointing

Streaming writes need checkpointing to commit data. Configure it before submitting the write job:

SET 'execution.runtime-mode' = 'streaming';
SET 'execution.checkpointing.interval' = '10 s';

A query reads committed snapshots. If newly written records are not visible, check that the writer has completed a checkpoint. Changing a SQL client setting affects subsequently submitted jobs; it does not reconfigure an already running job.

Use Flink Managed Memory

By default, each sink task manages its own heap memory for writer buffers. Enable managed memory to let Flink allocate a share of the slot's managed memory to the Paimon writer.

Option	Default	Meaning
sink.use-managed-memory-allocator	false	Use Flink managed memory for the writer's merge tree buffers.
sink.managed.writer-buffer-memory	256 MiB	Weight used by Flink to allocate managed memory to the writer. The actual allocation depends on the slot's managed memory and the operators sharing it.

The writer-buffer setting is a weight, not a fixed allocation of that many bytes.

INSERT INTO paimon_table /*+ OPTIONS(
    'sink.use-managed-memory-allocator' = 'true',
    'sink.managed.writer-buffer-memory' = '256 mb'
) */
SELECT * FROM source_table;

Managed Memory for an Explicit Writer Resource Group

When setting sink.writer-memory to configure the writer slot sharing group's task heap, use sink.writer-managed-memory to also request managed memory for that group. This optional budget must be positive and requires sink.writer-memory; sink.writer-cpu retains its default of one core unless explicitly configured. See Flink's fine-grained resource management for the deployment requirements and limitations of explicit slot sharing group resources.

INSERT INTO paimon_table /*+ OPTIONS(
    'sink.writer-memory' = '512 mb',
    'sink.writer-managed-memory' = '1 gb',
    'sink.use-managed-memory-allocator' = 'true',
    'sink.managed.writer-buffer-memory' = '256 mb'
) */
SELECT * FROM source_table;

The managed budget is a resource request for each shared slot of the writer group, not a total across all writer subtasks or a guaranteed allocation to the writer alone. Actual allocation depends on Flink's resource management mode, available resources, and managed memory consumers. The writer-buffer setting remains a weight: 256 mb in this example does not cap the writer's buffer at 256 MiB. Flink determines each writer subtask's share of the slot's managed memory.

Setting the budget does not enable the managed allocator. With the allocator disabled, writer buffers still use heap memory. The existing downstream slot sharing and explicit committer resource settings are unchanged. Leaving the new option unset preserves existing resource configuration, including default-slot managed allocation.

Setting dynamic options

Dynamic options tune a query or a subsequently submitted job without changing the options stored in the catalog.

Scope	Syntax	Effect
One statement	/*+ OPTIONS('key' = 'value') */ on the table	Applies to that table reference.
Session, all tables	SET 'key' = 'value'	Supplies a global dynamic option.
Session, selected tables	SET 'paimon.catalog.database.table.key' = 'value'	Overrides the corresponding global dynamic option for matching tables.

In a table-scoped key, the catalog, database, or table component can be *. Avoid setting conflicting values in overlapping table patterns. Reset session options after using them so they do not affect later queries.

-- Global default for queries in this session.
SET 'scan.timestamp-millis' = '1697018249001';

-- Override it for one table.
SET 'paimon.mycatalog.default.T.scan.timestamp-millis' = '1697018249000';
SELECT * FROM mycatalog.default.T;

RESET 'paimon.mycatalog.default.T.scan.timestamp-millis';
RESET 'scan.timestamp-millis';

-- Match default.T in any catalog.
SET 'paimon.*.default.T.scan.timestamp-millis' = '1697018249000';
SELECT * FROM mycatalog.default.T;
RESET 'paimon.*.default.T.scan.timestamp-millis';

For symptom-based checks, start with Troubleshooting.

Related Tuning

Concern	Guide
Source parallelism and split planning	SQL Query
Writer buffers, parallelism, and compaction	Write Performance
Missing lookup matches or a large dimension table	Lookup Joins
Progress retention and restarting readers	Consumer ID
Upgrading or restoring a writer	Savepoint