Configuration

Use the scope that matches the setting you want to change:

Scope	How to set it	Example
Catalog connection	Startup --conf spark.sql.catalog.<name>.<key>=<value>	spark.sql.catalog.paimon.warehouse=file:/tmp/paimon
Default options for newly created tables	Catalog option table-default.<key>	spark.sql.catalog.paimon.table-default.file.format=parquet
Persistent table options	CREATE TABLE ... TBLPROPERTIES or ALTER TABLE ... SET TBLPROPERTIES	'bucket' = '4'
Session options	SET spark.paimon.<key>=<value>	spark.paimon.scan.snapshot-id=1
Table-specific session options	SET spark.paimon.<catalog>.<database>.<table>.<key>=<value>	spark.paimon.paimon.default.t.scan.snapshot-id=1
Individual reads or writes	DataFrame .option("<key>", "<value>") for options supported by that path	.option("scan.snapshot-id", "1")

Session and operation options are temporary; use table properties for persistent settings. Some connector options are session-only. Follow the scope in the option description and the relevant feature guide, such as Schema Evolution on Write.

Set and Reset Session Options

The SET command sets a property, returns the value of an existing property or returns all SQLConf properties with value and meaning. The RESET command resets runtime configurations specific to the current session which were set via the SET command to their default values.

To set dynamic options globally, you need add the spark.paimon. prefix. You can also set dynamic table options at this format: spark.paimon.${catalogName}.${dbName}.${tableName}.${config_key}. The catalogName/dbName/tableName can be *, which means matching all the specific parts. Dynamic table options will override global options if there are conflicts.

-- set spark conf
SET spark.sql.sources.partitionOverwriteMode=dynamic;

-- set paimon conf
SET spark.paimon.file.block-size=512M;

-- reset conf
RESET spark.paimon.file.block-size;

-- set scan.snapshot-id=1 for the table default.T in any catalogs
SET spark.paimon.*.default.T.scan.snapshot-id=1;
SELECT * FROM default.T;

-- set scan.snapshot-id=1 for the table T in any databases and catalogs
SET spark.paimon.*.*.T.scan.snapshot-id=1;
SELECT * FROM default.T;

-- set scan.snapshot-id=2 for the table default.T1 in any catalogs and scan.snapshot-id=1 on other tables
SET spark.paimon.scan.snapshot-id=1;
SET spark.paimon.*.default.T1.scan.snapshot-id=2;
SELECT * FROM default.T1 JOIN default.T2 ON xxxx;

Spark Catalog Options

Key	Default	Type	Description
catalog.create-underlying-session-catalog	false	Boolean	If true, create and use an underlying session catalog instead of default session catalog when use SparkGenericCatalog.
defaultDatabase	"default"	String	The default database name.
disable-create-table-in-default-db	false	Boolean	If true, creating table in default database is not allowed. Default is false.
v1Function.enabled	true	Boolean	Whether to enable v1 function.

Spark Connector Options

Streaming admission limits operate on whole splits. See Triggers and Read Limits for how byte, row, and file thresholds are applied.

Key	Default	Type	Description
format-table.repair.collect-statistics	false	Boolean	Whether MSCK REPAIR TABLE on a Format Table also measures the partitions it finds. Off by default: measuring lists the files inside every partition, not only the partition directories.
format-table.statistics.parallelism	8	Integer	How many requests MSCK REPAIR TABLE and ANALYZE TABLE use at once to measure Format Table partitions, so that a large table does not burst them at storage.
legacy-timestamp-mapping.enabled	false	Boolean	If true, map Paimon TIMESTAMP to Spark TIMESTAMP instead of TIMESTAMP_NTZ.
read.allow.fullScan	true	Boolean	Whether to allow full scan when reading a partitioned table.
read.changelog	false	Boolean	Whether to read row in the form of changelog (add rowkind column in row to represent its change type).
read.stream.maxBytesPerTrigger	(none)	Long	The maximum number of bytes returned in a single batch.
read.stream.maxFilesPerTrigger	(none)	Integer	The maximum number of files returned in a single batch.
read.stream.maxRowsPerTrigger	(none)	Long	The maximum number of rows returned in a single batch.
read.stream.maxTriggerDelayMs	(none)	Long	The maximum delay between two adjacent batches, which used to create MinRowsReadLimit with read.stream.minRowsPerTrigger together.
read.stream.minRowsPerTrigger	(none)	Long	The minimum number of rows returned in a single batch, which used to create MinRowsReadLimit with read.stream.maxTriggerDelayMs together.
requiredSparkConfsCheck.enabled	true	Boolean	Whether to verify SparkSession is initialized with required configurations.
source.split.target-size-with-column-pruning	false	Boolean	Whether to adjust the target split size based on pruned (projected) columns. If enabled, split size estimation uses only the columns actually being read.
vector-search.lateral-join.parallelism	16	Integer	Parallelism used to repartition a single-partition LIMIT input before executing a lateral vector search.
write.data-evolution.update-conflict-retry.max-attempts	20	Integer	Maximum attempts for Spark V1 UPDATE on data-evolution tables when concurrent partial-column updates conflict on the same row-id range and update columns. Values less than 2 disable retry.
write.data-evolution.update-conflict-retry.wait-ms	10	Long	Wait time in milliseconds between retry attempts for Spark V1 UPDATE on data-evolution tables after row-id range update conflicts.
write.hive-style-dynamic-partition.enabled	false	Boolean	If true, positional SQL inserts with explicit dynamic partitions use Hive's column order, with non-dynamic columns followed by dynamic partition columns. If false, the query output follows the table schema order.
write.merge-schema	false	Boolean	If true, evolve the table schema to accept new columns from the incoming data. Existing column types are preserved and incoming values are cast to them; to also widen existing types, enable 'write.merge-schema.type-widening'.
write.merge-schema.explicit-cast	false	Boolean	Only effective when 'write.merge-schema.type-widening' is true. If true, also allow lossy type changes between compatible types (e.g. BIGINT -> INT, STRING -> DATE).
write.merge-schema.type-widening	false	Boolean	Only effective when 'write.merge-schema' is true. If true, widen an existing column type when the incoming data has a wider compatible type (e.g. INT -> BIGINT, DECIMAL precision increase). Lossy changes are still rejected unless 'write.merge-schema.explicit-cast' is also true.
write.use-v2-write	false	Boolean	If true, v2 write will be used. Currently, only HASH_FIXED and BUCKET_UNAWARE bucket modes are supported. Will fall back to v1 write for other bucket modes. Currently, Spark V2 write does not support TableCapability.STREAMING_WRITE.

Shared Paimon Options

See CatalogOptions for catalog settings and CoreOptions for storage, scan, retention, and other table options.