This documentation is for an unreleased version of Apache Paimon. We recommend you use the latest stable version.

Catalog #

Paimon provides a Catalog abstraction to manage the table of contents and metadata. The Catalog abstraction provides a series of ways to help you better integrate with computing engines. We always recommend that you use Catalog to access the Paimon table.

Catalogs #

Paimon catalogs currently support three types of metastores:

filesystem metastore (default), which stores both metadata and table files in filesystems.
hive metastore, which additionally stores metadata in Hive metastore. Users can directly access the tables from Hive.
jdbc metastore, which additionally stores metadata in relational databases such as MySQL, Postgres, etc.

Filesystem Catalog #

Metadata and table files are stored under hdfs:///path/to/warehouse.

-- Flink SQL
CREATE CATALOG my_catalog WITH (
    'type' = 'paimon',
    'warehouse' = 'hdfs:///path/to/warehouse'
);

Hive Catalog #

By using Paimon Hive catalog, changes to the catalog will directly affect the corresponding Hive metastore. Tables created in such catalog can also be accessed directly from Hive. Metadata and table files are stored under hdfs:///path/to/warehouse. In addition, schema is also stored in Hive metastore.

-- Flink SQL
CREATE CATALOG my_hive WITH (
    'type' = 'paimon',
    'metastore' = 'hive',
    -- 'warehouse' = 'hdfs:///path/to/warehouse', default use 'hive.metastore.warehouse.dir' in HiveConf
);

By default, Paimon does not synchronize newly created partitions into Hive metastore. Users will see an unpartitioned table in Hive. Partition push-down will be carried out by filter push-down instead.

If you want to see a partitioned table in Hive and also synchronize newly created partitions into Hive metastore, please set the table option metastore.partitioned-table to true.

JDBC Catalog #

By using the Paimon JDBC catalog, changes to the catalog will be directly stored in relational databases such as SQLite, MySQL, postgres, etc.

-- Flink SQL
CREATE CATALOG my_jdbc WITH (
    'type' = 'paimon',
    'metastore' = 'jdbc',
    'uri' = 'jdbc:mysql://<host>:<port>/<databaseName>',
    'jdbc.user' = '...', 
    'jdbc.password' = '...', 
    'catalog-key'='jdbc',
    'warehouse' = 'hdfs:///path/to/warehouse'
);