This documentation is for an unreleased version of Apache Paimon. We recommend you use the latest stable version.

DLF Token #

DLF (Data Lake Formation) building is a fully-managed platform for unified metadata and data storage and management, aiming to provide customers with functions such as metadata management, storage management, permission management, storage analysis, and storage optimization.

DLF provides multiple authentication methods for different environments.

The 'warehouse' is your catalog instance name on the server, not the path.

Use the access key #

CREATE CATALOG `paimon-rest-catalog`
WITH (
    'type' = 'paimon',
    'uri' = '<catalog server url>',
    'metastore' = 'rest',
    'warehouse' = 'my_instance_name',
    'token.provider' = 'dlf',
    'dlf.access-key-id'='<access-key-id>',
    'dlf.access-key-secret'='<access-key-secret>',
);

You can grant specific permissions to a RAM user and use the RAM user’s access key for long-term access to your DLF resources. Compared to using the Alibaba Cloud account access key, accessing DLF resources with a RAM user access key is more secure.

Use the STS temporary access token #

Through the STS service, you can generate temporary access tokens for users, allowing them to access DLF resources restricted by policies within the validity period.

CREATE CATALOG `paimon-rest-catalog`
WITH (
    'type' = 'paimon',
    'uri' = '<catalog server url>',
    'metastore' = 'rest',
    'warehouse' = 'my_instance_name',
    'token.provider' = 'dlf',
    'dlf.access-key-id'='<access-key-id>',
    'dlf.access-key-secret'='<access-key-secret>',
    'dlf.security-token'='<security-token>'
);

In some environments, temporary access token can be periodically refreshed by using a local file:

CREATE CATALOG `paimon-rest-catalog`
WITH (
    'type' = 'paimon',
    'uri' = '<catalog server url>',
    'metastore' = 'rest',
    'warehouse' = 'my_instance_name',
    'token.provider' = 'dlf',
    'dlf.token-path' = 'my_token_path_in_disk'
);

Use the STS token from aliyun ecs role #

An instance RAM role refers to a RAM role granted to an ECS instance. This RAM role is a standard service role with the trusted entity being the cloud server. By using an instance RAM role, it is possible to obtain temporary access token (STS Token) within the ECS instance without configuring an AccessKey.

CREATE CATALOG `paimon-rest-catalog`
WITH (
    'type' = 'paimon',
    'uri' = '<catalog server url>',
    'metastore' = 'rest',
    'warehouse' = 'my_instance_name',
    'token.provider' = 'dlf',
    'dlf.token-loader' = 'ecs'
    -- optional, loader can obtain it through ecs metadata service
    -- 'dlf.token-ecs-role-name' = 'my_ecs_role_name'
);