Catalog#

Interface#

class Catalog#

This interface is responsible for reading and writing metadata such as database/table from a paimon catalog.

Public Functions

virtual Status CreateDatabase(const std::string &name, const std::map<std::string, std::string> &options, bool ignore_if_exists) = 0#

Creates a database with the specified properties.

Parameters:

  • name – Name of the database to be created.

  • options – Additional properties associated with the database.

  • ignore_if_exists – If true, no action is taken if the database already exists. If false, an error status is returned if the database exists.

Returns:

A status indicating success or failure.

virtual Status CreateTable(const Identifier &identifier, ArrowSchema *c_schema, const std::vector<std::string> &partition_keys, const std::vector<std::string> &primary_keys, const std::map<std::string, std::string> &options, bool ignore_if_exists) = 0#

Creates a new table in the catalog.

Note

System tables cannot be created using this method.

Parameters:

  • identifier – Identifier of the table to be created.

  • c_schema – The schema of the table to be created.

  • partition_keys – List of columns that should be used as partition keys for the table.

  • primary_keys – List of columns that should be used as primary keys for the table.

  • options – Additional table-specific options.

  • ignore_if_exists – If true, no action is taken if the table already exists. If false, an error status is returned if the table exists.

Returns:

A status indicating success or failure.

virtual Result<std::vector<std::string>> ListDatabases() const = 0#

Lists all the databases available in the catalog.

Returns:

A result containing a vector of database names, or an error status.

virtual Result<std::vector<std::string>> ListTables(const std::string &db_name) const = 0#

Lists all the tables within a specified database.

Note

System tables will not be listed.

Parameters:

db_name – The name of the database to list tables from.

Returns:

A result containing a vector of table names in the specified database, or an error status.

virtual Status DropDatabase(const std::string &name, bool ignore_if_not_exists, bool cascade) = 0#

Drops a database.

Parameters:

  • name – Name of the database to be dropped.

  • ignore_if_not_exists – If true, no action is taken if the database does not exist.

  • cascade – If true, drops all tables and functions in the database before dropping the database.

Returns:

A status indicating success or failure.

virtual Status DropTable(const Identifier &identifier, bool ignore_if_not_exists) = 0#

Drops a table.

Parameters:

  • identifier – Identifier of the table to drop.

  • ignore_if_not_exists – If true, no action is taken if the table does not exist.

Returns:

A status indicating success or failure.

virtual Status RenameTable(const Identifier &from_table, const Identifier &to_table, bool ignore_if_not_exists) = 0#

Renames a table.

Parameters:

  • from_table – Current identifier of the table.

  • to_table – New identifier for the table.

  • ignore_if_not_exists – If true, no action is taken if the table does not exist.

Returns:

A status indicating success or failure.

virtual Result<std::shared_ptr<Table>> GetTable(const Identifier &identifier) const = 0#

Gets a table.

Parameters:

identifier – Identifier of the table to get.

Returns:

A result containing the table, or an error status.

virtual Result<bool> DatabaseExists(const std::string &db_name) const = 0#

Checks whether a database with the specified name exists in the catalog.

Parameters:

db_name – The name of the database to check for existence.

Returns:

A result containing true if the database exists, false otherwise, or an error status.

virtual Result<bool> TableExists(const Identifier &identifier) const = 0#

Checks whether a table with the specified identifier exists in the catalog.

Parameters:

identifier – The identifier of the table to check for existence.

Returns:

A result containing true if the table exists, false otherwise, or an error status.

virtual Result<std::string> GetDatabaseLocation(const std::string &db_name) const = 0#

Returns the expected location of a specified database.

Parameters:

db_name – The name of the database to get the location for.

Returns:

A result containing the expected location of the database, or an error status on failure. An implementation that builds the location from the warehouse path, such as the file system catalog, answers without checking whether the database exists. One that resolves the location on a server, such as the REST catalog, propagates the server’s error and so fails for a database that does not exist.

virtual Result<std::string> GetTableLocation(const Identifier &identifier) const = 0#

Returns the expected location of a specified table.

Parameters:

identifier – The table identifier containing database and table name.

Returns:

A result containing the expected location of the table, or an error status on failure. Whether a missing table is an error depends on the implementation, in the same way as for GetDatabaseLocation.

virtual std::string GetRootPath() const = 0#

Returns the root path of the catalog.

Returns:

A string representing the root path of the catalog.

virtual std::shared_ptr<FileSystem> GetFileSystem() const = 0#

Returns the file system used by the catalog.

Returns:

A shared pointer to the file system instance.

inline virtual Result<std::shared_ptr<FileSystem>> GetTableFileSystem(const Identifier &identifier) const#

Returns the file system used to access the data of a specified table.

Note

A catalog that hands out per-table temporary credentials returns a file system that refreshes them, so the returned instance must be used for the table it was requested for. Pass it to ReadContextBuilder::WithFileSystem, ScanContextBuilder::WithFileSystem or WriteContextBuilder::WithFileSystem, or let a builder request it for you through ReadContextBuilder::WithCatalog, ScanContextBuilder::WithCatalog, WriteContextBuilder::WithCatalog or CommitContextBuilder::WithCatalog.

Parameters:

identifier – The identifier (database and table name) of the table.

Returns:

A shared pointer to the file system instance; the catalog-level file system by default.

virtual const std::map<std::string, std::string> &GetOptions() const = 0#

Returns the catalog-level options that were passed during catalog creation.

Returns:

A const reference to the map of catalog options (key-value pairs).

virtual Result<std::shared_ptr<Schema>> LoadTableSchema(const Identifier &identifier) const = 0#

Loads the latest schema of a specified table.

Note

System tables will not be supported.

Parameters:

identifier – The identifier (database and table name) of the table to load.

Returns:

A result containing table schema if the table exists, or an error status on failure.

virtual Result<std::vector<SnapshotInfo>> ListSnapshots(const Identifier &identifier, const std::string &branch = "") const = 0#

Lists all snapshots of the specified table, ordered by snapshot id.

Parameters:

  • identifier – The identifier (database and table name) of the table.

  • branch – Branch name; empty string means the main branch.

Returns:

A result containing a vector of SnapshotInfo ordered by snapshot id ascending, or an error status.

virtual bool SupportsVersionManagement() const#

Returns whether this catalog publishes snapshots through catalog commits.

Defaults to false.

Note

Implementations returning true must also implement the internal VersionManagedCatalog.

Result<std::shared_ptr<FormatTable>> GetFormatTable(const Identifier &identifier) const#

Gets a format table: a directory of data files laid out like a standard Hive table.

A format table carries no snapshots and no manifests, so it is loaded through its own method rather than GetTable(). Reading and writing it go through the same TableScan, TableRead, FileStoreWrite and FileStoreCommit entry points every other table uses: hand the table back to ScanContextBuilder, ReadContextBuilder, WriteContextBuilder or CommitContextBuilder, each of which takes one.

Parameters:

identifier – Identifier of the table to get.

Returns:

A result containing the format table, or an error status if the table does not exist or its type option is not format-table.

Public Static Functions

static Result<std::unique_ptr<Catalog>> Create(const std::string &root_path, const std::map<std::string, std::string> &options, const std::shared_ptr<FileSystem> &file_system = nullptr, const std::map<std::string, std::string> &fs_scheme_to_identifier_map = {})#

Factory method for creating a Catalog instance.

Parameters:

  • root_path – Path to the root directory where the catalog is located. For the REST catalog (CatalogOptions::METASTORE set to “rest”) this is instead the warehouse (instance) name registered on the server.

  • options – Configuration options for catalog initialization.

  • file_system – Specifies the file system for file operations. If not set, use default file system (configured in Options::FILE_SYSTEM)

  • fs_scheme_to_identifier_map – Maps a URI scheme (like “oss”) to the registered file system identifier that serves it, so a catalog that resolves several schemes, including the file systems it builds from per-table data tokens, keeps routing each scheme to its backend. Ignored when file_system is supplied, which serves every scheme itself.

Returns:

A result containing a unique pointer to a Catalog instance, or an error status.

class Identifier#

An identifier for a table containing database and table name.

Public Functions

explicit Identifier(const std::string &table)#
Identifier(const std::string &database, const std::string &table)#
Identifier(const std::string &database, const std::string &table, const std::string &branch)#

Names branch of table: every branch but the main one is addressed by the object name this builds, which joins them as tbl$branch_dev.

The branch is matched against the main branch ignoring case, as the Java client matches it, so MAIN names the main branch here rather than a branch of its own. A branch of only whitespace names it too, which the Java client does not do: this library reads a blank branch as the main branch everywhere.

Parameters:

  • database – Database holding the table.

  • table – Data table name, carrying no branch or system table suffix.

  • branch – Branch of table; any spelling of main, an empty and a blank name all name the main branch.

bool operator==(const Identifier &other) const#
const std::string &GetDatabaseName() const#
const std::string &GetTableName() const#
Result<std::string> GetDataTableName() const#
Result<std::optional<std::string>> GetBranchName() const#
Result<std::string> GetBranchNameOrDefault() const#
Result<std::optional<std::string>> GetSystemTableName() const#
Result<bool> IsSystemTable() const#
std::string GetFullName() const#
std::string ToString() const#
int32_t HashCode() const#

Public Static Functions

static Result<Identifier> FromString(const std::string &full_name)#

Public Static Attributes

static const char kUnknownDatabase[]#
static const char kSystemTableSplitter[]#
static const char kSystemBranchPrefix[]#
static const char kDefaultMainBranch[]#