Format Table#

Table#

class FormatTable#

A table that is a directory of data files of one format, laid out like a standard Hive table.

It carries no snapshots and no manifests: the files under its location are the table, and a partitioned table’s partitions are the key=value directories below that location, or the bare-value ones under format-table.partition-path-only-value. A table is a format table when its type option is format-table; file.format then names the format of every file in it, defaulting to parquet.

Writes only insert - there is nowhere to record an update or a delete - so a read fills _VALUE_KIND with inserts throughout.

Reading and writing go through the entry points every other table uses: TableScan, TableRead, FileStoreWrite and FileStoreCommit. Each recognises a format table from the schema under the table path, or takes one already loaded through the FormatTable constructor every context builder has, which is how a table whose schema lives in a metastore is reached.

See docs/source/user_guide/format_table.rst for what is not supported yet.

Public Types

enum class Format#

Formats a format table’s files can be in.

Values:

enumerator PARQUET#
enumerator ORC#

Public Functions

~FormatTable()#
inline const std::string &Location() const#

Directory the data files live in.

inline Format GetFormat() const#

Format of every data file in the directory.

const std::vector<std::string> &PartitionKeys() const#

Fields the table is partitioned by, in the order their directories nest.

inline const std::string &FileCompression() const#

Compression new data files are written with.

It is resolved from file.compression, then format-table.file.compression, then the bare compression key an engine’s own writer reads, then what the table’s format writes by default.

inline const std::string &PartitionDefaultName() const#

Directory name standing for a null partition value, from partition.default-name.

inline bool PartitionOnlyValueInPath() const#

Whether a partition directory is named by its value alone (2025/01/) instead of key=value (year=2025/month=01/), from format-table.partition-path-only-value.

The value-only layout carries no field names, so the nesting order of the table’s partition keys alone says which key a directory belongs to.

inline const std::map<std::string, std::string> &Options() const#

Table options: the ones stored in the schema, with any given at the call on top.

inline std::string Name() const#

A name to identify this table.

std::string FullName() const#

Full name of the table, database.tableName.

inline std::shared_ptr<DataSchema> LatestSchema() const#

Schema of the table, including its partition fields.

Result<std::unique_ptr<::ArrowSchema>> GetArrowSchema() const#

Schema of the table as an arrow schema, including its partition fields.

inline std::shared_ptr<FileSystem> GetFileSystem() const#

File system holding the table directory.

inline bool LocationCarriesPaimonMetadata() const#

Whether this table’s own metadata lives under its location, as told by whoever loaded it.

Only then are the schema and branch directories below the location table metadata rather than table content. For a table whose schema lives in a metastore they are data, and are read and written like any other directory.

Public Static Functions

static Result<Format> ParseFormat(const std::string &file_format)#

Parses the file.format option, case-insensitively.

A format this library has no reader for is rejected by name, instead of failing later with a missing-format-factory error.

static std::string FormatToString(Format format)#

The identifier of a format, as it appears in file.format and as a file extension.

static Result<std::shared_ptr<FormatTable>> Create(const std::shared_ptr<FileSystem> &file_system, const std::string &table_path, const Identifier &identifier, const std::map<std::string, std::string> &dynamic_options)#

Loads a format table from its directory, reading the schema stored under it.

This needs a schema file under the table directory, which a table created through SchemaManager or a file system catalog has. A table whose schema lives in a metastore has none, and is loaded through Catalog::GetFormatTable() instead.

Parameters:

  • file_system – File system holding the table directory.

  • table_path – Root path of the table, which is also its data location.

  • identifier – Logical table identifier, used for naming and error messages.

  • dynamic_options – Options given at the call, which win over the ones stored in the schema. Empty when the caller has none.

Returns:

A result containing the format table, or an error status.

static Result<std::shared_ptr<FormatTable>> Create(const std::shared_ptr<FileSystem> &file_system, const std::string &location, const Identifier &identifier, const std::shared_ptr<DataSchema> &schema, bool location_carries_paimon_metadata, const std::map<std::string, std::string> &dynamic_options)#

Builds a format table from a schema that is already loaded, for a caller that has one in hand, such as a catalog that just created the table.

Parameters:

  • location – Directory the data files live in. It may not be empty: every path this table reads or writes is checked against it, and an empty one is a prefix of nothing. A trailing separator names the same directory as none.

  • location_carries_paimon_metadata – See LocationCarriesPaimonMetadata(). Only the caller knows: a file system catalog puts metadata there, a REST or Hive catalog keeps it in the metastore.

  • dynamic_options – Options given at the call, which win over the ones stored in the schema. Empty when the caller has none.

Returns:

A result containing the format table, or an error status.

static Result<std::shared_ptr<FormatTable>> Copy(const std::shared_ptr<FormatTable> &table, const std::map<std::string, std::string> &dynamic_options)#

Copies table with dynamic_options on top of the options it already carries, which is the precedence every context builder promises for a table it was handed rather than loaded itself.

table comes back as it is when there is nothing to add.

type is not overridable: it is structural and is read from the schema alone.

Parameters:

  • table – The table to copy.

  • dynamic_options – Options given at the call.

Returns:

A result containing the copied table, or an error status.

Reading and writing a format table go through the generic entry points every other table uses: paimon::TableScan, paimon::TableRead, paimon::FileStoreWrite and paimon::FileStoreCommit. See Format Table.

Contexts#

Each of those entry points builds its context from a table path, and the schema under that path says what kind of table it is. A caller that already holds a FormatTable hands it over instead, through the constructor each context builder has for one:

This is the only way to reach a format table whose schema does not live under its own location, such as one a REST catalog serves: nothing under the location says that it is a format table, nor that what sits below it is data rather than metadata. The table knows both, so a setting that would answer either question again - SetTableSchema(), WithFileSystem() and a branch among them - is refused by these builders rather than quietly ignored. Options given at the call still win over the ones the table carries, as they do everywhere else.