org.apache.paimon.catalog

Class AbstractCatalog

java.lang.Object

org.apache.paimon.catalog.AbstractCatalog

All Implemented Interfaces:

AutoCloseable, Catalog

Direct Known Subclasses:

FileSystemCatalog, HiveCatalog, JdbcCatalog

public abstract class AbstractCatalog
extends Object
implements Catalog

Common implementation of Catalog.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	AbstractCatalog.TableMeta Table metadata.

Nested classes/interfaces inherited from interface org.apache.paimon.catalog.Catalog

Catalog.ColumnAlreadyExistException, Catalog.ColumnNotExistException, Catalog.DatabaseAlreadyExistException, Catalog.DatabaseNotEmptyException, Catalog.DatabaseNotExistException, Catalog.Loader, Catalog.PartitionNotExistException, Catalog.ProcessSystemDatabaseException, Catalog.TableAlreadyExistException, Catalog.TableNotExistException, Catalog.ViewAlreadyExistException, Catalog.ViewNotExistException

Field Summary

Fields

Modifier and Type	Field and Description
protected Options	catalogOptions
protected FileIO	fileIO
protected LineageMetaFactory	lineageMetaFactory
protected Map<String,String>	tableDefaultOptions

Fields inherited from interface org.apache.paimon.catalog.Catalog

COMMENT_PROP, DB_LOCATION_PROP, DB_SUFFIX, DEFAULT_DATABASE, HIVE_LAST_UPDATE_TIME_PROP, LAST_UPDATE_TIME_PROP, NUM_FILES_PROP, NUM_ROWS_PROP, OWNER_PROP, SYSTEM_BRANCH_PREFIX, SYSTEM_DATABASE_NAME, SYSTEM_TABLE_SPLITTER, TABLE_DEFAULT_OPTION_PREFIX, TOTAL_SIZE_PROP

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	AbstractCatalog(FileIO fileIO)
protected	AbstractCatalog(FileIO fileIO, Options options)

Method Summary

Modifier and Type	Method and Description
protected boolean	allowCustomTablePath()
boolean	allowUpperCase() Return a boolean that indicates whether this catalog allow upper case.
Map<String,Map<String,Path>>	allTablePaths()
void	alterTable(Identifier identifier, List<SchemaChange> changes, boolean ignoreIfNotExists) Modify an existing table from SchemaChanges.
protected abstract void	alterTableImpl(Identifier identifier, List<SchemaChange> changes)
protected void	assertMainBranch(Identifier identifier)
protected static void	checkNotBranch(Identifier identifier, String method)
protected void	checkNotSystemDatabase(String database) Validate database cannot be a system database.
protected static void	checkNotSystemTable(Identifier identifier, String method)
void	createDatabase(String name, boolean ignoreIfExists, Map<String,String> properties) Create a database with properties.
protected abstract void	createDatabaseImpl(String name, Map<String,String> properties)
void	createFormatTable(Identifier identifier, Schema schema) Create a FormatTable identified by the given Identifier.
void	createPartition(Identifier identifier, Map<String,String> partitionSpec) Create the partition of the specify table.
void	createTable(Identifier identifier, Schema schema, boolean ignoreIfExists) Create a new table.
protected abstract void	createTableImpl(Identifier identifier, Schema schema)
Optional<CatalogLockFactory>	defaultLockFactory()
void	dropDatabase(String name, boolean ignoreIfNotExists, boolean cascade) Drop a database.
protected abstract void	dropDatabaseImpl(String name)
void	dropPartition(Identifier identifier, Map<String,String> partitionSpec) Drop the partition of the specify table.
void	dropTable(Identifier identifier, boolean ignoreIfNotExists) Drop a table.
protected abstract void	dropTableImpl(Identifier identifier)
FileIO	fileIO()
Database	getDatabase(String name) Return a Database identified by the given name.
protected abstract Database	getDatabaseImpl(String name)
protected Table	getDataOrFormatTable(Identifier identifier)
protected AbstractCatalog.TableMeta	getDataTableMeta(Identifier identifier)
protected abstract TableSchema	getDataTableSchema(Identifier identifier)
Table	getTable(Identifier identifier) Return a Table identified by the given Identifier.
Path	getTableLocation(Identifier identifier) Get the table location in this catalog.
static boolean	isSpecifiedSystemTable(Identifier identifier)
static boolean	isSystemDatabase(String database)
protected static boolean	isTableInSystemDatabase(Identifier identifier)
protected List<String>	listDatabasesInFileSystem(Path warehouse)
List<PartitionEntry>	listPartitions(Identifier identifier) Get PartitionEntry of all partitions of the table.
List<String>	listTables(String databaseName) Get names of all tables under this database.
protected abstract List<String>	listTablesImpl(String databaseName)
protected List<String>	listTablesInFileSystem(Path databasePath)
Optional<CatalogLockContext>	lockContext()
protected boolean	lockEnabled()
Optional<CatalogLockFactory>	lockFactory()
Optional<MetastoreClient.Factory>	metastoreClientFactory(Identifier identifier, TableSchema schema) Get metastore client factory for the table specified by identifier.
Path	newDatabasePath(String database) Get warehouse path for specified database.
static Path	newDatabasePath(String warehouse, String database)
static Path	newTableLocation(String warehouse, Identifier identifier)
protected FileIO	objectFileIO(String objectLocation) Catalog implementation may override this method to provide FileIO to object table.
Map<String,String>	options() Catalog options.
void	renameTable(Identifier fromTable, Identifier toTable, boolean ignoreIfNotExists) Rename a table.
protected abstract void	renameTableImpl(Identifier fromTable, Identifier toTable)
protected boolean	tableExistsInFileSystem(Path tablePath, String branchName)
Optional<TableSchema>	tableSchemaInFileSystem(Path tablePath, String branchName)
protected void	validateFieldNameCaseInsensitive(List<String> fieldNames)
protected void	validateIdentifierNameCaseInsensitive(Identifier identifier)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.paimon.catalog.Catalog

alterTable, createDatabase, createView, dropView, getView, invalidateTable, listDatabases, listViews, renameView, repairCatalog, repairDatabase, repairTable, tableDefaultOptions, validateCaseInsensitive, validateCaseInsensitive, warehouse

Methods inherited from interface java.lang.AutoCloseable

close

Field Detail

fileIO

protected final FileIO fileIO

tableDefaultOptions

protected final Map<String,String> tableDefaultOptions

catalogOptions

protected final Options catalogOptions

lineageMetaFactory

@Nullable
protected final LineageMetaFactory lineageMetaFactory

Constructor Detail

AbstractCatalog

protected AbstractCatalog(FileIO fileIO)

AbstractCatalog

protected AbstractCatalog(FileIO fileIO,
                          Options options)

Method Detail

options

public Map<String,String> options()

Description copied from interface: Catalog

Catalog options.

Specified by:

options in interface Catalog

fileIO

public FileIO fileIO()

Specified by:

fileIO in interface Catalog

lockFactory

public Optional<CatalogLockFactory> lockFactory()

defaultLockFactory

public Optional<CatalogLockFactory> defaultLockFactory()

lockContext

public Optional<CatalogLockContext> lockContext()

lockEnabled

protected boolean lockEnabled()

allowUpperCase

public boolean allowUpperCase()

Description copied from interface: Catalog

Return a boolean that indicates whether this catalog allow upper case.

Specified by:

allowUpperCase in interface Catalog

allowCustomTablePath

protected boolean allowCustomTablePath()

createDatabase

public void createDatabase(String name,
                           boolean ignoreIfExists,
                           Map<String,String> properties)
                    throws Catalog.DatabaseAlreadyExistException

Description copied from interface: Catalog

Create a database with properties.

Specified by:

createDatabase in interface Catalog

Parameters:

name - Name of the database to be created

ignoreIfExists - Flag to specify behavior when a database with the given name already exists: if set to false, throw a DatabaseAlreadyExistException, if set to true, do nothing.

properties - properties to be associated with the database

Throws:

Catalog.DatabaseAlreadyExistException - if the given database already exists and ignoreIfExists is false

getDatabase

public Database getDatabase(String name)
                     throws Catalog.DatabaseNotExistException

Description copied from interface: Catalog

Return a Database identified by the given name.

Specified by:

getDatabase in interface Catalog

Parameters:

name - Database name

Returns:

The requested Database

Throws:

Catalog.DatabaseNotExistException - if the requested database does not exist

getDatabaseImpl

protected abstract Database getDatabaseImpl(String name)
                                     throws Catalog.DatabaseNotExistException

Throws:

Catalog.DatabaseNotExistException

createPartition

public void createPartition(Identifier identifier,
                            Map<String,String> partitionSpec)
                     throws Catalog.TableNotExistException

Description copied from interface: Catalog

Create the partition of the specify table.

Only catalog with metastore can support this method, and only table with 'metastore.partitioned-table' can support this method.

Specified by:

createPartition in interface Catalog

Parameters:

identifier - path of the table to drop partition

partitionSpec - the partition to be created

Throws:

Catalog.TableNotExistException - if the table does not exist

dropPartition

public void dropPartition(Identifier identifier,
                          Map<String,String> partitionSpec)
                   throws Catalog.TableNotExistException

Description copied from interface: Catalog

Drop the partition of the specify table.

Specified by:

dropPartition in interface Catalog

Parameters:

identifier - path of the table to drop partition

partitionSpec - the partition to be deleted

Throws:

Catalog.TableNotExistException - if the table does not exist

listPartitions

public List<PartitionEntry> listPartitions(Identifier identifier)
                                    throws Catalog.TableNotExistException

Description copied from interface: Catalog

Get PartitionEntry of all partitions of the table.

Specified by:

listPartitions in interface Catalog

Parameters:

identifier - path of the table to list partitions

Throws:

Catalog.TableNotExistException - if the table does not exist

createDatabaseImpl

protected abstract void createDatabaseImpl(String name,
                                           Map<String,String> properties)

dropDatabase

public void dropDatabase(String name,
                         boolean ignoreIfNotExists,
                         boolean cascade)
                  throws Catalog.DatabaseNotExistException,
                         Catalog.DatabaseNotEmptyException

Description copied from interface: Catalog

Drop a database.

Specified by:

dropDatabase in interface Catalog

Parameters:

name - Name of the database to be dropped.

ignoreIfNotExists - Flag to specify behavior when the database does not exist: if set to false, throw an exception, if set to true, do nothing.

cascade - Flag to specify behavior when the database contains table or function: if set to true, delete all tables and functions in the database and then delete the database, if set to false, throw an exception.

Throws:

Catalog.DatabaseNotEmptyException - if the given database is not empty and isRestrict is true

Catalog.DatabaseNotExistException

dropDatabaseImpl

protected abstract void dropDatabaseImpl(String name)

listTables

public List<String> listTables(String databaseName)
                        throws Catalog.DatabaseNotExistException

Description copied from interface: Catalog

Get names of all tables under this database. An empty list is returned if none exists.

NOTE: System tables will not be listed.

Specified by:

listTables in interface Catalog

Returns:

a list of the names of all tables in this database

Throws:

Catalog.DatabaseNotExistException - if the database does not exist

listTablesImpl

protected abstract List<String> listTablesImpl(String databaseName)

dropTable

public void dropTable(Identifier identifier,
                      boolean ignoreIfNotExists)
               throws Catalog.TableNotExistException

Description copied from interface: Catalog

Drop a table.

NOTE: System tables can not be dropped.

Specified by:

dropTable in interface Catalog

Parameters:

identifier - Path of the table to be dropped

ignoreIfNotExists - Flag to specify behavior when the table does not exist: if set to false, throw an exception, if set to true, do nothing.

Throws:

Catalog.TableNotExistException - if the table does not exist

dropTableImpl

protected abstract void dropTableImpl(Identifier identifier)

createTable

public void createTable(Identifier identifier,
                        Schema schema,
                        boolean ignoreIfExists)
                 throws Catalog.TableAlreadyExistException,
                        Catalog.DatabaseNotExistException

Description copied from interface: Catalog

Create a new table.

NOTE: System tables can not be created.

Specified by:

createTable in interface Catalog

Parameters:

identifier - path of the table to be created

schema - the table definition

ignoreIfExists - flag to specify behavior when a table already exists at the given path: if set to false, it throws a TableAlreadyExistException, if set to true, do nothing.

Throws:

Catalog.TableAlreadyExistException - if table already exists and ignoreIfExists is false

Catalog.DatabaseNotExistException - if the database in identifier doesn't exist

createTableImpl

protected abstract void createTableImpl(Identifier identifier,
                                        Schema schema)

renameTable

public void renameTable(Identifier fromTable,
                        Identifier toTable,
                        boolean ignoreIfNotExists)
                 throws Catalog.TableNotExistException,
                        Catalog.TableAlreadyExistException

Description copied from interface: Catalog

Rename a table.

NOTE: If you use object storage, such as S3 or OSS, please use this syntax carefully, because the renaming of object storage is not atomic, and only partial files may be moved in case of failure.

NOTE: System tables can not be renamed.

Specified by:

renameTable in interface Catalog

Parameters:

fromTable - the name of the table which need to rename

toTable - the new table

ignoreIfNotExists - Flag to specify behavior when the table does not exist: if set to false, throw an exception, if set to true, do nothing.

Throws:

Catalog.TableNotExistException - if the fromTable does not exist

Catalog.TableAlreadyExistException - if the toTable already exists

renameTableImpl

protected abstract void renameTableImpl(Identifier fromTable,
                                        Identifier toTable)

alterTable

public void alterTable(Identifier identifier,
                       List<SchemaChange> changes,
                       boolean ignoreIfNotExists)
                throws Catalog.TableNotExistException,
                       Catalog.ColumnAlreadyExistException,
                       Catalog.ColumnNotExistException

Description copied from interface: Catalog

Modify an existing table from SchemaChanges.

NOTE: System tables can not be altered.

Specified by:

alterTable in interface Catalog

Parameters:

identifier - path of the table to be modified

changes - the schema changes

ignoreIfNotExists - flag to specify behavior when the table does not exist: if set to false, throw an exception, if set to true, do nothing.

Throws:

Catalog.TableNotExistException - if the table does not exist

Catalog.ColumnAlreadyExistException

Catalog.ColumnNotExistException

alterTableImpl

protected abstract void alterTableImpl(Identifier identifier,
                                       List<SchemaChange> changes)
                                throws Catalog.TableNotExistException,
                                       Catalog.ColumnAlreadyExistException,
                                       Catalog.ColumnNotExistException

Throws:

Catalog.TableNotExistException

Catalog.ColumnAlreadyExistException

Catalog.ColumnNotExistException

getTable

public Table getTable(Identifier identifier)
               throws Catalog.TableNotExistException

Description copied from interface: Catalog

Return a Table identified by the given Identifier.

System tables can be got by '$' splitter.

Specified by:

getTable in interface Catalog

Parameters:

identifier - Path of the table

Returns:

The requested table

Throws:

Catalog.TableNotExistException - if the target does not exist

getDataOrFormatTable

protected Table getDataOrFormatTable(Identifier identifier)
                              throws Catalog.TableNotExistException

Throws:

Catalog.TableNotExistException

objectFileIO

protected FileIO objectFileIO(String objectLocation)

Catalog implementation may override this method to provide FileIO to object table.

createFormatTable

public void createFormatTable(Identifier identifier,
                              Schema schema)

Create a FormatTable identified by the given Identifier.

Parameters:

identifier - Path of the table

schema - Schema of the table

newDatabasePath

public Path newDatabasePath(String database)

Get warehouse path for specified database. If a catalog would like to provide individual path for each database, this method can be `Override` in that catalog.

Parameters:

database - The given database name

Returns:

The warehouse path for the database

allTablePaths

public Map<String,Map<String,Path>> allTablePaths()

getDataTableMeta

protected AbstractCatalog.TableMeta getDataTableMeta(Identifier identifier)
                                              throws Catalog.TableNotExistException

Throws:

Catalog.TableNotExistException

getDataTableSchema

protected abstract TableSchema getDataTableSchema(Identifier identifier)
                                           throws Catalog.TableNotExistException

Throws:

Catalog.TableNotExistException

metastoreClientFactory

public Optional<MetastoreClient.Factory> metastoreClientFactory(Identifier identifier,
                                                                TableSchema schema)

Get metastore client factory for the table specified by identifier.

getTableLocation

public Path getTableLocation(Identifier identifier)

Description copied from interface: Catalog

Get the table location in this catalog. If the table exists, return the location of the table; If the table does not exist, construct the location for table.

Specified by:

getTableLocation in interface Catalog

Returns:

the table location

checkNotBranch

protected static void checkNotBranch(Identifier identifier,
                                     String method)

assertMainBranch

protected void assertMainBranch(Identifier identifier)

isSpecifiedSystemTable

public static boolean isSpecifiedSystemTable(Identifier identifier)

isTableInSystemDatabase

protected static boolean isTableInSystemDatabase(Identifier identifier)

checkNotSystemTable

protected static void checkNotSystemTable(Identifier identifier,
                                          String method)

newTableLocation

public static Path newTableLocation(String warehouse,
                                    Identifier identifier)

newDatabasePath

public static Path newDatabasePath(String warehouse,
                                   String database)

isSystemDatabase

public static boolean isSystemDatabase(String database)

checkNotSystemDatabase

protected void checkNotSystemDatabase(String database)

Validate database cannot be a system database.

validateIdentifierNameCaseInsensitive

protected void validateIdentifierNameCaseInsensitive(Identifier identifier)

validateFieldNameCaseInsensitive

protected void validateFieldNameCaseInsensitive(List<String> fieldNames)

listDatabasesInFileSystem

protected List<String> listDatabasesInFileSystem(Path warehouse)
                                          throws IOException

Throws:

IOException

listTablesInFileSystem

protected List<String> listTablesInFileSystem(Path databasePath)
                                       throws IOException

Throws:

IOException

tableExistsInFileSystem

protected boolean tableExistsInFileSystem(Path tablePath,
                                          String branchName)

tableSchemaInFileSystem

public Optional<TableSchema> tableSchemaInFileSystem(Path tablePath,
                                                     String branchName)