org.apache.paimon.catalog

Class CachingCatalog

java.lang.Object

org.apache.paimon.catalog.DelegateCatalog

org.apache.paimon.catalog.CachingCatalog

All Implemented Interfaces:

AutoCloseable, Catalog

public class CachingCatalog
extends DelegateCatalog

A Catalog to cache databases and tables and manifests.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.paimon.catalog.Catalog

Catalog.ColumnAlreadyExistException, Catalog.ColumnNotExistException, Catalog.DatabaseAlreadyExistException, Catalog.DatabaseNotEmptyException, Catalog.DatabaseNotExistException, Catalog.Loader, Catalog.PartitionNotExistException, Catalog.ProcessSystemDatabaseException, Catalog.TableAlreadyExistException, Catalog.TableNotExistException, Catalog.ViewAlreadyExistException, Catalog.ViewNotExistException

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.paimon.shade.caffeine2.com.github.benmanes.caffeine.cache.Cache<String,Database>	databaseCache
protected SegmentsCache<Path>	manifestCache
protected org.apache.paimon.shade.caffeine2.com.github.benmanes.caffeine.cache.Cache<Identifier,List<PartitionEntry>>	partitionCache
protected org.apache.paimon.shade.caffeine2.com.github.benmanes.caffeine.cache.Cache<Identifier,Table>	tableCache

Fields inherited from class org.apache.paimon.catalog.DelegateCatalog

wrapped

Fields inherited from interface org.apache.paimon.catalog.Catalog

COMMENT_PROP, DB_LOCATION_PROP, DB_SUFFIX, DEFAULT_DATABASE, HIVE_LAST_UPDATE_TIME_PROP, LAST_UPDATE_TIME_PROP, NUM_FILES_PROP, NUM_ROWS_PROP, OWNER_PROP, SYSTEM_BRANCH_PREFIX, SYSTEM_DATABASE_NAME, SYSTEM_TABLE_SPLITTER, TABLE_DEFAULT_OPTION_PREFIX, TOTAL_SIZE_PROP

Constructor Summary

Constructors

Constructor and Description
CachingCatalog(Catalog wrapped)
CachingCatalog(Catalog wrapped, java.time.Duration expirationInterval, MemorySize manifestMaxMemory, long manifestCacheThreshold, long cachedPartitionMaxNum, int snapshotMaxNumPerTable)
CachingCatalog(Catalog wrapped, java.time.Duration expirationInterval, MemorySize manifestMaxMemory, long manifestCacheThreshold, long cachedPartitionMaxNum, int snapshotMaxNumPerTable, org.apache.paimon.shade.caffeine2.com.github.benmanes.caffeine.cache.Ticker ticker)

Method Summary

Modifier and Type	Method and Description
void	alterTable(Identifier identifier, List<SchemaChange> changes, boolean ignoreIfNotExists) Modify an existing table from SchemaChanges.
void	dropDatabase(String name, boolean ignoreIfNotExists, boolean cascade) Drop a database.
void	dropPartition(Identifier identifier, Map<String,String> partitions) Drop the partition of the specify table.
void	dropTable(Identifier identifier, boolean ignoreIfNotExists) Drop a table.
Database	getDatabase(String databaseName) Return a Database identified by the given name.
Table	getTable(Identifier identifier) Return a Table identified by the given Identifier.
void	invalidateTable(Identifier identifier) Invalidate cached table metadata for an identifier.
List<PartitionEntry>	listPartitions(Identifier identifier) Get PartitionEntry of all partitions of the table.
void	refreshPartitions(Identifier identifier)
void	renameTable(Identifier fromTable, Identifier toTable, boolean ignoreIfNotExists) Rename a table.
static Catalog	tryToCreate(Catalog catalog, Options options)

Methods inherited from class org.apache.paimon.catalog.DelegateCatalog

allowUpperCase, close, createDatabase, createPartition, createTable, createView, dropView, fileIO, getTableLocation, getView, listDatabases, listTables, listViews, options, renameView, repairCatalog, repairDatabase, repairTable, warehouse, wrapped

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.paimon.catalog.Catalog

alterTable, createDatabase, tableDefaultOptions, validateCaseInsensitive, validateCaseInsensitive

Field Detail

databaseCache

protected final org.apache.paimon.shade.caffeine2.com.github.benmanes.caffeine.cache.Cache<String,Database> databaseCache

tableCache

protected final org.apache.paimon.shade.caffeine2.com.github.benmanes.caffeine.cache.Cache<Identifier,Table> tableCache

manifestCache

@Nullable
protected final SegmentsCache<Path> manifestCache

partitionCache

@Nullable
protected final org.apache.paimon.shade.caffeine2.com.github.benmanes.caffeine.cache.Cache<Identifier,List<PartitionEntry>> partitionCache

Constructor Detail

CachingCatalog

public CachingCatalog(Catalog wrapped)

CachingCatalog

public CachingCatalog(Catalog wrapped,
                      java.time.Duration expirationInterval,
                      MemorySize manifestMaxMemory,
                      long manifestCacheThreshold,
                      long cachedPartitionMaxNum,
                      int snapshotMaxNumPerTable)

CachingCatalog

public CachingCatalog(Catalog wrapped,
                      java.time.Duration expirationInterval,
                      MemorySize manifestMaxMemory,
                      long manifestCacheThreshold,
                      long cachedPartitionMaxNum,
                      int snapshotMaxNumPerTable,
                      org.apache.paimon.shade.caffeine2.com.github.benmanes.caffeine.cache.Ticker ticker)

Method Detail

tryToCreate

public static Catalog tryToCreate(Catalog catalog,
                                  Options options)

getDatabase

public Database getDatabase(String databaseName)
                     throws Catalog.DatabaseNotExistException

Description copied from interface: Catalog

Return a Database identified by the given name.

Specified by:

getDatabase in interface Catalog

Overrides:

getDatabase in class DelegateCatalog

Parameters:

databaseName - Database name

Returns:

The requested Database

Throws:

Catalog.DatabaseNotExistException - if the requested database does not exist

dropDatabase

public void dropDatabase(String name,
                         boolean ignoreIfNotExists,
                         boolean cascade)
                  throws Catalog.DatabaseNotExistException,
                         Catalog.DatabaseNotEmptyException

Description copied from interface: Catalog

Drop a database.

Specified by:

dropDatabase in interface Catalog

Overrides:

dropDatabase in class DelegateCatalog

Parameters:

name - Name of the database to be dropped.

ignoreIfNotExists - Flag to specify behavior when the database does not exist: if set to false, throw an exception, if set to true, do nothing.

cascade - Flag to specify behavior when the database contains table or function: if set to true, delete all tables and functions in the database and then delete the database, if set to false, throw an exception.

Throws:

Catalog.DatabaseNotEmptyException - if the given database is not empty and isRestrict is true

Catalog.DatabaseNotExistException

dropTable

public void dropTable(Identifier identifier,
                      boolean ignoreIfNotExists)
               throws Catalog.TableNotExistException

Description copied from interface: Catalog

Drop a table.

NOTE: System tables can not be dropped.

Specified by:

dropTable in interface Catalog

Overrides:

dropTable in class DelegateCatalog

Parameters:

identifier - Path of the table to be dropped

ignoreIfNotExists - Flag to specify behavior when the table does not exist: if set to false, throw an exception, if set to true, do nothing.

Throws:

Catalog.TableNotExistException - if the table does not exist

renameTable

public void renameTable(Identifier fromTable,
                        Identifier toTable,
                        boolean ignoreIfNotExists)
                 throws Catalog.TableNotExistException,
                        Catalog.TableAlreadyExistException

Description copied from interface: Catalog

Rename a table.

NOTE: If you use object storage, such as S3 or OSS, please use this syntax carefully, because the renaming of object storage is not atomic, and only partial files may be moved in case of failure.

NOTE: System tables can not be renamed.

Specified by:

renameTable in interface Catalog

Overrides:

renameTable in class DelegateCatalog

Parameters:

fromTable - the name of the table which need to rename

toTable - the new table

ignoreIfNotExists - Flag to specify behavior when the table does not exist: if set to false, throw an exception, if set to true, do nothing.

Throws:

Catalog.TableNotExistException - if the fromTable does not exist

Catalog.TableAlreadyExistException - if the toTable already exists

alterTable

public void alterTable(Identifier identifier,
                       List<SchemaChange> changes,
                       boolean ignoreIfNotExists)
                throws Catalog.TableNotExistException,
                       Catalog.ColumnAlreadyExistException,
                       Catalog.ColumnNotExistException

Description copied from interface: Catalog

Modify an existing table from SchemaChanges.

NOTE: System tables can not be altered.

Specified by:

alterTable in interface Catalog

Overrides:

alterTable in class DelegateCatalog

Parameters:

identifier - path of the table to be modified

changes - the schema changes

ignoreIfNotExists - flag to specify behavior when the table does not exist: if set to false, throw an exception, if set to true, do nothing.

Throws:

Catalog.TableNotExistException - if the table does not exist

Catalog.ColumnAlreadyExistException

Catalog.ColumnNotExistException

getTable

public Table getTable(Identifier identifier)
               throws Catalog.TableNotExistException

Description copied from interface: Catalog

Return a Table identified by the given Identifier.

System tables can be got by '$' splitter.

Specified by:

getTable in interface Catalog

Overrides:

getTable in class DelegateCatalog

Parameters:

identifier - Path of the table

Returns:

The requested table

Throws:

Catalog.TableNotExistException - if the target does not exist

listPartitions

public List<PartitionEntry> listPartitions(Identifier identifier)
                                    throws Catalog.TableNotExistException

Description copied from interface: Catalog

Get PartitionEntry of all partitions of the table.

Specified by:

listPartitions in interface Catalog

Overrides:

listPartitions in class DelegateCatalog

Parameters:

identifier - path of the table to list partitions

Throws:

Catalog.TableNotExistException - if the table does not exist

dropPartition

public void dropPartition(Identifier identifier,
                          Map<String,String> partitions)
                   throws Catalog.TableNotExistException,
                          Catalog.PartitionNotExistException

Description copied from interface: Catalog

Drop the partition of the specify table.

Specified by:

dropPartition in interface Catalog

Overrides:

dropPartition in class DelegateCatalog

Parameters:

identifier - path of the table to drop partition

partitions - the partition to be deleted

Throws:

Catalog.TableNotExistException - if the table does not exist

Catalog.PartitionNotExistException - if the partition does not exist

invalidateTable

public void invalidateTable(Identifier identifier)

Description copied from interface: Catalog

Invalidate cached table metadata for an identifier.

If the table is already loaded or cached, drop cached data. If the table does not exist or is not cached, do nothing. Calling this method should not query remote services.

Parameters:

identifier - a table identifier

refreshPartitions

public void refreshPartitions(Identifier identifier)
                       throws Catalog.TableNotExistException

Throws:

Catalog.TableNotExistException