org.apache.paimon.hive

Class HiveCatalog

java.lang.Object

org.apache.paimon.catalog.AbstractCatalog

org.apache.paimon.hive.HiveCatalog

All Implemented Interfaces:

AutoCloseable, Catalog

public class HiveCatalog
extends AbstractCatalog

A catalog implementation for Hive.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.paimon.catalog.AbstractCatalog

AbstractCatalog.TableMeta

Nested classes/interfaces inherited from interface org.apache.paimon.catalog.Catalog

Catalog.ColumnAlreadyExistException, Catalog.ColumnNotExistException, Catalog.DatabaseAlreadyExistException, Catalog.DatabaseNotEmptyException, Catalog.DatabaseNotExistException, Catalog.Loader, Catalog.PartitionNotExistException, Catalog.ProcessSystemDatabaseException, Catalog.TableAlreadyExistException, Catalog.TableNotExistException, Catalog.ViewAlreadyExistException, Catalog.ViewNotExistException

Field Summary

Fields

Modifier and Type	Field and Description
static String	HIVE_SITE_FILE
static String	PAIMON_TABLE_IDENTIFIER
static String	TABLE_TYPE_PROP

Fields inherited from class org.apache.paimon.catalog.AbstractCatalog

catalogOptions, fileIO, tableDefaultOptions

Fields inherited from interface org.apache.paimon.catalog.Catalog

COMMENT_PROP, DB_LOCATION_PROP, DB_SUFFIX, DEFAULT_DATABASE, HIVE_LAST_UPDATE_TIME_PROP, LAST_UPDATE_TIME_PROP, NUM_FILES_PROP, NUM_ROWS_PROP, OWNER_PROP, SYSTEM_BRANCH_PREFIX, SYSTEM_DATABASE_NAME, SYSTEM_TABLE_SPLITTER, TABLE_DEFAULT_OPTION_PREFIX, TOTAL_SIZE_PROP

Constructor Summary

Constructors

Constructor and Description
HiveCatalog(FileIO fileIO, org.apache.hadoop.hive.conf.HiveConf hiveConf, String clientClassName, Options options, String warehouse)
HiveCatalog(FileIO fileIO, org.apache.hadoop.hive.conf.HiveConf hiveConf, String clientClassName, String warehouse)

Method Summary

Modifier and Type	Method and Description
protected boolean	allowCustomTablePath()
boolean	allowUpperCase() Return a boolean that indicates whether this catalog allow upper case.
protected void	alterTableImpl(Identifier identifier, List<SchemaChange> changes)
void	close()
protected void	createDatabaseImpl(String name, Map<String,String> properties)
void	createFormatTable(Identifier identifier, Schema schema) Create a FormatTable identified by the given Identifier.
static Catalog	createHiveCatalog(CatalogContext context)
static org.apache.hadoop.hive.conf.HiveConf	createHiveConf(CatalogContext context)
static org.apache.hadoop.hive.conf.HiveConf	createHiveConf(String hiveConfDir, String hadoopConfDir, org.apache.hadoop.conf.Configuration defaultHadoopConf)
protected void	createTableImpl(Identifier identifier, Schema schema)
void	createView(Identifier identifier, View view, boolean ignoreIfExists) Create a new view.
Optional<CatalogLockFactory>	defaultLockFactory()
protected void	dropDatabaseImpl(String name)
void	dropPartition(Identifier identifier, Map<String,String> partitionSpec) Drop the partition of the specify table.
protected void	dropTableImpl(Identifier identifier)
void	dropView(Identifier identifier, boolean ignoreIfNotExists) Drop a view.
Database	getDatabaseImpl(String name)
Table	getDataOrFormatTable(Identifier identifier)
protected AbstractCatalog.TableMeta	getDataTableMeta(Identifier identifier)
TableSchema	getDataTableSchema(Identifier identifier)
org.apache.hadoop.hive.metastore.IMetaStoreClient	getHmsClient()
org.apache.hadoop.hive.metastore.api.Table	getHmsTable(Identifier identifier)
Path	getTableLocation(Identifier identifier) Get the table location in this catalog.
View	getView(Identifier identifier) Return a View identified by the given Identifier.
static boolean	isView(org.apache.hadoop.hive.metastore.api.Table table)
List<String>	listDatabases() Get the names of all databases in this catalog.
protected List<String>	listTablesImpl(String databaseName)
List<String>	listViews(String databaseName) Get names of all views under this database.
Optional<CatalogLockContext>	lockContext()
Optional<MetastoreClient.Factory>	metastoreClientFactory(Identifier identifier, TableSchema schema) Get metastore client factory for the table specified by identifier.
static String	possibleHiveConfPath()
protected void	renameTableImpl(Identifier fromTable, Identifier toTable)
void	renameView(Identifier fromView, Identifier toView, boolean ignoreIfNotExists) Rename a view.
void	repairCatalog()
void	repairDatabase(String databaseName)
void	repairTable(Identifier identifier)
boolean	syncAllProperties()
String	warehouse() Warehouse root path containing all database directories in this catalog.

Methods inherited from class org.apache.paimon.catalog.AbstractCatalog

allTablePaths, alterTable, assertMainBranch, checkNotBranch, checkNotSystemDatabase, checkNotSystemTable, createDatabase, createPartition, createTable, dropDatabase, dropTable, fileIO, getDatabase, getTable, isSpecifiedSystemTable, isSystemDatabase, isTableInSystemDatabase, listDatabasesInFileSystem, listPartitions, listTables, listTablesInFileSystem, lockEnabled, lockFactory, newDatabasePath, newDatabasePath, newTableLocation, objectFileIO, options, renameTable, tableExistsInFileSystem, tableSchemaInFileSystem, validateFieldNameCaseInsensitive, validateIdentifierNameCaseInsensitive

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.paimon.catalog.Catalog

alterTable, createDatabase, invalidateTable, tableDefaultOptions, validateCaseInsensitive, validateCaseInsensitive

Field Detail

TABLE_TYPE_PROP

public static final String TABLE_TYPE_PROP

See Also:

Constant Field Values

PAIMON_TABLE_IDENTIFIER

public static final String PAIMON_TABLE_IDENTIFIER

See Also:

Constant Field Values

HIVE_SITE_FILE

public static final String HIVE_SITE_FILE

See Also:

Constant Field Values

Constructor Detail

HiveCatalog

public HiveCatalog(FileIO fileIO,
                   org.apache.hadoop.hive.conf.HiveConf hiveConf,
                   String clientClassName,
                   String warehouse)

HiveCatalog

public HiveCatalog(FileIO fileIO,
                   org.apache.hadoop.hive.conf.HiveConf hiveConf,
                   String clientClassName,
                   Options options,
                   String warehouse)

Method Detail

defaultLockFactory

public Optional<CatalogLockFactory> defaultLockFactory()

Overrides:

defaultLockFactory in class AbstractCatalog

lockContext

public Optional<CatalogLockContext> lockContext()

Overrides:

lockContext in class AbstractCatalog

metastoreClientFactory

public Optional<MetastoreClient.Factory> metastoreClientFactory(Identifier identifier,
                                                                TableSchema schema)

Description copied from class: AbstractCatalog

Get metastore client factory for the table specified by identifier.

Overrides:

metastoreClientFactory in class AbstractCatalog

getTableLocation

public Path getTableLocation(Identifier identifier)

Description copied from interface: Catalog

Get the table location in this catalog. If the table exists, return the location of the table; If the table does not exist, construct the location for table.

Specified by:

getTableLocation in interface Catalog

Overrides:

getTableLocation in class AbstractCatalog

Returns:

the table location

listDatabases

public List<String> listDatabases()

Description copied from interface: Catalog

Get the names of all databases in this catalog.

Returns:

a list of the names of all databases

createDatabaseImpl

protected void createDatabaseImpl(String name,
                                  Map<String,String> properties)

Specified by:

createDatabaseImpl in class AbstractCatalog

getDatabaseImpl

public Database getDatabaseImpl(String name)
                         throws Catalog.DatabaseNotExistException

Specified by:

getDatabaseImpl in class AbstractCatalog

Throws:

Catalog.DatabaseNotExistException

dropPartition

public void dropPartition(Identifier identifier,
                          Map<String,String> partitionSpec)
                   throws Catalog.TableNotExistException

Description copied from interface: Catalog

Drop the partition of the specify table.

Specified by:

dropPartition in interface Catalog

Overrides:

dropPartition in class AbstractCatalog

Parameters:

identifier - path of the table to drop partition

partitionSpec - the partition to be deleted

Throws:

Catalog.TableNotExistException - if the table does not exist

dropDatabaseImpl

protected void dropDatabaseImpl(String name)

Specified by:

dropDatabaseImpl in class AbstractCatalog

listTablesImpl

protected List<String> listTablesImpl(String databaseName)

Specified by:

listTablesImpl in class AbstractCatalog

getDataTableMeta

protected AbstractCatalog.TableMeta getDataTableMeta(Identifier identifier)
                                              throws Catalog.TableNotExistException

Overrides:

getDataTableMeta in class AbstractCatalog

Throws:

Catalog.TableNotExistException

getDataTableSchema

public TableSchema getDataTableSchema(Identifier identifier)
                               throws Catalog.TableNotExistException

Specified by:

getDataTableSchema in class AbstractCatalog

Throws:

Catalog.TableNotExistException

getView

public View getView(Identifier identifier)
             throws Catalog.ViewNotExistException

Description copied from interface: Catalog

Return a View identified by the given Identifier.

Parameters:

identifier - Path of the view

Returns:

The requested view

Throws:

Catalog.ViewNotExistException - if the target does not exist

createView

public void createView(Identifier identifier,
                       View view,
                       boolean ignoreIfExists)
                throws Catalog.ViewAlreadyExistException,
                       Catalog.DatabaseNotExistException

Description copied from interface: Catalog

Create a new view.

Parameters:

identifier - path of the view to be created

view - the view definition

ignoreIfExists - flag to specify behavior when a view already exists at the given path: if set to false, it throws a ViewAlreadyExistException, if set to true, do nothing.

Throws:

Catalog.ViewAlreadyExistException - if view already exists and ignoreIfExists is false

Catalog.DatabaseNotExistException - if the database in identifier doesn't exist

dropView

public void dropView(Identifier identifier,
                     boolean ignoreIfNotExists)
              throws Catalog.ViewNotExistException

Description copied from interface: Catalog

Drop a view.

Parameters:

identifier - Path of the view to be dropped

ignoreIfNotExists - Flag to specify behavior when the view does not exist: if set to false, throw an exception, if set to true, do nothing.

Throws:

Catalog.ViewNotExistException - if the view does not exist

listViews

public List<String> listViews(String databaseName)
                       throws Catalog.DatabaseNotExistException

Description copied from interface: Catalog

Get names of all views under this database. An empty list is returned if none exists.

Returns:

a list of the names of all views in this database

Throws:

Catalog.DatabaseNotExistException - if the database does not exist

renameView

public void renameView(Identifier fromView,
                       Identifier toView,
                       boolean ignoreIfNotExists)
                throws Catalog.ViewNotExistException,
                       Catalog.ViewAlreadyExistException

Description copied from interface: Catalog

Rename a view.

Parameters:

fromView - identifier of the view to rename

toView - new view identifier

Throws:

Catalog.ViewNotExistException - if the fromView does not exist

Catalog.ViewAlreadyExistException - if the toView already exists

getDataOrFormatTable

public Table getDataOrFormatTable(Identifier identifier)
                           throws Catalog.TableNotExistException

Overrides:

getDataOrFormatTable in class AbstractCatalog

Throws:

Catalog.TableNotExistException

createFormatTable

public void createFormatTable(Identifier identifier,
                              Schema schema)

Description copied from class: AbstractCatalog

Create a FormatTable identified by the given Identifier.

Overrides:

createFormatTable in class AbstractCatalog

Parameters:

identifier - Path of the table

schema - Schema of the table

dropTableImpl

protected void dropTableImpl(Identifier identifier)

Specified by:

dropTableImpl in class AbstractCatalog

createTableImpl

protected void createTableImpl(Identifier identifier,
                               Schema schema)

Specified by:

createTableImpl in class AbstractCatalog

renameTableImpl

protected void renameTableImpl(Identifier fromTable,
                               Identifier toTable)

Specified by:

renameTableImpl in class AbstractCatalog

alterTableImpl

protected void alterTableImpl(Identifier identifier,
                              List<SchemaChange> changes)
                       throws Catalog.TableNotExistException,
                              Catalog.ColumnAlreadyExistException,
                              Catalog.ColumnNotExistException

Specified by:

alterTableImpl in class AbstractCatalog

Throws:

Catalog.TableNotExistException

Catalog.ColumnAlreadyExistException

Catalog.ColumnNotExistException

allowUpperCase

public boolean allowUpperCase()

Description copied from interface: Catalog

Return a boolean that indicates whether this catalog allow upper case.

Specified by:

allowUpperCase in interface Catalog

Overrides:

allowUpperCase in class AbstractCatalog

allowCustomTablePath

protected boolean allowCustomTablePath()

Overrides:

allowCustomTablePath in class AbstractCatalog

syncAllProperties

public boolean syncAllProperties()

repairCatalog

public void repairCatalog()

repairDatabase

public void repairDatabase(String databaseName)

repairTable

public void repairTable(Identifier identifier)
                 throws Catalog.TableNotExistException

Throws:

Catalog.TableNotExistException

close

public void close()
           throws Exception

Throws:

Exception

warehouse

public String warehouse()

Description copied from interface: Catalog

Warehouse root path containing all database directories in this catalog.

getHmsTable

public org.apache.hadoop.hive.metastore.api.Table getHmsTable(Identifier identifier)
                                                       throws Catalog.TableNotExistException

Throws:

Catalog.TableNotExistException

isView

public static boolean isView(org.apache.hadoop.hive.metastore.api.Table table)

getHmsClient

@VisibleForTesting
public org.apache.hadoop.hive.metastore.IMetaStoreClient getHmsClient()

createHiveConf

public static org.apache.hadoop.hive.conf.HiveConf createHiveConf(@Nullable
                                                                  String hiveConfDir,
                                                                  @Nullable
                                                                  String hadoopConfDir,
                                                                  org.apache.hadoop.conf.Configuration defaultHadoopConf)

createHiveCatalog

public static Catalog createHiveCatalog(CatalogContext context)

createHiveConf

public static org.apache.hadoop.hive.conf.HiveConf createHiveConf(CatalogContext context)

possibleHiveConfPath

public static String possibleHiveConfPath()