org.apache.paimon.hive

Class HiveCatalog

java.lang.Object

org.apache.paimon.catalog.AbstractCatalog

org.apache.paimon.hive.HiveCatalog

All Implemented Interfaces:

AutoCloseable, Catalog

public class HiveCatalog
extends AbstractCatalog

A catalog implementation for Hive.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.paimon.catalog.Catalog

Catalog.ColumnAlreadyExistException, Catalog.ColumnNotExistException, Catalog.DatabaseAlreadyExistException, Catalog.DatabaseNotEmptyException, Catalog.DatabaseNotExistException, Catalog.Loader, Catalog.PartitionNotExistException, Catalog.ProcessSystemDatabaseException, Catalog.TableAlreadyExistException, Catalog.TableNotExistException

Field Summary

Fields

Modifier and Type	Field and Description
static String	HIVE_SITE_FILE
static String	PAIMON_TABLE_TYPE_VALUE
static String	TABLE_TYPE_PROP

Fields inherited from class org.apache.paimon.catalog.AbstractCatalog

catalogOptions, fileIO, lineageMetaFactory, tableDefaultOptions

Fields inherited from interface org.apache.paimon.catalog.Catalog

COMMENT_PROP, DB_LOCATION_PROP, DB_SUFFIX, DEFAULT_DATABASE, SYSTEM_BRANCH_PREFIX, SYSTEM_DATABASE_NAME, SYSTEM_TABLE_SPLITTER, TABLE_DEFAULT_OPTION_PREFIX

Constructor Summary

Constructors

Constructor and Description
HiveCatalog(FileIO fileIO, org.apache.hadoop.hive.conf.HiveConf hiveConf, String clientClassName, Options options, String warehouse)
HiveCatalog(FileIO fileIO, org.apache.hadoop.hive.conf.HiveConf hiveConf, String clientClassName, String warehouse)

Method Summary

Modifier and Type	Method and Description
boolean	allowUpperCase() Return a boolean that indicates whether this catalog allow upper case.
protected void	alterTableImpl(Identifier identifier, List<SchemaChange> changes)
void	close()
protected void	createDatabaseImpl(String name, Map<String,String> properties)
static Catalog	createHiveCatalog(CatalogContext context)
static org.apache.hadoop.hive.conf.HiveConf	createHiveConf(CatalogContext context)
static org.apache.hadoop.hive.conf.HiveConf	createHiveConf(String hiveConfDir, String hadoopConfDir, org.apache.hadoop.conf.Configuration defaultHadoopConf)
protected void	createTableImpl(Identifier identifier, Schema schema)
Optional<CatalogLockFactory>	defaultLockFactory()
protected void	dropDatabaseImpl(String name)
void	dropPartition(Identifier identifier, Map<String,String> partitionSpec) Drop the partition of the specify table.
protected void	dropTableImpl(Identifier identifier)
TableSchema	getDataTableSchema(Identifier identifier)
FormatTable	getFormatTable(Identifier identifier) Return a FormatTable identified by the given Identifier.
static org.apache.hadoop.conf.Configuration	getHadoopConfiguration(String hadoopConfDir) Returns a new Hadoop Configuration object using the path to the hadoop conf configured.
org.apache.hadoop.hive.metastore.IMetaStoreClient	getHmsClient()
Path	getTableLocation(Identifier identifier) Get the table location in this catalog.
List<String>	listDatabases() Get the names of all databases in this catalog.
protected List<String>	listTablesImpl(String databaseName)
Map<String,String>	loadDatabasePropertiesImpl(String name)
Optional<CatalogLockContext>	lockContext() Get lock context for lock factory to create a lock.
Optional<MetastoreClient.Factory>	metastoreClientFactory(Identifier identifier) Get metastore client factory for the table specified by identifier.
static String	possibleHiveConfPath()
protected void	renameTableImpl(Identifier fromTable, Identifier toTable)
void	repairCatalog()
void	repairDatabase(String databaseName)
void	repairTable(Identifier identifier)
boolean	syncAllProperties()
boolean	tableExists(Identifier identifier) Check if a table exists in this catalog.
String	warehouse() Warehouse root path containing all database directories in this catalog.

Methods inherited from class org.apache.paimon.catalog.AbstractCatalog

allTablePaths, alterTable, assertMainBranch, checkNotBranch, checkNotSystemDatabase, checkNotSystemTable, createDatabase, createTable, dropDatabase, dropTable, fileIO, getTable, isSpecifiedSystemTable, isSystemDatabase, isSystemTable, listDatabasesInFileSystem, listTables, listTablesInFileSystem, loadDatabaseProperties, lockEnabled, lockFactory, newDatabasePath, newDatabasePath, newTableLocation, options, renameTable, tableExistsInFileSystem, tableSchemaInFileSystem, validateFieldNameCaseInsensitive, validateIdentifierNameCaseInsensitive

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.paimon.catalog.Catalog

alterTable, createDatabase, databaseExists, invalidateTable, tableDefaultOptions, validateCaseInsensitive, validateCaseInsensitive

Field Detail

TABLE_TYPE_PROP

public static final String TABLE_TYPE_PROP

See Also:

Constant Field Values

PAIMON_TABLE_TYPE_VALUE

public static final String PAIMON_TABLE_TYPE_VALUE

See Also:

Constant Field Values

HIVE_SITE_FILE

public static final String HIVE_SITE_FILE

See Also:

Constant Field Values

Constructor Detail

HiveCatalog

public HiveCatalog(FileIO fileIO,
                   org.apache.hadoop.hive.conf.HiveConf hiveConf,
                   String clientClassName,
                   String warehouse)

HiveCatalog

public HiveCatalog(FileIO fileIO,
                   org.apache.hadoop.hive.conf.HiveConf hiveConf,
                   String clientClassName,
                   Options options,
                   String warehouse)

Method Detail

defaultLockFactory

public Optional<CatalogLockFactory> defaultLockFactory()

Overrides:

defaultLockFactory in class AbstractCatalog

lockContext

public Optional<CatalogLockContext> lockContext()

Description copied from interface: Catalog

Get lock context for lock factory to create a lock.

Specified by:

lockContext in interface Catalog

Overrides:

lockContext in class AbstractCatalog

metastoreClientFactory

public Optional<MetastoreClient.Factory> metastoreClientFactory(Identifier identifier)

Description copied from interface: Catalog

Get metastore client factory for the table specified by identifier.

getTableLocation

public Path getTableLocation(Identifier identifier)

Description copied from interface: Catalog

Get the table location in this catalog. If the table exists, return the location of the table; If the table does not exist, construct the location for table.

Specified by:

getTableLocation in interface Catalog

Overrides:

getTableLocation in class AbstractCatalog

Returns:

the table location

listDatabases

public List<String> listDatabases()

Description copied from interface: Catalog

Get the names of all databases in this catalog.

Returns:

a list of the names of all databases

createDatabaseImpl

protected void createDatabaseImpl(String name,
                                  Map<String,String> properties)

Specified by:

createDatabaseImpl in class AbstractCatalog

loadDatabasePropertiesImpl

public Map<String,String> loadDatabasePropertiesImpl(String name)
                                              throws Catalog.DatabaseNotExistException

Specified by:

loadDatabasePropertiesImpl in class AbstractCatalog

Throws:

Catalog.DatabaseNotExistException

dropPartition

public void dropPartition(Identifier identifier,
                          Map<String,String> partitionSpec)
                   throws Catalog.TableNotExistException

Description copied from interface: Catalog

Drop the partition of the specify table.

Specified by:

dropPartition in interface Catalog

Overrides:

dropPartition in class AbstractCatalog

Parameters:

identifier - path of the table to drop partition

partitionSpec - the partition to be deleted

Throws:

Catalog.TableNotExistException - if the table does not exist

dropDatabaseImpl

protected void dropDatabaseImpl(String name)

Specified by:

dropDatabaseImpl in class AbstractCatalog

listTablesImpl

protected List<String> listTablesImpl(String databaseName)

Specified by:

listTablesImpl in class AbstractCatalog

tableExists

public boolean tableExists(Identifier identifier)

Description copied from interface: Catalog

Check if a table exists in this catalog.

Parameters:

identifier - Path of the table

Returns:

true if the given table exists in the catalog false otherwise

getDataTableSchema

public TableSchema getDataTableSchema(Identifier identifier)
                               throws Catalog.TableNotExistException

Specified by:

getDataTableSchema in class AbstractCatalog

Throws:

Catalog.TableNotExistException

getFormatTable

public FormatTable getFormatTable(Identifier identifier)
                           throws Catalog.TableNotExistException

Description copied from class: AbstractCatalog

Return a FormatTable identified by the given Identifier.

Overrides:

getFormatTable in class AbstractCatalog

Parameters:

identifier - Path of the table

Returns:

The requested table

Throws:

Catalog.TableNotExistException - if the target does not exist

dropTableImpl

protected void dropTableImpl(Identifier identifier)

Specified by:

dropTableImpl in class AbstractCatalog

createTableImpl

protected void createTableImpl(Identifier identifier,
                               Schema schema)

Specified by:

createTableImpl in class AbstractCatalog

renameTableImpl

protected void renameTableImpl(Identifier fromTable,
                               Identifier toTable)

Specified by:

renameTableImpl in class AbstractCatalog

alterTableImpl

protected void alterTableImpl(Identifier identifier,
                              List<SchemaChange> changes)
                       throws Catalog.TableNotExistException,
                              Catalog.ColumnAlreadyExistException,
                              Catalog.ColumnNotExistException

Specified by:

alterTableImpl in class AbstractCatalog

Throws:

Catalog.TableNotExistException

Catalog.ColumnAlreadyExistException

Catalog.ColumnNotExistException

allowUpperCase

public boolean allowUpperCase()

Description copied from interface: Catalog

Return a boolean that indicates whether this catalog allow upper case.

Specified by:

allowUpperCase in interface Catalog

Overrides:

allowUpperCase in class AbstractCatalog

syncAllProperties

public boolean syncAllProperties()

repairCatalog

public void repairCatalog()

repairDatabase

public void repairDatabase(String databaseName)

repairTable

public void repairTable(Identifier identifier)
                 throws Catalog.TableNotExistException

Throws:

Catalog.TableNotExistException

close

public void close()
           throws Exception

Throws:

Exception

warehouse

public String warehouse()

Description copied from interface: Catalog

Warehouse root path containing all database directories in this catalog.

getHmsClient

@VisibleForTesting
public org.apache.hadoop.hive.metastore.IMetaStoreClient getHmsClient()

createHiveConf

public static org.apache.hadoop.hive.conf.HiveConf createHiveConf(@Nullable
                                                                  String hiveConfDir,
                                                                  @Nullable
                                                                  String hadoopConfDir,
                                                                  org.apache.hadoop.conf.Configuration defaultHadoopConf)

createHiveCatalog

public static Catalog createHiveCatalog(CatalogContext context)

createHiveConf

public static org.apache.hadoop.hive.conf.HiveConf createHiveConf(CatalogContext context)

getHadoopConfiguration

public static org.apache.hadoop.conf.Configuration getHadoopConfiguration(String hadoopConfDir)

Returns a new Hadoop Configuration object using the path to the hadoop conf configured.

Parameters:

hadoopConfDir - Hadoop conf directory path.

Returns:

A Hadoop configuration instance.

possibleHiveConfPath

public static String possibleHiveConfPath()