org.apache.paimon.hive

Class HiveCatalog

java.lang.Object

org.apache.paimon.catalog.AbstractCatalog

org.apache.paimon.hive.HiveCatalog

All Implemented Interfaces:

AutoCloseable, Catalog

public class HiveCatalog
extends AbstractCatalog

A catalog implementation for Hive.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.paimon.catalog.Catalog

Catalog.BranchAlreadyExistException, Catalog.BranchNotExistException, Catalog.ColumnAlreadyExistException, Catalog.ColumnNotExistException, Catalog.DatabaseAlreadyExistException, Catalog.DatabaseNoPermissionException, Catalog.DatabaseNotEmptyException, Catalog.DatabaseNotExistException, Catalog.DialectAlreadyExistException, Catalog.DialectNotExistException, Catalog.ProcessSystemDatabaseException, Catalog.TableAlreadyExistException, Catalog.TableNoPermissionException, Catalog.TableNotExistException, Catalog.TagNotExistException, Catalog.ViewAlreadyExistException, Catalog.ViewNotExistException

Field Summary

Fields

Modifier and Type	Field and Description
static String	HIVE_SITE_FILE
static String	PAIMON_TABLE_IDENTIFIER
static String	TABLE_TYPE_PROP

Fields inherited from class org.apache.paimon.catalog.AbstractCatalog

catalogOptions, fileIO, tableDefaultOptions

Fields inherited from interface org.apache.paimon.catalog.Catalog

COMMENT_PROP, DB_LOCATION_PROP, DB_SUFFIX, DEFAULT_DATABASE, LAST_UPDATE_TIME_PROP, NUM_FILES_PROP, NUM_ROWS_PROP, OWNER_PROP, SYSTEM_BRANCH_PREFIX, SYSTEM_DATABASE_NAME, SYSTEM_TABLE_SPLITTER, TABLE_DEFAULT_OPTION_PREFIX, TOTAL_SIZE_PROP

Constructor Summary

Constructors

Constructor and Description
HiveCatalog(FileIO fileIO, org.apache.hadoop.hive.conf.HiveConf hiveConf, String clientClassName, Options options, String warehouse)
HiveCatalog(FileIO fileIO, org.apache.hadoop.hive.conf.HiveConf hiveConf, String clientClassName, String warehouse)

Method Summary

Modifier and Type	Method and Description
protected boolean	allowCustomTablePath()
protected void	alterDatabaseImpl(String name, List<PropertyChange> changes)
void	alterPartitions(Identifier identifier, List<PartitionStatistics> partitions) Alter partitions of the specify table.
protected void	alterTableImpl(Identifier identifier, List<SchemaChange> changes)
boolean	caseSensitive() Return a boolean that indicates whether this catalog is case-sensitive.
CatalogLoader	catalogLoader() Serializable loader to create catalog.
void	close()
protected void	createDatabaseImpl(String name, Map<String,String> properties)
void	createFormatTable(Identifier identifier, Schema schema) Create a FormatTable identified by the given Identifier.
static Catalog	createHiveCatalog(CatalogContext context)
static org.apache.hadoop.hive.conf.HiveConf	createHiveConf(CatalogContext context)
static org.apache.hadoop.hive.conf.HiveConf	createHiveConf(String hiveConfDir, String hadoopConfDir, org.apache.hadoop.conf.Configuration defaultHadoopConf)
void	createPartitions(Identifier identifier, List<Map<String,String>> partitions) Create partitions of the specify table.
protected void	createTableImpl(Identifier identifier, Schema schema)
void	createView(Identifier identifier, View view, boolean ignoreIfExists) Create a new view.
Optional<CatalogLockFactory>	defaultLockFactory()
protected void	dropDatabaseImpl(String name)
void	dropPartitions(Identifier identifier, List<Map<String,String>> partitions) Drop partitions of the specify table.
protected void	dropTableImpl(Identifier identifier, List<Path> externalPaths)
void	dropView(Identifier identifier, boolean ignoreIfNotExists) Drop a view.
int	getBatchGetTableSize()
Database	getDatabaseImpl(String name)
org.apache.hadoop.hive.metastore.IMetaStoreClient	getHmsClient()
org.apache.hadoop.hive.metastore.api.Table	getHmsTable(Identifier identifier)
Path	getTableLocation(Identifier identifier)
View	getView(Identifier identifier) Return a View identified by the given Identifier.
static boolean	isView(org.apache.hadoop.hive.metastore.api.Table table)
List<String>	listDatabases() Get the names of all databases in this catalog.
List<Partition>	listPartitions(Identifier identifier) Get Partition of all partitions of the table.
protected List<String>	listTablesImpl(String databaseName)
PagedList<View>	listViewDetailsPaged(String databaseName, Integer maxResults, String pageToken) Get paged list view details under this database.
List<String>	listViews(String databaseName) Get names of all views under this database.
protected TableMetadata	loadTableMetadata(Identifier identifier)
TableSchema	loadTableSchema(Identifier identifier)
Optional<CatalogLockContext>	lockContext()
void	markDonePartitions(Identifier identifier, List<Map<String,String>> partitions) Mark partitions done of the specify table.
static String	possibleHiveConfPath()
protected void	renameTableImpl(Identifier fromTable, Identifier toTable)
void	renameView(Identifier fromView, Identifier toView, boolean ignoreIfNotExists) Rename a view.
void	repairCatalog() Repair the entire Catalog, repair the metadata in the metastore consistent with the metadata in the filesystem, register missing tables in the metastore.
void	repairDatabase(String databaseName) Repair the entire database, repair the metadata in the metastore consistent with the metadata in the filesystem, register missing tables in the metastore.
void	repairTable(Identifier identifier) Repair the table, repair the metadata in the metastore consistent with the metadata in the filesystem.
<T> T	runWithLock(Identifier identifier, Callable<T> callable)
boolean	syncAllProperties()
String	warehouse()

Methods inherited from class org.apache.paimon.catalog.AbstractCatalog

alterDatabase, alterTable, assertMainBranch, commitSnapshot, createBranch, createDatabase, createTable, dropBranch, dropDatabase, dropTable, fastForward, fileIO, fileIO, getDatabase, getTable, listBranches, listDatabasesInFileSystem, listDatabasesPaged, listPartitionsPaged, listTableDetailsPaged, listTableDetailsPagedImpl, listTables, listTablesInFileSystem, listTablesPaged, loadSnapshot, lockEnabled, lockFactory, newDatabasePath, newDatabasePath, newTableLocation, options, renameTable, rollbackTo, supportsListObjectsPaged, supportsVersionManagement, tableExistsInFileSystem, tableSchemaInFileSystem

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.paimon.catalog.Catalog

alterTable, alterView, createDatabase, invalidateTable, listViewsPaged

Field Detail

TABLE_TYPE_PROP

public static final String TABLE_TYPE_PROP

See Also:

Constant Field Values

PAIMON_TABLE_IDENTIFIER

public static final String PAIMON_TABLE_IDENTIFIER

See Also:

Constant Field Values

HIVE_SITE_FILE

public static final String HIVE_SITE_FILE

See Also:

Constant Field Values

Constructor Detail

HiveCatalog

public HiveCatalog(FileIO fileIO,
                   org.apache.hadoop.hive.conf.HiveConf hiveConf,
                   String clientClassName,
                   String warehouse)

HiveCatalog

public HiveCatalog(FileIO fileIO,
                   org.apache.hadoop.hive.conf.HiveConf hiveConf,
                   String clientClassName,
                   Options options,
                   String warehouse)

Method Detail

defaultLockFactory

public Optional<CatalogLockFactory> defaultLockFactory()

Overrides:

defaultLockFactory in class AbstractCatalog

lockContext

public Optional<CatalogLockContext> lockContext()

Overrides:

lockContext in class AbstractCatalog

getTableLocation

public Path getTableLocation(Identifier identifier)

Overrides:

getTableLocation in class AbstractCatalog

listDatabases

public List<String> listDatabases()

Description copied from interface: Catalog

Get the names of all databases in this catalog.

Returns:

a list of the names of all databases

createDatabaseImpl

protected void createDatabaseImpl(String name,
                                  Map<String,String> properties)

Specified by:

createDatabaseImpl in class AbstractCatalog

getDatabaseImpl

public Database getDatabaseImpl(String name)
                         throws Catalog.DatabaseNotExistException

Specified by:

getDatabaseImpl in class AbstractCatalog

Throws:

Catalog.DatabaseNotExistException

createPartitions

public void createPartitions(Identifier identifier,
                             List<Map<String,String>> partitions)
                      throws Catalog.TableNotExistException

Description copied from interface: Catalog

Create partitions of the specify table. Ignore existing partitions.

Specified by:

createPartitions in interface Catalog

Overrides:

createPartitions in class AbstractCatalog

Parameters:

identifier - path of the table to create partitions

partitions - partitions to be created

Throws:

Catalog.TableNotExistException - if the table does not exist

dropPartitions

public void dropPartitions(Identifier identifier,
                           List<Map<String,String>> partitions)
                    throws Catalog.TableNotExistException

Description copied from interface: Catalog

Drop partitions of the specify table. Ignore non-existent partitions.

Specified by:

dropPartitions in interface Catalog

Overrides:

dropPartitions in class AbstractCatalog

Parameters:

identifier - path of the table to drop partitions

partitions - partitions to be deleted

Throws:

Catalog.TableNotExistException - if the table does not exist

alterPartitions

public void alterPartitions(Identifier identifier,
                            List<PartitionStatistics> partitions)
                     throws Catalog.TableNotExistException

Description copied from interface: Catalog

Alter partitions of the specify table. For non-existent partitions, partitions will be created directly.

Specified by:

alterPartitions in interface Catalog

Overrides:

alterPartitions in class AbstractCatalog

Parameters:

identifier - path of the table to alter partitions

partitions - partitions to be altered

Throws:

Catalog.TableNotExistException - if the table does not exist

markDonePartitions

public void markDonePartitions(Identifier identifier,
                               List<Map<String,String>> partitions)
                        throws Catalog.TableNotExistException

Description copied from interface: Catalog

Mark partitions done of the specify table. For non-existent partitions, partitions will be created directly.

Specified by:

markDonePartitions in interface Catalog

Overrides:

markDonePartitions in class AbstractCatalog

Parameters:

identifier - path of the table to mark done partitions

partitions - partitions to be marked done

Throws:

Catalog.TableNotExistException - if the table does not exist

listPartitions

public List<Partition> listPartitions(Identifier identifier)
                               throws Catalog.TableNotExistException

Description copied from interface: Catalog

Get Partition of all partitions of the table.

Specified by:

listPartitions in interface Catalog

Overrides:

listPartitions in class AbstractCatalog

Parameters:

identifier - path of the table to list partitions

Throws:

Catalog.TableNotExistException - if the table does not exist

dropDatabaseImpl

protected void dropDatabaseImpl(String name)

Specified by:

dropDatabaseImpl in class AbstractCatalog

alterDatabaseImpl

protected void alterDatabaseImpl(String name,
                                 List<PropertyChange> changes)

Specified by:

alterDatabaseImpl in class AbstractCatalog

listTablesImpl

protected List<String> listTablesImpl(String databaseName)

Specified by:

listTablesImpl in class AbstractCatalog

loadTableMetadata

protected TableMetadata loadTableMetadata(Identifier identifier)
                                   throws Catalog.TableNotExistException

Overrides:

loadTableMetadata in class AbstractCatalog

Throws:

Catalog.TableNotExistException

loadTableSchema

public TableSchema loadTableSchema(Identifier identifier)
                            throws Catalog.TableNotExistException

Specified by:

loadTableSchema in class AbstractCatalog

Throws:

Catalog.TableNotExistException

getView

public View getView(Identifier identifier)
             throws Catalog.ViewNotExistException

Description copied from interface: Catalog

Return a View identified by the given Identifier.

Parameters:

identifier - Path of the view

Returns:

The requested view

Throws:

Catalog.ViewNotExistException - if the target does not exist

createView

public void createView(Identifier identifier,
                       View view,
                       boolean ignoreIfExists)
                throws Catalog.ViewAlreadyExistException,
                       Catalog.DatabaseNotExistException

Description copied from interface: Catalog

Create a new view.

Parameters:

identifier - path of the view to be created

view - the view definition

ignoreIfExists - flag to specify behavior when a view already exists at the given path: if set to false, it throws a ViewAlreadyExistException, if set to true, do nothing.

Throws:

Catalog.ViewAlreadyExistException - if view already exists and ignoreIfExists is false

Catalog.DatabaseNotExistException - if the database in identifier doesn't exist

dropView

public void dropView(Identifier identifier,
                     boolean ignoreIfNotExists)
              throws Catalog.ViewNotExistException

Description copied from interface: Catalog

Drop a view.

Parameters:

identifier - Path of the view to be dropped

ignoreIfNotExists - Flag to specify behavior when the view does not exist: if set to false, throw an exception, if set to true, do nothing.

Throws:

Catalog.ViewNotExistException - if the view does not exist

listViews

public List<String> listViews(String databaseName)
                       throws Catalog.DatabaseNotExistException

Description copied from interface: Catalog

Get names of all views under this database. An empty list is returned if none exists.

Returns:

a list of the names of all views in this database

Throws:

Catalog.DatabaseNotExistException - if the database does not exist

listViewDetailsPaged

public PagedList<View> listViewDetailsPaged(String databaseName,
                                            Integer maxResults,
                                            String pageToken)
                                     throws Catalog.DatabaseNotExistException

Description copied from interface: Catalog

Get paged list view details under this database. An empty list is returned if none view exists.

Parameters:

databaseName - Name of the database to list views.

maxResults - Optional parameter indicating the maximum number of results to include in the result. If maxResults is not specified or set to 0, will return the default number of max results.

pageToken - Optional parameter indicating the next page token allows list to be start from a specific point.

Returns:

a list of the view details with provided page size (@param maxResults) in this database and next page token, or a list of the details of all views in this database if the catalog does not Catalog.supportsListObjectsPaged().

Throws:

Catalog.DatabaseNotExistException - if the database does not exist

renameView

public void renameView(Identifier fromView,
                       Identifier toView,
                       boolean ignoreIfNotExists)
                throws Catalog.ViewNotExistException,
                       Catalog.ViewAlreadyExistException

Description copied from interface: Catalog

Rename a view.

Parameters:

fromView - identifier of the view to rename

toView - new view identifier

Throws:

Catalog.ViewNotExistException - if the fromView does not exist

Catalog.ViewAlreadyExistException - if the toView already exists

createFormatTable

public void createFormatTable(Identifier identifier,
                              Schema schema)

Description copied from class: AbstractCatalog

Create a FormatTable identified by the given Identifier.

Overrides:

createFormatTable in class AbstractCatalog

Parameters:

identifier - Path of the table

schema - Schema of the table

dropTableImpl

protected void dropTableImpl(Identifier identifier,
                             List<Path> externalPaths)

Specified by:

dropTableImpl in class AbstractCatalog

createTableImpl

protected void createTableImpl(Identifier identifier,
                               Schema schema)

Specified by:

createTableImpl in class AbstractCatalog

renameTableImpl

protected void renameTableImpl(Identifier fromTable,
                               Identifier toTable)

Specified by:

renameTableImpl in class AbstractCatalog

alterTableImpl

protected void alterTableImpl(Identifier identifier,
                              List<SchemaChange> changes)
                       throws Catalog.TableNotExistException,
                              Catalog.ColumnAlreadyExistException,
                              Catalog.ColumnNotExistException

Specified by:

alterTableImpl in class AbstractCatalog

Throws:

Catalog.TableNotExistException

Catalog.ColumnAlreadyExistException

Catalog.ColumnNotExistException

caseSensitive

public boolean caseSensitive()

Description copied from interface: Catalog

Return a boolean that indicates whether this catalog is case-sensitive.

allowCustomTablePath

protected boolean allowCustomTablePath()

Overrides:

allowCustomTablePath in class AbstractCatalog

syncAllProperties

public boolean syncAllProperties()

repairCatalog

public void repairCatalog()

Description copied from interface: Catalog

Repair the entire Catalog, repair the metadata in the metastore consistent with the metadata in the filesystem, register missing tables in the metastore.

repairDatabase

public void repairDatabase(String databaseName)

Description copied from interface: Catalog

Repair the entire database, repair the metadata in the metastore consistent with the metadata in the filesystem, register missing tables in the metastore.

repairTable

public void repairTable(Identifier identifier)
                 throws Catalog.TableNotExistException

Description copied from interface: Catalog

Repair the table, repair the metadata in the metastore consistent with the metadata in the filesystem.

Throws:

Catalog.TableNotExistException

close

public void close()
           throws Exception

Throws:

Exception

warehouse

public String warehouse()

Specified by:

warehouse in class AbstractCatalog

catalogLoader

public CatalogLoader catalogLoader()

Description copied from interface: Catalog

Serializable loader to create catalog.

getHmsTable

public org.apache.hadoop.hive.metastore.api.Table getHmsTable(Identifier identifier)
                                                       throws Catalog.TableNotExistException

Throws:

Catalog.TableNotExistException

isView

public static boolean isView(org.apache.hadoop.hive.metastore.api.Table table)

getHmsClient

@VisibleForTesting
public org.apache.hadoop.hive.metastore.IMetaStoreClient getHmsClient()

runWithLock

public <T> T runWithLock(Identifier identifier,
                         Callable<T> callable)
                  throws Exception

Throws:

Exception

createHiveConf

public static org.apache.hadoop.hive.conf.HiveConf createHiveConf(@Nullable
                                                                  String hiveConfDir,
                                                                  @Nullable
                                                                  String hadoopConfDir,
                                                                  org.apache.hadoop.conf.Configuration defaultHadoopConf)

createHiveCatalog

public static Catalog createHiveCatalog(CatalogContext context)

createHiveConf

public static org.apache.hadoop.hive.conf.HiveConf createHiveConf(CatalogContext context)

possibleHiveConfPath

public static String possibleHiveConfPath()

getBatchGetTableSize

public int getBatchGetTableSize()