org.apache.paimon.spark

Class SparkUtils

java.lang.Object

org.apache.paimon.spark.SparkUtils

public class SparkUtils
extends Object

Utils for Spark.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	SparkUtils.CatalogAndIdentifier This mimics a class inside of Spark which is private inside of LookupCatalog.

Constructor Summary

Constructors

Constructor and Description
SparkUtils()

Method Summary

Modifier and Type	Method and Description
static <C,T> Pair<C,T>	catalogAndIdentifier(List<String> nameParts, java.util.function.Function<String,C> catalogProvider, java.util.function.BiFunction<String[],String,T> identifierProvider, C currentCatalog, String[] currentNamespace) A modified version of Spark's LookupCatalog.CatalogAndIdentifier.unapply Attempts to find the catalog and identifier a multipart identifier represents.
static SparkUtils.CatalogAndIdentifier	catalogAndIdentifier(org.apache.spark.sql.SparkSession spark, List<String> nameParts, org.apache.spark.sql.connector.catalog.CatalogPlugin defaultCatalog) A modified version of Spark's LookupCatalog.CatalogAndIdentifier.unapply Attempts to find the catalog and identifier a multipart identifier represents.
static SparkUtils.CatalogAndIdentifier	catalogAndIdentifier(org.apache.spark.sql.SparkSession spark, String name, org.apache.spark.sql.connector.catalog.CatalogPlugin defaultCatalog)
static SparkUtils.CatalogAndIdentifier	catalogAndIdentifier(String description, org.apache.spark.sql.SparkSession spark, String name, org.apache.spark.sql.connector.catalog.CatalogPlugin defaultCatalog)
static IOManager	createIOManager()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SparkUtils

public SparkUtils()

Method Detail

createIOManager

public static IOManager createIOManager()

catalogAndIdentifier

public static <C,T> Pair<C,T> catalogAndIdentifier(List<String> nameParts,
                                                   java.util.function.Function<String,C> catalogProvider,
                                                   java.util.function.BiFunction<String[],String,T> identifierProvider,
                                                   C currentCatalog,
                                                   String[] currentNamespace)

A modified version of Spark's LookupCatalog.CatalogAndIdentifier.unapply Attempts to find the catalog and identifier a multipart identifier represents.

Parameters:

nameParts - Multipart identifier representing a table

Returns:

The CatalogPlugin and Identifier for the table

catalogAndIdentifier

public static SparkUtils.CatalogAndIdentifier catalogAndIdentifier(org.apache.spark.sql.SparkSession spark,
                                                                   List<String> nameParts,
                                                                   org.apache.spark.sql.connector.catalog.CatalogPlugin defaultCatalog)

A modified version of Spark's LookupCatalog.CatalogAndIdentifier.unapply Attempts to find the catalog and identifier a multipart identifier represents.

Parameters:

spark - Spark session to use for resolution

nameParts - Multipart identifier representing a table

defaultCatalog - Catalog to use if none is specified

Returns:

The CatalogPlugin and Identifier for the table

catalogAndIdentifier

public static SparkUtils.CatalogAndIdentifier catalogAndIdentifier(org.apache.spark.sql.SparkSession spark,
                                                                   String name,
                                                                   org.apache.spark.sql.connector.catalog.CatalogPlugin defaultCatalog)
                                                            throws org.apache.spark.sql.catalyst.parser.ParseException

Throws:

org.apache.spark.sql.catalyst.parser.ParseException

catalogAndIdentifier

public static SparkUtils.CatalogAndIdentifier catalogAndIdentifier(String description,
                                                                   org.apache.spark.sql.SparkSession spark,
                                                                   String name,
                                                                   org.apache.spark.sql.connector.catalog.CatalogPlugin defaultCatalog)