Skip to main content

Installation

Match the connector to the Spark minor version and Scala binary version of your distribution. The Java versions below describe how the connector artifacts are built; your Spark distribution may require a newer Java runtime. Check its runtime requirements before starting Spark.

  • Spark 4.x (including 4.1, 4.0) : Pre-built with Java 17 and Scala 2.13

  • Spark 3.x (including 3.5, 3.4, 3.3, 3.2) : Pre-built with Java 8 and Scala 2.12/2.13

Download the Connector​

Choose the JAR matching your Spark and Scala versions:

VersionJar (Scala 2.12)Jar (Scala 2.13)
Spark 4.1-JAR
Spark 4.0-JAR
Spark 3.5JARJAR
Spark 3.4JARJAR
Spark 3.3JARJAR
Spark 3.2JARJAR

Build from Source​

You can also build the bundled JAR from source.

To build from source code, clone the git repository, then build the bundled jar with the following command.

# build paimon spark 3.5 with scala 2.12
mvn clean package -DskipTests -pl paimon-spark/paimon-spark-3.5 -am -Pspark3

# build paimon spark 3.5 with scala 2.13
mvn clean package -DskipTests -pl paimon-spark/paimon-spark-3.5 -am -Pspark3,scala-2.13

# build paimon spark 4.0
mvn clean package -DskipTests -pl paimon-spark/paimon-spark-4.0 -am -Pspark4

# build paimon spark 4.1
mvn clean package -DskipTests -pl paimon-spark/paimon-spark-4.1 -am -Pspark4

For Spark 3.5, you can find the bundled jar in ./paimon-spark/paimon-spark-3.5/target/paimon-spark-3.5_2.12-2.2-SNAPSHOT.jar.

Add the Connector to Spark​

Pass --jars /path/to/paimon-connector.jar when starting spark-sql or spark-shell, or put the JAR in your Spark installation's jars directory. For published releases, Maven coordinates are another option:

spark-sql --packages org.apache.paimon:paimon-spark-3.5_2.12:2.2-SNAPSHOT

Snapshot artifacts are hosted in the Apache snapshot repository linked above. Download the snapshot JAR and use --jars for the local quick start.

Configure Storage and a Catalog​

Use the complete command in Quick Start for local storage, or configure Catalogs for your deployment. When using HDFS, set HADOOP_HOME or HADOOP_CONF_DIR so Spark can load the Hadoop configuration. Add the filesystem dependencies required by your warehouse as described in Filesystems.