Overview #
Apache Paimon utilizes the same pluggable file systems as Apache Flink. Users can follow the standard plugin mechanism to configure the plugin structure if using Flink as compute engine. However, for other engines like Spark or Hive, the provided opt jars (by Flink) may get conflicts and cannot be used directly. It is not convenient for users to fix class conflicts, thus Paimon provides the self-contained and engine-unified FileSystem pluggable jars for user to query tables from Spark/Hive side.
Supported FileSystems #
| FileSystem | URI Scheme | Pluggable | Description |
|---|---|---|---|
| Local File System | file:// | N | Built-in Support |
| HDFS | hdfs:// | N | Built-in Support, ensure that the cluster is in the hadoop environment |
| Aliyun OSS | oss:// | Y | |
| S3 | s3:// | Y |
Dependency #
We recommend you to download the jar directly: Download Link.
You can also manually build bundled jar from the source code.
To build from source code, clone the git repository.
Build shaded jar with the following command.
mvn clean install -DskipTests
You can find the shaded jars under
./paimon-filesystems/paimon-${fs}/target/paimon-${fs}-0.7.0-incubating.jar.