//paimon.apache.org/docs/1.4/flink/ Recent content in Engine Flink on Apache Paimon Hugo -- gohugo.io en-us //paimon.apache.org/docs/1.4/flink/quick-start/ Mon, 01 Jan 0001 00:00:00 +0000 //paimon.apache.org/docs/1.4/flink/quick-start/ Quick Start # This documentation is a guide for using Paimon in Flink. Jars # Paimon currently supports Flink 2.2, 2.1, 2.0, 1.20, 1.19, 1.18, 1.17, 1.16. We recommend the latest Flink version for a better experience. Download the jar file with corresponding version. Currently, paimon provides two types jar: one of which(the bundled jar) is used for read/write data, and the other(action jar) for operations such as manually compaction, Version Type Jar Flink 2. //paimon.apache.org/docs/1.4/flink/sql-ddl/ Mon, 01 Jan 0001 00:00:00 +0000 //paimon.apache.org/docs/1.4/flink/sql-ddl/ SQL DDL # Create Catalog # Paimon catalogs currently support three types of metastores: filesystem metastore (default), which stores both metadata and table files in filesystems. hive metastore, which additionally stores metadata in Hive metastore. Users can directly access the tables from Hive. jdbc metastore, which additionally stores metadata in relational databases such as MySQL, Postgres, etc. See CatalogOptions for detailed options when creating a catalog. //paimon.apache.org/docs/1.4/flink/sql-write/ Mon, 01 Jan 0001 00:00:00 +0000 //paimon.apache.org/docs/1.4/flink/sql-write/ SQL Write # Syntax # INSERT { INTO | OVERWRITE } table_identifier [ part_spec ] [ column_list ] { value_expr | query }; For more information, please check the syntax document: Flink INSERT Statement INSERT INTO # Use INSERT INTO to apply records and changes to tables. INSERT INTO my_table SELECT ... INSERT INTO supports both batch and streaming mode. In Streaming mode, by default, it will also perform compaction, snapshot expiration, and even partition expiration in Flink Sink (if it is configured). //paimon.apache.org/docs/1.4/flink/sql-query/ Mon, 01 Jan 0001 00:00:00 +0000 //paimon.apache.org/docs/1.4/flink/sql-query/ SQL Query # Just like all other tables, Paimon tables can be queried with SELECT statement. Batch Query # Paimon’s batch read returns all the data in a snapshot of the table. By default, batch reads return the latest snapshot. -- Flink SQL SET 'execution.runtime-mode' = 'batch'; Batch Time Travel # Paimon batch reads with time travel can specify a snapshot or a tag and read the corresponding data. //paimon.apache.org/docs/1.4/flink/consumer-id/ Mon, 01 Jan 0001 00:00:00 +0000 //paimon.apache.org/docs/1.4/flink/consumer-id/ Consumer ID # Consumer id can help you accomplish the following two things: Safe consumption: When deciding whether a snapshot has expired, Paimon looks at all the consumers of the table in the file system, and if there are consumers that still depend on this snapshot, then this snapshot will not be deleted by expiration. Resume from breakpoint: When previous job is stopped, the newly started job can continue to consume from the previous progress without resuming from the state. //paimon.apache.org/docs/1.4/flink/sql-lookup/ Mon, 01 Jan 0001 00:00:00 +0000 //paimon.apache.org/docs/1.4/flink/sql-lookup/ Lookup Joins # Lookup Joins are a type of join in streaming queries. It is used to enrich a table with data that is queried from Paimon. The join requires one table to have a processing time attribute and the other table to be backed by a lookup source connector. Paimon supports lookup joins on tables with primary keys and append tables in Flink. The following example illustrates this feature. //paimon.apache.org/docs/1.4/flink/sql-alter/ Mon, 01 Jan 0001 00:00:00 +0000 //paimon.apache.org/docs/1.4/flink/sql-alter/ Altering Tables # Changing/Adding Table Properties # The following SQL sets write-buffer-size table property to 256 MB. ALTER TABLE my_table SET ( 'write-buffer-size' = '256 MB' ); Removing Table Properties # The following SQL removes write-buffer-size table property. ALTER TABLE my_table RESET ('write-buffer-size'); Changing/Adding Table Comment # The following SQL changes comment of table my_table to table comment. ALTER TABLE my_table SET ( 'comment' = 'table comment' ); Removing Table Comment # The following SQL removes table comment. //paimon.apache.org/docs/1.4/flink/default-value/ Mon, 01 Jan 0001 00:00:00 +0000 //paimon.apache.org/docs/1.4/flink/default-value/ Default Value # Paimon allows specifying default values for columns. When users write to these tables without explicitly providing values for certain columns, Paimon automatically generates default values for these columns. Create Table # Flink SQL does not have native support for default values, so we can only create a table without default values: CREATE TABLE my_table ( a BIGINT, b STRING, c INT, tags ARRAY<STRING>, properties MAP<STRING, STRING>, nested ROW<x INT, y STRING> ); We support the procedure of modifying column default values in Flink. //paimon.apache.org/docs/1.4/flink/procedures/ Mon, 01 Jan 0001 00:00:00 +0000 //paimon.apache.org/docs/1.4/flink/procedures/ Procedures # Flink 1.18 and later versions support Call Statements, which make it easier to manipulate data and metadata of Paimon table by writing SQLs instead of submitting Flink jobs. In 1.18, the procedure only supports passing arguments by position. You must pass all arguments in order, and if you don’t want to pass some arguments, you must use '' as placeholder. For example, if you want to compact table default. //paimon.apache.org/docs/1.4/flink/action-jars/ Mon, 01 Jan 0001 00:00:00 +0000 //paimon.apache.org/docs/1.4/flink/action-jars/ Action Jars # After the Flink Local Cluster has been started, you can execute the action jar by using the following command. <FLINK_HOME>/bin/flink run \ /path/to/paimon-flink-action-1.4.0.jar \ <action> <args> The following command is used to compact a table. <FLINK_HOME>/bin/flink run \ /path/to/paimon-flink-action-1.4.0.jar \ compact \ --path <TABLE_PATH> Merging into table # Paimon supports “MERGE INTO” via submitting the ‘merge_into’ job through flink run. //paimon.apache.org/docs/1.4/flink/savepoint/ Mon, 01 Jan 0001 00:00:00 +0000 //paimon.apache.org/docs/1.4/flink/savepoint/ Savepoint # Paimon has its own snapshot management, this may conflict with Flink’s checkpoint management, causing exceptions when restoring from savepoint (don’t worry, it will not cause the storage to be damaged). It is recommended that you use the following methods to savepoint: Use Flink Stop with savepoint. Use Paimon Tag with Flink Savepoint, and rollback-to-tag before restoring from savepoint. Stop with savepoint # This feature of Flink ensures that the last checkpoint is fully processed, which means there will be no more uncommitted metadata left.