Flink
Use Flink to ingest streams into Paimon, query table snapshots, follow changes, and enrich events with lookup joins. Start with Quick Start for a complete local example or Installation to prepare an existing cluster.
Streaming writers commit data as checkpoints complete. Batch readers select a table snapshot; streaming readers can load a snapshot and then follow changes. Lookup joins use a Paimon table to enrich records from another stream. The table's layout and changelog configuration determine which read paths and change semantics are available.
Find the Right Guide
| What you want to do | Start here | Continue with |
|---|---|---|
| Run your first pipeline | Quick Start | Installation |
| Define a catalog, keys, and partitions | SQL DDL | SQL Alter, Default Value |
| Ingest or modify records | SQL Write | CDC Ingestion |
| Read current or historical data | SQL Query | Snapshots, Tags |
| Enrich a stream | Lookup Joins | Query Service |
| Tune and operate a job | Runtime Configuration | Consumer ID, Savepoint, Data Lineage |
| Run a maintenance operation | Procedures | Action Jars |
| Diagnose unexpected results or recovery failures | Troubleshooting | Runtime Configuration |
| Develop with Java | Flink API | Table Concepts |
Choose the Table Semantics
- Append tables: store rows without merging by primary key. Start here for append-only events and logs.
- Primary-key tables: merge changes to the same key. Choose the merge engine and changelog producer for CDC and upsert workloads.
- Data Evolution: use row tracking and column-level updates for supported append-table workloads, including multimodal data.
SQL or Action Jar?
Use SQL for table definitions and read/write queries. On Flink 1.18+, use CALL for the
operations listed in Procedures. Named procedure arguments require Flink 1.19+.
Use an action jar when submitting the corresponding operation from the command line.
Both interfaces can start Flink jobs, so select the runtime mode and options for the operation.