Data Lineage
Paimon implements FLIP-314 to expose lineage information through Flink's native LineageVertexProvider interface. This allows lineage consumers (such as OpenLineage) to automatically discover Paimon datasets in the Flink job graph.
Data Lineage support is available only for Flink 2.0 and above.
Table API / SQL
When using Paimon tables via Flink SQL or Table API, lineage is automatically reported for both source and sink tables.
Apart from the table lineage information provided by Flink natively (name, schema), Paimon enriches each dataset with a namespace and a config facet containing core table options and Iceberg metadata options.
Full Table API lineage support requires this upstream Flink change which is pending merge. Without it, Flink will not call getLineageVertex() for DataStreamScanProvider and DataStreamSinkProvider, which can lead to potentially missing Paimon input or output datasets. This applies to all Flink 2.x versions that do not yet include this change.
DataStream API
Paimon also supports lineage for the DataStream API. Lineage is automatically reported for sources built via FlinkSourceBuilder and for all sinks that extend FlinkSink.
Similar to the Table API, it reports all lineage information including name, namespace, schema facet, and config facet.
Configuration
To consume lineage events, a JobStatusChangedListener must be added in your Flink configuration.
For example, to configure with OpenLineage:
execution.job-status-changed-listeners = io.openlineage.flink.listener.OpenLineageJobStatusChangedListenerFactory