org.apache.paimon.flink.action.cdc

Class MessageQueueSyncTableActionBase

java.lang.Object

org.apache.paimon.flink.action.ActionBase

org.apache.paimon.flink.action.cdc.SynchronizationActionBase

org.apache.paimon.flink.action.cdc.SyncTableActionBase

org.apache.paimon.flink.action.cdc.MessageQueueSyncTableActionBase

All Implemented Interfaces:

Action

Direct Known Subclasses:

KafkaSyncTableAction, PulsarSyncTableAction

public abstract class MessageQueueSyncTableActionBase
extends SyncTableActionBase

Base Action for synchronizing one message queue topic into one Paimon table.

If the specified Paimon table does not exist, this action will automatically create the table. Its schema will be derived from all specified topics. If the Paimon table already exists, its schema will be compared against the schema of all specified topics.

This action supports a limited number of schema changes. Unsupported schema changes will be ignored. Currently supported schema changes includes:

• Adding columns.
• Altering column types. More specifically,
  • altering from a string type (char, varchar, text) to another string type with longer length,
  • altering from a binary type (binary, varbinary, blob) to another binary type with longer length,
  • altering from an integer type (tinyint, smallint, int, bigint) to another integer type with wider range,
  • altering from a floating-point type (float, double) to another floating-point type with wider range,
  are supported. Other type changes will cause exceptions.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.paimon.flink.action.cdc.SyncTableActionBase

SyncTableActionBase.SchemaRetrievalException

Field Summary

Fields inherited from class org.apache.paimon.flink.action.cdc.SyncTableActionBase

computedColumnArgs, computedColumns, fileStoreTable, partitionKeys, primaryKeys, table

Fields inherited from class org.apache.paimon.flink.action.cdc.SynchronizationActionBase

allowUpperCase, cdcSourceConfig, database, metadataConverters, syncJobHandler, tableConfig, typeMapping

Fields inherited from class org.apache.paimon.flink.action.ActionBase

batchTEnv, catalog, catalogName, catalogOptions, env, flinkCatalog

Constructor Summary

Constructors

Constructor and Description
MessageQueueSyncTableActionBase(String warehouse, String database, String table, Map<String,String> catalogConfig, Map<String,String> mqConfig, SyncJobHandler.SourceType sourceType)

Method Summary

Modifier and Type	Method and Description
protected Schema	buildPaimonSchema(Schema retrievedSchema)
protected CdcTimestampExtractor	createCdcTimestampExtractor()
protected Schema	retrieveSchema()

Methods inherited from class org.apache.paimon.flink.action.cdc.SyncTableActionBase

beforeBuildingSourceSink, buildEventParserFactory, buildSink, fileStoreTable, recordParse, validateCaseSensitivity, withComputedColumnArgs, withPartitionKeys, withPartitionKeys, withPrimaryKeys, withPrimaryKeys

Methods inherited from class org.apache.paimon.flink.action.cdc.SynchronizationActionBase

alterTableOptions, build, buildSource, run, tableConfig, validateRuntimeExecutionMode, withMetadataColumns, withTableConfig, withTypeMapping

Methods inherited from class org.apache.paimon.flink.action.ActionBase

catalogConfig, catalogLoader, compatibleCheck, execute, initFlinkCatalog, initFlinkEnv, initPaimonCatalog, toPaimonTypes, withStreamExecutionEnvironment

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

MessageQueueSyncTableActionBase

public MessageQueueSyncTableActionBase(String warehouse,
                                       String database,
                                       String table,
                                       Map<String,String> catalogConfig,
                                       Map<String,String> mqConfig,
                                       SyncJobHandler.SourceType sourceType)

Method Detail

retrieveSchema

protected Schema retrieveSchema()
                         throws Exception

Specified by:

retrieveSchema in class SyncTableActionBase

Throws:

Exception

createCdcTimestampExtractor

protected CdcTimestampExtractor createCdcTimestampExtractor()

Overrides:

createCdcTimestampExtractor in class SynchronizationActionBase

buildPaimonSchema

protected Schema buildPaimonSchema(Schema retrievedSchema)

Overrides:

buildPaimonSchema in class SyncTableActionBase