org.apache.paimon.flink.action

Class RemoveUnexistingFilesAction

java.lang.Object

org.apache.paimon.flink.action.ActionBase

org.apache.paimon.flink.action.TableActionBase

org.apache.paimon.flink.action.RemoveUnexistingFilesAction

All Implemented Interfaces:

Action

public class RemoveUnexistingFilesAction
extends TableActionBase

Action to remove unexisting data files from manifest entries. It has the following use cases:

• There is currently a known case when unexisting data file might be written into manifest. Consider a write-only job (W) plus a dedicated compaction job (C):
  1. W commits a snapshot with file F. Then W constantly fails and restarts, each time before we can retry the commit (or W is stopped, creating Flink savepoint S).
  2. C compacts F into a larger file, so F is now deleted from the manifest.
  3. Before the compact snapshot expires, but after all snapshots created by W expires, W comes back to normal (or restarted from savepoint S). As W cannot find its previous snapshot, it assumes that this snapshot has not been committed (see FileStoreCommitImpl.filterCommitted(java.util.List<org.apache.paimon.manifest.ManifestCommittable>) for more detail), so file F is committed to the manifest once again.
  4. When the compact snapshot expires, file F will be deleted from the file system. Now F is in the manifest, but not on the file system. With this situation, user might want to remove F from the manifest to continue reading the table.
• User deletes a data file by mistake (for example, by incorrectly setting the time threshold for orphan files cleaning). If the user can tolerate skipping some records when consuming this table, he can also use this action to remove the file from manifest.

Note that user is on his own risk using this procedure, which may cause data loss when used outside from the use cases above.

Field Summary

Fields inherited from class org.apache.paimon.flink.action.TableActionBase

identifier, table

Fields inherited from class org.apache.paimon.flink.action.ActionBase

batchTEnv, catalog, catalogName, catalogOptions, env, flinkCatalog

Constructor Summary

Constructors

Constructor and Description
RemoveUnexistingFilesAction(String databaseName, String tableName, Map<String,String> catalogConfig)

Method Summary

Modifier and Type	Method and Description
void	build() Builds the action within the given Flink Stream Execution Environment.
org.apache.flink.streaming.api.datastream.DataStream<String>	buildDataStream()
RemoveUnexistingFilesAction	dryRun()
void	run() The execution method of the action.
RemoveUnexistingFilesAction	withParallelism(int parallelism)

Methods inherited from class org.apache.paimon.flink.action.TableActionBase

batchSink

Methods inherited from class org.apache.paimon.flink.action.ActionBase

catalogConfig, catalogLoader, compatibleCheck, execute, initFlinkCatalog, initFlinkEnv, initPaimonCatalog, toPaimonTypes, withStreamExecutionEnvironment

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

RemoveUnexistingFilesAction

public RemoveUnexistingFilesAction(String databaseName,
                                   String tableName,
                                   Map<String,String> catalogConfig)

Method Detail

dryRun

public RemoveUnexistingFilesAction dryRun()

withParallelism

public RemoveUnexistingFilesAction withParallelism(int parallelism)

build

public void build()
           throws Exception

Description copied from interface: Action

Builds the action within the given Flink Stream Execution Environment.

This method is responsible for setting up any necessary configurations or resources needed for the action to run. It is called before the `run` method to prepare the environment for execution.

By default, this method is empty and can be overridden by subclasses to provide custom setup logic.

Throws:

Exception - If an error occurs during the build process.

buildDataStream

public org.apache.flink.streaming.api.datastream.DataStream<String> buildDataStream()
                                                                             throws Exception

Throws:

Exception

run

public void run()
         throws Exception

Description copied from interface: Action

The execution method of the action.

Throws:

Exception