org.apache.paimon.operation

Class OrphanFilesClean

java.lang.Object

org.apache.paimon.operation.OrphanFilesClean

public class OrphanFilesClean
extends Object

To remove the data files and metadata files that are not used by table (so-called "orphan files").

It will ignore exception when listing all files because it's OK to not delete unread files.

To avoid deleting newly written files, it only deletes orphan files older than 1 day by default. The interval can be modified by olderThan(java.lang.String).

To avoid conflicting with snapshot expiration, tag deletion and rollback, it will skip the snapshot/tag when catching FileNotFoundException in the process of listing used files.

To avoid deleting files that are used but not read by mistaken, it will stop removing process when failed to read used files.

Constructor Summary

Constructors

Constructor and Description
OrphanFilesClean(FileStoreTable table)

Method Summary

Modifier and Type	Method and Description
List<Path>	clean()
static List<OrphanFilesClean>	createOrphanFilesCleans(Catalog catalog, Map<String,String> tableConfig, String databaseName, String tableName)
static String[]	executeOrphanFilesClean(List<OrphanFilesClean> tableCleans)
OrphanFilesClean	fileCleaner(java.util.function.Consumer<Path> fileCleaner)
OrphanFilesClean	olderThan(String timestamp)
static List<String>	showDeletedFiles(List<Path> deleteFiles, int showLimit)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

OrphanFilesClean

public OrphanFilesClean(FileStoreTable table)

Method Detail

olderThan

public OrphanFilesClean olderThan(String timestamp)

fileCleaner

public OrphanFilesClean fileCleaner(java.util.function.Consumer<Path> fileCleaner)

clean

public List<Path> clean()
                 throws IOException,
                        ExecutionException,
                        InterruptedException

Throws:

IOException

ExecutionException

InterruptedException

showDeletedFiles

public static List<String> showDeletedFiles(List<Path> deleteFiles,
                                            int showLimit)

createOrphanFilesCleans

public static List<OrphanFilesClean> createOrphanFilesCleans(Catalog catalog,
                                                             Map<String,String> tableConfig,
                                                             String databaseName,
                                                             @Nullable
                                                             String tableName)
                                                      throws Catalog.DatabaseNotExistException,
                                                             Catalog.TableNotExistException

Throws:

Catalog.DatabaseNotExistException

Catalog.TableNotExistException

executeOrphanFilesClean

public static String[] executeOrphanFilesClean(List<OrphanFilesClean> tableCleans)