org.apache.paimon.operation

Class OrphanFilesClean

java.lang.Object

org.apache.paimon.operation.OrphanFilesClean

All Implemented Interfaces:

Serializable

Direct Known Subclasses:

FlinkOrphanFilesClean, LocalOrphanFilesClean

public abstract class OrphanFilesClean
extends Object
implements Serializable

To remove the data files and metadata files that are not used by table (so-called "orphan files").

It will ignore exception when listing all files because it's OK to not delete unread files.

To avoid deleting newly written files, it only deletes orphan files older than olderThanMillis (1 day by default).

To avoid conflicting with snapshot expiration, tag deletion and rollback, it will skip the snapshot/tag when catching FileNotFoundException in the process of listing used files.

To avoid deleting files that are used but not read by mistaken, it will stop removing process when failed to read used files.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static interface	OrphanFilesClean.ReaderWithIOException<T> A helper functional interface for method retryReadingFiles(org.apache.paimon.operation.OrphanFilesClean.ReaderWithIOException<T>, T).

Field Summary

Fields

Modifier and Type	Field and Description
protected SerializableConsumer<Path>	fileCleaner
protected FileIO	fileIO
protected Path	location
protected static org.slf4j.Logger	LOG
protected long	olderThanMillis
protected int	partitionKeysNum
protected static int	READ_FILE_RETRY_INTERVAL
protected static int	READ_FILE_RETRY_NUM
protected FileStoreTable	table

Constructor Summary

Constructors

Constructor and Description
OrphanFilesClean(FileStoreTable table, long olderThanMillis, SerializableConsumer<Path> fileCleaner)

Method Summary

Modifier and Type	Method and Description
protected void	cleanSnapshotDir(List<String> branches, java.util.function.Consumer<Path> deletedFileConsumer)
protected void	collectWithoutDataFile(String branch, Snapshot snapshot, java.util.function.Consumer<String> usedFileConsumer, java.util.function.Consumer<String> manifestConsumer)
protected void	collectWithoutDataFileWithManifestFlag(String branch, Snapshot snapshot, java.util.function.Consumer<Pair<String,Boolean>> usedFileWithFlagConsumer)
static SerializableConsumer<Path>	createFileCleaner(Catalog catalog, Boolean dryRun)
protected List<Path>	listPaimonFileDirs() List directories that contains data files and manifest files.
protected boolean	oldEnough(FileStatus status)
static long	olderThanMillis(String olderThan)
protected static <T> T	retryReadingFiles(OrphanFilesClean.ReaderWithIOException<T> reader, T defaultValue) Retry reading files when IOException was thrown by the reader.
protected Set<Snapshot>	safelyGetAllSnapshots(String branch)
protected List<FileStatus>	tryBestListingDirs(Path dir) If failed to list directory, just return an empty result because it's OK to not delete them.
protected List<String>	validBranches()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

protected static final org.slf4j.Logger LOG

READ_FILE_RETRY_NUM

protected static final int READ_FILE_RETRY_NUM

See Also:

Constant Field Values

READ_FILE_RETRY_INTERVAL

protected static final int READ_FILE_RETRY_INTERVAL

See Also:

Constant Field Values

table

protected final FileStoreTable table

fileIO

protected final FileIO fileIO

olderThanMillis

protected final long olderThanMillis

fileCleaner

protected final SerializableConsumer<Path> fileCleaner

partitionKeysNum

protected final int partitionKeysNum

location

protected final Path location

Constructor Detail

OrphanFilesClean

public OrphanFilesClean(FileStoreTable table,
                        long olderThanMillis,
                        SerializableConsumer<Path> fileCleaner)

Method Detail

validBranches

protected List<String> validBranches()

cleanSnapshotDir

protected void cleanSnapshotDir(List<String> branches,
                                java.util.function.Consumer<Path> deletedFileConsumer)

safelyGetAllSnapshots

protected Set<Snapshot> safelyGetAllSnapshots(String branch)
                                       throws IOException

Throws:

IOException

collectWithoutDataFile

protected void collectWithoutDataFile(String branch,
                                      Snapshot snapshot,
                                      java.util.function.Consumer<String> usedFileConsumer,
                                      java.util.function.Consumer<String> manifestConsumer)
                               throws IOException

Throws:

IOException

collectWithoutDataFileWithManifestFlag

protected void collectWithoutDataFileWithManifestFlag(String branch,
                                                      Snapshot snapshot,
                                                      java.util.function.Consumer<Pair<String,Boolean>> usedFileWithFlagConsumer)
                                               throws IOException

Throws:

IOException

listPaimonFileDirs

protected List<Path> listPaimonFileDirs()

List directories that contains data files and manifest files.

tryBestListingDirs

protected List<FileStatus> tryBestListingDirs(Path dir)

If failed to list directory, just return an empty result because it's OK to not delete them.

retryReadingFiles

protected static <T> T retryReadingFiles(OrphanFilesClean.ReaderWithIOException<T> reader,
                                         T defaultValue)
                                  throws IOException

Retry reading files when IOException was thrown by the reader. If the exception is FileNotFoundException, return default value. Finally, if retry times reaches the limits, rethrow the IOException.

Throws:

IOException

oldEnough

protected boolean oldEnough(FileStatus status)

createFileCleaner

public static SerializableConsumer<Path> createFileCleaner(Catalog catalog,
                                                           @Nullable
                                                           Boolean dryRun)

olderThanMillis

public static long olderThanMillis(@Nullable
                                   String olderThan)