org.apache.paimon.operation

Class FileDeletionBase<T extends Snapshot>

java.lang.Object

org.apache.paimon.operation.FileDeletionBase<T>

Direct Known Subclasses:

ChangelogDeletion, SnapshotDeletion, TagDeletion

public abstract class FileDeletionBase<T extends Snapshot>
extends Object

Base class for file deletion including methods for clean data files, manifest files and empty data directories.

Field Summary

Fields

Modifier and Type	Field and Description
protected boolean	changelogDecoupled
protected Map<BinaryRow,Set<Integer>>	deletionBuckets
protected FileIO	fileIO
protected IndexFileHandler	indexFileHandler
protected ManifestFile	manifestFile
protected ManifestList	manifestList
protected FileStorePathFactory	pathFactory
protected StatsFileHandler	statsFileHandler

Constructor Summary

Constructors

Constructor and Description
FileDeletionBase(FileIO fileIO, FileStorePathFactory pathFactory, ManifestFile manifestFile, ManifestList manifestList, IndexFileHandler indexFileHandler, StatsFileHandler statsFileHandler, boolean cleanEmptyDirectories, int deleteFileThreadNum)

Method Summary

Modifier and Type	Method and Description
protected void	addMergedDataFiles(Map<BinaryRow,Map<Integer,Set<String>>> dataFiles, Snapshot snapshot) NOTE: This method is used for building data file skipping set.
void	cleanEmptyDirectories() Try to delete data directories that may be empty after data file deletion.
void	cleanUnusedDataFiles(String manifestList, java.util.function.Predicate<ManifestEntry> skipper)
abstract void	cleanUnusedDataFiles(T snapshot, java.util.function.Predicate<ManifestEntry> skipper) Clean data files that will not be used anymore in the snapshot.
void	cleanUnusedIndexManifests(Snapshot snapshot, Set<String> skippingSet)
void	cleanUnusedManifestList(String manifestName, Set<String> skippingSet)
protected void	cleanUnusedManifests(Snapshot snapshot, Set<String> skippingSet, boolean deleteDataManifestLists, boolean deleteChangelog)
abstract void	cleanUnusedManifests(T snapshot, Set<String> skippingSet) Clean metadata files that will not be used anymore of a snapshot, including data manifests, index manifests and manifest lists.
void	cleanUnusedStatisticsManifests(Snapshot snapshot, Set<String> skippingSet)
protected boolean	containsDataFile(Map<BinaryRow,Map<Integer,Set<String>>> dataFiles, ManifestEntry testee)
java.util.function.Predicate<ManifestEntry>	createDataFileSkipperForTags(List<Snapshot> taggedSnapshots, long expiringSnapshotId)
void	deleteAddedDataFiles(String manifestListName) Delete added file in the manifest list files.
protected <F> void	deleteFiles(Collection<F> files, java.util.function.Consumer<F> deletion)
protected void	doCleanUnusedDataFile(Map<Path,Pair<ManifestEntry,List<Path>>> dataFileToDelete, java.util.function.Predicate<ManifestEntry> skipper)
protected void	getDataFileToDelete(Map<Path,Pair<ManifestEntry,List<Path>>> dataFileToDelete, List<ManifestEntry> dataFileEntries)
Set<String>	manifestSkippingSet(List<Snapshot> skippingSnapshots)
Set<String>	manifestSkippingSet(Snapshot skippingSnapshot) Changelogs were not checked.
protected List<String>	readManifestFileNames(List<ManifestFileMeta> manifestFileMetas)
protected Collection<ManifestEntry>	readMergedDataFiles(Snapshot snapshot)
protected void	recordDeletionBuckets(ManifestEntry entry)
void	setChangelogDecoupled(boolean changelogDecoupled)
protected List<String>	tryReadDataManifests(Snapshot snapshot)
protected List<ManifestFileMeta>	tryReadManifestList(String manifestListName) It is possible that a job was killed during expiration and some manifest files have been deleted, so if the clean methods need to get manifests of a snapshot to be cleaned, we should try to read manifests and return empty list if failed instead of calling ManifestList.readDataManifests(org.apache.paimon.Snapshot) directly.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

fileIO

protected final FileIO fileIO

pathFactory

protected final FileStorePathFactory pathFactory

manifestFile

protected final ManifestFile manifestFile

manifestList

protected final ManifestList manifestList

indexFileHandler

protected final IndexFileHandler indexFileHandler

statsFileHandler

protected final StatsFileHandler statsFileHandler

deletionBuckets

protected final Map<BinaryRow,Set<Integer>> deletionBuckets

changelogDecoupled

protected boolean changelogDecoupled

Constructor Detail

FileDeletionBase

public FileDeletionBase(FileIO fileIO,
                        FileStorePathFactory pathFactory,
                        ManifestFile manifestFile,
                        ManifestList manifestList,
                        IndexFileHandler indexFileHandler,
                        StatsFileHandler statsFileHandler,
                        boolean cleanEmptyDirectories,
                        int deleteFileThreadNum)

Method Detail

cleanUnusedDataFiles

public abstract void cleanUnusedDataFiles(T snapshot,
                                          java.util.function.Predicate<ManifestEntry> skipper)

Clean data files that will not be used anymore in the snapshot.

Parameters:

snapshot - Snapshot that will be cleaned

skipper - if the test result of a data file is true, it will be skipped when deleting; else it will be deleted

cleanUnusedManifests

public abstract void cleanUnusedManifests(T snapshot,
                                          Set<String> skippingSet)

Clean metadata files that will not be used anymore of a snapshot, including data manifests, index manifests and manifest lists.

Parameters:

snapshot - Snapshot that will be cleaned

skippingSet - manifests that should not be deleted

setChangelogDecoupled

public void setChangelogDecoupled(boolean changelogDecoupled)

cleanEmptyDirectories

public void cleanEmptyDirectories()

Try to delete data directories that may be empty after data file deletion.

recordDeletionBuckets

protected void recordDeletionBuckets(ManifestEntry entry)

cleanUnusedDataFiles

public void cleanUnusedDataFiles(String manifestList,
                                 java.util.function.Predicate<ManifestEntry> skipper)

doCleanUnusedDataFile

protected void doCleanUnusedDataFile(Map<Path,Pair<ManifestEntry,List<Path>>> dataFileToDelete,
                                     java.util.function.Predicate<ManifestEntry> skipper)

getDataFileToDelete

protected void getDataFileToDelete(Map<Path,Pair<ManifestEntry,List<Path>>> dataFileToDelete,
                                   List<ManifestEntry> dataFileEntries)

deleteAddedDataFiles

public void deleteAddedDataFiles(String manifestListName)

Delete added file in the manifest list files. Added files marked as "ADD" in manifests.

Parameters:

manifestListName - name of manifest list

cleanUnusedStatisticsManifests

public void cleanUnusedStatisticsManifests(Snapshot snapshot,
                                           Set<String> skippingSet)

cleanUnusedIndexManifests

public void cleanUnusedIndexManifests(Snapshot snapshot,
                                      Set<String> skippingSet)

cleanUnusedManifestList

public void cleanUnusedManifestList(String manifestName,
                                    Set<String> skippingSet)

cleanUnusedManifests

protected void cleanUnusedManifests(Snapshot snapshot,
                                    Set<String> skippingSet,
                                    boolean deleteDataManifestLists,
                                    boolean deleteChangelog)

createDataFileSkipperForTags

public java.util.function.Predicate<ManifestEntry> createDataFileSkipperForTags(List<Snapshot> taggedSnapshots,
                                                                                long expiringSnapshotId)
                                                                         throws Exception

Throws:

Exception

tryReadManifestList

protected List<ManifestFileMeta> tryReadManifestList(String manifestListName)

It is possible that a job was killed during expiration and some manifest files have been deleted, so if the clean methods need to get manifests of a snapshot to be cleaned, we should try to read manifests and return empty list if failed instead of calling ManifestList.readDataManifests(org.apache.paimon.Snapshot) directly.

tryReadDataManifests

protected List<String> tryReadDataManifests(Snapshot snapshot)

readManifestFileNames

protected List<String> readManifestFileNames(List<ManifestFileMeta> manifestFileMetas)

addMergedDataFiles

protected void addMergedDataFiles(Map<BinaryRow,Map<Integer,Set<String>>> dataFiles,
                                  Snapshot snapshot)
                           throws IOException

NOTE: This method is used for building data file skipping set. If failed to read some manifests, it will throw exception which callers must handle.

Throws:

IOException

readMergedDataFiles

protected Collection<ManifestEntry> readMergedDataFiles(Snapshot snapshot)
                                                 throws IOException

Throws:

IOException

containsDataFile

protected boolean containsDataFile(Map<BinaryRow,Map<Integer,Set<String>>> dataFiles,
                                   ManifestEntry testee)

manifestSkippingSet

public Set<String> manifestSkippingSet(Snapshot skippingSnapshot)

Changelogs were not checked. Let the subclass determine whether to delete them.

manifestSkippingSet

public Set<String> manifestSkippingSet(List<Snapshot> skippingSnapshots)

deleteFiles

protected <F> void deleteFiles(Collection<F> files,
                               java.util.function.Consumer<F> deletion)