org.apache.paimon.utils

Class SnapshotManager

java.lang.Object

org.apache.paimon.utils.SnapshotManager

All Implemented Interfaces:

Serializable

public class SnapshotManager
extends Object
implements Serializable

Manager for Snapshot, providing utility methods related to paths and snapshot hints.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static String	EARLIEST
static String	LATEST

Constructor Summary

Constructors

Constructor and Description
SnapshotManager(FileIO fileIO, Path tablePath)
SnapshotManager(FileIO fileIO, Path tablePath, String branchName) Specify the default branch for data writing.
SnapshotManager(FileIO fileIO, Path tablePath, String branchName, org.apache.paimon.shade.caffeine2.com.github.benmanes.caffeine.cache.Cache<Path,Snapshot> cache)

Method Summary

Modifier and Type	Method and Description
String	branch()
Changelog	changelog(long snapshotId)
Path	changelogDirectory()
Iterator<Changelog>	changelogs()
void	commitChangelog(Changelog changelog, long id)
void	commitEarliestHint(long snapshotId)
void	commitLatestHint(long snapshotId)
void	commitLongLivedChangelogEarliestHint(long snapshotId)
void	commitLongLivedChangelogLatestHint(long snapshotId)
SnapshotManager	copyWithBranch(String branchName)
void	deleteLatestHint()
Snapshot	earlierOrEqualTimeMills(long timestampMills) Returns a Snapshot whoes commit time is earlier than or equal to given timestamp mills.
Long	earlierThanTimeMills(long timestampMills, boolean startFromChangelog) Returns the latest snapshot earlier than the timestamp mills.
Long	earliestLongLivedChangelogId()
Snapshot	earliestSnapshot()
Long	earliestSnapshotId()
FileIO	fileIO()
static int	findPreviousOrEqualSnapshot(List<Snapshot> sortedSnapshots, long targetSnapshotId)
static int	findPreviousSnapshot(List<Snapshot> sortedSnapshots, long targetSnapshotId)
List<Snapshot>	findSnapshotsForIdentifiers(String user, List<Long> identifiers) Find the snapshot of the specified identifiers written by the specified user.
void	invalidateCache()
Snapshot	laterOrEqualTimeMills(long timestampMills) Returns a Snapshot whoes commit time is later than or equal to given timestamp mills.
Snapshot	laterOrEqualWatermark(long watermark)
Long	latestChangelogId()
Long	latestLongLivedChangelogId()
Snapshot	latestSnapshot()
Long	latestSnapshotId()
Optional<Snapshot>	latestSnapshotOfUser(String user)
Changelog	longLivedChangelog(long snapshotId)
boolean	longLivedChangelogExists(long snapshotId)
Path	longLivedChangelogPath(long snapshotId)
Long	pickOrLatest(java.util.function.Predicate<Snapshot> predicate)
Long	readHint(String fileName)
Long	readHint(String fileName, Path dir)
List<Changelog>	safelyGetAllChangelogs()
List<Snapshot>	safelyGetAllSnapshots() If FileNotFoundException is thrown when reading the snapshot file, this snapshot may be deleted by other processes, so just skip this snapshot.
Snapshot	snapshot(long snapshotId)
long	snapshotCount()
Path	snapshotDirectory()
boolean	snapshotExists(long snapshotId)
Path	snapshotPath(long snapshotId)
List<Path>	snapshotPaths(java.util.function.Predicate<Long> predicate)
Iterator<Snapshot>	snapshots()
Iterator<Snapshot>	snapshotsWithId(List<Long> snapshotIds)
Iterator<Snapshot>	snapshotsWithinRange(Optional<Long> optionalMaxSnapshotId, Optional<Long> optionalMinSnapshotId)
Path	tablePath()
Snapshot	traversalSnapshotsFromLatestSafely(Filter<Snapshot> checker) Traversal snapshots from latest to earliest safely, this is applied on the writer side because the committer may delete obsolete snapshots, which may cause the writer to encounter unreadable snapshots.
List<Path>	tryGetNonChangelogFiles(java.util.function.Predicate<FileStatus> fileStatusFilter)
List<Path>	tryGetNonSnapshotFiles(java.util.function.Predicate<FileStatus> fileStatusFilter) Try to get non snapshot files.
Snapshot	tryGetSnapshot(long snapshotId)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

EARLIEST

public static final String EARLIEST

See Also:

Constant Field Values

LATEST

public static final String LATEST

See Also:

Constant Field Values

Constructor Detail

SnapshotManager

public SnapshotManager(FileIO fileIO,
                       Path tablePath)

SnapshotManager

public SnapshotManager(FileIO fileIO,
                       Path tablePath,
                       @Nullable
                       String branchName)

Specify the default branch for data writing.

SnapshotManager

public SnapshotManager(FileIO fileIO,
                       Path tablePath,
                       @Nullable
                       String branchName,
                       @Nullable
                       org.apache.paimon.shade.caffeine2.com.github.benmanes.caffeine.cache.Cache<Path,Snapshot> cache)

Method Detail

copyWithBranch

public SnapshotManager copyWithBranch(String branchName)

fileIO

public FileIO fileIO()

tablePath

public Path tablePath()

branch

public String branch()

changelogDirectory

public Path changelogDirectory()

longLivedChangelogPath

public Path longLivedChangelogPath(long snapshotId)

snapshotPath

public Path snapshotPath(long snapshotId)

snapshotDirectory

public Path snapshotDirectory()

invalidateCache

public void invalidateCache()

snapshot

public Snapshot snapshot(long snapshotId)

tryGetSnapshot

public Snapshot tryGetSnapshot(long snapshotId)
                        throws FileNotFoundException

Throws:

FileNotFoundException

changelog

public Changelog changelog(long snapshotId)

longLivedChangelog

public Changelog longLivedChangelog(long snapshotId)

snapshotExists

public boolean snapshotExists(long snapshotId)

longLivedChangelogExists

public boolean longLivedChangelogExists(long snapshotId)

latestSnapshot

@Nullable
public Snapshot latestSnapshot()

latestSnapshotId

@Nullable
public Long latestSnapshotId()

earliestSnapshot

@Nullable
public Snapshot earliestSnapshot()

earliestSnapshotId

@Nullable
public Long earliestSnapshotId()

earliestLongLivedChangelogId

@Nullable
public Long earliestLongLivedChangelogId()

latestLongLivedChangelogId

@Nullable
public Long latestLongLivedChangelogId()

latestChangelogId

@Nullable
public Long latestChangelogId()

pickOrLatest

@Nullable
public Long pickOrLatest(java.util.function.Predicate<Snapshot> predicate)

earlierThanTimeMills

@Nullable
public Long earlierThanTimeMills(long timestampMills,
                                           boolean startFromChangelog)

Returns the latest snapshot earlier than the timestamp mills. A non-existent snapshot may be returned if all snapshots are equal to or later than the timestamp mills.

earlierOrEqualTimeMills

@Nullable
public Snapshot earlierOrEqualTimeMills(long timestampMills)

Returns a Snapshot whoes commit time is earlier than or equal to given timestamp mills. If there is no such a snapshot, returns null.

laterOrEqualTimeMills

@Nullable
public Snapshot laterOrEqualTimeMills(long timestampMills)

Returns a Snapshot whoes commit time is later than or equal to given timestamp mills. If there is no such a snapshot, returns null.

laterOrEqualWatermark

@Nullable
public Snapshot laterOrEqualWatermark(long watermark)

snapshotCount

public long snapshotCount()
                   throws IOException

Throws:

IOException

snapshots

public Iterator<Snapshot> snapshots()
                             throws IOException

Throws:

IOException

snapshotPaths

public List<Path> snapshotPaths(java.util.function.Predicate<Long> predicate)
                         throws IOException

Throws:

IOException

snapshotsWithId

public Iterator<Snapshot> snapshotsWithId(List<Long> snapshotIds)

snapshotsWithinRange

public Iterator<Snapshot> snapshotsWithinRange(Optional<Long> optionalMaxSnapshotId,
                                               Optional<Long> optionalMinSnapshotId)

changelogs

public Iterator<Changelog> changelogs()
                               throws IOException

Throws:

IOException

safelyGetAllSnapshots

public List<Snapshot> safelyGetAllSnapshots()
                                     throws IOException

If FileNotFoundException is thrown when reading the snapshot file, this snapshot may be deleted by other processes, so just skip this snapshot.

Throws:

IOException

safelyGetAllChangelogs

public List<Changelog> safelyGetAllChangelogs()
                                       throws IOException

Throws:

IOException

tryGetNonSnapshotFiles

public List<Path> tryGetNonSnapshotFiles(java.util.function.Predicate<FileStatus> fileStatusFilter)

Try to get non snapshot files. If any error occurred, just ignore it and return an empty result.

tryGetNonChangelogFiles

public List<Path> tryGetNonChangelogFiles(java.util.function.Predicate<FileStatus> fileStatusFilter)

latestSnapshotOfUser

public Optional<Snapshot> latestSnapshotOfUser(String user)

findSnapshotsForIdentifiers

public List<Snapshot> findSnapshotsForIdentifiers(@Nonnull
                                                  String user,
                                                  List<Long> identifiers)

Find the snapshot of the specified identifiers written by the specified user.

commitChangelog

public void commitChangelog(Changelog changelog,
                            long id)
                     throws IOException

Throws:

IOException

traversalSnapshotsFromLatestSafely

@Nullable
public Snapshot traversalSnapshotsFromLatestSafely(Filter<Snapshot> checker)

Traversal snapshots from latest to earliest safely, this is applied on the writer side because the committer may delete obsolete snapshots, which may cause the writer to encounter unreadable snapshots.

readHint

public Long readHint(String fileName)

readHint

public Long readHint(String fileName,
                     Path dir)

findPreviousSnapshot

public static int findPreviousSnapshot(List<Snapshot> sortedSnapshots,
                                       long targetSnapshotId)

findPreviousOrEqualSnapshot

public static int findPreviousOrEqualSnapshot(List<Snapshot> sortedSnapshots,
                                              long targetSnapshotId)

deleteLatestHint

public void deleteLatestHint()
                      throws IOException

Throws:

IOException

commitLatestHint

public void commitLatestHint(long snapshotId)
                      throws IOException

Throws:

IOException

commitLongLivedChangelogLatestHint

public void commitLongLivedChangelogLatestHint(long snapshotId)
                                        throws IOException

Throws:

IOException

commitLongLivedChangelogEarliestHint

public void commitLongLivedChangelogEarliestHint(long snapshotId)
                                          throws IOException

Throws:

IOException

commitEarliestHint

public void commitEarliestHint(long snapshotId)
                        throws IOException

Throws:

IOException