org.apache.paimon.utils

Class SnapshotManager

java.lang.Object

org.apache.paimon.utils.SnapshotManager

All Implemented Interfaces:

Serializable

public class SnapshotManager
extends Object
implements Serializable

Manager for Snapshot, providing utility methods related to paths and snapshot hints.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static String	EARLIEST
static String	LATEST

Constructor Summary

Constructors

Constructor and Description
SnapshotManager(FileIO fileIO, Path tablePath)
SnapshotManager(FileIO fileIO, Path tablePath, String branchName) Specify the default branch for data writing.

Method Summary

Modifier and Type	Method and Description
String	branch()
Changelog	changelog(long snapshotId)
Path	changelogDirectory()
Iterator<Changelog>	changelogs()
void	commitChangelog(Changelog changelog, long id)
void	commitEarliestHint(long snapshotId)
void	commitLatestHint(long snapshotId)
void	commitLongLivedChangelogEarliestHint(long snapshotId)
void	commitLongLivedChangelogLatestHint(long snapshotId)
SnapshotManager	copyWithBranch(String branchName)
void	deleteLatestHint()
Snapshot	earlierOrEqualTimeMills(long timestampMills) Returns a Snapshot whoes commit time is earlier than or equal to given timestamp mills.
Long	earlierThanTimeMills(long timestampMills, boolean startFromChangelog) Returns the latest snapshot earlier than the timestamp mills.
Long	earliestLongLivedChangelogId()
Snapshot	earliestSnapshot()
Long	earliestSnapshotId()
FileIO	fileIO()
static List<Snapshot>	findOverlappedSnapshots(List<Snapshot> sortedSnapshots, long beginInclusive, long endExclusive) Find the overlapping snapshots between sortedSnapshots and range of [beginInclusive, endExclusive).
static int	findPreviousSnapshot(List<Snapshot> sortedSnapshots, long targetSnapshotId)
List<Snapshot>	findSnapshotsForIdentifiers(String user, List<Long> identifiers) Find the snapshot of the specified identifiers written by the specified user.
Snapshot	laterOrEqualTimeMills(long timestampMills) Returns a Snapshot whoes commit time is later than or equal to given timestamp mills.
Snapshot	laterOrEqualWatermark(long watermark)
Long	latestChangelogId()
Long	latestLongLivedChangelogId()
Snapshot	latestSnapshot()
Long	latestSnapshotId()
Optional<Snapshot>	latestSnapshotOfUser(String user)
Changelog	longLivedChangelog(long snapshotId)
boolean	longLivedChangelogExists(long snapshotId)
Path	longLivedChangelogPath(long snapshotId)
Long	pickOrLatest(java.util.function.Predicate<Snapshot> predicate)
Long	readHint(String fileName)
Long	readHint(String fileName, Path dir)
List<Changelog>	safelyGetAllChangelogs()
List<Snapshot>	safelyGetAllSnapshots() If FileNotFoundException is thrown when reading the snapshot file, this snapshot may be deleted by other processes, so just skip this snapshot.
Snapshot	snapshot(long snapshotId)
long	snapshotCount()
Path	snapshotDirectory()
boolean	snapshotExists(long snapshotId)
Path	snapshotPath(long snapshotId)
Iterator<Snapshot>	snapshots()
Iterator<Snapshot>	snapshotsWithinRange(Optional<Long> optionalMaxSnapshotId, Optional<Long> optionalMinSnapshotId)
Path	tablePath()
Snapshot	traversalSnapshotsFromLatestSafely(Filter<Snapshot> checker) Traversal snapshots from latest to earliest safely, this is applied on the writer side because the committer may delete obsolete snapshots, which may cause the writer to encounter unreadable snapshots.
List<Path>	tryGetNonChangelogFiles(java.util.function.Predicate<FileStatus> fileStatusFilter)
List<Path>	tryGetNonSnapshotFiles(java.util.function.Predicate<FileStatus> fileStatusFilter) Try to get non snapshot files.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

EARLIEST

public static final String EARLIEST

See Also:

Constant Field Values

LATEST

public static final String LATEST

See Also:

Constant Field Values

Constructor Detail

SnapshotManager

public SnapshotManager(FileIO fileIO,
                       Path tablePath)

SnapshotManager

public SnapshotManager(FileIO fileIO,
                       Path tablePath,
                       String branchName)

Specify the default branch for data writing.

Method Detail

copyWithBranch

public SnapshotManager copyWithBranch(String branchName)

fileIO

public FileIO fileIO()

tablePath

public Path tablePath()

branch

public String branch()

changelogDirectory

public Path changelogDirectory()

longLivedChangelogPath

public Path longLivedChangelogPath(long snapshotId)

snapshotPath

public Path snapshotPath(long snapshotId)

snapshotDirectory

public Path snapshotDirectory()

snapshot

public Snapshot snapshot(long snapshotId)

changelog

public Changelog changelog(long snapshotId)

longLivedChangelog

public Changelog longLivedChangelog(long snapshotId)

snapshotExists

public boolean snapshotExists(long snapshotId)

longLivedChangelogExists

public boolean longLivedChangelogExists(long snapshotId)

latestSnapshot

@Nullable
public Snapshot latestSnapshot()

latestSnapshotId

@Nullable
public Long latestSnapshotId()

earliestSnapshot

@Nullable
public Snapshot earliestSnapshot()

earliestSnapshotId

@Nullable
public Long earliestSnapshotId()

earliestLongLivedChangelogId

@Nullable
public Long earliestLongLivedChangelogId()

latestLongLivedChangelogId

@Nullable
public Long latestLongLivedChangelogId()

latestChangelogId

@Nullable
public Long latestChangelogId()

pickOrLatest

@Nullable
public Long pickOrLatest(java.util.function.Predicate<Snapshot> predicate)

earlierThanTimeMills

@Nullable
public Long earlierThanTimeMills(long timestampMills,
                                           boolean startFromChangelog)

Returns the latest snapshot earlier than the timestamp mills. A non-existent snapshot may be returned if all snapshots are equal to or later than the timestamp mills.

earlierOrEqualTimeMills

@Nullable
public Snapshot earlierOrEqualTimeMills(long timestampMills)

Returns a Snapshot whoes commit time is earlier than or equal to given timestamp mills. If there is no such a snapshot, returns null.

laterOrEqualTimeMills

@Nullable
public Snapshot laterOrEqualTimeMills(long timestampMills)

Returns a Snapshot whoes commit time is later than or equal to given timestamp mills. If there is no such a snapshot, returns null.

laterOrEqualWatermark

@Nullable
public Snapshot laterOrEqualWatermark(long watermark)

snapshotCount

public long snapshotCount()
                   throws IOException

Throws:

IOException

snapshots

public Iterator<Snapshot> snapshots()
                             throws IOException

Throws:

IOException

snapshotsWithinRange

public Iterator<Snapshot> snapshotsWithinRange(Optional<Long> optionalMaxSnapshotId,
                                               Optional<Long> optionalMinSnapshotId)
                                        throws IOException

Throws:

IOException

changelogs

public Iterator<Changelog> changelogs()
                               throws IOException

Throws:

IOException

safelyGetAllSnapshots

public List<Snapshot> safelyGetAllSnapshots()
                                     throws IOException

If FileNotFoundException is thrown when reading the snapshot file, this snapshot may be deleted by other processes, so just skip this snapshot.

Throws:

IOException

safelyGetAllChangelogs

public List<Changelog> safelyGetAllChangelogs()
                                       throws IOException

Throws:

IOException

tryGetNonSnapshotFiles

public List<Path> tryGetNonSnapshotFiles(java.util.function.Predicate<FileStatus> fileStatusFilter)

Try to get non snapshot files. If any error occurred, just ignore it and return an empty result.

tryGetNonChangelogFiles

public List<Path> tryGetNonChangelogFiles(java.util.function.Predicate<FileStatus> fileStatusFilter)

latestSnapshotOfUser

public Optional<Snapshot> latestSnapshotOfUser(String user)

findSnapshotsForIdentifiers

public List<Snapshot> findSnapshotsForIdentifiers(@Nonnull
                                                  String user,
                                                  List<Long> identifiers)

Find the snapshot of the specified identifiers written by the specified user.

commitChangelog

public void commitChangelog(Changelog changelog,
                            long id)
                     throws IOException

Throws:

IOException

traversalSnapshotsFromLatestSafely

@Nullable
public Snapshot traversalSnapshotsFromLatestSafely(Filter<Snapshot> checker)

Traversal snapshots from latest to earliest safely, this is applied on the writer side because the committer may delete obsolete snapshots, which may cause the writer to encounter unreadable snapshots.

readHint

public Long readHint(String fileName)

readHint

public Long readHint(String fileName,
                     Path dir)

findOverlappedSnapshots

public static List<Snapshot> findOverlappedSnapshots(List<Snapshot> sortedSnapshots,
                                                     long beginInclusive,
                                                     long endExclusive)

Find the overlapping snapshots between sortedSnapshots and range of [beginInclusive, endExclusive).

findPreviousSnapshot

public static int findPreviousSnapshot(List<Snapshot> sortedSnapshots,
                                       long targetSnapshotId)

deleteLatestHint

public void deleteLatestHint()
                      throws IOException

Throws:

IOException

commitLatestHint

public void commitLatestHint(long snapshotId)
                      throws IOException

Throws:

IOException

commitLongLivedChangelogLatestHint

public void commitLongLivedChangelogLatestHint(long snapshotId)
                                        throws IOException

Throws:

IOException

commitLongLivedChangelogEarliestHint

public void commitLongLivedChangelogEarliestHint(long snapshotId)
                                          throws IOException

Throws:

IOException

commitEarliestHint

public void commitEarliestHint(long snapshotId)
                        throws IOException

Throws:

IOException