org.apache.paimon.table

Class AppendOnlyFileStoreTable

java.lang.Object

org.apache.paimon.table.AppendOnlyFileStoreTable

All Implemented Interfaces:

Serializable, DataTable, FileStoreTable, InnerTable, Table

public class AppendOnlyFileStoreTable
extends Object

FileStoreTable for append table.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected CatalogEnvironment	catalogEnvironment
protected FileIO	fileIO
protected SegmentsCache<Path>	manifestCache
protected Path	path
protected org.apache.paimon.shade.caffeine2.com.github.benmanes.caffeine.cache.Cache<Path,Snapshot>	snapshotCache
protected org.apache.paimon.shade.caffeine2.com.github.benmanes.caffeine.cache.Cache<String,Statistics>	statsCache
protected TableSchema	tableSchema

Constructor Summary

Constructors

Constructor and Description
AppendOnlyFileStoreTable(FileIO fileIO, Path path, TableSchema tableSchema, CatalogEnvironment catalogEnvironment)

Method Summary

Modifier and Type	Method and Description
BranchManager	branchManager()
CatalogEnvironment	catalogEnvironment()
ChangelogManager	changelogManager()
FileStoreTable	copy(Map<String,String> dynamicOptions) Copy this table with adding dynamic options.
FileStoreTable	copy(TableSchema newTableSchema)
protected FileStoreTable	copyInternal(Map<String,String> dynamicOptions, boolean tryTimeTravel)
FileStoreTable	copyWithLatestSchema() TODO: this method is weird, old options will overwrite new options.
FileStoreTable	copyWithoutTimeTravel(Map<String,String> dynamicOptions) Doesn't change table schema even when there exists time travel scan options.
CoreOptions	coreOptions()
void	createBranch(String branchName) Create an empty branch.
void	createBranch(String branchName, String tagName) Create a branch from given tag.
RowKeyExtractor	createRowKeyExtractor()
void	createTag(String tagName) Create a tag from latest snapshot.
void	createTag(String tagName, java.time.Duration timeRetained)
void	createTag(String tagName, long fromSnapshotId) Create a tag from given snapshot.
void	createTag(String tagName, long fromSnapshotId, java.time.Duration timeRetained)
String	currentBranch()
void	deleteBranch(String branchName) Delete a branch by branchName.
void	deleteTag(String tagName) Delete a tag by name.
boolean	equals(Object o)
void	fastForward(String branchName) Merge a branch to main branch.
FileIO	fileIO() File io of this table.
Snapshot	findSnapshot(long fromSnapshotId)
String	fullName() Full name of the table, default is database.tableName.
Identifier	identifier()
SimpleFileReader<IndexManifestEntry>	indexManifestFileReader() Reader to read index manifest entry from index manifest file.
Optional<Snapshot>	latestSnapshot() Get the latest snapshot for this table, or empty if there are no snapshots.
Path	location()
SimpleFileReader<ManifestEntry>	manifestFileReader() Reader to read manifest entry from manifest file.
SimpleFileReader<ManifestFileMeta>	manifestListReader() Reader to read manifest file meta from manifest list file.
String	name() A name to identify this table.
TableCommitImpl	newCommit(String commitUser)
ExpireSnapshots	newExpireChangelog()
ExpireSnapshots	newExpireSnapshots() Manually expire snapshots, parameters can be controlled independently of table options.
LocalTableQuery	newLocalTableQuery()
InnerTableRead	newRead()
DataTableBatchScan	newScan()
SnapshotReader	newSnapshotReader()
StreamDataTableScan	newStreamScan()
TableWriteImpl<InternalRow>	newWrite(String commitUser)
TableWriteImpl<InternalRow>	newWrite(String commitUser, ManifestCacheFilter manifestFilter)
Optional<WriteSelector>	newWriteSelector()
protected java.util.function.BiConsumer<FileStoreScan,Predicate>	nonPartitionFilterConsumer()
void	renameTag(String tagName, String targetTagName)
void	replaceTag(String tagName, Long fromSnapshotId, java.time.Duration timeRetained) Replace a tag with new snapshot id and new time retained.
void	rollbackTo(long snapshotId) Rollback table's state to a specific snapshot.
void	rollbackTo(String tagName) Rollback table's state to a specific tag.
protected RowKindGenerator	rowKindGenerator()
TableSchema	schema()
SchemaManager	schemaManager()
void	setManifestCache(SegmentsCache<Path> manifestCache)
void	setSnapshotCache(org.apache.paimon.shade.caffeine2.com.github.benmanes.caffeine.cache.Cache<Path,Snapshot> cache)
void	setStatsCache(org.apache.paimon.shade.caffeine2.com.github.benmanes.caffeine.cache.Cache<String,Statistics> cache)
Snapshot	snapshot(long snapshotId) Get the Snapshot from snapshot id.
SnapshotManager	snapshotManager()
protected SplitGenerator	splitGenerator()
Optional<Statistics>	statistics() Optional statistics of this table.
AppendOnlyFileStore	store()
boolean	supportStreamingReadOverwrite() Currently, the streaming read of overwrite is implemented by reversing the RowKind of overwrote records to RowKind.DELETE, so only tables that have primary key support it.
FileStoreTable	switchToBranch(String branchName) Get DataTable with branch identified by branchName.
TagManager	tagManager()
String	uuid() UUID of the table, metastore can provide the true UUID of this table, default is the full name.

Methods inherited from class java.lang.Object

clone, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.paimon.table.FileStoreTable

bucketMode, bucketSpec, comment, options, partitionKeys, primaryKeys, rowType

Methods inherited from interface org.apache.paimon.table.InnerTable

newBatchWriteBuilder, newReadBuilder, newStreamWriteBuilder

Methods inherited from interface org.apache.paimon.table.Table

deleteBranches, deleteTags

Field Detail

fileIO

protected final FileIO fileIO

path

protected final Path path

tableSchema

protected final TableSchema tableSchema

catalogEnvironment

protected final CatalogEnvironment catalogEnvironment

manifestCache

@Nullable
protected transient SegmentsCache<Path> manifestCache

snapshotCache

@Nullable
protected transient org.apache.paimon.shade.caffeine2.com.github.benmanes.caffeine.cache.Cache<Path,Snapshot> snapshotCache

statsCache

@Nullable
protected transient org.apache.paimon.shade.caffeine2.com.github.benmanes.caffeine.cache.Cache<String,Statistics> statsCache

Constructor Detail

AppendOnlyFileStoreTable

public AppendOnlyFileStoreTable(FileIO fileIO,
                                Path path,
                                TableSchema tableSchema,
                                CatalogEnvironment catalogEnvironment)

Method Detail

store

public AppendOnlyFileStore store()

splitGenerator

protected SplitGenerator splitGenerator()

supportStreamingReadOverwrite

public boolean supportStreamingReadOverwrite()

Currently, the streaming read of overwrite is implemented by reversing the RowKind of overwrote records to RowKind.DELETE, so only tables that have primary key support it.

nonPartitionFilterConsumer

protected java.util.function.BiConsumer<FileStoreScan,Predicate> nonPartitionFilterConsumer()

newRead

public InnerTableRead newRead()

newWrite

public TableWriteImpl<InternalRow> newWrite(String commitUser)

newWrite

public TableWriteImpl<InternalRow> newWrite(String commitUser,
                                            ManifestCacheFilter manifestFilter)

newLocalTableQuery

public LocalTableQuery newLocalTableQuery()

currentBranch

public String currentBranch()

setManifestCache

public void setManifestCache(SegmentsCache<Path> manifestCache)

Specified by:

setManifestCache in interface FileStoreTable

setSnapshotCache

public void setSnapshotCache(org.apache.paimon.shade.caffeine2.com.github.benmanes.caffeine.cache.Cache<Path,Snapshot> cache)

Specified by:

setSnapshotCache in interface FileStoreTable

setStatsCache

public void setStatsCache(org.apache.paimon.shade.caffeine2.com.github.benmanes.caffeine.cache.Cache<String,Statistics> cache)

Specified by:

setStatsCache in interface FileStoreTable

latestSnapshot

public Optional<Snapshot> latestSnapshot()

Description copied from interface: Table

Get the latest snapshot for this table, or empty if there are no snapshots.

Specified by:

latestSnapshot in interface Table

snapshot

public Snapshot snapshot(long snapshotId)

Description copied from interface: Table

Get the Snapshot from snapshot id.

Specified by:

snapshot in interface Table

manifestListReader

public SimpleFileReader<ManifestFileMeta> manifestListReader()

Description copied from interface: Table

Reader to read manifest file meta from manifest list file.

Specified by:

manifestListReader in interface Table

manifestFileReader

public SimpleFileReader<ManifestEntry> manifestFileReader()

Description copied from interface: Table

Reader to read manifest entry from manifest file.

Specified by:

manifestFileReader in interface Table

indexManifestFileReader

public SimpleFileReader<IndexManifestEntry> indexManifestFileReader()

Description copied from interface: Table

Reader to read index manifest entry from index manifest file.

Specified by:

indexManifestFileReader in interface Table

name

public String name()

Description copied from interface: Table

A name to identify this table.

Specified by:

name in interface Table

fullName

public String fullName()

Description copied from interface: Table

Full name of the table, default is database.tableName.

Specified by:

fullName in interface Table

identifier

public Identifier identifier()

uuid

public String uuid()

Description copied from interface: Table

UUID of the table, metastore can provide the true UUID of this table, default is the full name.

Specified by:

uuid in interface Table

statistics

public Optional<Statistics> statistics()

Description copied from interface: Table

Optional statistics of this table.

Specified by:

statistics in interface Table

newWriteSelector

public Optional<WriteSelector> newWriteSelector()

Specified by:

newWriteSelector in interface InnerTable

catalogEnvironment

public CatalogEnvironment catalogEnvironment()

Specified by:

catalogEnvironment in interface FileStoreTable

createRowKeyExtractor

public RowKeyExtractor createRowKeyExtractor()

Specified by:

createRowKeyExtractor in interface FileStoreTable

newSnapshotReader

public SnapshotReader newSnapshotReader()

Specified by:

newSnapshotReader in interface DataTable

newScan

public DataTableBatchScan newScan()

Specified by:

newScan in interface DataTable

Specified by:

newScan in interface InnerTable

newStreamScan

public StreamDataTableScan newStreamScan()

Specified by:

newStreamScan in interface InnerTable

copy

public FileStoreTable copy(Map<String,String> dynamicOptions)

Description copied from interface: Table

Copy this table with adding dynamic options.

Specified by:

copy in interface FileStoreTable

Specified by:

copy in interface Table

copyWithoutTimeTravel

public FileStoreTable copyWithoutTimeTravel(Map<String,String> dynamicOptions)

Description copied from interface: FileStoreTable

Doesn't change table schema even when there exists time travel scan options.

Specified by:

copyWithoutTimeTravel in interface FileStoreTable

copyInternal

protected FileStoreTable copyInternal(Map<String,String> dynamicOptions,
                                      boolean tryTimeTravel)

copyWithLatestSchema

public FileStoreTable copyWithLatestSchema()

Description copied from interface: FileStoreTable

TODO: this method is weird, old options will overwrite new options.

Specified by:

copyWithLatestSchema in interface FileStoreTable

copy

public FileStoreTable copy(TableSchema newTableSchema)

Specified by:

copy in interface FileStoreTable

schemaManager

public SchemaManager schemaManager()

Specified by:

schemaManager in interface DataTable

coreOptions

public CoreOptions coreOptions()

Specified by:

coreOptions in interface DataTable

fileIO

public FileIO fileIO()

Description copied from interface: Table

File io of this table.

Specified by:

fileIO in interface Table

location

public Path location()

Specified by:

location in interface DataTable

schema

public TableSchema schema()

Specified by:

schema in interface FileStoreTable

snapshotManager

public SnapshotManager snapshotManager()

Specified by:

snapshotManager in interface DataTable

changelogManager

public ChangelogManager changelogManager()

Specified by:

changelogManager in interface DataTable

newExpireSnapshots

public ExpireSnapshots newExpireSnapshots()

Description copied from interface: Table

Manually expire snapshots, parameters can be controlled independently of table options.

Specified by:

newExpireSnapshots in interface Table

newExpireChangelog

public ExpireSnapshots newExpireChangelog()

Specified by:

newExpireChangelog in interface Table

newCommit

public TableCommitImpl newCommit(String commitUser)

Specified by:

newCommit in interface FileStoreTable

Specified by:

newCommit in interface InnerTable

rollbackTo

public void rollbackTo(long snapshotId)

Description copied from interface: Table

Rollback table's state to a specific snapshot.

Specified by:

rollbackTo in interface Table

findSnapshot

public Snapshot findSnapshot(long fromSnapshotId)
                      throws SnapshotNotExistException

Throws:

SnapshotNotExistException

createTag

public void createTag(String tagName,
                      long fromSnapshotId)

Description copied from interface: Table

Create a tag from given snapshot.

Specified by:

createTag in interface Table

createTag

public void createTag(String tagName,
                      long fromSnapshotId,
                      java.time.Duration timeRetained)

Specified by:

createTag in interface Table

createTag

public void createTag(String tagName)

Description copied from interface: Table

Create a tag from latest snapshot.

Specified by:

createTag in interface Table

createTag

public void createTag(String tagName,
                      java.time.Duration timeRetained)

Specified by:

createTag in interface Table

renameTag

public void renameTag(String tagName,
                      String targetTagName)

Specified by:

renameTag in interface Table

replaceTag

public void replaceTag(String tagName,
                       @Nullable
                       Long fromSnapshotId,
                       @Nullable
                       java.time.Duration timeRetained)

Description copied from interface: Table

Replace a tag with new snapshot id and new time retained.

Specified by:

replaceTag in interface Table

deleteTag

public void deleteTag(String tagName)

Description copied from interface: Table

Delete a tag by name.

Specified by:

deleteTag in interface Table

createBranch

public void createBranch(String branchName)

Description copied from interface: Table

Create an empty branch.

Specified by:

createBranch in interface Table

createBranch

public void createBranch(String branchName,
                         String tagName)

Description copied from interface: Table

Create a branch from given tag.

Specified by:

createBranch in interface Table

deleteBranch

public void deleteBranch(String branchName)

Description copied from interface: Table

Delete a branch by branchName.

Specified by:

deleteBranch in interface Table

fastForward

public void fastForward(String branchName)

Description copied from interface: Table

Merge a branch to main branch.

Specified by:

fastForward in interface Table

rollbackTo

public void rollbackTo(String tagName)

Description copied from interface: Table

Rollback table's state to a specific tag.

Specified by:

rollbackTo in interface Table

tagManager

public TagManager tagManager()

Specified by:

tagManager in interface DataTable

branchManager

public BranchManager branchManager()

Specified by:

branchManager in interface DataTable

switchToBranch

public FileStoreTable switchToBranch(String branchName)

Description copied from interface: FileStoreTable

Get DataTable with branch identified by branchName. Note that this method does not keep dynamic options in current table.

Specified by:

switchToBranch in interface DataTable

Specified by:

switchToBranch in interface FileStoreTable

rowKindGenerator

protected RowKindGenerator rowKindGenerator()

equals

public boolean equals(Object o)

Overrides:

equals in class Object