Modifier and Type | Method and Description |
---|---|
List<DataFileMeta> |
BucketedAppendCompactManager.allFiles() |
List<DataFileMeta> |
UnawareAppendCompactionTask.compactAfter() |
List<DataFileMeta> |
UnawareAppendCompactionTask.compactBefore() |
Collection<DataFileMeta> |
AppendOnlyWriter.dataFiles() |
static Comparator<DataFileMeta> |
BucketedAppendCompactManager.fileComparator(boolean ignoreOverlap)
New files may be created during the compaction process, then the results of the compaction
may be put after the new files, and this order will be disrupted.
|
List<DataFileMeta> |
BucketedAppendCompactManager.CompactRewriter.rewrite(List<DataFileMeta> compactBefore) |
Modifier and Type | Method and Description |
---|---|
void |
BucketedAppendCompactManager.addNewFile(DataFileMeta file) |
Modifier and Type | Method and Description |
---|---|
void |
AppendOnlyWriter.addNewFiles(List<DataFileMeta> files) |
List<DataFileMeta> |
BucketedAppendCompactManager.CompactRewriter.rewrite(List<DataFileMeta> compactBefore) |
Modifier and Type | Method and Description |
---|---|
List<DataFileMeta> |
CompactResult.after() |
List<DataFileMeta> |
NoopCompactManager.allFiles() |
Collection<DataFileMeta> |
CompactManager.allFiles() |
List<DataFileMeta> |
CompactResult.before() |
List<DataFileMeta> |
CompactResult.changelog() |
List<DataFileMeta> |
CompactUnit.files() |
Modifier and Type | Method and Description |
---|---|
void |
NoopCompactManager.addNewFile(DataFileMeta file) |
void |
CompactManager.addNewFile(DataFileMeta file)
Add a new file.
|
Modifier and Type | Method and Description |
---|---|
static CompactUnit |
CompactUnit.fromFiles(int outputLevel,
List<DataFileMeta> files) |
protected String |
CompactTask.logMetric(long startMillis,
List<DataFileMeta> compactBefore,
List<DataFileMeta> compactAfter) |
protected String |
CompactTask.logMetric(long startMillis,
List<DataFileMeta> compactBefore,
List<DataFileMeta> compactAfter) |
Constructor and Description |
---|
CompactResult(DataFileMeta before,
DataFileMeta after) |
Constructor and Description |
---|
CompactResult(List<DataFileMeta> before,
List<DataFileMeta> after) |
CompactResult(List<DataFileMeta> before,
List<DataFileMeta> after) |
CompactResult(List<DataFileMeta> before,
List<DataFileMeta> after,
List<DataFileMeta> changelog) |
CompactResult(List<DataFileMeta> before,
List<DataFileMeta> after,
List<DataFileMeta> changelog) |
CompactResult(List<DataFileMeta> before,
List<DataFileMeta> after,
List<DataFileMeta> changelog) |
Modifier and Type | Method and Description |
---|---|
static DeletionVector.Factory |
DeletionVector.factory(FileIO fileIO,
List<DataFileMeta> files,
List<DeletionFile> deletionFiles) |
Modifier and Type | Method and Description |
---|---|
DataFileMeta |
RewriteFileIndexSink.FileIndexProcessor.process(BinaryRow partition,
int bucket,
DataFileMeta dataFileMeta) |
Modifier and Type | Method and Description |
---|---|
DataFileMeta |
RewriteFileIndexSink.FileIndexProcessor.process(BinaryRow partition,
int bucket,
DataFileMeta dataFileMeta) |
Modifier and Type | Method and Description |
---|---|
void |
StoreSinkWrite.notifyNewFiles(long snapshotId,
BinaryRow partition,
int bucket,
List<DataFileMeta> files) |
void |
StoreSinkWriteImpl.notifyNewFiles(long snapshotId,
BinaryRow partition,
int bucket,
List<DataFileMeta> files) |
Modifier and Type | Method and Description |
---|---|
List<DataFileMeta> |
PlaceholderSplit.beforeFiles() |
List<DataFileMeta> |
PlaceholderSplit.dataFiles() |
Modifier and Type | Method and Description |
---|---|
protected abstract boolean |
AbstractIcebergCommitCallback.shouldAddFileToIceberg(DataFileMeta meta) |
protected boolean |
PrimaryKeyIcebergCommitCallback.shouldAddFileToIceberg(DataFileMeta meta) |
protected boolean |
AppendOnlyIcebergCommitCallback.shouldAddFileToIceberg(DataFileMeta meta) |
Modifier and Type | Method and Description |
---|---|
DataFileMeta |
DataFileMeta.copy(byte[] newEmbeddedIndex) |
DataFileMeta |
DataFileMeta.copy(List<String> newExtraFiles) |
DataFileMeta |
DataFileMeta08Serializer.deserialize(DataInputView in) |
static DataFileMeta |
DataFileMeta.forAppend(String fileName,
long fileSize,
long rowCount,
SimpleStats rowStats,
long minSequenceNumber,
long maxSequenceNumber,
long schemaId,
FileSource fileSource) |
static DataFileMeta |
DataFileMeta.forAppend(String fileName,
long fileSize,
long rowCount,
SimpleStats rowStats,
long minSequenceNumber,
long maxSequenceNumber,
long schemaId,
List<String> extraFiles,
byte[] embeddedIndex,
FileSource fileSource) |
DataFileMeta |
DataFileMetaSerializer.fromRow(InternalRow row) |
DataFileMeta |
DataFileMeta.rename(String newFileName) |
DataFileMeta |
KeyValueDataFileWriter.result() |
DataFileMeta |
RowDataFileWriter.result() |
DataFileMeta |
DataFileMeta.upgrade(int newLevel) |
Modifier and Type | Method and Description |
---|---|
List<DataFileMeta> |
DataIncrement.changelogFiles() |
List<DataFileMeta> |
CompactIncrement.changelogFiles() |
List<DataFileMeta> |
CompactIncrement.compactAfter() |
List<DataFileMeta> |
CompactIncrement.compactBefore() |
RollingFileWriter<KeyValue,DataFileMeta> |
KeyValueFileWriterFactory.createRollingChangelogFileWriter(int level) |
RollingFileWriter<KeyValue,DataFileMeta> |
KeyValueFileWriterFactory.createRollingMergeTreeFileWriter(int level,
FileSource fileSource) |
List<DataFileMeta> |
DataIncrement.deletedFiles() |
List<DataFileMeta> |
DataFileMeta08Serializer.deserializeList(DataInputView source) |
List<DataFileMeta> |
DataIncrement.newFiles() |
Modifier and Type | Method and Description |
---|---|
RecordReader<T> |
FileReaderFactory.createRecordReader(DataFileMeta file) |
RecordReader<KeyValue> |
KeyValueFileReaderFactory.createRecordReader(DataFileMeta file) |
void |
DataFileMeta08Serializer.serialize(DataFileMeta meta,
DataOutputView target) |
static boolean |
FileIndexSkipper.skip(FileIO fileIO,
TableSchema dataSchema,
List<Predicate> dataFilter,
DataFilePathFactory dataFilePathFactory,
DataFileMeta file) |
InternalRow |
DataFileMetaSerializer.toRow(DataFileMeta meta) |
Modifier and Type | Method and Description |
---|---|
static long |
DataFileMeta.getMaxSequenceNumber(List<DataFileMeta> fileMetas) |
void |
DataFileMeta08Serializer.serializeList(List<DataFileMeta> records,
DataOutputView target) |
Constructor and Description |
---|
CompactIncrement(List<DataFileMeta> compactBefore,
List<DataFileMeta> compactAfter,
List<DataFileMeta> changelogFiles) |
CompactIncrement(List<DataFileMeta> compactBefore,
List<DataFileMeta> compactAfter,
List<DataFileMeta> changelogFiles) |
CompactIncrement(List<DataFileMeta> compactBefore,
List<DataFileMeta> compactAfter,
List<DataFileMeta> changelogFiles) |
DataIncrement(List<DataFileMeta> newFiles,
List<DataFileMeta> deletedFiles,
List<DataFileMeta> changelogFiles) |
DataIncrement(List<DataFileMeta> newFiles,
List<DataFileMeta> deletedFiles,
List<DataFileMeta> changelogFiles) |
DataIncrement(List<DataFileMeta> newFiles,
List<DataFileMeta> deletedFiles,
List<DataFileMeta> changelogFiles) |
Modifier and Type | Method and Description |
---|---|
DataFileMeta |
ManifestEntry.file() |
Constructor and Description |
---|
ManifestEntry(FileKind kind,
BinaryRow partition,
int bucket,
int totalBuckets,
DataFileMeta file) |
Modifier and Type | Method and Description |
---|---|
DataFileMeta |
DataFileReader.remoteFile() |
DataFileMeta |
LookupFile.remoteFile() |
Modifier and Type | Method and Description |
---|---|
List<DataFileMeta> |
Levels.allFiles() |
Collection<DataFileMeta> |
MergeTreeWriter.dataFiles() |
List<DataFileMeta> |
SortedRun.files() |
TreeSet<DataFileMeta> |
Levels.level0() |
Modifier and Type | Method and Description |
---|---|
void |
Levels.addLevel0File(DataFileMeta file) |
static SortedRun |
SortedRun.fromSingle(DataFileMeta file) |
Constructor and Description |
---|
DataFileReader(File localFile,
DataFileMeta remoteFile,
LookupStoreReader reader) |
LookupFile(File localFile,
DataFileMeta remoteFile,
LookupStoreReader reader,
Runnable callback) |
Constructor and Description |
---|
Levels(Comparator<InternalRow> keyComparator,
List<DataFileMeta> inputFiles,
int numLevels) |
LookupLevels(Levels levels,
Comparator<InternalRow> keyComparator,
RowType keyType,
LookupLevels.ValueProcessor<T> valueProcessor,
IOFunction<DataFileMeta,RecordReader<KeyValue>> fileReaderFactory,
java.util.function.Function<String,File> localFileFactory,
LookupStoreFactory lookupStoreFactory,
java.util.function.Function<Long,BloomFilter.Builder> bfGenerator,
org.apache.paimon.shade.caffeine2.com.github.benmanes.caffeine.cache.Cache<String,LookupFile> lookupFileCache) |
Modifier and Type | Method and Description |
---|---|
List<DataFileMeta> |
MergeTreeCompactManager.allFiles() |
protected static List<DataFileMeta> |
AbstractCompactRewriter.extractFilesFromSections(List<List<SortedRun>> sections) |
Modifier and Type | Method and Description |
---|---|
void |
MergeTreeCompactManager.addNewFile(DataFileMeta file) |
CompactResult |
CompactRewriter.upgrade(int outputLevel,
DataFileMeta file)
Upgrade file to new level, usually file data is not rewritten, only the metadata is updated.
|
CompactResult |
AbstractCompactRewriter.upgrade(int outputLevel,
DataFileMeta file) |
CompactResult |
ChangelogMergeTreeRewriter.upgrade(int outputLevel,
DataFileMeta file) |
protected ChangelogMergeTreeRewriter.UpgradeStrategy |
LookupMergeTreeCompactRewriter.upgradeStrategy(int outputLevel,
DataFileMeta file) |
protected ChangelogMergeTreeRewriter.UpgradeStrategy |
FullChangelogMergeTreeCompactRewriter.upgradeStrategy(int outputLevel,
DataFileMeta file) |
protected abstract ChangelogMergeTreeRewriter.UpgradeStrategy |
ChangelogMergeTreeRewriter.upgradeStrategy(int outputLevel,
DataFileMeta file) |
Modifier and Type | Method and Description |
---|---|
protected String |
MergeTreeCompactTask.logMetric(long startMillis,
List<DataFileMeta> compactBefore,
List<DataFileMeta> compactAfter) |
protected String |
MergeTreeCompactTask.logMetric(long startMillis,
List<DataFileMeta> compactBefore,
List<DataFileMeta> compactAfter) |
protected void |
MergeTreeCompactRewriter.notifyRewriteCompactBefore(List<DataFileMeta> files) |
protected void |
LookupMergeTreeCompactRewriter.notifyRewriteCompactBefore(List<DataFileMeta> files) |
Constructor and Description |
---|
IntervalPartition(List<DataFileMeta> inputFiles,
Comparator<InternalRow> keyComparator) |
Modifier and Type | Method and Description |
---|---|
static List<DataFileMeta> |
FileMetaUtils.construct(FileIO fileIO,
String format,
String location,
Table paimonTable,
java.util.function.Predicate<FileStatus> filter,
Path dir,
Map<Path,Path> rollback) |
Modifier and Type | Method and Description |
---|---|
static CommitMessage |
FileMetaUtils.commitFile(BinaryRow partition,
List<DataFileMeta> dataFileMetas) |
Modifier and Type | Field and Description |
---|---|
protected List<DataFileMeta> |
FileStoreWrite.State.dataFiles |
Modifier and Type | Method and Description |
---|---|
List<DataFileMeta> |
AppendOnlyFileStoreWrite.compactRewrite(BinaryRow partition,
int bucket,
java.util.function.Function<String,DeletionVector> dvFactory,
List<DataFileMeta> toCompact) |
static Map<BinaryRow,Map<Integer,List<DataFileMeta>>> |
FileStoreScan.Plan.groupByPartFiles(List<ManifestEntry> files)
Return a map group by partition and bucket.
|
Modifier and Type | Method and Description |
---|---|
List<DataFileMeta> |
AppendOnlyFileStoreWrite.compactRewrite(BinaryRow partition,
int bucket,
java.util.function.Function<String,DeletionVector> dvFactory,
List<DataFileMeta> toCompact) |
RecordReader<KeyValue> |
MergeFileSplitRead.createMergeReader(BinaryRow partition,
int bucket,
List<DataFileMeta> files,
List<DeletionFile> deletionFiles,
boolean keepDelete) |
RecordReader<KeyValue> |
MergeFileSplitRead.createNoMergeReader(BinaryRow partition,
int bucket,
List<DataFileMeta> files,
List<DeletionFile> deletionFiles,
boolean onlyFilterKey) |
RecordReader<InternalRow> |
RawFileSplitRead.createReader(BinaryRow partition,
int bucket,
List<DataFileMeta> files,
List<IOExceptionSupplier<DeletionVector>> dvFactories) |
protected RecordWriter<InternalRow> |
AppendOnlyFileStoreWrite.createWriter(Long snapshotId,
BinaryRow partition,
int bucket,
List<DataFileMeta> restoredFiles,
long restoredMaxSeqNumber,
CommitIncrement restoreIncrement,
ExecutorService compactExecutor,
DeletionVectorsMaintainer dvMaintainer) |
protected MergeTreeWriter |
KeyValueFileStoreWrite.createWriter(Long snapshotId,
BinaryRow partition,
int bucket,
List<DataFileMeta> restoreFiles,
long restoredMaxSeqNumber,
CommitIncrement restoreIncrement,
ExecutorService compactExecutor,
DeletionVectorsMaintainer dvMaintainer) |
protected abstract RecordWriter<T> |
AbstractFileStoreWrite.createWriter(Long snapshotId,
BinaryRow partition,
int bucket,
List<DataFileMeta> restoreFiles,
long restoredMaxSeqNumber,
CommitIncrement restoreIncrement,
ExecutorService compactExecutor,
DeletionVectorsMaintainer deletionVectorsMaintainer) |
void |
FileStoreWrite.notifyNewFiles(long snapshotId,
BinaryRow partition,
int bucket,
List<DataFileMeta> files)
Notify that some new files are created at given snapshot in given bucket.
|
void |
AbstractFileStoreWrite.notifyNewFiles(long snapshotId,
BinaryRow partition,
int bucket,
List<DataFileMeta> files) |
Constructor and Description |
---|
State(BinaryRow partition,
int bucket,
long baseSnapshotId,
long lastModifiedCommitIdentifier,
Collection<DataFileMeta> dataFiles,
long maxSequenceNumber,
IndexMaintainer<T> indexMaintainer,
DeletionVectorsMaintainer deletionVectorsMaintainer,
CommitIncrement commitIncrement) |
Modifier and Type | Method and Description |
---|---|
default SimpleStats |
FileStoreTable.getSchemaFieldStats(DataFileMeta dataFileMeta) |
Modifier and Type | Method and Description |
---|---|
void |
LocalTableQuery.refreshFiles(BinaryRow partition,
int bucket,
List<DataFileMeta> beforeFiles,
List<DataFileMeta> dataFiles) |
void |
LocalTableQuery.refreshFiles(BinaryRow partition,
int bucket,
List<DataFileMeta> beforeFiles,
List<DataFileMeta> dataFiles) |
Modifier and Type | Method and Description |
---|---|
void |
TableWriteImpl.notifyNewFiles(long snapshotId,
BinaryRow partition,
int bucket,
List<DataFileMeta> files)
Notify that some new files are created at given snapshot in given bucket.
|
Modifier and Type | Field and Description |
---|---|
List<DataFileMeta> |
SplitGenerator.SplitGroup.files |
Modifier and Type | Method and Description |
---|---|
List<DataFileMeta> |
DataSplit.beforeFiles() |
List<DataFileMeta> |
DataSplit.dataFiles() |
Modifier and Type | Method and Description |
---|---|
List<DataFileMeta> |
FileMonitorTable.FileChange.beforeFiles() |
List<DataFileMeta> |
FileMonitorTable.FileChange.dataFiles() |
Constructor and Description |
---|
FileChange(long snapshotId,
BinaryRow partition,
int bucket,
List<DataFileMeta> beforeFiles,
List<DataFileMeta> dataFiles) |
FileChange(long snapshotId,
BinaryRow partition,
int bucket,
List<DataFileMeta> beforeFiles,
List<DataFileMeta> dataFiles) |
Modifier and Type | Method and Description |
---|---|
Collection<DataFileMeta> |
RecordWriter.dataFiles()
Get all data files maintained by this writer.
|
Modifier and Type | Method and Description |
---|---|
void |
RecordWriter.addNewFiles(List<DataFileMeta> files)
Add files to the internal
CompactManager . |
Copyright © 2023–2024 The Apache Software Foundation. All rights reserved.