public class MergeTreeWriter extends Object implements RecordWriter<KeyValue>, MemoryOwner
RecordWriter
to write records and generate CompactIncrement
.Constructor and Description |
---|
MergeTreeWriter(boolean writeBufferSpillable,
MemorySize maxDiskSize,
int sortMaxFan,
CompressOptions sortCompression,
IOManager ioManager,
CompactManager compactManager,
long maxSequenceNumber,
Comparator<InternalRow> keyComparator,
MergeFunction<KeyValue> mergeFunction,
KeyValueFileWriterFactory writerFactory,
boolean commitForceCompact,
CoreOptions.ChangelogProducer changelogProducer,
CommitIncrement increment,
FieldsComparator userDefinedSeqComparator) |
Modifier and Type | Method and Description |
---|---|
void |
addNewFiles(List<DataFileMeta> files)
Add files to the internal
CompactManager . |
void |
close()
Close this writer, the call will delete newly generated but not committed files.
|
void |
compact(boolean fullCompaction)
Compact files related to the writer.
|
Collection<DataFileMeta> |
dataFiles()
Get all data files maintained by this writer.
|
void |
flushMemory()
Flush memory of owner, release memory.
|
boolean |
isCompacting()
Check if a compaction is in progress, or if a compaction result remains to be fetched.
|
long |
maxSequenceNumber()
Get max sequence number of records written by this writer.
|
long |
memoryOccupancy()
Memory occupancy size of this owner.
|
CommitIncrement |
prepareCommit(boolean waitCompaction)
Prepare for a commit.
|
void |
setMemoryPool(MemorySegmentPool memoryPool)
Set
MemorySegmentPool for the owner. |
void |
sync()
Sync the writer.
|
void |
withInsertOnly(boolean insertOnly)
This method is called when the insert only status of the records changes.
|
void |
write(KeyValue kv)
Add a key-value element to the writer.
|
public MergeTreeWriter(boolean writeBufferSpillable, MemorySize maxDiskSize, int sortMaxFan, CompressOptions sortCompression, IOManager ioManager, CompactManager compactManager, long maxSequenceNumber, Comparator<InternalRow> keyComparator, MergeFunction<KeyValue> mergeFunction, KeyValueFileWriterFactory writerFactory, boolean commitForceCompact, CoreOptions.ChangelogProducer changelogProducer, @Nullable CommitIncrement increment, @Nullable FieldsComparator userDefinedSeqComparator)
public void setMemoryPool(MemorySegmentPool memoryPool)
MemoryOwner
MemorySegmentPool
for the owner.setMemoryPool
in interface MemoryOwner
public void write(KeyValue kv) throws Exception
RecordWriter
write
in interface RecordWriter<KeyValue>
Exception
public void compact(boolean fullCompaction) throws Exception
RecordWriter
compact
in interface RecordWriter<KeyValue>
fullCompaction
- whether to trigger full compaction or just normal compactionException
public void addNewFiles(List<DataFileMeta> files)
RecordWriter
CompactManager
.addNewFiles
in interface RecordWriter<KeyValue>
files
- files to addpublic Collection<DataFileMeta> dataFiles()
RecordWriter
dataFiles
in interface RecordWriter<KeyValue>
public long maxSequenceNumber()
RecordWriter
maxSequenceNumber
in interface RecordWriter<KeyValue>
public long memoryOccupancy()
MemoryOwner
memoryOccupancy
in interface MemoryOwner
public void flushMemory() throws Exception
MemoryOwner
flushMemory
in interface MemoryOwner
Exception
public CommitIncrement prepareCommit(boolean waitCompaction) throws Exception
RecordWriter
prepareCommit
in interface RecordWriter<KeyValue>
waitCompaction
- if this method need to wait for current compaction to completeException
public boolean isCompacting()
RecordWriter
isCompacting
in interface RecordWriter<KeyValue>
public void sync() throws Exception
RecordWriter
sync
in interface RecordWriter<KeyValue>
Exception
public void withInsertOnly(boolean insertOnly)
RecordWriter
withInsertOnly
in interface RecordWriter<KeyValue>
insertOnly
- If true, all the following records would be of RowKind.INSERT
, and no two records would have the same primary
key.public void close() throws Exception
RecordWriter
close
in interface RecordWriter<KeyValue>
Exception
Copyright © 2023–2024 The Apache Software Foundation. All rights reserved.