org.apache.paimon.hash

Class BytesHashMap<K>

java.lang.Object

org.apache.paimon.hash.BytesMap<K,BinaryRow>

org.apache.paimon.hash.BytesHashMap<K>

public class BytesHashMap<K>
extends BytesMap<K,BinaryRow>

Bytes based hash map. It can be used for performing aggregations where the aggregated values are fixed-width, because the data is stored in continuous memory, AggBuffer of variable length cannot be applied to this HashMap. The KeyValue form in hash map is designed to reduce the cost of key fetching in lookup. The memory is divided into two areas:

Bucket area: pointer + hashcode.

• Bytes 0 to 4: a pointer to the record in the record area
• Bytes 4 to 8: key's full 32-bit hashcode

Record area: the actual data in linked list records, a record has four parts:

• Bytes 0 to 4: len(k)
• Bytes 4 to 4 + len(k): key data
• Bytes 4 + len(k) to 8 + len(k): len(v)
• Bytes 8 + len(k) to 8 + len(k) + len(v): value data

BytesHashMap are influenced by Apache Spark BytesToBytesMap.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.paimon.hash.BytesMap

BytesMap.LookupInfo<K,V>

Field Summary

Fields

Modifier and Type	Field and Description
protected PagedTypeSerializer<K>	keySerializer Used to serialize map key into RecordArea's MemorySegments.

Fields inherited from class org.apache.paimon.hash.BytesMap

BUCKET_SIZE, BUCKET_SIZE_BITS, bucketSegments, ELEMENT_POINT_LENGTH, END_OF_LIST, growthThreshold, INIT_BUCKET_MEMORY_IN_BYTES, lastBucketPosition, LOAD_FACTOR, log2NumBuckets, memoryPool, numBucketsMask, numBucketsMask2, numBucketsPerSegment, numBucketsPerSegmentBits, numBucketsPerSegmentMask, numElements, numSpillFiles, RECORD_EXTRA_LENGTH, recordArea, reservedNumBuffers, reusedKey, reusedValue, segmentSize, spillInBytes, STEP_INCREMENT

Constructor Summary

Constructors

Constructor and Description
BytesHashMap(MemorySegmentPool memoryPool, PagedTypeSerializer<K> keySerializer, int valueArity)

Method Summary

Modifier and Type	Method and Description
BinaryRow	append(BytesMap.LookupInfo<K,BinaryRow> lookupInfo, BinaryRow value) Append an value into the hash map's record area.
void	free()
List<MemorySegment>	getBucketAreaMemorySegments()
KeyValueIterator<K,BinaryRow>	getEntryIterator(boolean requiresCopy) Returns an iterator for iterating over the entries of this map.
int	getNumElements()
long	getNumKeys() Returns the number of keys in this map.
long	getNumSpillFiles()
ArrayList<MemorySegment>	getRecordAreaMemorySegments()
long	getSpillInBytes()
long	getUsedMemoryInBytes()
void	reset() reset the map's record and bucket area's memory segments for reusing.

Methods inherited from class org.apache.paimon.hash.BytesMap

calcSecondHashCode, growAndRehash, initBucketSegments, lookup, resetBucketSegments, returnSegments

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

keySerializer

protected final PagedTypeSerializer<K> keySerializer

Used to serialize map key into RecordArea's MemorySegments.

Constructor Detail

BytesHashMap

public BytesHashMap(MemorySegmentPool memoryPool,
                    PagedTypeSerializer<K> keySerializer,
                    int valueArity)

Method Detail

getNumKeys

public long getNumKeys()

Description copied from class: BytesMap

Returns the number of keys in this map.

Specified by:

getNumKeys in class BytesMap<K,BinaryRow>

append

public BinaryRow append(BytesMap.LookupInfo<K,BinaryRow> lookupInfo,
                        BinaryRow value)
                 throws IOException

Append an value into the hash map's record area.

Returns:

An BinaryRow mapping to the memory segments in the map's record area belonging to the newly appended value.

Throws:

EOFException - if the map can't allocate much more memory.

IOException

getNumSpillFiles

public long getNumSpillFiles()

Overrides:

getNumSpillFiles in class BytesMap<K,BinaryRow>

getUsedMemoryInBytes

public long getUsedMemoryInBytes()

getSpillInBytes

public long getSpillInBytes()

Overrides:

getSpillInBytes in class BytesMap<K,BinaryRow>

getNumElements

public int getNumElements()

Overrides:

getNumElements in class BytesMap<K,BinaryRow>

getEntryIterator

public KeyValueIterator<K,BinaryRow> getEntryIterator(boolean requiresCopy)

Returns an iterator for iterating over the entries of this map.

getRecordAreaMemorySegments

public ArrayList<MemorySegment> getRecordAreaMemorySegments()

Returns:

the underlying memory segments of the hash map's record area

getBucketAreaMemorySegments

public List<MemorySegment> getBucketAreaMemorySegments()

free

public void free()

Overrides:

free in class BytesMap<K,BinaryRow>

reset

public void reset()

reset the map's record and bucket area's memory segments for reusing.

Overrides:

reset in class BytesMap<K,BinaryRow>