org.apache.paimon.fileindex.bitmap

Class BitmapFileIndexMetaV2

java.lang.Object

org.apache.paimon.fileindex.bitmap.BitmapFileIndexMeta

org.apache.paimon.fileindex.bitmap.BitmapFileIndexMetaV2

public class BitmapFileIndexMetaV2
extends BitmapFileIndexMeta

When the bitmap-indexed column cardinality is high, using the first version of the bitmap index format will take a lot of time to read the entire dictionary. But in fact we don't need a full dictionary when dealing with a small number of predicates, the performance of predicate hits on the bitmap can be improved by creating a secondary index on the dictionary.

 Bitmap file index format (V2)
 +-------------------------------------------------+-----------------
 ｜ version (1 byte) = 2                           ｜
 +-------------------------------------------------+
 ｜ row count (4 bytes int)                        ｜
 +-------------------------------------------------+
 ｜ non-null value bitmap number (4 bytes int)     ｜
 +-------------------------------------------------+
 ｜ has null value (1 byte)                        ｜
 +-------------------------------------------------+
 ｜ null value offset (4 bytes if has null value)  ｜       HEAD
 +-------------------------------------------------+
 ｜ null bitmap length (4 bytes if has null value) ｜
 +-------------------------------------------------+
 ｜ bitmap index block number (4 bytes int)        ｜
 +-------------------------------------------------+
 ｜ value 1 | offset 1                             ｜
 +-------------------------------------------------+
 ｜ value 2 | offset 2                             ｜
 +-------------------------------------------------+
 ｜ ...                                            ｜
 +-------------------------------------------------+
 ｜ bitmap blocks offset (4 bytes int)             ｜
 +-------------------------------------------------+-----------------
 ｜ bitmap index block 1                           ｜
 +-------------------------------------------------+
 ｜ bitmap index block 2                           ｜  INDEX BLOCKS
 +-------------------------------------------------+
 ｜ ...                                            ｜
 +-------------------------------------------------+-----------------
 ｜ serialized bitmap 1                            ｜
 +-------------------------------------------------+
 ｜ serialized bitmap 2                            ｜
 +-------------------------------------------------+  BITMAP BLOCKS
 ｜ serialized bitmap 3                            ｜
 +-------------------------------------------------+
 ｜ ...                                            ｜
 +-------------------------------------------------+-----------------

 index block format:
 +-------------------------------------------------+
 ｜ entry number (4 bytes int)                     ｜
 +-------------------------------------------------+
 ｜ value 1 | offset 1 | length 1                  ｜
 +-------------------------------------------------+
 ｜ value 2 | offset 2 | length 2                  ｜
 +-------------------------------------------------+
 ｜ ...                                            ｜
 +-------------------------------------------------+
 

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.paimon.fileindex.bitmap.BitmapFileIndexMeta

BitmapFileIndexMeta.DataTypeVisitorAdapter<R>, BitmapFileIndexMeta.Entry, BitmapFileIndexMeta.ThrowableConsumer, BitmapFileIndexMeta.ThrowableSupplier

Field Summary

Fields inherited from class org.apache.paimon.fileindex.bitmap.BitmapFileIndexMeta

bitmapLengths, bitmapOffsets, bodyStart, dataType, hasNullValue, nonNullBitmapNumber, nullValueOffset, options, rowCount

Constructor Summary

Constructors

Constructor and Description
BitmapFileIndexMetaV2(DataType dataType, Options options)
BitmapFileIndexMetaV2(DataType dataType, Options options, int rowCount, int nonNullBitmapNumber, boolean hasNullValue, int nullValueOffset, int nullBitmapLength, LinkedHashMap<Object,Integer> bitmapOffsets, int finalOffset)

Method Summary

Modifier and Type	Method and Description
void	deserialize(SeekableInputStream seekableInputStream)
BitmapFileIndexMeta.Entry	findEntry(Object bitmapId) Find entry for bitmap.
static Comparator<Object>	getComparator(DataType dataType)
void	serialize(DataOutput out)

Methods inherited from class org.apache.paimon.fileindex.bitmap.BitmapFileIndexMeta

getBodyStart, getRowCount, getSerializeSizeMeasure, getValueReader, getValueWriter

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BitmapFileIndexMetaV2

public BitmapFileIndexMetaV2(DataType dataType,
                             Options options)

BitmapFileIndexMetaV2

public BitmapFileIndexMetaV2(DataType dataType,
                             Options options,
                             int rowCount,
                             int nonNullBitmapNumber,
                             boolean hasNullValue,
                             int nullValueOffset,
                             int nullBitmapLength,
                             LinkedHashMap<Object,Integer> bitmapOffsets,
                             int finalOffset)

Method Detail

getComparator

public static Comparator<Object> getComparator(DataType dataType)

findEntry

public BitmapFileIndexMeta.Entry findEntry(Object bitmapId)

Description copied from class: BitmapFileIndexMeta

Find entry for bitmap.

Overrides:

findEntry in class BitmapFileIndexMeta

Parameters:

bitmapId - the bitmap identifier to be searched.

Returns:

an BitmapFileIndexMeta.Entry which contains offset and length of bitmap if it is contained in the index meta; otherwise `null`.

serialize

public void serialize(DataOutput out)
               throws Exception

Overrides:

serialize in class BitmapFileIndexMeta

Throws:

Exception

deserialize

public void deserialize(SeekableInputStream seekableInputStream)
                 throws Exception

Overrides:

deserialize in class BitmapFileIndexMeta

Throws:

Exception