org.apache.paimon.format.parquet.newreader

Class VectorizedRleValuesReader

java.lang.Object

org.apache.parquet.column.values.ValuesReader

org.apache.paimon.format.parquet.newreader.VectorizedRleValuesReader

All Implemented Interfaces:

VectorizedValuesReader

public final class VectorizedRleValuesReader
extends org.apache.parquet.column.values.ValuesReader
implements VectorizedValuesReader

A values reader for Parquet's run-length encoded data. This is based off of the version in parquet-mr with these changes: - Supports the vectorized interface. - Works on byte arrays(byte[]) instead of making byte streams.

This encoding is used in multiple places: - Definition/Repetition levels - Dictionary ids. - Boolean type values of Parquet DataPageV2

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.paimon.format.parquet.newreader.VectorizedValuesReader

VectorizedValuesReader.ByteBufferOutputWriter, VectorizedValuesReader.IntegerOutputWriter

Constructor Summary

Constructors

Constructor and Description
VectorizedRleValuesReader()
VectorizedRleValuesReader(int bitWidth)
VectorizedRleValuesReader(int bitWidth, boolean readLength)

Method Summary

Modifier and Type	Method and Description
void	initFromPage(int valueCount, org.apache.parquet.bytes.ByteBufferInputStream in)
void	readBatch(ParquetReadState state, WritableColumnVector values, WritableIntVector defLevels, VectorizedValuesReader valueReader, ParquetVectorUpdater updater) Reads a batch of definition levels and values into vector 'defLevels' and 'values' respectively.
void	readBatchRepeated(ParquetReadState state, WritableIntVector repLevels, VectorizedRleValuesReader defLevelsReader, WritableIntVector defLevels, WritableColumnVector values, VectorizedValuesReader valueReader, ParquetVectorUpdater updater) Reads a batch of repetition levels, definition levels and values into 'repLevels', 'defLevels' and 'values' respectively.
void	readBatchRepeatedInternal(ParquetReadState state, WritableIntVector repLevels, VectorizedRleValuesReader defLevelsReader, WritableIntVector defLevels, WritableColumnVector values, WritableColumnVector nulls, boolean valuesReused, VectorizedValuesReader valueReader, ParquetVectorUpdater updater) Keep reading repetition level values from the page until either: 1) we've read enough top-level rows to fill the current batch, or 2) we've drained the data page completely.
org.apache.parquet.io.api.Binary	readBinary(int len)
void	readBinary(int total, WritableBytesVector c, int rowId)
boolean	readBoolean()
void	readBooleans(int total, WritableBooleanVector c, int rowId)
byte	readByte()
void	readBytes(int total, WritableByteVector c, int rowId)
void	readDoubles(int total, WritableDoubleVector c, int rowId)
void	readFloats(int total, WritableFloatVector c, int rowId)
int	readInteger()
void	readIntegers(int total, WritableIntVector c, int rowId)
void	readIntegers(ParquetReadState state, WritableColumnVector values, WritableColumnVector nulls, WritableIntVector defLevels, VectorizedValuesReader valueReader) Decoding for dictionary ids.
void	readIntegersRepeated(ParquetReadState state, WritableIntVector repLevels, VectorizedRleValuesReader defLevelsReader, WritableIntVector defLevels, WritableColumnVector values, WritableColumnVector nulls, VectorizedValuesReader valueReader) Reads a batch of repetition levels, definition levels and integer values into 'repLevels', 'defLevels', 'values' and 'nulls' respectively.
void	readLongs(int total, WritableLongVector c, int rowId)
short	readShort()
void	readShorts(int total, WritableShortVector c, int rowId)
int	readValueDictionaryId()
void	skip()
void	skipBinary(int total)
void	skipBooleans(int total)
void	skipBytes(int total)
void	skipDoubles(int total)
void	skipFixedLenByteArray(int total, int len)
void	skipFloats(int total)
void	skipIntegers(int total)
void	skipLongs(int total)
void	skipShorts(int total)

Methods inherited from class org.apache.parquet.column.values.ValuesReader

getNextOffset, initFromPage, initFromPage, readBytes, readDouble, readFloat, readLong, skip, updateNextOffset

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.paimon.format.parquet.newreader.VectorizedValuesReader

readDouble, readFloat, readLong

Constructor Detail

VectorizedRleValuesReader

public VectorizedRleValuesReader()

VectorizedRleValuesReader

public VectorizedRleValuesReader(int bitWidth)

VectorizedRleValuesReader

public VectorizedRleValuesReader(int bitWidth,
                                 boolean readLength)

Method Detail

initFromPage

public void initFromPage(int valueCount,
                         org.apache.parquet.bytes.ByteBufferInputStream in)
                  throws IOException

Overrides:

initFromPage in class org.apache.parquet.column.values.ValuesReader

Throws:

IOException

readBoolean

public boolean readBoolean()

Specified by:

readBoolean in interface VectorizedValuesReader

Overrides:

readBoolean in class org.apache.parquet.column.values.ValuesReader

skip

public void skip()

Specified by:

skip in class org.apache.parquet.column.values.ValuesReader

readValueDictionaryId

public int readValueDictionaryId()

Overrides:

readValueDictionaryId in class org.apache.parquet.column.values.ValuesReader

readInteger

public int readInteger()

Specified by:

readInteger in interface VectorizedValuesReader

Overrides:

readInteger in class org.apache.parquet.column.values.ValuesReader

readBatch

public void readBatch(ParquetReadState state,
                      WritableColumnVector values,
                      WritableIntVector defLevels,
                      VectorizedValuesReader valueReader,
                      ParquetVectorUpdater updater)

Reads a batch of definition levels and values into vector 'defLevels' and 'values' respectively. The values are read using 'valueReader'.

The related states such as row index, offset, number of values left in the batch and page, are tracked by 'state'. The type-specific 'updater' is used to update or skip values.

This reader reads the definition levels and then will read from 'valueReader' for the non-null values. If the value is null, 'values' will be populated with null value.

readIntegers

public void readIntegers(ParquetReadState state,
                         WritableColumnVector values,
                         WritableColumnVector nulls,
                         WritableIntVector defLevels,
                         VectorizedValuesReader valueReader)

Decoding for dictionary ids. The IDs are populated into 'values' and the nullability is populated into 'nulls'.

readBatchRepeated

public void readBatchRepeated(ParquetReadState state,
                              WritableIntVector repLevels,
                              VectorizedRleValuesReader defLevelsReader,
                              WritableIntVector defLevels,
                              WritableColumnVector values,
                              VectorizedValuesReader valueReader,
                              ParquetVectorUpdater updater)

Reads a batch of repetition levels, definition levels and values into 'repLevels', 'defLevels' and 'values' respectively. The definition levels and values are read via 'defLevelsReader' and 'valueReader' respectively.

The related states such as row index, offset, number of rows left in the batch and page, are tracked by 'state'. The type-specific 'updater' is used to update or skip values.

readIntegersRepeated

public void readIntegersRepeated(ParquetReadState state,
                                 WritableIntVector repLevels,
                                 VectorizedRleValuesReader defLevelsReader,
                                 WritableIntVector defLevels,
                                 WritableColumnVector values,
                                 WritableColumnVector nulls,
                                 VectorizedValuesReader valueReader)

Reads a batch of repetition levels, definition levels and integer values into 'repLevels', 'defLevels', 'values' and 'nulls' respectively. The definition levels and values are read via 'defLevelsReader' and 'valueReader' respectively.

The 'values' vector is used to hold non-null values, while 'nulls' vector is used to hold null values.

The related states such as row index, offset, number of rows left in the batch and page, are tracked by 'state'.

Unlike 'readBatchRepeated', this is used to decode dictionary indices in dictionary encoding.

readBatchRepeatedInternal

public void readBatchRepeatedInternal(ParquetReadState state,
                                      WritableIntVector repLevels,
                                      VectorizedRleValuesReader defLevelsReader,
                                      WritableIntVector defLevels,
                                      WritableColumnVector values,
                                      WritableColumnVector nulls,
                                      boolean valuesReused,
                                      VectorizedValuesReader valueReader,
                                      ParquetVectorUpdater updater)

Keep reading repetition level values from the page until either: 1) we've read enough top-level rows to fill the current batch, or 2) we've drained the data page completely.

Parameters:

valuesReused - whether 'values' vector is reused for 'nulls'

readIntegers

public void readIntegers(int total,
                         WritableIntVector c,
                         int rowId)

Specified by:

readIntegers in interface VectorizedValuesReader

readByte

public byte readByte()

Specified by:

readByte in interface VectorizedValuesReader

readShort

public short readShort()

Specified by:

readShort in interface VectorizedValuesReader

readBytes

public void readBytes(int total,
                      WritableByteVector c,
                      int rowId)

Specified by:

readBytes in interface VectorizedValuesReader

readShorts

public void readShorts(int total,
                       WritableShortVector c,
                       int rowId)

Specified by:

readShorts in interface VectorizedValuesReader

readLongs

public void readLongs(int total,
                      WritableLongVector c,
                      int rowId)

Specified by:

readLongs in interface VectorizedValuesReader

readBinary

public void readBinary(int total,
                       WritableBytesVector c,
                       int rowId)

Specified by:

readBinary in interface VectorizedValuesReader

readBooleans

public void readBooleans(int total,
                         WritableBooleanVector c,
                         int rowId)

Specified by:

readBooleans in interface VectorizedValuesReader

readFloats

public void readFloats(int total,
                       WritableFloatVector c,
                       int rowId)

Specified by:

readFloats in interface VectorizedValuesReader

readDoubles

public void readDoubles(int total,
                        WritableDoubleVector c,
                        int rowId)

Specified by:

readDoubles in interface VectorizedValuesReader

readBinary

public org.apache.parquet.io.api.Binary readBinary(int len)

Specified by:

readBinary in interface VectorizedValuesReader

skipIntegers

public void skipIntegers(int total)

Specified by:

skipIntegers in interface VectorizedValuesReader

skipBooleans

public void skipBooleans(int total)

Specified by:

skipBooleans in interface VectorizedValuesReader

skipBytes

public void skipBytes(int total)

Specified by:

skipBytes in interface VectorizedValuesReader

skipShorts

public void skipShorts(int total)

Specified by:

skipShorts in interface VectorizedValuesReader

skipLongs

public void skipLongs(int total)

Specified by:

skipLongs in interface VectorizedValuesReader

skipFloats

public void skipFloats(int total)

Specified by:

skipFloats in interface VectorizedValuesReader

skipDoubles

public void skipDoubles(int total)

Specified by:

skipDoubles in interface VectorizedValuesReader

skipBinary

public void skipBinary(int total)

Specified by:

skipBinary in interface VectorizedValuesReader

skipFixedLenByteArray

public void skipFixedLenByteArray(int total,
                                  int len)

Specified by:

skipFixedLenByteArray in interface VectorizedValuesReader