org.apache.paimon.memory

Class MemorySegmentUtils

java.lang.Object

org.apache.paimon.memory.MemorySegmentUtils

public class MemorySegmentUtils
extends Object

Utils for MemorySegment.

Field Summary

Fields

Modifier and Type	Field and Description
static int	BIT_BYTE_INDEX_MASK

Constructor Summary

Constructors

Constructor and Description
MemorySegmentUtils()

Method Summary

Modifier and Type	Method and Description
static byte[]	allocateReuseBytes(int length) Allocate bytes that is only for temporary usage, it should not be stored in somewhere else.
static char[]	allocateReuseChars(int length)
static boolean	bitGet(MemorySegment[] segments, int baseOffset, int index) read bit from segments.
static boolean	bitGet(MemorySegment segment, int baseOffset, int index) read bit.
static void	bitSet(MemorySegment[] segments, int baseOffset, int index) set bit from segments.
static void	bitSet(MemorySegment segment, int baseOffset, int index) set bit.
static void	bitUnSet(MemorySegment[] segments, int baseOffset, int index) unset bit from segments.
static void	bitUnSet(MemorySegment segment, int baseOffset, int index) unset bit.
static int	byteIndex(int bitIndex) Given a bit index, return the byte index containing it.
static void	copyFromBytes(MemorySegment[] segments, int offset, byte[] bytes, int bytesOffset, int numBytes) Copy target segments from source byte[].
static void	copyMultiSegmentsToBytes(MemorySegment[] segments, int offset, byte[] bytes, int bytesOffset, int numBytes)
static byte[]	copyToBytes(MemorySegment[] segments, int offset, byte[] bytes, int bytesOffset, int numBytes) Copy segments to target byte[].
static byte[]	copyToBytes(MemorySegment[] segments, int offset, int numBytes) Copy segments to a new byte[].
static void	copyToUnsafe(MemorySegment[] segments, int offset, Object target, int pointer, int numBytes) Copy segments to target unsafe pointer.
static void	copyToView(MemorySegment[] segments, int offset, int sizeInBytes, DataOutputView target) Copy bytes of segments to output view.
static boolean	equals(MemorySegment[] segments1, int offset1, MemorySegment[] segments2, int offset2, int len) Equals two memory segments regions.
static int	find(MemorySegment[] segments1, int offset1, int numBytes1, MemorySegment[] segments2, int offset2, int numBytes2) Find equal segments2 in segments1.
static boolean	getBoolean(MemorySegment[] segments, int offset) get boolean from segments.
static byte	getByte(MemorySegment[] segments, int offset) get byte from segments.
static byte[]	getBytes(MemorySegment[] segments, int baseOffset, int sizeInBytes) Maybe not copied, if want copy, please use copyTo.
static double	getDouble(MemorySegment[] segments, int offset) get double from segments.
static float	getFloat(MemorySegment[] segments, int offset) get float from segments.
static int	getInt(MemorySegment[] segments, int offset) get int from segments.
static long	getLong(MemorySegment[] segments, int offset) get long from segments.
static short	getShort(MemorySegment[] segments, int offset) get short from segments.
static int	hash(MemorySegment[] segments, int offset, int numBytes) hash segments to int.
static int	hashByWords(MemorySegment[] segments, int offset, int numBytes) hash segments to int, numBytes must be aligned to 4 bytes.
static InternalArray	readArrayData(MemorySegment[] segments, int baseOffset, long offsetAndSize) Gets an instance of InternalArray from underlying MemorySegment.
static byte[]	readBinary(MemorySegment[] segments, int baseOffset, int fieldOffset, long variablePartOffsetAndLen) Get binary, if len less than 8, will be include in variablePartOffsetAndLen.
static BinaryString	readBinaryString(MemorySegment[] segments, int baseOffset, int fieldOffset, long variablePartOffsetAndLen) Get binary string, if len less than 8, will be include in variablePartOffsetAndLen.
static Decimal	readDecimal(MemorySegment[] segments, int baseOffset, long offsetAndSize, int precision, int scale) Gets an instance of Decimal from underlying MemorySegment.
static InternalMap	readMapData(MemorySegment[] segments, int baseOffset, long offsetAndSize) Gets an instance of InternalMap from underlying MemorySegment.
static InternalRow	readRowData(MemorySegment[] segments, int numFields, int baseOffset, long offsetAndSize) Gets an instance of InternalRow from underlying MemorySegment.
static Timestamp	readTimestampData(MemorySegment[] segments, int baseOffset, long offsetAndNanos) Gets an instance of Timestamp from underlying MemorySegment.
static void	setBoolean(MemorySegment[] segments, int offset, boolean value) set boolean from segments.
static void	setByte(MemorySegment[] segments, int offset, byte value) set byte from segments.
static void	setDouble(MemorySegment[] segments, int offset, double value) set double from segments.
static void	setFloat(MemorySegment[] segments, int offset, float value) set float from segments.
static void	setInt(MemorySegment[] segments, int offset, int value) set int from segments.
static void	setLong(MemorySegment[] segments, int offset, long value) set long from segments.
static void	setShort(MemorySegment[] segments, int offset, short value) set short from segments.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

BIT_BYTE_INDEX_MASK

public static final int BIT_BYTE_INDEX_MASK

See Also:

Constant Field Values

Constructor Detail

MemorySegmentUtils

public MemorySegmentUtils()

Method Detail

allocateReuseBytes

public static byte[] allocateReuseBytes(int length)

Allocate bytes that is only for temporary usage, it should not be stored in somewhere else. Use a ThreadLocal to reuse bytes to avoid overhead of byte[] new and gc.

If there are methods that can only accept a byte[], instead of a MemorySegment[] parameter, we can allocate a reuse bytes and copy the MemorySegment data to byte[], then call the method. Such as String deserialization.

allocateReuseChars

public static char[] allocateReuseChars(int length)

copyToView

public static void copyToView(MemorySegment[] segments,
                              int offset,
                              int sizeInBytes,
                              DataOutputView target)
                       throws IOException

Copy bytes of segments to output view.

Note: It just copies the data in, not include the length.

Parameters:

segments - source segments

offset - offset for segments

sizeInBytes - size in bytes

target - target output view

Throws:

IOException

copyFromBytes

public static void copyFromBytes(MemorySegment[] segments,
                                 int offset,
                                 byte[] bytes,
                                 int bytesOffset,
                                 int numBytes)

Copy target segments from source byte[].

Parameters:

segments - target segments.

offset - target segments offset.

bytes - source byte[].

bytesOffset - source byte[] offset.

numBytes - the number bytes to copy.

getBytes

public static byte[] getBytes(MemorySegment[] segments,
                              int baseOffset,
                              int sizeInBytes)

Maybe not copied, if want copy, please use copyTo.

equals

public static boolean equals(MemorySegment[] segments1,
                             int offset1,
                             MemorySegment[] segments2,
                             int offset2,
                             int len)

Equals two memory segments regions.

Parameters:

segments1 - Segments 1

offset1 - Offset of segments1 to start equaling

segments2 - Segments 2

offset2 - Offset of segments2 to start equaling

len - Length of the equaled memory region

Returns:

true if equal, false otherwise

copyToBytes

public static byte[] copyToBytes(MemorySegment[] segments,
                                 int offset,
                                 int numBytes)

Copy segments to a new byte[].

Parameters:

segments - Source segments.

offset - Source segments offset.

numBytes - the number bytes to copy.

copyToBytes

public static byte[] copyToBytes(MemorySegment[] segments,
                                 int offset,
                                 byte[] bytes,
                                 int bytesOffset,
                                 int numBytes)

Copy segments to target byte[].

Parameters:

segments - Source segments.

offset - Source segments offset.

bytes - target byte[].

bytesOffset - target byte[] offset.

numBytes - the number bytes to copy.

copyMultiSegmentsToBytes

public static void copyMultiSegmentsToBytes(MemorySegment[] segments,
                                            int offset,
                                            byte[] bytes,
                                            int bytesOffset,
                                            int numBytes)

copyToUnsafe

public static void copyToUnsafe(MemorySegment[] segments,
                                int offset,
                                Object target,
                                int pointer,
                                int numBytes)

Copy segments to target unsafe pointer.

Parameters:

segments - Source segments.

offset - The position where the bytes are started to be read from these memory segments.

target - The unsafe memory to copy the bytes to.

pointer - The position in the target unsafe memory to copy the chunk to.

numBytes - the number bytes to copy.

hashByWords

public static int hashByWords(MemorySegment[] segments,
                              int offset,
                              int numBytes)

hash segments to int, numBytes must be aligned to 4 bytes.

Parameters:

segments - Source segments.

offset - Source segments offset.

numBytes - the number bytes to hash.

hash

public static int hash(MemorySegment[] segments,
                       int offset,
                       int numBytes)

hash segments to int.

Parameters:

segments - Source segments.

offset - Source segments offset.

numBytes - the number bytes to hash.

find

public static int find(MemorySegment[] segments1,
                       int offset1,
                       int numBytes1,
                       MemorySegment[] segments2,
                       int offset2,
                       int numBytes2)

Find equal segments2 in segments1.

Parameters:

segments1 - segs to find.

segments2 - sub segs.

Returns:

Return the found offset, return -1 if not find.

byteIndex

public static int byteIndex(int bitIndex)

Given a bit index, return the byte index containing it.

Parameters:

bitIndex - the bit index.

Returns:

the byte index.

bitUnSet

public static void bitUnSet(MemorySegment segment,
                            int baseOffset,
                            int index)

unset bit.

Parameters:

segment - target segment.

baseOffset - bits base offset.

index - bit index from base offset.

bitSet

public static void bitSet(MemorySegment segment,
                          int baseOffset,
                          int index)

set bit.

Parameters:

segment - target segment.

baseOffset - bits base offset.

index - bit index from base offset.

bitGet

public static boolean bitGet(MemorySegment segment,
                             int baseOffset,
                             int index)

read bit.

Parameters:

segment - target segment.

baseOffset - bits base offset.

index - bit index from base offset.

bitUnSet

public static void bitUnSet(MemorySegment[] segments,
                            int baseOffset,
                            int index)

unset bit from segments.

Parameters:

segments - target segments.

baseOffset - bits base offset.

index - bit index from base offset.

bitSet

public static void bitSet(MemorySegment[] segments,
                          int baseOffset,
                          int index)

set bit from segments.

Parameters:

segments - target segments.

baseOffset - bits base offset.

index - bit index from base offset.

bitGet

public static boolean bitGet(MemorySegment[] segments,
                             int baseOffset,
                             int index)

read bit from segments.

Parameters:

segments - target segments.

baseOffset - bits base offset.

index - bit index from base offset.

getBoolean

public static boolean getBoolean(MemorySegment[] segments,
                                 int offset)

get boolean from segments.

Parameters:

segments - target segments.

offset - value offset.

setBoolean

public static void setBoolean(MemorySegment[] segments,
                              int offset,
                              boolean value)

set boolean from segments.

Parameters:

segments - target segments.

offset - value offset.

getByte

public static byte getByte(MemorySegment[] segments,
                           int offset)

get byte from segments.

Parameters:

segments - target segments.

offset - value offset.

setByte

public static void setByte(MemorySegment[] segments,
                           int offset,
                           byte value)

set byte from segments.

Parameters:

segments - target segments.

offset - value offset.

getInt

public static int getInt(MemorySegment[] segments,
                         int offset)

get int from segments.

Parameters:

segments - target segments.

offset - value offset.

setInt

public static void setInt(MemorySegment[] segments,
                          int offset,
                          int value)

set int from segments.

Parameters:

segments - target segments.

offset - value offset.

getLong

public static long getLong(MemorySegment[] segments,
                           int offset)

get long from segments.

Parameters:

segments - target segments.

offset - value offset.

setLong

public static void setLong(MemorySegment[] segments,
                           int offset,
                           long value)

set long from segments.

Parameters:

segments - target segments.

offset - value offset.

getShort

public static short getShort(MemorySegment[] segments,
                             int offset)

get short from segments.

Parameters:

segments - target segments.

offset - value offset.

setShort

public static void setShort(MemorySegment[] segments,
                            int offset,
                            short value)

set short from segments.

Parameters:

segments - target segments.

offset - value offset.

getFloat

public static float getFloat(MemorySegment[] segments,
                             int offset)

get float from segments.

Parameters:

segments - target segments.

offset - value offset.

setFloat

public static void setFloat(MemorySegment[] segments,
                            int offset,
                            float value)

set float from segments.

Parameters:

segments - target segments.

offset - value offset.

getDouble

public static double getDouble(MemorySegment[] segments,
                               int offset)

get double from segments.

Parameters:

segments - target segments.

offset - value offset.

setDouble

public static void setDouble(MemorySegment[] segments,
                             int offset,
                             double value)

set double from segments.

Parameters:

segments - target segments.

offset - value offset.

readDecimal

public static Decimal readDecimal(MemorySegment[] segments,
                                  int baseOffset,
                                  long offsetAndSize,
                                  int precision,
                                  int scale)

Gets an instance of Decimal from underlying MemorySegment.

readTimestampData

public static Timestamp readTimestampData(MemorySegment[] segments,
                                          int baseOffset,
                                          long offsetAndNanos)

Gets an instance of Timestamp from underlying MemorySegment.

Parameters:

segments - the underlying MemorySegments

baseOffset - the base offset of current instance of TimestampData

offsetAndNanos - the offset of milli-seconds part and nanoseconds

Returns:

an instance of Timestamp

readBinary

public static byte[] readBinary(MemorySegment[] segments,
                                int baseOffset,
                                int fieldOffset,
                                long variablePartOffsetAndLen)

Get binary, if len less than 8, will be include in variablePartOffsetAndLen.

Note: Need to consider the ByteOrder.

Parameters:

baseOffset - base offset of composite binary format.

fieldOffset - absolute start offset of 'variablePartOffsetAndLen'.

variablePartOffsetAndLen - a long value, real data or offset and len.

readBinaryString

public static BinaryString readBinaryString(MemorySegment[] segments,
                                            int baseOffset,
                                            int fieldOffset,
                                            long variablePartOffsetAndLen)

Get binary string, if len less than 8, will be include in variablePartOffsetAndLen.

Note: Need to consider the ByteOrder.

Parameters:

baseOffset - base offset of composite binary format.

fieldOffset - absolute start offset of 'variablePartOffsetAndLen'.

variablePartOffsetAndLen - a long value, real data or offset and len.

readMapData

public static InternalMap readMapData(MemorySegment[] segments,
                                      int baseOffset,
                                      long offsetAndSize)

Gets an instance of InternalMap from underlying MemorySegment.

readArrayData

public static InternalArray readArrayData(MemorySegment[] segments,
                                          int baseOffset,
                                          long offsetAndSize)

Gets an instance of InternalArray from underlying MemorySegment.

readRowData

public static InternalRow readRowData(MemorySegment[] segments,
                                      int numFields,
                                      int baseOffset,
                                      long offsetAndSize)

Gets an instance of InternalRow from underlying MemorySegment.