This documentation is for an unreleased version of Apache Paimon. We recommend you use the latest stable version.

Manifest #

Manifest List #

├── manifest
    └── manifest-list-51c16f7b-421c-4bc0-80a0-17677f343358-1

Manifest List includes meta of several manifest files. Its name contains UUID, it is a avro file, the schema is:

1. _FILE_NAME: STRING, manifest file name.
2. _FILE_SIZE: BIGINT, manifest file size.
3. _NUM_ADDED_FILES: BIGINT, number added files in manifest.
4. _NUM_DELETED_FILES: BIGINT, number deleted files in manifest.
5. _PARTITION_STATS: SimpleStats, partition stats, the minimum and maximum values of partition fields in this manifest are beneficial for skipping certain manifest files during queries, it is a SimpleStats.
6. _SCHEMA_ID: BIGINT, schema id when writing this manifest file.

Manifest #

Manifest includes meta of several data files or changelog files or table-index files. Its name contains UUID, it is an avro file.

The changes of the file are saved in the manifest, and the file can be added or deleted. Manifests should be in an orderly manner, and the same file may be added or deleted multiple times. The last version should be read. This design can make commit lighter to support file deletion generated by compaction.

Data Manifest #

Data Manifest includes meta of several data files or changelog files.

├── manifest
    └── manifest-6758823b-2010-4d06-aef0-3b1b597723d6-0

The schema is:

1. _KIND: TINYINT, ADD or DELETE,
2. _PARTITION: BYTES, partition spec, a BinaryRow.
3. _BUCKET: INT, bucket of this file.
4. _TOTAL_BUCKETS: INT, total buckets when write this file, it is used for verification after bucket changes.
5. _FILE: data file meta.

The data file meta is:

1. _FILE_NAME: STRING, file name.
2. _FILE_SIZE: BIGINT, file size.
3. _ROW_COUNT: BIGINT, total number of rows (including add & delete) in this file.
4. _MIN_KEY: STRING, the minimum key of this file.
5. _MAX_KEY: STRING, the maximum key of this file.
6. _KEY_STATS: SimpleStats, the statistics of the key.
7. _VALUE_STATS: SimpleStats, the statistics of the value.
8. _MIN_SEQUENCE_NUMBER: BIGINT, the minimum sequence number.
9. _MAX_SEQUENCE_NUMBER: BIGINT, the maximum sequence number.
10. _SCHEMA_ID: BIGINT, schema id when write this file.
11. _LEVEL: INT, level of this file, in LSM.
12. _EXTRA_FILES: ARRAY, extra files for this file, for example, data file index file.
13. _CREATION_TIME: TIMESTAMP_MILLIS, creation time of this file.
14. _DELETE_ROW_COUNT: BIGINT, rowCount = addRowCount + deleteRowCount.
15. _EMBEDDED_FILE_INDEX: BYTES, if data file index is too small, store the index in manifest.
16. _FILE_SOURCE: TINYINT, indicate whether this file is generated as an APPEND or COMPACT file
17. _VALUE_STATS_COLS: ARRAY, statistical column in metadata

Index Manifest #

Index Manifest includes meta of several table-index files.

├── manifest
    └── index-manifest-5d670043-da25-4265-9a26-e31affc98039-0

The schema is:

1. _KIND: TINYINT, ADD or DELETE,
2. _PARTITION: BYTES, partition spec, a BinaryRow.
3. _BUCKET: INT, bucket of this file.
4. _INDEX_TYPE: STRING, “HASH” or “DELETION_VECTORS”.
5. _FILE_NAME: STRING, file name.
6. _FILE_SIZE: BIGINT, file size.
7. _ROW_COUNT: BIGINT, total number of rows.
8. _DELETIONS_VECTORS_RANGES: Metadata only used by “DELETION_VECTORS”, is an array of deletion vector meta, the schema of each deletion vector meta is:
   1. f0: the data file name corresponding to this deletion vector.
   2. f1: the starting offset of this deletion vector in the index file.
   3. f2: the length of this deletion vector in the index file.
   4. _CARDINALITY: the number of deleted rows.

Appendix #

SimpleStats #

SimpleStats is nested row, the schema is:

1. _MIN_VALUES: BYTES, BinaryRow, the minimum values of the columns.
2. _MAX_VALUES: BYTES, BinaryRow, the maximum values of the columns.
3. _NULL_COUNTS: ARRAY, the number of nulls of the columns.

BinaryRow #

BinaryRow is backed by bytes instead of Object. It can significantly reduce the serialization/deserialization of Java objects.

A Row has two part: Fixed-length part and variable-length part. Fixed-length part contains 1 byte header and null bit set and field values. Null bit set is used for null tracking and is aligned to 8-byte word boundaries. Field values holds fixed-length primitive types and variable-length values which can be stored in 8 bytes inside. If it do not fit the variable-length field, then store the length and offset of variable-length part.