Mosaic

A columnar-bucket hybrid format optimized for wide tables.

Rust Java Python C/C++

Overview

Mosaic is a columnar-bucket hybrid format optimized for wide tables (10,000+ columns). Columns are sorted by name and evenly distributed into buckets using range-based assignment, stored column-oriented within each bucket, and independently compressed. This enables efficient projection pushdown at bucket granularity — reading 10 columns out of 10,000 only decompresses the buckets that contain those 10 columns. Range-based assignment ensures that columns with similar name prefixes land in the same bucket, improving both compression ratio and projection locality.

Mosaic is implemented as a Rust core library with bindings for Java (via JNI), Python (via ctypes FFI), and C/C++ (via FFI), enabling high-performance read and write access across multiple language ecosystems.

Key Features

Columnar-Bucket Hybrid

Columns sorted by name are distributed into buckets via range-based assignment, enabling projection pushdown at bucket granularity. Similar name prefixes land in the same bucket.

Adaptive Encoding

Each column is automatically encoded as ALL_NULL, CONST, DICT, or PLAIN based on its data distribution.

Zstd Compression

Optional Zstandard compression per bucket and schema block, with configurable compression level. Each bucket is independently compressed.

BPE Name Compression

Byte Pair Encoding compresses column names in the schema block, reducing metadata overhead for wide tables.

Rich Type System

18 data types from Boolean to TimestampLtz, with support for fixed-width and variable-length encodings.

Multi-Language

Rust core with Java JNI bindings, Python ctypes bindings, and C/C++ FFI headers. Write once in Rust, use everywhere.

Supported Types

Type	Width	Description
`Boolean`	1	true / false
`TinyInt`	1	Signed 8-bit integer
`SmallInt`	2	Signed 16-bit integer
`Integer`	4	Signed 32-bit integer
`BigInt`	8	Signed 64-bit integer
`Float`	4	32-bit IEEE 754
`Double`	8	64-bit IEEE 754
`Date`	4	Days since epoch
`Time`	4	Milliseconds since midnight
`Char(n)`	variable	Fixed-length string
`VarChar(n)`	variable	Variable-length string with max length
`String`	variable	Unbounded UTF-8 string
`Binary(n)`	variable	Fixed-length byte array
`VarBinary(n)`	variable	Variable-length byte array with max length
`Bytes`	variable	Unbounded byte array
`Decimal(p, s)`	8 or variable	Exact numeric; compact (p≤18) or large
`Timestamp(p)`	8 or 12	Millis (p≤3), micros (p≤6), or millis + nanos (p>6)
`TimestampLtz(p)`	8 or 12	Same as Timestamp, with local timezone

Benchmark

Test setup: 10,000 columns (90% STRING, 10% INT), column names ~80 bytes each, Zstd compression (level 9).

File Size (10 rows)

Format	Size	vs Mosaic
Parquet	9,696 KB	14.8x
ORC	6,377 KB	9.7x
Mosaic	654 KB	1x

Projection Read (500 rows)

File size — Parquet: 57.4 MB, ORC: 95.4 MB, Mosaic: 11.5 MB

Projected Columns	Parquet	ORC	Mosaic
10 / 10,000	53,170 us	72,729 us	25,081 us
1 / 10,000	50,919 us	70,712 us	2,374 us

Projection Read (4,500 rows)

File size — Parquet: 458.4 MB, ORC: 827.9 MB, Mosaic: 100.2 MB

Projected Columns	Parquet	ORC	Mosaic
10 / 10,000	369,627 us	89,344 us	67,314 us
1 / 10,000	360,458 us	81,934 us	26,924 us

Why is Mosaic faster for projection? Projection pushdown operates at bucket granularity. With 10,000 columns distributed across 100 buckets, reading 1 column only decompresses the 1 bucket that contains it — roughly 1% of the file. Range-based assignment keeps columns with similar name prefixes co-located, so typical projections touch very few buckets.

Status

Mosaic is under active development as part of the Apache Paimon ecosystem. Both the write path and read path are fully implemented with round-trip test coverage.