Parquet Metadata Cache#

Overview#

Paimon C++ can cache serialized Parquet metadata footer bytes for Parquet data files. The cache is used by ParquetReaderBuilder before opening the Arrow Parquet reader. On a cache miss, Paimon C++ loads the Parquet file metadata, serializes it as a complete metadata footer, and stores those bytes in the public Cache abstraction. On a cache hit, Paimon C++ parses the cached footer bytes into parquet::FileMetaData and passes the metadata to the Parquet reader.

The cache stores serialized metadata footer bytes instead of caching a parquet::FileMetaData instance. This keeps the cache value compact and similar to manifest cache values: the cache weight follows the actual cached bytes, while the Parquet library still owns metadata parsing and validation.

This optimization is useful when the same Parquet files are opened repeatedly in the same process, for example repeated get or scan requests over the same snapshot. On a cache hit, the read path avoids reading the Parquet footer bytes from the filesystem again. Paimon C++ still parses the cached footer bytes into parquet::FileMetaData for each reader open. ColumnIndex and OffsetIndex bytes also use CacheKind::DATA_FILE_FOOTER with their actual positions and lengths. Data pages and column chunks are not stored in this metadata cache.

Configuration#

Parquet metadata caching is disabled by default. Embedding applications that need it can provide a custom Cache implementation and inject it through ScanContextBuilder or ReadContextBuilder. Parquet reader builders receive the cache from the read context and create cache keys with CacheKind::DATA_FILE_FOOTER internally.

The cache key represents the file footer and is created from the file URI with position -1 and length -1. Callers do not need to construct this key directly; they only need to route CacheKind::DATA_FILE_FOOTER entries to an appropriate cache backend.

Example:

class RoutingCache : public paimon::Cache {
 public:
  RoutingCache(std::shared_ptr<paimon::Cache> default_cache,
               std::shared_ptr<paimon::Cache> parquet_metadata_cache)
      : default_cache_(std::move(default_cache)),
        parquet_metadata_cache_(std::move(parquet_metadata_cache)) {}

  paimon::Result<std::shared_ptr<paimon::CacheValue>> Get(
      const std::shared_ptr<paimon::CacheKey>& key,
      std::function<paimon::Result<std::shared_ptr<paimon::CacheValue>>(
          const std::shared_ptr<paimon::CacheKey>&)> supplier) override {
    return Select(key)->Get(key, std::move(supplier));
  }

  // Put(), Invalidate(), InvalidateAll(), and Size() route in the same way.

 private:
  std::shared_ptr<paimon::Cache> Select(
      const std::shared_ptr<paimon::CacheKey>& key) const {
    return key && key->GetKind() == paimon::CacheKind::DATA_FILE_FOOTER
               ? parquet_metadata_cache_
               : default_cache_;
  }

  std::shared_ptr<paimon::Cache> default_cache_;
  std::shared_ptr<paimon::Cache> parquet_metadata_cache_;
};

auto cache = std::make_shared<RoutingCache>(
    std::make_shared<MyDefaultCache>(),
    std::make_shared<MyParquetMetadataCache>());

paimon::ScanContextBuilder scan_builder(table_path);
scan_builder.WithCache(cache);

paimon::ReadContextBuilder read_builder(table_path);
read_builder.WithCache(cache);

Passing nullptr or omitting WithCache() leaves Parquet metadata caching disabled. If a file URI cannot be obtained, Paimon C++ also bypasses the cache and opens the Parquet file normally.

Reader-Local Page Indexes#

Set parquet.read.enable-offset-index-cache=true to reuse parsed OffsetIndex objects during bitmap trimming, page-range planning and filtered decoding within one file reader. The option defaults to false; it does not require WithCache(). Enable it only after measuring the CPU/memory trade-off for the workload, especially when projecting many columns.

These objects are not stored in the caller’s shared Cache. They are released with their owning row-group index reader; the existing limit of 1,024 retained row-group readers is a count limit, not a parsed-index byte budget. Restricted predicate index readers remain separate so their column hints do not restrict later projected-column reads.

Parsed page locations require approximately retained row groups x accessed columns x pages per column x sizeof(PageLocation) bytes, plus vector/map/object overhead, in addition to serialized index buffers. For example, with a 24-byte PageLocation, 1,024 retained row groups, 10 accessed columns and 1,000 pages per column require about 234 MiB for page locations alone. There is no fixed byte upper bound; memory scales with the file’s index sizes.

This is independent of ReadAheadCache and its per-file FileBlockCache, which reuse bytes rather than parsed objects. That block cache survives resets of the prefetch plan, but does not provide reuse across independent file-cache lifetimes. This optimization introduces no data-cache option or shared data cache.

Filesystem or read-ahead byte caching does not replace this optimization: Arrow deserializes the cached index bytes again on every GetOffsetIndex call. With the option disabled, Paimon retains the existing byte-buffer reuse without retaining parsed OffsetIndex objects.

Future Optimizations#

  • Add hit, miss, bypass, and eviction metrics for Parquet metadata cache.

  • Add single-flight loading for high-concurrency misses on the same Parquet file.

  • Evaluate sharing cached metadata footer bytes with page-index prefetch logic when those read paths can use the same cache abstraction.