RoboMIND ACT Storage Benchmark

This benchmark measures the same CPU LeRobot ACT training workload over an original RoboMIND AgileX HDF5 dataset or an already ingested and canonical-action-backfilled Paimon warehouse. Ingestion and backfill are outside the timed scope; see RoboMIND AgileX to build the warehouse.

The backends run independently. A resolved experiment document preserves the shared configuration, normalization, seed, episode selection, Paimon snapshot, and logical window sequence. Result comparison verifies that contract before it calculates performance ratios.

Install

Python 3.10 or newer is required.

pip install 'pypaimon[act,hdf5]'

1. Ingest and publish the table group

python -m pypaimon.benchmark.act ingest \
  --input /data/RoboMIND/h5_agilex_3rgb \
  --warehouse /data/warehouse \
  --statistics-version act-release-1

Use a new warehouse. This writes exactly four tables: info, episode, frame, and stat. The stat row contains raw feature moments and the nested stats.act normalization scope; no separate ACT statistics table is created. --statistics-version is also the immutable Tag shared by all group members; info is published last. Original images remain in the frame table. The ingestion contract describes the schemas and source mapping.

2. Prepare the experiment

python -m pypaimon.benchmark.act prepare \
  --input /data/RoboMIND/h5_agilex_3rgb \
  --warehouse /data/warehouse \
  --statistics-version act-release-1 \
  --output /data/results/experiment.json

Preparation is not timed. It verifies that HDF5 discovery matches the Paimon episode table at the published Tag, checks versioned action statistics against train-only HDF5 moments from stat.stats.act, selects eligible train and validation episodes, pins the frames snapshot resolved from that same Tag, and materializes deterministic measurement, training, and validation window indices.

Without --experiment, preparation starts from the packaged default_experiment.json. --experiment replaces that definition, so a custom JSON file must contain every required field. Command-line options then override individual values:

python -m pypaimon.benchmark.act prepare \
  --experiment my-experiment.json \
  --input /data/RoboMIND/h5_agilex_3rgb \
  --warehouse /data/warehouse \
  --action-horizon 32 \
  --batch-size 2 \
  --fetch-batches 8 \
  --rounds 3 \
  --statistics-version act-release-1 \
  --output /data/results/experiment.json

The resolved experiment embeds the effective parameters as well as:

• portable source episode metadata and its SHA-256;
• normalization values, scope, version, frame count, and SHA-256;
• selected train and validation episode IDs;
• every logical window index, the window-plan SHA-256, and the episode-qualified sample-sequence SHA-256;
• the Paimon database, group Tag, frames table, and pinned snapshot ID.

Only valid, successful episodes are eligible for training and normalization. The source manifest still retains excluded episodes. The comparison requires all frames in the selected training/eval episodes to be valid: a dirty selected episode is rejected instead of silently changing the HDF5/Paimon window sequence. Opening a group also verifies that every registered member has the published Tag, including the stat table.

3. Run each backend

python -m pypaimon.benchmark.act run \
  --backend hdf5 \
  --experiment /data/results/experiment.json \
  --input /data/RoboMIND/h5_agilex_3rgb \
  --results-dir /data/results

python -m pypaimon.benchmark.act run \
  --backend paimon \
  --experiment /data/results/experiment.json \
  --warehouse /data/warehouse \
  --results-dir /data/results

Use --output to choose an exact result path. Otherwise the command writes an automatically named JSON file below --results-dir and prints its absolute path as a compact JSON object.

Each result contains the complete resolved experiment and experiment SHA-256, backend identity, runtime environment, model metadata, planned-sample tensor fingerprint, three or more raw measurement rounds, and median/minimum/maximum summary metrics.

Both adapters produce the same shared sample contract. State and camera images come from the anchor frame; action covers the complete horizon. HDF5 reads a window on demand from one episode file. Paimon uses a lazy, snapshot-pinned ContiguousWindowDataset grouped by episode_index and ordered by frame_index. It resolves the source episode ID from the episode table for HDF5 comparison. Image columns are anchor-only, and plural __getitems__ access coalesces multiple logical batches into a physical fetch before splitting them back into the unchanged model batch size.

4. Compare results

Compare explicit files:

python -m pypaimon.benchmark.act compare \
  /data/results/robomind-act-hdf5-20260901T010000Z-a1b2c3d4.json \
  /data/results/robomind-act-paimon-20260901T011000Z-e5f6a7b8.json \
  --output /data/results/comparison.json

Or discover all ACT result documents in a directory:

python -m pypaimon.benchmark.act compare \
  --results-dir /data/results \
  --output /data/results/comparison.json

Directory discovery ignores experiment and prior comparison JSON files. Results are grouped by experiment SHA-256. Different experiments remain separate entries in one comparison artifact; only compatible repeated results for the same experiment and backend are aggregated.

Within one experiment group, comparison requires identical runtime environment, model metadata, tensor fingerprint, train-loss trace, and validation-loss trace. An environment mismatch marks the group INCOMPATIBLE; a model, tensor, or loss mismatch marks it FAILED. Neither case produces performance ratios.

For compatible HDF5 and Paimon results, higher-is-better metrics report paimon_over_hdf5. Lower-is-better latency, time, and memory metrics report hdf5_over_paimon, which is the Paimon speedup or reduction factor.

Measurements

Every backend repeat records:

• dataset construction time;
• first-batch latency after construction;
• batch-fetch samples per second after warm-up;
• end-to-end fixed ACT optimizer-step time, including dataset fetch;
• per-step loss and compute time after each training batch has been fetched;
• validation loss;
• total measured wall time;
• Python peak allocation from a separate dataset-first-batch replay.

The shared harness resets Python, NumPy, and Torch random generators before model construction and enables deterministic Torch algorithms. The logical window plan is explicit rather than delegated to a streaming reader.

Python peak allocation uses tracemalloc after wall-clock measurement so tracing overhead does not distort throughput. It does not include every native Arrow or Torch allocation. The benchmark does not drop the OS page cache. GPU, multi-worker loading, distributed training, recovery, and policy quality remain outside this benchmark.

Code organization

• benchmark.act.robomind_agilex: source discovery, ingestion, action backfill, contract schemas, and table-group publication;
• benchmark.act.harness: shared ACT tensors, model, trainer, window plan, and measurement lifecycle;
• benchmark.act.hdf5: HDF5 window dataset and train normalization moments;
• benchmark.act.paimon: Paimon adapter, snapshot-pinned datasets, and versioned statistics access;
• benchmark.act.runner: experiment preparation and one-backend execution;
• benchmark.act.compare: result discovery, compatibility checks, grouping by experiment, and aggregation of compatible repeated runs;
• benchmark.act.__main__: the ingest, prepare, run, and compare command-line interface.