PyTorch
Read
This requires torch to be installed.
For multi-worker loading, set multiprocessing_context="spawn" and create and
iterate the DataLoader inside an if __name__ == "__main__": guarded entry
point. Native row iteration uses multithreaded Polars; forking after a read can
inherit locked worker state and hang. See the
Polars multiprocessing guide.
Define custom transforms and collators at module scope so workers can import them.
You can read all the data into a torch.utils.data.Dataset or torch.utils.data.IterableDataset:
from torch.utils.data import DataLoader
table_read = read_builder.new_read()
dataset = table_read.to_torch(splits, streaming=True, prefetch_concurrency=2)
dataloader = DataLoader(
dataset,
batch_size=2,
num_workers=2, # Concurrency to read data
multiprocessing_context="spawn",
shuffle=False
)
# Collect all data from dataloader
for batch_idx, batch_data in enumerate(dataloader):
print(batch_data)
# output:
# {'user_id': tensor([1, 2]), 'behavior': ['a', 'b']}
# {'user_id': tensor([3, 4]), 'behavior': ['c', 'd']}
# {'user_id': tensor([5, 6]), 'behavior': ['e', 'f']}
# {'user_id': tensor([7, 8]), 'behavior': ['g', 'h']}
When the streaming parameter is true, it will iteratively read. When it is
false, eligible data-evolution reads fetch each DataLoader batch lazily by row
ID; other reads retain an Arrow table in memory for map-style access.
In map-style mode, __getitems__(indices) preserves input order and repeated
positions, including sampling with replacement. For example, [2, 0, 2]
returns [dataset[2], dataset[0], dataset[2]]. DataLoader uses this batch
interface for sampler output. Empty input returns an empty list; out-of-range
positions raise IndexError. These indices are Dataset positions, not Paimon
_ROW_ID values. To fetch an ordered batch of Paimon row IDs directly, use
MultimodalTable.take_batch.
prefetch_concurrency (default: 1): In streaming row mode, controls
reader threads per DataLoader worker. It has no effect in non-streaming mode.
Named nested projections
Map-style datasets support named ROW fields and literal MAP-key projections, including projections mixed with top-level columns:
read_builder = table.new_read_builder().with_projection({
"user_id": "id",
"city": "profile.city",
"source": "attributes['source']",
})
splits = read_builder.new_scan().plan().splits()
dataset = read_builder.new_read().to_torch(splits, streaming=False, batch_format="row")
sample = dataset[0]
samples = dataset.__getitems__([2, 0, 2])
Each sample uses the specified aliases in projection order. These requests
materialize their Arrow result in memory when the dataset is constructed,
including on Data Evolution tables. Indexing preserves the requested order
and duplicate indices. Values and supported reads follow to_arrow, including
its table, format, and authorization restrictions.
Map-style row datasets also support the existing float32 variant_get and
try_variant_get projections, including mixtures with ordinary columns and
nested paths:
read_builder = table.new_read_builder().with_projection({
"user_id": "id",
"score": "try_variant_get(payload, '$.score', 'float')",
})
splits = read_builder.new_scan().plan().splits()
dataset = read_builder.new_read().to_torch(splits, streaming=False, batch_format="row")
sample = dataset[0]
samples = dataset.__getitems__([2, 0, 2])
VARIANT extractions require a compatible Native reader. These requests
materialize through to_arrow when the dataset is constructed, including on
Data Evolution tables; they do not use lazy row-ID reads. Aliases, output order,
repeated sources, extracted values and nulls follow the same Arrow request.
variant_get cast errors propagate during construction, while try_variant_get
returns null for failed casts. Initialization retains the Arrow result in memory.
Batch formats ("pyarrow" and "torch") still require streaming=True.
Distributed Sharding
Streaming reads shard splits across DDP ranks and DataLoader workers:
def main():
dataset = table_read.to_torch(
splits,
streaming=True,
auto_detect_rank=True,
)
dataloader = DataLoader(
dataset,
batch_size=32,
num_workers=2,
multiprocessing_context="spawn",
)
with model.join():
for batch in dataloader:
train(batch)
if __name__ == "__main__":
main()
Automatic rank sharding is opt-in. Enable it only when every rank receives the same ordered, complete splits from one snapshot; leave it disabled for splits already sharded by the application. Automatic detection uses the default process group. For subgroup DDP, resolve the context from the group before creating the DataLoader:
import torch.distributed as dist
dataset = table_read.to_torch(
splits,
streaming=True,
sharding_rank=dist.get_rank(ddp_group),
sharding_world_size=dist.get_world_size(ddp_group),
)
With multi-worker DDP, use spawn (or forkserver) and create and iterate the
DataLoader through an if __name__ == "__main__": guarded entry point.
A rank may receive fewer rows because splits have different sizes; join()
keeps DDP collectives aligned while preserving every row without duplication.
A limit that may truncate the input is rejected when multiple ranks are active.
Batch Streaming
For batch-oriented training, make the streaming dataset yield batches directly:
dataset = table_read.to_torch(
splits,
streaming=True,
batch_format="torch",
batch_size=1024,
)
dataloader = DataLoader(dataset, batch_size=None, num_workers=2,
multiprocessing_context="spawn")
for batch in dataloader:
train(batch["features"], batch["label"])
batch_format="pyarrow" yields PyArrow RecordBatch objects instead;
batch_format="torch" yields dictionaries of tensors. The default Tensor
converter supports non-null numeric, boolean, and numeric fixed-size-list
columns. Use to_tensor_fn for other types or custom conversion.
Without shuffle, omit batch_size to preserve native reader batches. Otherwise,
batches are combined or sliced to the requested size. Use
DataLoader(batch_size=None) to disable a second batching step.
Numeric tensors may share read-only Arrow buffers; clone them before in-place
mutation. Batch formats currently require prefetch_concurrency=1.
Shuffled Batch Streaming
Set shuffle=True to mix rows across input batches while keeping values in
Arrow until the final Tensor conversion. The same options work with
batch_format="pyarrow" and custom to_tensor_fn converters:
dataset = table_read.to_torch(
splits,
streaming=True,
batch_format="torch",
batch_size=256,
shuffle=True,
seed=42,
buffer_size=4096,
max_buffer_input_splits=4,
)
loader = DataLoader(dataset, batch_size=None, num_workers=2,
multiprocessing_context="spawn")
for epoch in range(10):
dataset.set_epoch(epoch)
for batch in loader:
train(batch["features"], batch["label"])
Each worker retains at most buffer_size rows in a rolling shuffle buffer and
replaces randomly selected slots with rows from incoming Arrow blocks. Input
blocks contain at most buffer_size rows; gathering replacements, output
batching, and each open format reader use additional memory. This is a row
bound, not a byte bound. The buffer drains early if combining Arrow blocks
would overflow a 32-bit offset. Without batch_size, output blocks contain at
most buffer_size rows.
max_buffer_input_splits bounds the number of interleaved split readers per
worker; 1 reads splits in order. A binding limit keeps the existing ordered
read path so it selects the same rows before shuffling. Filters and distributed
worker/rank sharding also precede shuffle. Each selected row is emitted once.
The shuffle is local to each worker's buffer, not a uniform permutation of the
whole dataset. The same seed, epoch, input batches and worker/rank configuration
reproduce the same order; set_epoch() also reaches persistent workers. Changing
reader batching or worker configuration can change the order. Arrow and Tensor
formats share the same shuffle path, while row-format split interleaving can
produce a different order. The existing prefetch_concurrency=1 requirement
also applies to shuffled batches.
Video frame descriptors
For a multimodal frame table, use the higher-level scan API:
dataset = (
frames.scan()
.select(["episode_id", "state", "action", "video"])
.to_torch(streaming=True)
)
The .video column yields serialized VideoFrameDescriptor values whose
embedded frame ordinals keep frame mapping out of the normal data file. Use
pypaimon.multimodal.VideoFrameCollator as the DataLoader collate_fn to open
physical video ranges and cache decoder sessions per worker. See
Multimodal API: Video Frame Storage
for the write path and a complete decoder example.
Contiguous Windows
Use a map-style ContiguousWindowDataset to construct training samples from
one materialized frame table. Each field can have its own history or future
window, and windows never cross a sequence boundary. The dataset builds an index
from only the group column, order column, and Paimon row IDs. Projected values,
including BLOB payloads, are read from the pinned snapshot when a sample is
requested; they are not retained in the index.
from functools import partial
import torch
from torch.utils.data import DataLoader
from pypaimon.multimodal.window_transforms import images_to_tensor, to_tensor
dataset = (
frames.scan()
.to_contiguous_window_dataset(
columns=["state", "image", "action"],
frame_offsets={"state": [-2, -1, 0], "action": list(range(16))},
group_key="episode_index",
order_key="frame_index",
boundary="pad",
column_transforms={
"state": partial(to_tensor, dtype=torch.float32),
"action": partial(to_tensor, dtype=torch.float32),
"image": images_to_tensor,
},
)
)
loader = DataLoader(dataset, batch_size=32, num_workers=4, shuffle=True,
multiprocessing_context="spawn")
Each raw item contains scalar group/order keys, a list for every selected column,
and a Boolean <column>_is_pad tensor where True marks padding. Unspecified
field offsets default to [0]. The example produces state/action tensors with
time lengths 3/16 and an image tensor of shape (1, C, H, W); DataLoader adds the
batch dimension. pad_values can replace endpoint repetition for individual
columns, before any transforms. adapter can normalize or rename fields and
produce a model-specific mapping. Keep callbacks picklable for multiple workers.
Scheduled anchors start at each group's first row and advance by stride
(default 1). boundary="drop" (default) omits anchors incomplete for any
field, boundary="pad" pads either end, and boundary="error" rejects any
scheduled incomplete window. delta_timestamps with an explicit fps is an
alternative to integer offsets, with frame-grid alignment checked against
tolerance_s. Existing window_size/anchor_columns/tail calls retain their
single is_pad output. See multimodal reading
for the full sample and conversion contract.
Rows are sorted by order_key inside each group_key value. Order values must
be integers which increase by exactly one; duplicates and missing steps are
rejected, and windows never cross groups. The resolved Paimon
snapshot is pinned for the lifetime of the dataset, so later commits cannot
change its index or sample contents. A dataset pinned through tag_name fails
its reads if the tag is moved to another snapshot, rather than mixing rows from
the two snapshots.
Use normal PyTorch samplers for shuffling and distributed training. For DDP,
pass DistributedSampler(dataset) to DataLoader, omit shuffle=True, and call
the sampler's set_epoch(epoch) each epoch. Its padding or drop behavior still
applies when the sample count is not divisible by the number of ranks. Dataset
version pinning does not save sampler progress or random augmentation state.
Columns configured by video-frame-field are decoded after the requested rows
have been coalesced. Their descriptor metadata remains intact, overlapping
windows reuse the same decoded rows, and column_transforms={"video": torch.stack} converts the raw list of CHW frames to TCHW. Install
pypaimon[torch,video] for PyAV, or install TorchCodec separately. Select the
backend with video_backend="pyav" or "torchcodec"; omitting the option
selects an available backend.
Task-local context and anchor selection
The scan query defines the readable context, including its filters and limit.
For a task or rank-local dataset, restrict that query to complete episodes.
Use anchor_row_ids to select training anchors without removing the history
or future frames needed by their windows:
context = frames.scan(tag_name="train-v1").where("task_id = 'pick'")
eligible = (
frames.scan(tag_name="train-v1")
.where("task_id = 'pick' AND eligible = true")
.select(["episode_index", "frame_index"])
.with_row_id()
.to_arrow()
)
dataset = context.to_contiguous_window_dataset(
columns=["state", "image", "action"],
frame_offsets={"state": [-2, -1, 0], "action": list(range(16))},
anchor_row_ids=eligible.column("_ROW_ID").to_pylist(),
boundary="pad",
)
Both scans must resolve to the same fixed snapshot. Row IDs outside the
context raise ValueError. Selection intersects the original stride schedule;
it neither changes its starting position nor orders or repeats samples.
Duplicates in anchor_row_ids are ignored. An empty selection produces an
empty dataset. Boundary handling applies only to selected scheduled anchors.
The context is planned once for the index and all requested value columns. Only index columns are read at construction; payloads remain lazy. All projections and DataLoader workers reuse that plan's split metadata and apply the authorization rules resolved for the complete request at construction. There is no per-projection full-table planning when the first batch is read.
A context filter or limit that cuts an episode changes its boundaries. It
does not automatically supply missing history or future frames. Use anchor
selection for frame eligibility; keep complete episodes in the context when
preserving their original boundaries and stride. Windows can still read across
physical file/split boundaries inside those episodes. If context has already
been divided among ranks, use a local sampler instead of sharding it again
with a world-size DistributedSampler; coordinate rank step counts separately.
Search hits as training samples and window anchors
On data-evolution tables, vector, full-text, and hybrid queries can directly
call to_torch() or to_contiguous_window_dataset(). Both resolve the search
and its post-filtered row IDs eagerly, then read training values lazily from
that same snapshot. Reusing the query creates a new Dataset at the next
execution's snapshot; an existing Dataset keeps its original read view.
hits = (
frames.search(query_embedding, column="embedding", tag_name="train-v1")
.pre_filter("task_id = 'pick'")
.where("eligible = true")
.select(["state", "image", "action"])
.limit(1000)
)
# Train on the matching rows, with ordinary scan().to_torch() options.
rows = hits.to_torch(streaming=True)
# Use the matching rows as anchors while retaining their surrounding frames.
windows = hits.to_contiguous_window_dataset(
context_filter="task_id = 'pick'",
frame_offsets={"state": [-2, -1, 0], "action": list(range(16))},
boundary="pad",
)
For search windows, pre_filter(), where(), and limit() select anchors;
they do not restrict the context or truncate its groups. The context defaults
to the entire captured snapshot. Use context_filter to restrict it to complete
episodes for the task. A selected anchor outside that context raises
ValueError. The search supplies anchor_row_ids, so passing a separate
selection is rejected. Stride, boundary handling, per-column offsets,
transforms, and adapters retain the scan window semantics described above.
select() supplies the training projection; windows can override it with
columns. to_torch() returns rows in scan order, or in its requested shuffle
order. Windows retain group/order and stride order. Neither conversion accepts
with_score() or order_by_score(). Batch vector queries via search_vectors
return one Dataset per input query, in input-query order, with a shared
snapshot and context index. The context is read, validated, and sorted once
for the batch; each Dataset keeps its own anchor selection, read cache, and
video decoders. The search execution options execution, concurrency, and
ray_remote_args match to_arrow(); training reads use the local PyTorch
Dataset and its DataLoader workers. Retain the snapshot's files for the whole
training run, for example with tag_name="train-v1".
Resuming a CWD training run
Install torchdata>=0.11,<0.12 alongside your compatible PyTorch installation.
Use torchdata.stateful_dataloader.StatefulDataLoader in place of DataLoader.
With shuffle=True it uses TorchData's stateful random sampler. It records the
batch position delivered to training, including the worker states needed to
reconstruct prefetched batches. Do not checkpoint a sampler's current index as
the training position: prefetch can put it ahead of completed optimizer steps.
CWD implements state_dict() and load_state_dict() for this loader. Its state
checks the table, branch, resolved snapshot, actual context/anchor index, and
window settings. Rebuild the same dataset before loading; restoration does not
silently switch snapshots. Retain the snapshot files with a training tag for
as long as checkpoints may be resumed. A changed snapshot, context, anchors,
offsets or padding raises ValueError, even when the sample count is unchanged.
The index identity is cached after its first checkpoint; do not mutate dataset
configuration or index arrays during the run.
CWD also saves Python, NumPy and Torch CPU global RNG state in each process.
Transforms and adapters with private RNGs or other mutable state must implement
both state_dict() and load_state_dict(state). Their state is copied before
it is returned. Keep callback code, normalization and preprocessing settings
unchanged; CWD cannot infer these from an arbitrary callable. Custom collators
must be stateless apart from those global RNGs. Reader caches and video decoder
handles are rebuilt rather than checkpointed.
The training loop must save model, optimizer and loader state together, after
an optimizer update and before fetching another batch. The following pattern
uses make_dataset(snapshot_id) to rebuild your CWD with the same window and
preprocessing arguments, via frames.scan(snapshot_id=snapshot_id):
import os
import random
import numpy as np
import torch
from torchdata.stateful_dataloader import StatefulDataLoader
loader_config = dict(batch_size=32, shuffle=True, drop_last=False,
num_workers=4, prefetch_factor=2,
multiprocessing_context="spawn",
persistent_workers=True, in_order=True)
# Include the application's preprocessing/normalization version in this contract.
run_config = dict(loader=loader_config, preprocessing="train-v1", world_size=1,
sampler="torchdata.RandomSampler", seed=42)
generator = torch.Generator().manual_seed(42)
# Load only trusted checkpoint files, and choose their snapshot before planning.
saved = (torch.load(resume_path, map_location="cpu", weights_only=False)
if resume_path is not None else None)
snapshot_id = saved["snapshot_id"] if saved is not None else training_snapshot_id
dataset = make_dataset(snapshot_id)
loader = StatefulDataLoader(dataset, generator=generator, **loader_config)
epoch, global_step = 0, 0
# Restore model/optimizer/scheduler/scaler before restoring RNGs.
if saved is not None:
if saved["config"] != run_config or saved["snapshot_id"] != dataset.snapshot_id:
raise ValueError("Training checkpoint configuration changed")
model.load_state_dict(saved["model"])
optimizer.load_state_dict(saved["optimizer"])
loader.load_state_dict(saved["loader"])
epoch, global_step = saved["epoch"], saved["global_step"]
iterator = iter(loader) # Dataset/worker state is restored here, not at load_state_dict.
if saved is not None:
# Iterator construction consumes RNG; restore these after construction.
generator.set_state(saved["loader_generator"])
random.setstate(saved["python_rng"])
np.random.set_state(saved["numpy_rng"])
torch.set_rng_state(saved["torch_rng"])
if saved["cuda_rng"] is not None:
torch.cuda.set_rng_state_all(saved["cuda_rng"])
for batch in iterator:
optimizer.zero_grad(set_to_none=True)
loss = training_loss(model, batch)
loss.backward()
optimizer.step()
global_step += 1
if should_checkpoint(global_step):
checkpoint = dict(
config=run_config, snapshot_id=dataset.snapshot_id,
epoch=epoch, global_step=global_step,
model=model.state_dict(), optimizer=optimizer.state_dict(),
loader=loader.state_dict(), loader_generator=generator.get_state(),
python_rng=random.getstate(), numpy_rng=np.random.get_state(),
torch_rng=torch.get_rng_state(),
cuda_rng=torch.cuda.get_rng_state_all() if torch.cuda.is_available() else None,
)
# Include scheduler/scaler states here when the trainer uses them.
torch.save(checkpoint, checkpoint_path + ".tmp")
os.replace(checkpoint_path + ".tmp", checkpoint_path)
Alternatively, use the same retained tag in make_dataset and verify its
resolved ID. Move optimizer state to the training device as required by the
trainer. The example saves inside the optimizer-step loop,
including its last batch, before the iterator reports exhaustion. With TorchData
0.11, saving an already exhausted iterator outside that loop is not equivalent:
restoring it may immediately start a new epoch and consume another permutation.
After finishing the restored suffix, start the next epoch with iter(loader).
Keep batch size, drop_last, sampler configuration, worker count, prefetch and
persistent-worker settings unchanged, with in_order=True. Keep the same
PyTorch/TorchData versions; checkpoint compatibility across versions is not
promised. TorchData may regenerate sampling indices or replay prefetched work
on restore; CWD does not promise constant-time recovery or zero rereads.
For fixed-topology DDP, use TorchData's StatefulDistributedSampler, save one
loader/generator/RNG state per rank at the same completed global step, and keep
rank assignment and world size unchanged. Save its seed and epoch in the training
configuration and call sampler.set_epoch(saved_epoch) before restoring the
loader; the sampler's state does not itself contain the epoch. Cross-rank
checkpoint publication, gradient-accumulation boundaries, scheduler/scaler state
and deterministic model execution remain the trainer's responsibility.
File Format Metadata Cache
Reusable PyArrow Dataset metadata is cached across reads. Configure its estimated size limit in the catalog options:
catalog = CatalogFactory.create({
"warehouse": "file:///path/to/warehouse",
"file-format.metadata-cache.max-size": "50 mb",
})
table = catalog.get_table("database.table")
read_builder = table.new_read_builder()
The default limit is 50 MB; set it to 0 b to disable and clear the cache. The
cache is local to each process and benefits workers reused with
DataLoader(..., persistent_workers=True). The cache uses a conservative
per-entry memory estimate and an internal entry-count safeguard; actual native
PyArrow memory may still be higher. The cache assumes immutable Paimon data files.
Shuffle
PyPaimon supports streaming shuffle for PyTorch IterableDataset. The shuffle
pipeline can be composed of three layers:
- Chunk shuffle: split files into row chunks during scan planning and
shuffle the generated chunk splits. This is enabled by
TableScan.with_chunk_shuffle(seed, chunk_size). - Split interleave: read from multiple splits in round-robin order inside each DataLoader worker.
- Buffer shuffle: apply a reservoir-style row shuffle buffer before rows are yielded to PyTorch.
Chunk shuffle is a scan planning feature for append tables, including
Data Evolution append tables. For Data Evolution tables, chunk shuffle keeps
row-id-aligned data files and sidecar files together while slicing by row-id
range. Chunk shuffle should be used with file formats that support random
access. Currently, the random-access file formats are Vortex, Row, and
Blob. Primary-key tables and deletion-vector scans are not supported by
with_chunk_shuffle.
The second and third layers are Dataset features. They work on the splits you
pass to to_torch, so they can be used with either normal splits or
chunk-shuffled splits.
Use Dataset Shuffle Only
Use this when normal scan splits are enough and you only want split interleave plus row buffer shuffle:
from torch.utils.data import DataLoader
table_scan = read_builder.new_scan()
table_read = read_builder.new_read()
splits = table_scan.plan().splits()
dataset = table_read.to_torch(
splits,
streaming=True,
shuffle=True,
seed=42,
buffer_size=1000,
max_buffer_input_splits=10,
)
dataloader = DataLoader(
dataset,
batch_size=32,
num_workers=2,
multiprocessing_context="spawn",
shuffle=False,
)
buffer_size controls the row shuffle buffer. Larger values produce a better
approximation of global shuffle, at the cost of more memory. If
max_buffer_input_splits is 1, split interleave is skipped and only buffer
shuffle is applied. shuffle=True requires streaming=True and does not
support prefetch_concurrency > 1.
Use All Three Layers
For append tables, enable chunk shuffle during scan planning, then enable Dataset shuffle when converting to PyTorch:
from torch.utils.data import DataLoader
seed = 42
table_scan = read_builder.new_scan().with_chunk_shuffle(
seed=seed,
chunk_size=1000,
)
table_read = read_builder.new_read()
splits = table_scan.plan().splits()
dataset = table_read.to_torch(
splits,
streaming=True,
shuffle=True,
seed=seed,
buffer_size=1000,
max_buffer_input_splits=10,
)
dataloader = DataLoader(
dataset,
batch_size=32,
num_workers=2,
multiprocessing_context="spawn",
shuffle=False,
)
Call dataset.set_epoch(epoch) before creating or iterating a DataLoader for a
new training epoch if you want a different buffer-shuffle order for each epoch.