123D Gives Autonomous-Driving Datasets a Common Language — Here Is What It Establishes and Where It Stops

October 5, 2026 · 5 min read · Updated October 6, 2026

SentX SI

Explainer of 123D: Unifying Multi-Modal Autonomous Driving Data at Scale. SentX was not involved in this research.

123D is a data-infrastructure contribution, not a new perception or planning algorithm. It establishes one thing clearly: a common representation and API over the patchwork of autonomous-driving datasets — nuScenes, Waymo Open Perception, Argoverse 2 Sensor, PandaSet, KITTI-360, Waymo Open Motion, nuPlan and PAI-AV, plus the synthetic L3AD set recorded in CARLA — so downstream code reads one interface regardless of where the data came from. Its findings on model transfer are bounded: the authors report empirical benefits from data mixing in their defined experiments, and those results do not establish universal transfer gains. Radar, richer semantic annotations, web-scale ingestion, continuous streaming, and non-standard platforms sit outside the described framework, and the paper says so.

Source: 123D: Unifying Multi-Modal Autonomous Driving Data at Scale

What does the framework actually do?

Every recording is converted into timestamped event streams: one stream per modality — cameras, with support for multiple projection models, lidar, ego-states, 3D bounding boxes, traffic-light states, and user-defined types. Streams carry no prescribed sampling rate; each event keeps its own timestamp, and static metadata such as vehicle dimensions and sensor calibration lives in the file's schema rather than being repeated row by row. A separate synchronization table precomputes which rows across modalities refer to the same instant, with configurable strategies — preserve the original keyframes, choose a reference modality, or resample everything to a target rate — and the access layer can bypass the table for arbitrary-timestamp reads, which is what lets it serve both synchronous and asynchronous consumers. Original annotations are preserved rather than re-labeled.

Source: 123D: Unifying Multi-Modal Autonomous Driving Data at Scale

What did the paper evaluate?

Two worked applications: detection transfer and planning. In the detection study, standard detectors (PETR and BEVFormer-S, both with ResNet50 backbones) were trained individually on five selected sources — nuScenes, Waymo Open Perception, Argoverse 2 Sensor, nuPlan and CARLA — and on a uniform mixture of the same five, under a fixed 30,000-frame budget so diversity is isolated from raw volume. PandaSet, KITTI-360 and PAI-AV were excluded from training and used for held-out transfer evaluation, and storage constraints led the authors to use the nuPlan-mini and PAI-AV-NCore subsets. The detection evaluation is deliberately narrow: vehicle class only, within a 50-meter radius, scored with a modified NDS that drops velocity and attribute errors. The authors' own conclusion is measured — cross-domain and vehicle transfer remains challenging for both perception and behavior tasks, simple data mixing reduces the gap to some extent, and sim-to-real transfer proves no easier than cross-rig transfer between real datasets, even when the simulator replicates the target rig. The planning example, built on reinforcement learning, lands in the same place: behavior transfer stays difficult, with partial benefit from mixing. All reported numbers are the authors' measurements on their own harness; this explainer is a source-based interpretation of the paper's inspected sections, not an independent reproduction of the results.

Source: 123D: Unifying Multi-Modal Autonomous Driving Data at Scale

Where does the paper say it stops?

Three limitations are documented in the paper itself. Coverage is limited to common sensor modalities; additional sensors such as radar and richer semantic annotations are named development targets, not delivered features. There is no explicit support for web-scale datasets or continuous data streaming; the authors note that Arrow's cloud-access capabilities were only preliminarily tested. And the framework handles standard vehicles; trucks, mobile robots and other platforms are targeted for later expansion. The work is an author-submitted arXiv paper, and the description above reflects the sections available in that version.

Source: 123D: Unifying Multi-Modal Autonomous Driving Data at Scale

How to check a claim before trusting a summary

Consider a plausible-sounding summary sentence: "123D supports continuous live data streaming." The paper supports the machinery the sentence leans on — independent timestamped event streams with no prescribed rate — but that is a statement about how recordings are stored and indexed, not about ingesting a live feed. Each event keeping its own timestamp says nothing about whether the system can accept data arriving continuously in real time. The distinction matters because the two sections that settle it are exactly the ones a skimmer skips: the framework description shows streams as a static, timestamped representation with a precomputed synchronization table, and the limitations section explicitly names continuous data streaming as unsupported, with Arrow's cloud access only preliminarily tested. A summary claim survives contact with the paper only if you can locate the sentence that earns it.

Source: 123D: Unifying Multi-Modal Autonomous Driving Data at Scale

That habit is worth having for any model-assisted reading. SentX's research summarizer accepts supplied paper text and answers questions about methods, results and limitations, and its own page directs users to check important claims against the source. Used that way, it can prepare a draft source-checking note from the paper's public text; the manual review against the actual sections is still yours to do, and the tool is not a driving or perception product validating anything here. One reason for the caution: SentX's retained full-paper demonstration on a different paper (MIST) kept two substantive errors in its first answer — a false claim contradicted by Table 7 and an unsupported relative-percentage reading of the abstract. That demonstration concerns a different paper and establishes no result about 123D; it exists to show why summary claims need checking in the first place.

Source: SentX full-paper reading demonstration: retained prompt, first answer and claim review; SI Research Paper Summarizer

Should you build on 123D?

The following is editorial guidance based on this paper, not a deployment result:

1. Map your task to the described coverage. If you are running existing detection or planning pipelines across several heterogeneous driving datasets with common sensors — cameras, lidar, ego-state, 3D boxes, traffic-light state — and standard vehicles, the framework addresses that gap.Source: 123D: Unifying Multi-Modal Autonomous Driving Data at Scale
2. Check each dependency against the limitations. Radar, richer semantic annotations, web-scale ingestion, continuous streaming, trucks or other non-standard platforms: the paper names these as future work or leaves them out, so any claim that 123D covers them today is unsupported by this source.Source: 123D: Unifying Multi-Modal Autonomous Driving Data at Scale
3. Treat every number as author-reported. Results come from the authors' own harness on the named subsets, with the narrow detection metric scope described above; if your decision depends on certified accuracy, that question is unanswered by this paper.Source: 123D: Unifying Multi-Modal Autonomous Driving Data at Scale
4. Verify before you depend. For any specific feature you plan to rely on, open the framework description and the limitations section and find the supporting sentence — the same check applied to the streaming claim above.Source: 123D: Unifying Multi-Modal Autonomous Driving Data at Scale

The honest bottom line is the paper's own: consolidation makes mixing data easier, but teaching models to transfer across domains remains the hard part. 123D shrinks the plumbing problem, not the learning problem.

Source: 123D: Unifying Multi-Modal Autonomous Driving Data at Scale