A body, a task, a sequence, an outcome. Each belongs to a particular setup. To compare results, we need to know the conditions and procedures.
Research on reproducibility in robot learning identifies sources of variation such as stochastic environments and unreported hyperparameters. It also emphasizes evaluation metrics and statistical reporting. [1]
Start with the conditions.
- Hardware and observations. Robot, end effector, sensors, camera placement and the inputs available to the policy.
- Policy and execution settings. Checkpoint, known training exposure, adaptation procedure, software versions and control settings.
- Task and environment. Objects, starting conditions, lighting, reset procedure and the definition of success.
- Trial accounting. Number of attempts, retries, exclusions, human interventions and uncertainty in the reported estimate.
Follow what changes.
Environmental changes are already studied in robot evaluation. The Colosseum, for example, tests manipulation policies under perturbations including lighting, camera pose and object properties. Its findings concern the policies and conditions evaluated. [2]
Repeated tests can estimate performance across specified conditions. A single failure does not establish a general capability limit or explain its cause. Cross-embodiment comparisons also need to account for task feasibility and policy adaptation.
Make room to repeat the work.
Physical evaluation can require equipment access, calibration and resets. Simulation throughput depends on the workload and implementation; GPU-based systems such as Isaac Gym demonstrate that substantial parallelization is possible. [3]
Build the atlas together.
EmbodiAtlas aims to connect outcomes with their conditions and study where performance holds. The pilot is in development: we are defining the first protocol and supported setups. Reducing evaluation cost remains a goal to be tested.
The proposed atlas brings setup records, repeatable procedures, and execution evidence together. We invite researchers, environment builders, robotics teams, and supporting partners to shape this work. Fair comparisons and reliable estimates are requirements; the journal will document our choices.