Recording and Evaluating Experiments

Experiments can be recorded and evaluated afterwards with the cpm_eval command. Both are part of the CPM Lab installation.

User Data Directory

The CPM Lab tools keep user data outside the workspace, so the layout is the same for source builds and Debian installations:

~/cpm_lab/
├── experiments/    # recordings, the evaluations are stored inside each run
├── calibrations/   # IPS camera calibrations (cpm_calibrate)
├── ips/            # your IPS configurations
└── maps/           # your maps

Set CPM_LAB_HOME to use another root, for example a data disk. All tools then use $CPM_LAB_HOME/experiments, $CPM_LAB_HOME/calibrations, $CPM_LAB_HOME/ips and $CPM_LAB_HOME/maps, see User Data Directory. Explicit options take precedence: recording_root of the launch file, cpm_eval --root and cpm_calibrate --output-root. The directories are created on first use.

Note

The recordings are written by the control flow, so the directory is resolved for the user and environment that started the lab, for example on the central workstation.

Warning

In Docker, everything outside the mounted directories is lost when the container exits. The helper scripts of the images and the workspace mount ~/cpm_lab of the host, see User Data Directory. If you start a container with docker run yourself, mount it, e.g. -v ~/cpm_lab:/cpm_lab -e CPM_LAB_HOME=/cpm_lab.

Recordings made before this layout are stored in ~/experiments. cpm_eval list points to them, and they can be moved to the new location unchanged.

Recording an Experiment

The deploy step with Record experiment enabled and the recorder state

Enable Record experiment in the deploy step of the Control Center before you deploy. The control flow then starts the experiment recorder together with the bridge:

1. Deploy: the recorder prepares a new run and reports that it is armed. The experiment can only be started once the bridge is ready and the recorder is armed.

2. Start: the recording starts at the same timestamp as the experiment.

3. Stop: the recorder finalizes the run. Wait until the recording is reported as complete before you deploy the next experiment or evaluate the run.

If the recorder fails or stops responding, the control flow stops the experiment. Only one recorder can run per ROS domain.

To store the recordings in another directory for a single session, start the lab with the headless launch file:

ros2 launch cpm_lab cpm_lab_headless.launch.xml recording_root:=/data/experiments

Each run is stored in its own directory <YYYYmmddTHHMMSSZ>_<run-id>/:

<run>/
├── bag/            # rosbag2 (MCAP) with metadata.yaml
├── map/            # map used in the run (runtime_map.bin, original.osm)
├── config/         # parameter snapshots of the recorder and the nodes
├── manifest.json   # state, vehicles, planner, map, timestamps, ROS distribution and middleware
└── evaluations/    # created by cpm_eval

The bag contains the command, state and observation of each deployed vehicle, /lab/map, /lab/system_trigger, /tf, /tf_static and /diagnostics. Runs are never overwritten or deleted automatically.

Evaluating an Experiment

cpm_eval evaluates a recorded run offline. It only needs a sourced ROS2 environment; the lab does not need to be running.

List the recorded runs. The number in the first column can be used to select a run:

cpm_eval list

Print a quick summary of a run without writing any files:

cpm_eval evaluate 1 --quick

Create the full report:

cpm_eval evaluate 1

The report is written to a new directory <run>/evaluations/<timestamp>_<id>/ and contains report.html with all plots, the samples as CSV files, summary.json and the evaluation log. Open report.html in a browser and keep the directory together. Earlier evaluations are never overwritten.

The most important options are:

Option	Description
--root <path>	Directory with the runs. Default: $CPM_LAB_HOME/experiments or ~/cpm_lab/experiments.
--metrics <name> ...	Evaluate only the given metrics: topic_sanity, obs, states_speed, command_follow, vehicle_collision, boundary_violation, lanelet_collision. Default: all.
--vehicles <id> ...	Evaluate only the given vehicles. Default: all recorded vehicles.
--config <file>	YAML file overriding the default thresholds, for example the vehicle size.
--allow-incomplete	Evaluate runs that were interrupted or failed.
--output-root <path>	Write the report to another directory.
--quick	Print the summary only and write no files.
--bag <path>	Evaluate a bare rosbag2 directory from older recordings instead of a run.
--jobs <n>	Number of parallel jobs. Default: number of CPUs.
--verbose	Print the full reason for every check.

Examples:

# Run from another directory
cpm_eval evaluate 2 --root /data/experiments --quick

# Selected metrics and vehicles of a run given by its path
cpm_eval evaluate ~/cpm_lab/experiments/<run> --metrics states_speed command_follow --vehicles 1 3

# Custom thresholds
cpm_eval evaluate <run> --config ~/my_thresholds.yaml

The summary lists each check as PASS, FAIL, INSUFFICIENT, UNAVAILABLE or N/A. cpm_eval exits with 0 if all applicable checks pass, 2 if a check does not pass and 1 on errors, so it can be used in scripts.