Design: Recording and Evaluation

The evaluation repository has two packages: the experiment recorder (cpm_lab_experiment_recorder), a C++ component the control flow starts for recorded experiments, and the offline evaluation cpm_eval (cpm_lab_evaluation), a Python command that reads the recordings later without ROS nodes. The user’s view is in Recording and Evaluating Experiments and Evaluation Metrics and Own Analyses; the package README.md files describe every parameter and metric.

Recorder

The control flow starts experiment_recorder_node in the lab domain when an experiment is deployed with Record experiment, and talks to it through the services of cpm_lab_evaluation_msgs:

Interface

Role

/lab/evaluation/prepare_recording

Prepare a run: reserve the output directory, subscribe to the topics of the deployed vehicles, take the map and parameter snapshots and open the writer. The recorder then reports ARMED.

/lab/evaluation/begin_recording

Acknowledge Start; the control flow publishes START only afterwards.

/lab/evaluation/finish_recording

Finish (or cancel) the run after STOP; the recorder writes the manifest and reports COMPLETE or INTERRUPTED.

/lab/evaluation/recording_state

Heartbeat every 250 ms (reliable, transient local) with state, host, directory and counters.

 
        ---
config:
  layout: elk
  startOnLoad: false

---
sequenceDiagram
   participant CF as Control flow
   participant R as Recorder
   participant B as Bridge
   CF->>R: start process, Prepare(run_id, vehicles, planner, map)
   R-->>CF: state ARMED
   B-->>CF: READY
   Note over CF: user presses Start
   CF->>R: Begin(run_id)
   R-->>CF: acknowledged
   CF->>B: START (system_trigger)
   Note over CF: user presses Stop
   CF->>B: STOP (system_trigger)
   CF->>R: Finish(run_id, stamp, reason)
   R-->>CF: state COMPLETE

Recording sequence between control flow and recorder

Safety rules built into this sequence:

  • STOP is always published before the recording is finished or any process is stopped.

  • A writer failure, the exit of the recorder process or a missing heartbeat (recording_heartbeat_timeout_ms, 1.5 s) stops the experiment. These are operational controls, not a certified emergency stop.

  • Only one recorder may run per host and domain; the control flow refuses to deploy if another recorder service exists.

Each run is a new directory $CPM_LAB_HOME/experiments/<UTC time>_<run id>/ that is never overwritten:

<run>/
├── bag/                    # MCAP recording of the vehicle, trigger, map, TF, diagnostics and clock topics
├── map/runtime_map.bin     # the exact binary map the vehicles used (plus the original .osm, if known)
├── config/recorder.json    # recorder settings
├── config/nodes.json       # parameter snapshots of the running nodes
├── manifest.json           # run metadata, map SHA-256 and frame, result of the run
└── evaluations/            # one directory per cpm_eval run

The writer runs in its own thread with bounded queues (max_queued_messages, max_queued_bytes) and a write delay of 100 ms, so it can tell whether a message belongs inside the interval between START and STOP. Queue overflow, a full disk (minimum_free_bytes) or a stalled writer turn the run into FAILED.

Evaluation

cpm_eval (cpm_eval list, cpm_eval evaluate <run>) reads a finished run directly from its directory: the MCAP file with rosbag2_py, the map from map/runtime_map.bin with lanelet2_python and cpm_lab_lanelet2_extension_py. It needs the sourced workspace for the message types, but no running nodes.

Each metric is a module in cpm_lab_evaluation/metric_types/ (topic_sanity, states_speed, obs, command_follow, vehicle_collision, boundary_violation, lanelet_collision) and returns checks with PASS, FAIL, INSUFFICIENT, UNAVAILABLE or N/A. The settings are in cpm_lab_evaluation/default.yaml and can be overridden with --config. Every invocation writes a new directory under evaluations/ with the HTML report, CSV, PNG/SVG plots, the configuration used and a JSON summary.

To add a metric, add a module to metric_types/ with its mode name, register it in metric_types/registry.py, add its settings to default.yaml and add a test with a synthetic recording in test/.

Compatibility

Recordings contain the message types of the release they were made with. cpm_eval reads them with the installed message definitions, so a change of a recorded message can make old recordings unreadable. This is one reason to avoid message changes, see Changing Messages and Services. Note the version of the CPM Lab with every recording you keep; the manifest stores the build revision.