Evaluation Metrics and Own Analyses

This page explains what the cpm_eval checks measure and how to read a recorded run in Python for your own analyses and plots. See Recording and Evaluating Experiments for how to record a run and run cpm_eval.

Results of a Check

Every check reports one of the following results:

Result	Meaning
`PASS`	The check was evaluated on enough data and found no problem.
`FAIL`	The check found a problem, e.g. a collision. Collision and boundary checks fail as soon as one event is found, even if the coverage is low.
`INSUFFICIENT`	Not enough data could be evaluated, usually because the coverage is below 70 % (`minimum_coverage`).
`UNAVAILABLE`	Required input is missing, e.g. the recorded map.
`N/A`	The check does not apply, e.g. `command_follow` for a planner that only sends `DIRECT` commands.

The coverage is the share of samples that could be evaluated. Samples are skipped, for example, before the first command or outside the time range of a trajectory.

The Checks

topic_sanity: Counts the messages of every recorded topic. Fails if a topic has fewer than topic_sanity_min_messages (1) messages, which usually means a component was not running.
obs: Compares the vehicle state with the IPS observation at the same time (within 10 ms). Reports the position, lateral, longitudinal and heading error (RMSE, maximum and 95th percentile) per vehicle. A large error means the vehicle’s state estimate and the IPS disagree. There is no error threshold; the check only fails on insufficient data.
states_speed: Reports the minimum, maximum and mean speed per vehicle.
command_follow: Compares the vehicle state with the command that was active at that time, i.e. the latest command whose valid_after_stamp lies before the sample. For PATH_TRACKING the vehicle is projected onto the path, for TRAJECTORY the reference is interpolated at the sample time. Reports the lateral, longitudinal, heading and speed errors and the share of samples within 10 cm and 5°. DIRECT and SPEED_CURVATURE commands are not evaluated.
vehicle_collision: Checks every pair of vehicles for overlapping footprints at the same time. A footprint is a 0.25 m x 0.125 m rectangle. An overlap larger than 0.004 m² is a collision. Requires at least two vehicles.
boundary_violation: Checks whether a vehicle’s footprint leaves the drivable area, the union of all lanelets of the map, by more than 0.004 m².
lanelet_collision: Checks whether a vehicle stays within the lanelets of its own command, i.e. the lanelets along its path or trajectory and their routing connections. Leaving them by more than 0.004 m² counts as violation. This finds vehicles that cut corners or drift into the opposite lane even if they stay on the road.

All thresholds are defined in the default configuration of cpm_lab_evaluation and can be changed with cpm_eval evaluate <run> --config <file>, for example:

vehicle_length_m: 0.250
vehicle_width_m: 0.125
minimum_coverage: 0.7
minimum_overlap_area_m2: 0.004
within_lat_tol_m: 0.10
within_yaw_tol_deg: 5.0

Reading a Run in Python

The bag of a run is a standard rosbag2 in MCAP format in <run>/bag. You can inspect it with the ROS2 tools:

ros2 bag info ~/cpm_lab/experiments/<run>/bag

# Replay the run, e.g. to view it in RViz or Lichtblick
export ROS_DOMAIN_ID=21
ros2 bag play ~/cpm_lab/experiments/<run>/bag

For your own analysis, read the messages with rosbag2_py. The following script writes the speed of every vehicle over time to a CSV file. Source your ROS2 installation before running it:

import csv
import re
import sys
from pathlib import Path

import rosbag2_py
from rclpy.serialization import deserialize_message
from rosidl_runtime_py.utilities import get_message

run = Path(sys.argv[1])
reader = rosbag2_py.SequentialReader()
reader.open(
    rosbag2_py.StorageOptions(uri=str(run / 'bag'), storage_id=''),
    rosbag2_py.ConverterOptions('cdr', 'cdr'),
)
types = {topic.name: topic.type for topic in reader.get_all_topics_and_types()}
state_topic = re.compile(r'^/lab/vehicle_(\d+)/vehicleState$')

with open('speeds.csv', 'w', newline='') as file:
    writer = csv.writer(file)
    writer.writerow(['time_s', 'vehicle_id', 'x', 'y', 'speed'])
    while reader.has_next():
        topic, data, receive_time_ns = reader.read_next()
        if not state_topic.match(topic):
            continue
        state = deserialize_message(data, get_message(types[topic]))
        stamp = state.valid_after_stamp.sec + state.valid_after_stamp.nanosec * 1e-9
        writer.writerow([stamp, state.vehicle_id, state.pose.position.x,
                         state.pose.position.y, state.twist.linear.x])

Run it with python3 speeds.py ~/cpm_lab/experiments/<run> and plot the CSV file with your favorite tool.

The recorded topics are:

Topic	Type
`/lab/vehicle_<id>/command`	`cpm_lab_lab_msgs/msg/VehicleCommand`
`/lab/vehicle_<id>/vehicleState`	`cpm_lab_lab_msgs/msg/VehicleState`
`/lab/vehicle_<id>/vehicleObservation`	`cpm_lab_lab_msgs/msg/VehicleObservation`
`/lab/map`	`cpm_lab_map_msgs/msg/Lanelet2MapBin`
`/lab/system_trigger`	`cpm_lab_lab_msgs/msg/SystemTrigger`
`/tf`, `/tf_static`	`tf2_msgs/msg/TFMessage`
`/diagnostics`	`diagnostic_msgs/msg/DiagnosticArray`

The map used in the run is stored in <run>/map/runtime_map.bin and can be loaded with cpm_lab_lanelet2_extension_py.from_bin_msg(list(Path('runtime_map.bin').read_bytes())). manifest.json contains the vehicles, the planner, the start and stop time and the software versions of the run, which you should report in your thesis.

The time of a sample is valid_after_stamp, the time at which a state was valid or a command becomes active. Use it instead of the time the message was received.