Troubleshooting Guide

RViz is not showing anything

There is a problem with some rendering systems. If RViz is not showing the 3D world as expected, try disabling hardware acceleration.

# Manually for this terminal
export LIBGL_ALWAYS_SOFTWARE=1

# Automatically for all future terminals
echo "export LIBGL_ALWAYS_SOFTWARE=1" >> ~/.bashrc

Deploying fails with “No vehicle online”

The control flow deploys all vehicles that are online. A vehicle is online when its state reaches the lab domain within the last second. Check that the simulated vehicles are started or the muCars are switched on and in the same network and domain:

export ROS_DOMAIN_ID=21
ros2 topic echo /lab/control_flow/state --qos-durability transient_local --field online_vehicle_ids

The experiment cannot be started

The experiment can only be started when the bridge reports ready. This requires that the motion planner reported ready for every deployed vehicle, see Understanding the Motion Planner. The Deploy Checks show all checks after Deploy and the most common reasons why Start never becomes available. With an external motion planner, check that it runs in ROS_DOMAIN_ID=1 and was started with the vehicle IDs of all online vehicles. If the experiment is recorded, the recorder must be armed as well. The events of the control flow show which component is not ready:

ros2 topic echo --qos-durability transient_local /lab/control_flow/status

The catalog or state topics show nothing

The control flow publishes /lab/control_flow/catalog and /lab/control_flow/state only when they change. Add --qos-durability transient_local to ros2 topic echo to receive the last message.

Nodes on different computers do not see each other / the muCars do not come online

The muCars and every computer in the real lab use Zenoh, and Zenoh needs its router on the central workstation. The router is deliberately not started when the workstation boots; start it before the lab and keep it running:

source /opt/ros/jazzy/setup.bash
ros2 run rmw_zenoh_cpp rmw_zenohd      # or: make router in the lab workspace

Check that it listens on port 7447 (ss -ltnp | grep 7447) and that the firewall allows it, see Central Workstation. Then check in every terminal:

echo $RMW_IMPLEMENTATION prints rmw_zenoh_cpp. CycloneDDS or FastDDS cannot talk to the muCars, see Middleware.
On computers other than the workstation, ZENOH_CONFIG_OVERRIDE points to tcp/192.168.1.249:7447.
ROS_DOMAIN_ID is 21 for the lab and 1 for the planner.

A muCar connects to the router when its controller starts. If the router was started later, restart the controller on the car (sudo systemctl restart vehicle-controller.service) or switch the car off and on.

The muCars do not start the vehicle controller

On every boot, a car downloads http://192.168.1.249/raspberry3/package.tar.gz from the central workstation and starts it. On the car (ssh ubuntu@192.168.1.<20 + id>, staff only):

sudo journalctl -u vehicle-controller.service -b --no-pager -n 100

If the download fails, check on the central workstation:

systemctl status apache2                                   # web server running?
ls -l /var/www/html/raspberry3/package.tar.gz              # package deployed? (make deploy, see muCar)
sudo ufw status | grep -E "80|http"                        # port 80 open?
curl -sI http://192.168.1.249/raspberry3/package.tar.gz    # reachable in the lab network?

The controller also waits for a synchronized clock, see below. More checks are in muCar.

The IPS camera is not found or drops frames

Check that the camera is connected: lsusb | grep -i basler and /opt/pylon/bin/pylonviewer. Close the Pylon Viewer before starting the IPS; only one program can open the camera.

Basler USB 3 cameras need a larger USB buffer than the Linux default:

cat /sys/module/usbcore/parameters/usbfs_memory_mb      # should be 1000
echo 1000 | sudo tee /sys/module/usbcore/parameters/usbfs_memory_mb

Set it permanently with the kernel parameter usbcore.usbfs_memory_mb=1000.

Use a USB 3 port and, for cables longer than 3 m, an active cable.
In Docker, use the desktop image, which runs privileged with /dev mounted, see Docker Images.

The IPS positions are wrong or drift

Recalibrate the camera when it was moved or its focus was changed, and check that the IPS configuration of your lab is selected, see IPS Camera Calibration.

The clocks are not synchronized

The muCars synchronize their clocks with the NTP server (chrony) on the central workstation and only start the controller with a synchronized clock. On the central workstation:

chronyc tracking            # the workstation's own time source
sudo chronyc clients        # the muCars that asked for the time

On a muCar (staff only), which uses systemd-timesyncd:

timedatectl status                                   # "System clock synchronized: yes"
journalctl -u systemd-timesyncd --no-pager -n 50     # NTP errors
sudo timedatectl set-ntp false && sudo timedatectl set-ntp true    # force a resync

The workstation must serve the time even without internet (local stratum in the chrony configuration), see Central Workstation.

Known Limitations

Central routing finds valid solutions for up to about 10 vehicles. With more vehicles it may fail to plan.
Path tracking (the built-in planner) only drives one vehicle on the outer loop.
FastDDS may not deliver the map to RViz because of QoS incompatibilities. Use Zenoh, or CycloneDDS for simulations only, see Middleware.
Vehicle IDs are limited to 1 to 20.
Domain IDs are fixed to 21 for the lab and 1 for the planners when using the control flow.
Commands have no timeout. A vehicle keeps following its last command until it receives a new one or a stop, see Interface Reference.
Simulation does not model collisions, tire slip, battery discharge or network latency, see Vehicle Limits and Simulation.
TF frames of the vehicles are scaled by 18 for the display and must not be used to transform lab data.