Three ways of seeing
A self-driving car builds its picture of the world from sensors that measure different things. Cameras record light, like eyes, and give the richest detail: colour, text on signs, the state of a traffic light. Radar sends out radio waves and measures how long they take to return and how their frequency shifts, which gives distance and relative speed directly, in rain, fog and darkness, but with a blurry sense of shape. Lidar fires pulses of laser light and times their return to build a three-dimensional cloud of points, precise to centimetres, day or night, but degraded by heavy rain, snow and spray.
What each one misses
Cameras struggle with glare, darkness and depth: estimating how far away something is from a flat image is a learned skill, not a measurement, and it fails in unfamiliar situations. Radar has trouble telling a stationary car from a bridge or a sign, which is why early adaptive cruise systems ignored stationary objects altogether. Lidar sees shape but not colour, and cannot read a sign.
Because the failures are different, most robotaxi developers fuse all three so that one sensor covers for another. A Waymo vehicle carries a rooftop lidar plus several shorter-range units, an array of cameras and radars around the body, and microphones for sirens. Zoox, Cruise, Pony.ai and Baidu use similar combinations.
The camera-only bet
Tesla is the notable exception. It removed radar from its cars in 2021 and ultrasonic sensors in 2022, arguing that humans drive with two eyes and a brain, so a car with eight cameras and a large enough neural network should be able to do the same, at a fraction of the cost of lidar. Its FSD software is trained on video from its fleet.
Critics point out that human eyes come with a human brain, and that the cost argument has weakened as lidar prices have fallen from tens of thousands of dollars a unit to hundreds. The debate is, in the end, empirical, and it will be settled by safety records rather than by principle.
Putting it together
Raw sensor data is only the start. The perception software has to decide which measurements belong to the same object, track it over time, classify it and estimate where it will go next. That is the subject of the next article. The sensor choice shapes it, though: a car with lidar starts every frame knowing exactly how far away everything is; a camera-only car has to infer it.
Frequently asked questions
Why do Waymo cars have that spinning thing on the roof?
That is the main lidar unit. It spins to sweep its laser beams around the vehicle several times a second, building a 360-degree, three-dimensional map of everything within a few hundred metres.
Can a self-driving car see in the dark?
Lidar and radar do not need light at all, so a car with them sees as well at midnight as at noon. Camera-only systems depend on headlights and street lighting, like a human driver.