Perception, Prediction, Planning: How a Robotaxi Decides

For most of the field's history, an autonomous vehicle's software was a pipeline of separate modules. Perception turns sensor data into a list of objects with positions, sizes and classes: car, cyclist, pedestrian, cone. Prediction estimates where each of those objects will be over the next few seconds. Planning chooses a path and speed for the car that reaches its goal while keeping clear of everything the prediction module expects. A final control layer turns that plan into steering and pedal commands many times a second.

Detecting a pedestrian is a solved problem in good conditions. Knowing whether that pedestrian is about to step off the curb is not. Prediction has to reason about intent from posture, gaze, position and context, and it has to keep several possible futures open at once. Much of the difference between a smooth robotaxi and a hesitant one comes down to how well it predicts, and how gracefully its planner handles uncertainty.

Neural networks first replaced the hand-written parts of perception, then prediction, and increasingly planning. The extreme version is "end-to-end" driving: one large model trained to map camera input directly to driving actions, with no explicit list of objects in between. Tesla describes its FSD versions since late 2023 this way, and Wayve in the UK built its company on the idea. Waymo and others use large learned models too, but keep interpretable stages so engineers can inspect what the car believed and why.

A modular stack can be tested piece by piece and its failures explained: the car braked because perception reported a phantom object. An end-to-end model is harder to interrogate. Regulators investigating a crash want to know what the system saw and decided, and the industry is still working out how to answer that for a model that never wrote down a list of objects.