What Tesla Autopilot Sees in Paris

Tesla’s Autopilot is one of the most discussed features in the car. Every update, every new functionality is a matter of fierce discussions, especially if another Tesla is involved in a crash and the driver blames the Autopilot. Thanks to a posting we now get a new look into the inner workings of the Autopilot and what it sees and recognizes. This also seems to be the first investigation of an independent team into such a system (with the exception of Comma.ai and Open Source systems).   The hackers acquired on eBay an Autopilot Hardware Kit 2.5 and, as it turns out, was a developer version allowing full open access. Firmware 18.34 was installed and with the main camera recordings from a drive through Paris could be made with seeing what the car sees. We notice the different colors used for drivable roads, objects like cars, buses, bicyclists and motor bikers, as well as pedestrians and other objects on the road. Here is a drive through the Paris city center with the following highlights:

• 1:17 – Traffic cones shape driveable space;
• 01:31 – construction equipment recognized as a truck (shows they have quite a deep library of objects they train against? Though it’s not perfect, we saw some common objects not detected too. Notably a pedestrian pushing a cart (not present in this video);
• 02:23 – false positive, a container mistaken for a vehicle;
• 03:31 – a pedestrian in red(dish?) jacket is not detected at all. (note to self, don’t wear red jackets in Norway and California, where Teslas are everywhere);
• 04:12 – one example of lines showing right turn while there are no road markings of it;
• 06:52 – another false positive – poster mistaken for a pedestrian;
• 08:10 – another more prominent example of showing left turn lane with no actual road markings;
• 09:25 – close up cyclist;
• 11:44 – roller skater;
• 14:00 – we nearly got into accident with that car on the left. AP did not warn;
• 19:48 – 20 pedestrians at once;

Here a highway drive with the following highlights:

• 3:55 – even on “highways” gore area is apparently considered driveable? While technically true it’s probably not something that should be attempted.
• 4:08 – gore zone surrounded by bollards is correctly showing up as undriveable.
• 11:47 – you can see a bit of a hill crest with the path over it (Paris is not super hilly it appears so hard to demonstrate this on this particular footage)

This firmware-version seems to not use the side view cameras, with the exception of light levels. Also unknown are some of the color codings and metrics displayed. The following object types and rectangle/box colors are used:  

• purple: truck
• yellow: pedestrian
• green: bicycle
• blue: motor bike
• red: general color for vehicle

  The percentage number seems to be a confidence value of what object the Autopilot thinks it sees. The lane information and distance seem to come from the cameras and is not always accurate. The moving state of other objects comes from the radar. Estimates about distance and relative speed come from the camera without radar and seem to be accurate. Orange lines seem to indicate the direction the Autopilot intends to drive. It’s also displayed in 3D, which allows better driving on hilly road. This seems to be a reason why Autopilot 2.x performs better on hilly and mountain roads than Autopilot 1.0. This article has also been published in German.