With some delay, the Disengagement Report 2019 for California was published today. This report is an annual survey that the California Department of Motor Vehicles (DMV) requires all companies licensed to test autonomous cars on California’s public roads to submit to the DMV. The report must include the number and specifics of the test vehicles used, the miles driven on public roads, and disengagements and their causes. Disengagements are those events where a security driver, who is still required to be in the vehicle, takes control of the vehicle.
First, the numbers as reported:
A total of 36 companies with 676 vehicles drove a total of 2,880,612 miles and had 9,338 disengagements. Broken down by company – namely the 28 companies that already had to report last year – the order is as follows:
The vehicles of the Chinese internet giant Baidu are said to require an intervention every 18,050 miles, according to their own report, followed by Google’s sister company Waymo, which comes to 13,219 miles per intervention. After that come CRUISE, AutoX, Pony.ai, and then by far Nuro, Zoox and PlusAI. At the other end of the scale is Toyota, which comes in at just 0.4 miles per disengagement.
The two reporting German manufacturers, Mercedes Benz and BMW, show different levels of progress. Mercedes, for example, was able to increase from 1.4 miles to 6.9 miles compared to the previous year, BMW, on the other hand, decreased from 4.6 miles to 2.7.
If the report also includes the eight new companies, the diagram looks like this. The best performer is the Chinese ridesharing provider DiDi, which covers 1,535 miles per disengagement. Right behind them are ThorDrive with 189 and Gatik AI with 106 miles.
|Manufacturer||Number of Vehicles||Annual Total of Disengagements||Annual Total Miles||Miles per Disengagement||Disengagement Rate per Mile|
|SF Motors / Seres||2||140||3,493||24.9||0.0401|
Tesla also reported, but the company did not drive more than a few kilometers without a disengagement. For this reason – division by 0 is infinite – I removed Tesla from the graph and table.
What do the numbers tell us?
In particular, one figure stands out, which shows us how urgently needed are new metrics to measure progress in the development of autonomous cars. Baidu, which last year came in at just 206 miles per disengagement, claims to have improved by a factor of 86 to 18,050 miles in one year. That, with all due respect, seems extremely unlikely.
|Company||2015||2016||2017||2018||2019||Change over year|
Representatives from several companies such as Embark, Aurora, Voyage and now Waymo have also pointed out changes to improve key figures. A metric such as the disengagement rate can lead to risky behavior by security drivers if they are encouraged by superiors to intervene too little rather than too much.
Driving is more complex than just disengagement. Key figures in areas such as object classification, intention recognition, comfort, safety, for other road users and passengers, among many others, should be set up and tested in standard scenarios. Different autonomous systems should also be allowed to have different metrics. For example, trucks driving mainly on motorways should s be treated differently from robo-taxis in closed city districts and small delivery robots on sidewalks. This also includes a standard that allows data exchange so that companies can distort the progress of technology by independent third parties by means of such data, rather than by self-reported figures with a lot of leeway to distort the actual state of the art for the public.
I am looking at you! Don’t make a fool of yourself in front of the public. You just managed to lose our trust in everything what you say and do. Come clean now!
With the Disengagement Report, California is currently the only region in the world where this type of report is prescribed and published. This means that the interested public has no better and other data to inform them about the current state of development of autonomous cars.
What else was there?
Two license holders have not submitted a mandatory report and have therefore lost their test license. They are Roadstar.ai and Xmotors.ai. This means that 64 companies currently have a test license in California.
Another interesting detail: Apple has listed 70 test vehicles, but has only been on the road with 23 of them (in California).
This article was also published in German.
They should actually have a DMV person in the car and ask the car to navigate through certain areas and rate them. Not rely on the companies to report this. If the drive is all easy how do you compare that to a drive with huge complexity?
It wouldn’t be difficult to make these tests up and they could continue to get more complex as companies start to ace tests. It should be designed to make the car’s license require far high aptitude than a learner.
Others are as smart as you and had the same idea, the people at the DMV for example had that idea already years ago. And guess what? The DMV has a test procedure to verify that the cars are safe under certain circumstances and can master test scenarios and if they do, the companies get a license so that a specified number of their cars can go without a safety driver in a controlled public area.