Disengagement Report 2019

With some delay, the Disengagement Report 2019 for California was published today. This report is an annual survey that the California Department of Motor Vehicles (DMV) requires all companies licensed to test autonomous cars on California’s public roads to submit to the DMV. The report must include the number and specifics of the test vehicles used, the miles driven on public roads, and disengagements and their causes. Disengagements are those events where a security driver, who is still required to be in the vehicle, takes control of the vehicle.

First, the numbers as reported:

A total of 36 companies with 676 vehicles drove a total of 2,880,612 miles and had 9,338 disengagements. Broken down by company – namely the 28 companies that already had to report last year – the order is as follows:

Disengagement Report 2019 for license holders in California;
Companies that have reported once before.

The vehicles of the Chinese internet giant Baidu are said to require an intervention every 18,050 miles, according to their own report, followed by Google’s sister company Waymo, which comes to 13,219 miles per intervention. After that come CRUISE, AutoX, Pony.ai, and then by far Nuro, Zoox and PlusAI. At the other end of the scale is Toyota, which comes in at just 0.4 miles per disengagement.

The two reporting German manufacturers, Mercedes Benz and BMW, show different levels of progress. Mercedes, for example, was able to increase from 1.4 miles to 6.9 miles compared to the previous year, BMW, on the other hand, decreased from 4.6 miles to 2.7.

Disengagement Report 2019 for license holders in California;
including companies that filed their report for the first time.

If the report also includes the eight new companies, the diagram looks like this. The best performer is the Chinese ridesharing provider DiDi, which covers 1,535 miles per disengagement. Right behind them are ThorDrive with 189 and Gatik AI with 106 miles.

ManufacturerNumber of VehiclesAnnual Total of DisengagementsAnnual Total MilesMiles per DisengagementDisengagement Rate per Mile
Baidu46    108,300     18,050.0        0.0001
Waymo148110 1,454,137     13,219.4        0.0001
CRUISE22868    831,040     12,221.2        0.0001
AutoX83      32,054     10,684.7        0.0001
Pony.ai2227     174,845       6,475.8        0.0002
Nuro3334       68,762       2,022.4        0.0005
Zoox3242       67,015       1,595.6        0.0006
DiDi128    12,279.4       1,534.9        0.0007
PlusAI22         1,880          940.0        0.0011
Aimotive326          6,056          232.9        0.0043
ThorDrive227      5,089.0          188.5        0.0053
WeRide 839          5,917          151.7        0.0066
Apple 7064          7,544          117.9        0.0085
Gatik AI224      2,553.0          106.4        0.0094
Aurora6141       13,429            95.2        0.0105
Ambarella152      3,161.0            60.8        0.0165
Qualcomm337          2,164            58.5        0.0171
SAIC440          2,230            55.7        0.0179
Drive.ai875          3,974            53.0        0.0189
Nissan447          2,277            48.5        0.0206
Nullmax170          2,430            34.7        0.0288
Phantom AI143          1,125            26.2        0.0382
Lyft191667       42,931            25.8        0.0388
SF Motors / Seres2140    3,493           24.9       0.0401
NVIDIA10655          7,218            11.0        0.0907
Intel3165      1,295.0              7.8        0.1274
Telenav13               22              7.3        0.1364
Apex.Ai163          448.3              7.1        0.1405
Mercedes Benz182054       14,238              6.9        0.1443
BMW58               21              2.7        0.3745
Udelv4444             707              1.6        0.6284
RIDECELL16              8.7              1.4        0.6904
Valeo192             100              1.1        0.9241
Toyota62947          1,817              0.6        1.6219
Box Bot2109            38.5              0.4        2.8312

Tesla also reported, but the company did not drive more than a few kilometers without a disengagement. For this reason – division by 0 is infinite – I removed Tesla from the graph and table.

What do the numbers tell us?

In particular, one figure stands out, which shows us how urgently needed are new metrics to measure progress in the development of autonomous cars. Baidu, which last year came in at just 206 miles per disengagement, claims to have improved by a factor of 86 to 18,050 miles in one year. That, with all due respect, seems extremely unlikely.

Company20152016201720182019Change over year
Baidun.a.n.a.          41.60        205.60   18,050.008679%
Waymo     1,244.37     5,127.97     5,595.95   11,154.30   13,219.0019%
GM Cruisen.a.          54.01     1,254.06     5,204.90   12,221.00135%
Zooxn.a.n.a.        282.96     1,922.80     1,596.00-17%
Drive.ain.a.n.a.          43.59          83.00          53.00-36%
Nissan          14.01        141.34        208.63        210.50          48.50-77%
Mercedes Benz            2.17            2.00            1.29            1.50          11.10640%
NVIDIAn.a.n.a.            4.63          20.10          11.00-45%
Telenavn.a.n.a.          32.00            6.00            7.3022%
Delphi/Aptiv          41.14          17.57          22.35  
Bosch            1.50            0.68            2.41   
Disengagement rates per mile since 2015 for selected companies.

Representatives from several companies such as Embark, Aurora, Voyage and now Waymo have also pointed out changes to improve key figures. A metric such as the disengagement rate can lead to risky behavior by security drivers if they are encouraged by superiors to intervene too little rather than too much.

Driving is more complex than just disengagement. Key figures in areas such as object classification, intention recognition, comfort, safety, for other road users and passengers, among many others, should be set up and tested in standard scenarios. Different autonomous systems should also be allowed to have different metrics. For example, trucks driving mainly on motorways should s be treated differently from robo-taxis in closed city districts and small delivery robots on sidewalks. This also includes a standard that allows data exchange so that companies can distort the progress of technology by independent third parties by means of such data, rather than by self-reported figures with a lot of leeway to distort the actual state of the art for the public.

BAIDU!
I am looking at you! Don’t make a fool of yourself in front of the public. You just managed to lose our trust in everything what you say and do. Come clean now!

With the Disengagement Report, California is currently the only region in the world where this type of report is prescribed and published. This means that the interested public has no better and other data to inform them about the current state of development of autonomous cars.

What else was there?

Two license holders have not submitted a mandatory report and have therefore lost their test license. They are Roadstar.ai and Xmotors.ai. This means that 64 companies currently have a test license in California.

Another interesting detail: Apple has listed 70 test vehicles, but has only been on the road with 23 of them (in California).

This article was also published in German.

4 Comments

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s