Disengagement Report 2019

With some delay, the Disengagement Report 2019 for California was published today. This report is an annual survey that the California Department of Motor Vehicles (DMV) requires all companies licensed to test autonomous cars on California’s public roads to submit to the DMV. The report must include the number and specifics of the test vehicles used, the miles driven on public roads, and disengagements and their causes. Disengagements are those events where a security driver, who is still required to be in the vehicle, takes control of the vehicle.

First, the numbers as reported:

A total of 36 companies with 676 vehicles drove a total of 2,880,612 miles and had 9,338 disengagements. Broken down by company – namely the 28 companies that already had to report last year – the order is as follows:

The vehicles of the Chinese internet giant Baidu are said to require an intervention every 18,050 miles, according to their own report, followed by Google’s sister company Waymo, which comes to 13,219 miles per intervention. After that come CRUISE, AutoX, Pony.ai, and then by far Nuro, Zoox and PlusAI. At the other end of the scale is Toyota, which comes in at just 0.4 miles per disengagement.

The two reporting German manufacturers, Mercedes Benz and BMW, show different levels of progress. Mercedes, for example, was able to increase from 1.4 miles to 6.9 miles compared to the previous year, BMW, on the other hand, decreased from 4.6 miles to 2.7.

If the report also includes the eight new companies, the diagram looks like this. The best performer is the Chinese ridesharing provider DiDi, which covers 1,535 miles per disengagement. Right behind them are ThorDrive with 189 and Gatik AI with 106 miles.

Manufacturer	Number of Vehicles	Annual Total of Disengagements	Annual Total Miles	Miles per Disengagement	Disengagement Rate per Mile
Baidu	4	6	108,300	18,050.0	0.0001
Waymo	148	110	1,454,137	13,219.4	0.0001
CRUISE	228	68	831,040	12,221.2	0.0001
AutoX	8	3	32,054	10,684.7	0.0001
Pony.ai	22	27	174,845	6,475.8	0.0002
Nuro	33	34	68,762	2,022.4	0.0005
Zoox	32	42	67,015	1,595.6	0.0006
DiDi	12	8	12,279.4	1,534.9	0.0007
PlusAI	2	2	1,880	940.0	0.0011
Aimotive	3	26	6,056	232.9	0.0043
ThorDrive	2	27	5,089.0	188.5	0.0053
WeRide	8	39	5,917	151.7	0.0066
Apple	70	64	7,544	117.9	0.0085
Gatik AI	2	24	2,553.0	106.4	0.0094
Aurora	6	141	13,429	95.2	0.0105
Ambarella	1	52	3,161.0	60.8	0.0165
Qualcomm	3	37	2,164	58.5	0.0171
SAIC	4	40	2,230	55.7	0.0179
Drive.ai	8	75	3,974	53.0	0.0189
Nissan	4	47	2,277	48.5	0.0206
Nullmax	1	70	2,430	34.7	0.0288
Phantom AI	1	43	1,125	26.2	0.0382
Lyft	19	1667	42,931	25.8	0.0388
SF Motors / Seres	2	140	3,493	24.9	0.0401
NVIDIA	10	655	7,218	11.0	0.0907
Intel	3	165	1,295.0	7.8	0.1274
Telenav	1	3	22	7.3	0.1364
Apex.Ai	1	63	448.3	7.1	0.1405
Mercedes Benz	18	2054	14,238	6.9	0.1443
BMW	5	8	21	2.7	0.3745
Udelv	4	444	707	1.6	0.6284
RIDECELL	1	6	8.7	1.4	0.6904
Valeo	1	92	100	1.1	0.9241
Toyota	6	2947	1,817	0.6	1.6219
Box Bot	2	109	38.5	0.4	2.8312

Tesla also reported, but the company did not drive more than a few kilometers without a disengagement. For this reason – division by 0 is infinite – I removed Tesla from the graph and table.

What do the numbers tell us?

In particular, one figure stands out, which shows us how urgently needed are new metrics to measure progress in the development of autonomous cars. Baidu, which last year came in at just 206 miles per disengagement, claims to have improved by a factor of 86 to 18,050 miles in one year. That, with all due respect, seems extremely unlikely.

Company	2015	2016	2017	2018	2019	Change over year
Baidu	n.a.	n.a.	41.60	205.60	18,050.00	8679%
Waymo	1,244.37	5,127.97	5,595.95	11,154.30	13,219.00	19%
GM Cruise	n.a.	54.01	1,254.06	5,204.90	12,221.00	135%
Zoox	n.a.	n.a.	282.96	1,922.80	1,596.00	-17%
Drive.ai	n.a.	n.a.	43.59	83.00	53.00	-36%
Nissan	14.01	141.34	208.63	210.50	48.50	-77%
Mercedes Benz	2.17	2.00	1.29	1.50	11.10	640%
NVIDIA	n.a.	n.a.	4.63	20.10	11.00	-45%
Telenav	n.a.	n.a.	32.00	6.00	7.30	22%
Delphi/Aptiv	41.14	17.57	22.35
Bosch	1.50	0.68	2.41

Representatives from several companies such as Embark, Aurora, Voyage and now Waymo have also pointed out changes to improve key figures. A metric such as the disengagement rate can lead to risky behavior by security drivers if they are encouraged by superiors to intervene too little rather than too much.

Driving is more complex than just disengagement. Key figures in areas such as object classification, intention recognition, comfort, safety, for other road users and passengers, among many others, should be set up and tested in standard scenarios. Different autonomous systems should also be allowed to have different metrics. For example, trucks driving mainly on motorways should s be treated differently from robo-taxis in closed city districts and small delivery robots on sidewalks. This also includes a standard that allows data exchange so that companies can distort the progress of technology by independent third parties by means of such data, rather than by self-reported figures with a lot of leeway to distort the actual state of the art for the public.

BAIDU!
I am looking at you! Don’t make a fool of yourself in front of the public. You just managed to lose our trust in everything what you say and do. Come clean now!

With the Disengagement Report, California is currently the only region in the world where this type of report is prescribed and published. This means that the interested public has no better and other data to inform them about the current state of development of autonomous cars.

What else was there?

Two license holders have not submitted a mandatory report and have therefore lost their test license. They are Roadstar.ai and Xmotors.ai. This means that 64 companies currently have a test license in California.

Another interesting detail: Apple has listed 70 test vehicles, but has only been on the road with 23 of them (in California).

This article was also published in German.