Why The Ranking Of Autonomous Car Companies Is Just Plain Wrong

A favorite pastime of analyst companies is creating rankings, showing what companies are leading in what industry. Such lists are not just created for altruistic purposes. They serve to create attention for their very own analyst company and drive sales of lengthy reports for good money. And – so they hope – to get some projects from those very same companies with the aim to reconsider their evaluations by sticking them some more data of how far they really are and that they should move up the ranks. Gartner, Navigant, Deloitte and others are enjoying themselves in this work and publish such reports with regularity. The more controversial the results, the better. Especially when the controversy hits some newcomers and makes the incumbents look good.

A recent ranking by Navigant Research became a good contender for the Golden Raspberry Award for analyst groups. Published in January 2018, the report ranks 19 companies, of which some of them actually shouldn’t be in there, and some who should be there are not. The result is with little surprise – surprising. And plain wrong.

It looks like companies that build cars today where weighted higher, no matter if they actually have some self-driving initiatives.

Criteria

The result has to do with the weight of criteria that were used to answer the question what companies have today the best strategy and execution power to bring autonomous cars on the roads. And to stick it right from the beginning: Culture eats strategy for breakfast, to quote management guru Peter Drucker.

The weight of the criteria is crucial for the result – no surprise here – and it shows here. On the one hand they are calculating the efforts bey the companies in form of monetary efforts, on the other hand whether those companies are manufacturing cars today.

But both criteria have little to do with if you’ll be able to bring autonomous cars on the road. Carriage makers around 1900 had all the capacity and execution power to manufacture a large amount of carriages, and if necessary, equip them with engines. But they didn’t do that and the automotive pioneers back then took the initiative. This had to do with some new skills and knowledge around the engine, as well as a real interest to replace the old propulsion system ‘horse’ with the new one called ‘engine’.

And that’s where we return to culture. My discussion with the teams responsible at several traditional car makers showed the following: while they were all choosing the right words and messages around autonomous driving, electric vehicles, and sharing economy, their hearts were not in those new trends. And why should they? If you grew up with an interest in cars and driving them and which make some vroom-vroom sounds, the last you want to do is build an electric robotaxi. They can throw as much money at you as they want, if your motivation is not here, all is lost.

Detailed Analysis

Today 51 companies are registered in California, who received a permit for testing autonomous cars on all public roads. Combined, they have several hundred cars, with most of them roaming the Silicon Valley as their testing ground. On an average commute, Bay Area residents will encounter them several times a day.

Types of driving

We can categorize the autonomous test drives in several clusters: driving in city traffic, on highways and country roads, and on test tracks. Before those cars are allowed to hit public roads companies typically have to prove that they tested them for several thousand miles on closed tracks. The next step is highway driving, as the complexity is lower. Traffic is only one way, no pedestrians or signal lights that the cars have to deal with. In fact self-driving engineering nano-degree students from Udacity start with recognizing and processing lane markers on highways as their first course exercise, because it’s one of the simplest. City traffic is among the most complex scenario, because it comes with an almost infinite range of variations.

Those city drives can be made less complex by driving in suburbs. And the Financial District in San Francisco or Downtown Manhattan for sure are at the other end of the complexity spectrum for autonomous cars. Lots of people, traffic signals, traffic signs, many many other traffic participants that often ignore basic traffic rules, and for participants such as police cars and garbage trucks, autonomous cars need to understand and master the context (“Do they mean me, or can I continue driving?”).

In addition we are talking about level 4 or level 5 cars. Level 4 are cars that have to hand over control to a human, while Level 5 the car never expects to hand over control and master all situations. And even there we find differences. Does the autonomous vehicle drive a fixed route like a public bus, or can it vary the route and find the best path in the region it operates? French bus manufacturer Navya, for instance, manufactures autonomous mini-shuttles driving like a bus on pre-programed routes. Waymo’s vehicles, on the other hand, are supposedly fully autonomous covering all routes and streets in a given region or even the globe.

Maps

Even when the cars master driving, there is still the human element missing. But this time not as pedestrian on the road, but as passenger that has to be picked up, driven, and dropped off at the destination. And as it turns out: that’s a pretty challenging task. Addresses are surprisingly imprecise, and geo-location in areas with tall buildings make it difficult to ‘find’ the passenger. Every Uber-passenger in San Francisco of New York can tell stories of how they had to direct their Uber driver via phone call to find the right spot where they can be picked up. IIt doesn’t add to easy use of comfort with an Uber or an autonomous vehicle, when the official address is 100 meters from the house entrance and you have to cross to the car with heavy luggage or in a rainy weather.

Insofar, navigation maps containing that information become crucial. And there is the point where Google Maps is light years ahead of other map vendors such as Bing, Tomtom, Here or Apple Maps. We already had an extensive article on that.

Mileage

The last numbers for autonomous miles driven in real city traffic that Waymo has published were at 4 million miles. Uber recently reached 2 million miles. Even small startups such as Renovo.ai have more than 40,000 miles under their wheels. And Tesla has gathered hundreds of millions of miles driven by their customers in Autopilot mode on highways and country roads. Compare that to almost no publication of miles driven by most automotive manufacturers. At first, the reason was it was a secret, now the reason is more one of being ashamed of being so far behind.

Fleets

Those mile-numbers are proportional to the fleet sizes. The more cars I can use for testing, the faster the whole fleet gathers experience. Hundreds of active cars in a fleet are experiencing proportionally more traffic scenarios then just with a handful of vehicles.

Operators of such large active fleets driving in real city traffic are faster in increasing safety of their self-driving technology. Faster than those hesitant manufacturers, who mostly drive on test tracks and country roads.

If we are consulting the list of cities, where today autonomous taxi fleets are testing – and those are real taxi fleets with both external passengers or company employees asked to use them for running errands like GM Cruise is doing in San Francisco – then we’ll find those companies:

• Waymo
• Uber
• nuTonomy (Delphi)
• GMCruise
• Baidu
• Voyage
• Volvo
• and soon Optimus Ride

Please note: Voyage or nuTonomy are not listed in Navigant’s list of companies.

Who’s missing in this list are – with the rare exception of GM – all other traditional car manufacturers. I often hear from German car makers and suppliers that the Daimlers, BMWs and Volkswagen of this world do have have a larger number of autonomous cars, but there is almost no reporting on those cars being spotted on public streets. Even if, the sightings are almost always on highways.

There are also no reports of extensive testing of autonomous fleets of German manufacturers (and others) in cities. And even less so with passengers. The longer those companies are missing out, the less possible it will become to close today’s large gap to the leading companies.

With a dozen cities today having autonomous taxi fleets, we can expect 20+ cities by end of this year, and end of 2019 even 50 cities.

Simulation

Real-world driving is complemented by driving in simulators. In fact, Waymo-CEO John Krafcik stated that 80 percent of all improvements on autonomous vehicles is due to work in the simulator. Waymo and Uber have revealed their own simulators to the public. And Baidu offers in their Apollo-open-source-program a simulator, as well as the self-driving engineering program from Udacity.

Such simulators allow to test one traffic scenario in thousands of varying conditions. Thousands of cars, night and day, different weather and light, even adding new objects and traffic participants on the roads. And driving they do those scenarios. Waymo alone drives 1 billion miles per year in the simulator.

We know little about the simulators from German vendors. But even if, simulators are not replacing real-world testing.

Disengagements

Disengagements in autonomous driving parlance are those moments, where an autonomous driving car is confused and hands over control of the car to a safety driver. The metric to measure how advanced an autonomous vehicle is can be deducted by the number of miles a car can drive until it has to disengage. Unfortunately, the definition of what counts as disengagement is not completely clarified and is often based on the judgement of the company. The annual Disengagement Reports of the California DMV published end of January have given a first glimpse in the technology advancement in the last two years, even though the direct comparison between the companies proves to be tricky. Beginning 2017 Waymo had one disengagement every 5,000 miles. Given an annual mileage of an average driver in the US of 12,000 miles per year, this comes close to one disengagement every 5 to 6 months. All other companies had disengagement rates that were at least 10 to 100 times more common.

Safety Driver

Today, most states and regions require a so-called safety driver to be present when testing autonomous cars. Those are specially trained drivers who are in the driver seat and can take over control of the vehicle in case of problems. While California is close at passing a law that allows test drives without a safety driver in the vehicles, other states are already further. Waymo has been operating for several months its autonomous vehicles without a safety driver in Chandler, Arizona.

So far there is no other company that tests Level 5 driving without a driver on public roads and with passengers in a robotaxi fleet. GMCruise seems to come closest and has announced a specially designed car without a steering wheel for next year.

Sensor Technology

The best algorithms can’t do anything, when the sensors are not delivering prices and good data, and when the cars cannot execute on the data. It’s no surprise that hundreds of companies are working on sensor technologies, advancing in a rapid pace both the capabilities of the sensors while at the same time decreasing their prices, like we have seen it in the past with computers.

While most companies are using Lidar sensors from vendors such as Velodyne, Luminar and others, Google has put an effort in developing their own technology and succeeded to stay ahead of its specialist competitors. Uber as well put a focus on developing their own sensors, although seemingly not always with legal means, as has been demonstrated by the lawsuit filed around presumed IP-theft.

Security

A fear with autonomous cars is that ‘somebody is going to hack and do bad things with them.’ First, it turns out that Waymo is keeping its cars most of the time offline, without online-connection. Second, today there are are not enough bandwidth and stable wireless connections covering a large enough range to make this a viable path for autonomous car development.

Even if autonomous cars were online all the time, what manufacturer would you trust more with their security? Google or Apple, which both originate from the digital world and already today experience millions of cyber attacks every day that they have to fight off, identify and control; or traditional car manufacturers, which started only recently adding digital features to their cars?

Software

Developing an operation system for autonomous driving is a complex thing. Probably more complex than developing one for computers, smart phones and smart devices. Such operation system need development tools, so that developers can build new apps for them. Apps for autonomous cars, the next big thing.

All that requires familiarity with open standards, security, with documentation, and everything else that enables millions of software developers to build high quality software for autonomous cars without crashing the operation system.

What companies have decades of experience with building operation systems and can already today lean on large communities of software developers in their ecosystems? Google, Apple, Microsoft, Baidu & Co for sure, less so traditional car makers.

Laws & Regulations

The best efforts of companies and startups is leading nowhere, when the environment is not supportive and bringing laws and regulations enabling them. Multiple US states are competing with flexible laws around autonomous driving, and the US Senat and US Congress both have passed laws to support that development. In the next three years up to 100,000 autonomous cars can be brought on all public roads in the US without following the safety standards developed for human drivers. Not that the cars will make the roads less safe. Today’s regulations contain safety standards that have a human driver in mind. While back and side mirrors are in for human drivers, a robotaxi doesn’t need them. Its sensors are covering the whole range.

Other countries such as China, Germany, or generally the continent of Europe are lagging behind with their laws and regulations and thus slow down progress. Germany puts its focus on ethical problems, even installing an Ethics Commission, to solve problems and risks that autonomous cars may pose. Not so much on what problems autonomous cars may solve. Its all about mindset

Insofar American companies are getting an official push with the help of lawmakers and regulators. If you can’t legally test your technology in your own country, then you have a problem.

Vehicles

How quickly this technology becomes available to the public can be observed by how the technology is put into production cars. And Tesla for sure is top. Its Autopilot Hardware Kit 2 is aimed at autonomous driving. It’s been put into all Model S, X, and 3 since October 2016. Today, there are are over 130,000 cars in the hands of customers with that equipment. No other vendor has so many cars with hardware for autonomous driving built and delivered to customers.

While Tesla has certainly lost some speed with breaking off with Mobileye and internal difficulties in the Autopilot team haven’t helped either, recently the company seems to have intensified its efforts. Test drives from the west to the east coast have shown progress, even though under very controlled circumstances. Even that the company is currently lagging, the fact that Tesla has over 100,000 cars on the street equipped with that hardware makes the company a dormant giant in self-driving technology. once the first version of autonomous Autopilot-software turns all those cars into autonomous vehicles, the data gathering efforts and thus improvements on the self-driving capabilities with increase exponentially and quickly surpass the competition. Within a few months Tesla could become the absolute leader in self-driving.

Waymo may be leading the pack with many criteria, but the company has to ramp up its efforts to bring the technology to the masses and find partners that can help them.

GM has its own production and demonstrates intensive efforts, but the company seems to keep stumbling over its own history. As seen with the Chevrolet Bolt, which has turned out less of a project where their heart goes in, but more of hitting regulatory requirements.

Daimler keeps showcasing kind of cute prototypes, that stay the same over the years and the longer it takes they become more and more pitiable.  We keep hearing that “they have more than they show,” but this kind of stays a magic beast that nobody ever sees. Daimler appears as the Duke Nukem of car makers.

Volkswagen showcases through its technology lead Audi a car that can race a race track autonomously. And Audi introduced the A8 e-tron with Level 3 technology that was introduced by using a lot of technical acronyms that nobody understands and knows what it actually is good for. Now, with the help of startup Aurora the behemoth is aiming to test its first autonomous taxi fleets in up to five cities by 2021. Note: in three years, while we have already today a dozen fleets testing in cities.

BMW talks little, shows little, does little, and isn’t able? The company also decided to partner with Intel, the third runner up to the leading companies NVIDIA and Qualcomm. That’s not a good start.

Ford is putting in some effort and wants to introduce its first commercial taxi fleets by 2021, but the company is also stepping on its own toes all the time. The company is too much ingrained in old thinking patterns, even that the new CEO is a Silicon Valley veteran. With the help of Argo.ai the company is following a similar strategy as VW.

Lyft-Aptiv are new entrants in the game and I am pretty much skeptical about their position in Navigant’s Leadership-quadrant.

Baidu is aiming at beating Google with the Google tactics, which is open sourcing an operation system. The same playbook that Google followed with Android when Apple introduced the iPhone and iOS. Baidu could turn out to be pretty much successful, especially in China. They are the fully motivated underdog with a lot of cash to burn.

French automaker PSA has not much to show so far. They do have their demo-cars, but they are mostly driving assistance systems.

The fact that Jaguar-Landrover is ranked higher than Toyota or Uber and Tesla demonstrates the flaws of Navigant’s ranking. I guess a Navigant analysts must have put some commas on the wrong places in an Excel sheet, because really? Jaguar-Landrover being a serious contender?

Apple so far has 27 cars registered for testing autonomous cars in California, and money is not an issue here. But Apple appears surprisingly old industry in this field – and I am not meaning that in a good way. The efforts resemble more a brute force-attack than an elegant approach. Just look at their excessive Lidar arrangements on the roof. It doesn’t help that the company seems to make 3 steps forward and 2 steps back. The company’s commitment to developing this technology is flaky, always prone to stop and restart it. Very demotivating for a team.

Nissan so far has highlighted self-parking cars, self-parking office chairs and – I kid you not – self-parking slippers!

FCA appears in the infamous Navigant-list as leader. WHAT? Did I miss something? Are we talking about the same Fiat-Chrysler company? If they hadn’t thrown themselves in the arms of Waymo, they would have nothing at all to show. Nothing! Null! Zero! Zilch!

Companies that are missing in the list are Voyage, which is now starting its second autonomous taxi fleet in Florida. Also the likes of Zoox, Renovo, AutoX or retrofit kit startups like Kopernikusauto are prominent omissions in the list. Latter company is actually aiming at providing 10,000 retrofit kits for autonomous driving end of this year for Swiss car importer AMAG.

The Best Talents

Another critical issue that cannot be underestimated by traditional car makers is recruiting the best talent for those new technologies. Why should a computer vision specialist or AI-expert start with BMW, when they won’t find peers or a supportive climate in a culture dominated and managed by people who grew their career by bending metals? How quickly motivation tanks with young employees is something I experienced second hand through friends. After only two months in the Innovation Outposts in Silicon valley and development centers, and almost constant struggles with IT departments about the use of required software, power and competency struggles between the respective headquarters in Stuttgart, Munich, or Wolfsburg and outposts, as well as getting the vibe of doing all the work just for a quick and dirty image campaign for the board, those talents were pretty quickly turning their focus on finding positions at the new automotive industry with a different motivation to make the literal dent in the universe.

And one can only recommend them to do that. When developing  autonomous cars and electric vehicles, time served – pardon, spent – at VW, BMW or Daimler and having them in their CVs almost appears as a career obstacle that you have to explain. While your peers changed the world at Waymo & Co, you wasted your time fighting internal battles at the traditional car makers? It almost invites the question, if you weren’t good enough for those new automotive pioneers and didn’t have alternatives?

Summary

Today, we don’t know much about the real development status for each and every company vying for supremacy in the autonomous vehicle space. Some signals give us clues on the current status. The number of cars on the road and miles driven is one clue. Another are the disengagement deports, being published annually by the California DMV (we are expecting the new one coming next week). And then we know a bit about (hardware-)technology being already put into production cars, like at Tesla.

The Navigant report’s distinction in Leaders, Contenders, Challenger and Followers seems by chance only, also the ranking itself. The report seems to be kissing up to traditional makers to get some business from that industry. Were the report published in 1900, it would ensure the coach makers that they have the better chances for building cars than those fools we call ‘automotive pioneers.’ We know how that ended.

This report from Navigant hasn’t really much to do with reality.

This article has also been published in German.