How Waymo and Google Photo Search Work Together

A number of start-ups developing self-propelled cars regularly post videos of their vehicles navigating through various traffic situations and reacting to objects. We also have some understanding of how objects are detected. However, the number of objects and situations that can be recognized increases exponentially when combined.

Take a skateboard as an example. This can be carried by a person under the arm, or a person can be on it, or it can simply be parked somewhere. Depending on how it is used, a self-propelled car must be able to assess the situation and react to it.

Waymo has now given a small, non-technical insight into this issue in a blog post. In collaboration with Google Search, the blog draws on the data treasure that this department has been working on for years within the Alphabet Group. In Google Photo Search, for example, all kinds of objects are identified and catalogued in pictures. Even though Waymo himself has collected a large amount of data, the use of Google Photo Searc data allows a leap in accuracy.

All image data collected by Waymo during its test drives is broken down from rough categories to an increasingly granular object classification. For example, in the case of a truck, the inscriptions are also read using character recognition (OCR) and their contents analysed. Thus a truck can not only be classified as a heavy goods vehicle, but also the make and type. This makes it possible to predict the corresponding driving behaviour that can be expected from such vehicles. For example, such a heavy-duty transporter moves slower or needs a larger turning circle to turn, and first make a turn against the turning direction to create the turning circle.

Thanks to this additional data, Waymo’s neural network will be better trained, and therefore more responsive to driving situations. And one vehicle can share this with all the others, so the entire fleet will successively drive better. Waymo has prepared a small animated graphic to show this analysis of the image data.

This article was also published in German.