04/11/2022
If a human can do it with two eyes then can computers do it too? Many of the big players in the automotive space are betting on an amalgamation of different, but complementary, types of sensors (including lidar and radar) to develop self-driving cars. Tesla, however, have famously long believed that just cameras and computer vision software are enough. But is this actually true behind the scenes?
The idea sounds simple enough. Computer vision software is fed live video images captured by cameras around the vehicle (Tesla use eight of them). The software implements a neural network, that has been trained using lots of pre-prepared image data, to detect roads, cars, objects and people in the video feed. Of course things are not so simple in real life, but if the neural network can be made complex enough and the training data set big enough, some researchers hope they will end up with a system capable of replicating a human driver and dealing with the large (possibly infinite) tail of edge cases that vehicles can face on the roads in the real world.
To explore what Tesla are doing in this field let’s take a look at some of their US patents and applications. In particular, let’s explore the recently introduced International Patent Classification (IPC) G06V, which relates to “image or video recognition or understanding”. This IPC class was only introduced in January 2022 and so reflects mostly recent patents and patent applications published since then.
Tesla’s US patent application US 2022/0108130 relates to autonomous vehicle computer vision and particularly how to train machine learning models by taking into account the properties of the cameras used to capture the training images. This fits in with the public face of Tesla’s approach to autonomous driving.
However, Tesla still seem to acknowledge that lidar, or similar technologies, will be required in autonomous vehicles, at least initially.
In 2019 Tesla acquired computer vision start-up DeepScale. With that purchase came US patent US 11361457, which relates to training computer models for autonomous control systems. The problem it addresses is labelling of training data for training AI systems. Typically a data set would be labelled with human assistance and this labelled data would be used to train the AI. This is easy for a human to do when the training data is in a familiar format, such as images or videos captured by an optical camera. It becomes much harder when the data to be labelled is in an unfamiliar format such as lidar sensor data. To solve this problem, the patent suggests using a computer program to take a first set of annotations, generated by a human, of the output of a first sensor (e.g. a camera), to automatically identify a second set of annotations to apply to the output of a second sensor (e.g. lidar or radar). This makes use of alternative sensors at the training stage, whilst still relying on cameras in the end vehicle.
Along similar lines, Tesla’s application US 2022/0284712 seeks to improve training data by fusing camera data with auxiliary sensor data, such as lidar. The lidar data is associated with objects identified from the vision data. The superior object distance information provided by lidar is correlated with the vision data to allow accurate determination of distance to the object within the training data set. The hope would be that a vision based system trained on such data would be more accurate at determining object distances.
So it can be seen that although Tesla are patenting technology related to vision only autonomous systems, they still expect other sensor input such as lidar or radar to be important in the future, at least in the model training stage.
What about other players in this field? The major OEMs are all involved in developing autonomous driving systems and, unsurprisingly, are filing patent applications for their inventions. Whilst they may be using sensor technologies such as lidar and radar, this does not mean that they are ignoring computer vision solutions. In fact, as can be seen from the graph below, they are actively investing in IP in this area.
It is surprising that Tesla’s patent filings in this area are behind other manufacturers by some margin, especially given that they are pinning their efforts on camera based technology. Perhaps this can be explained, to an extent, by a difference in approach to IP. Toyota and Honda historically file lots of patent applications. This is one way to ensure value is captured and helps to mitigate the risks associated with infringing third party patents. Tesla, on the other hand, have adopted an “open source” strategy for several years making its patented technology available to other parties (provided they reciprocate of course). Admittedly Toyota operate a similar model, but only for a limited set of technologies (which does not yet include their computer vision patents).
Another reason could be that Tesla are relying on alternative forms of protection for their computer vision innovations. Trade secrets protect valuable commercial information and don’t require the publication of any details around the innovation, unlike patents. Tesla may have made the decision to protect some of their software innovations in this manner, especially if it would be difficult to determine if their competitors are using their techniques to train an autonomous driving model. Trade secrets can be powerful, but rely on robust and swift enforcement to prevent secrets leaking out (see, for example, Tesla’s ongoing 2020 suit against Rivian, who recruited former Tesla employees). Normally a balance is needed between trade secrets, patents and other forms of protection.
However you look at it, a complimentary sensor system involving computer vision and lidar, perhaps with radar and ultrasonic sensors as well, seems to be the direction of travel for most players in the autonomous driving field. In the long run, will Tesla’s alternative vision be proven right? We shall see.
The autonomous and electric vehicle group at Reddie & Grose LLP has a great breadth of experience dealing with the different technological and commercial challenges faced by companies of all sizes, and can provide tailored advice to you and your company on how best to protect your ideas in this exciting field.
This article is for general information only. Its content is not a statement of the law on any subject and does not constitute advice. Please contact Reddie & Grose LLP for advice before taking any action in reliance on it.
