An autonomous vehicle cannot be released on the market unless it is safe, but how the industry makes this assertion with confidence comes down to testing. And lots of it. Just as software has been developed to enable the car to drive, computing is also being used to generate highly accurate simulations of real-world driving scenarios.
As much as 95% of self-driving vehicle testing can be carried out via simulation, according to Elektrobit, and experts are confident that simulation can enable a significant increase in testing efficiency. Millions of kilometres of autonomous driving can be tested virtually with limited physical investment, and otherwise rare ‘corner cases’ can be repeated in high volumes.
Current estimations suggest that at least 240 million kilometres of autonomous driving must be carried out – accident free – in order for a system to be deemed ‘safe’. In a recent Automotive World webinar, Elektrobit’s Simon Tiedemann, Product Manager, EB Assist, explained how this ‘mammoth’ task can be greatly assisted by simulation. “You cannot drive 240 million kilometres,” he stated. “But we can do 5% of that real driving, equating to 12 million kilometres.” In the real world, 200 test cars on public roads can record 275km (175 miles) a day, and operate for 220 working days each year. Each car produces ten terabytes of data daily, with the entire fleet generating two petabytes (one thousand million) of data per day.
The remaining 95% of driving can be carried out via simulation software, equating to 228 million virtual test kilometres. This requires 10,000 high performance computers running for nearly 16 days. By comparison, it has taken Waymo – formerly the Google self-driving project – since 2009 to complete seven million real-world miles of testing. Through simulation, Waymo completed 2.7 billion miles of driving in 2017 alone.
A plethora of factors are taken into account in an effort to repeatedly test what would be rare scenarios in the real world. As Tiedemann put it, “some of these scenarios you are unlikely to experience more than once or twice in your lifetime.” Issues such as animals or children wandering into the road are common examples, but there are also hazardous conditions such as storms or floods to consider, which may alter the physics of a car, hide lane markings, or obscure traffic signs and sensors. Parked and moving cars will also appear differently to sensors in heavy snow. Automakers must ensure that fully autonomous systems are designed to handle all driving situations, and simulation can run through more of these situations than would be possible in the real world.
Software is used to log data gathered from vehicle sensors to record real-world test drives. However, this data alone is not sufficient to create all simulation scenarios. Tiedemann estimates that 140,000 scenarios need to be generated, and this cannot be done completely manually. “As you want to cover real test drives, the simulation has to be physically correct,” he added. “The behaviour of other cars, pedestrians and objects must be realistic, and of course simulation of the sensors must be as good as possible to replace real recordings with simulation.”
If 95% of testing can be carried out via simulation, some could question whether the remaining 5% of real-world driving is even necessary, but Tiedemann was quick to argue otherwise. “If you simulate, it is close to reality, but it is not reality. I do not think it is possible for 100% simulation,” he said.
Elsewhere in the automotive industry, simulation is widely used to project crash characteristics and vehicle design, among other things. However, it is currently unclear how accurate these autonomous driving simulations are. One way to test this is to run a simulation algorithm based on real data, say 100,000km of physical driving, and then compare the results with a simulation based on a software-generated scenario. “There are ways to find out if the simulation is acceptable and as close to reality as we need it,” he affirmed, but added: “This is just a theory, and testing the accuracy of simulations is a big task – it will not work with a small number of kilometres. [The accuracy] is not yet clear, but understanding this is absolutely necessary if we are to rely on simulation for testing.”
Autonomous vehicle testing has taken place on public roads for years now, and various European, US and Asian governments have welcomed such activities. As of 20 June 2018, 56 autonomous vehicle testing permits have been issued in California alone, counting the likes of Apple, Bosch, Delphi and Mercedes-Benz. In China, self-driving car licenses have been issued in Beijing, Chongqing, Shanghai and Shenzhen. Sinagpore has also emerged as a key test bed.
