California law now requires that all companies testing autonomous vehicles (AVs) in the state provide data to the California Department of Motor Vehicles (CDMV) on the number of miles driven autonomously, along with details on each and every disengagement. These in turn are made public, with results for 2018 released in February 2019. There were few surprises: of the 48 companies listed, Waymo is still king of the road with 1.2 million miles logged. A total of 114 disengagements means a rate of 0.09 disengagements per 1,000 miles. GM Cruise came second, logging 448,000 miles and a disengagement rate of 0.19 per 1,000 miles.
But one figure in the latest data demonstrates perfectly the flaws in the California system: Apple, which came third in miles driven with just under 80,000, logged 871.65 disengagements per 1,000 miles. Far from lagging behind its competitors, it emerged that Apple was reporting every instance in which the driver was taking control of the vehicle, including non-emergencies. This was later changed, such that these disengagements were removed from the count.
These stats underline the fact that the CDMV’s definition of disengagement lacks clarity, and is open to interpretation. It reads, for example, that a safety driver taking control of a vehicle in self-driving mode to ensure safe operation qualifies as a disengagement. In November 2017, a vehicle run by GM Cruise, nicknamed ‘Pickle’, stopped on a crosswalk in San Francisco as a traffic light changed from yellow to red. The safety driver took control, so that the car didn’t block the crosswalk. This is arguably an example of the CDMV’s definition, given the presence of pedestrians at the crosswalk, yet a report by Jalopnik in 2018 revealed the incident was not recorded.
I don’t think you’ll find anybody inside the community that thinks these numbers are a great measure of progress
Furthermore, Apple’s sudden change of tactics raises further questions: what exactly constitutes an emergency? Previous suggestions from the tech giant have failed to expand much upon the state’s ideas: “A disengagement should be defined as an unexpected event or failure that requires the safety driver to take control of the vehicle in order to prevent a crash or traffic violation,” it wrote in a 2017 letter to the CDMV. However, the company was clear on what it didn’t want: disengagements should not include incidents where a system error leads to a dropout where it wasn’t necessary, operational constraints where the driver has been trained to disengage the system (such as navigating a construction site), or scenarios such as the end of a test.
The system has drawn a fair number of critics as a result. “TuSimple does not view these numbers as great measures of progress,” said Chuck Price, Chief Product Officer at TuSimple: “I don’t think you’ll find anybody inside the community that does see them as a great measure of progress.” Instead, the self-driving truck developer is one of several advocating the concept of ‘meaningful miles’, and lessening scope for companies to simply rack them up along routes with low potential for learning, such as a highway at night.
It’s all we’ve got!
That said, there are those who believe that whilst the system is imperfect, it still holds value. Jeff Blackburn, Head of Business Development at Metamoto, a simulation platform being used by AV developers, believes that with some tweaking, regulatory agencies such as the National Highway Traffic Safety Administration (NHTSA) could roll out such a system to help the industry tackle one of the biggest obstacles to AVs: public acceptance.
Underlining the challenge ahead was this year’s report from the American Automobile Association (AAA). Its latest survey reports that 71% of participants are afraid of the idea of riding in fully self-driving vehicles, up 8% on the previous year’s survey. This follows a year of high profile self-driving vehicle accidents, including the death of Elaine Herzberg in Tempe, Arizona. Meanwhile, the CDMV reported collisions involving AVs more than doubled in 2018 to 67, up from 29 in 2017: a natural consequence of having more companies test on the road perhaps, but a statistic that’s unlikely to soften concerns. “Automated vehicle technology is evolving on a very public stage and, as a result, it is affecting how consumers feel about it,” said Greg Bannon, AAA’s Director of Automotive Engineering and Industry Relations, in a statement.
The company was clear on what it didn’t want: disengagements should not include incidents where a system error leads to a dropout where it wasn’t necessary, operational constraints where the driver has been trained to disengage the system, or scenarios such as the end of a test
Developers are keenly aware of the need to get the public on their side, and have expressed support for California’s goals if not its approach. “Apple believes that public acceptance is essential to the advancement of automated vehicles,” wrote the tech company, adding that “access to transparent and intuitive data on the safety of the vehicles being tested will be central to gaining public acceptance.”
What’s needed, says Blackburn, is better definitions. Otherwise, he argues, testers could potentially game the system. “If the authorities could come up with a structure which defined what an autonomous vehicle disengagement was,” he suggests, “and how these should be reported, I believe that would yield data transparency, which in turn would gain public trust in AVs.”
Faulty though California’s methods might be, it is worth remembering they are also among the most developed in the world, the product of many years of experience in its role as the cradle of life for the self-driving agenda. They will not disappear over night, and could yet prove the foundation upon which all other nations, let alone the rest of the states, develop their own AV regulations.
The industry will need to consider carefully how to produce fair and comparable reports
The state is therefore in a good position to set worldwide standards to make AV testing data as transparent and meaningful as possible, and as such it must continue to refine its processes. Recent data from China has underlined the importance of this: the Beijing Municipal Commission of Transport released the country’s first self-driving test reports, detailing 150,000km as logged by eight companies working in the capital. Of these, Baidu is the largest, accounting for more than 90% of the miles driven.
A separate report by the Beijing Innovation Center for Mobility Intelligent also covered the testing, mentioning 23 disengagements. However, the disengagements per kilometre driven metric is missing from both reports. On the one hand, this could be a cause for concern, as it means there is little way to compare the Chinese tech giants with those in the US. On the other, it might speak to the inefficacy of the metric: driving conditions in Beijing are very different to California test-sites, with arguably more potential for disengagement as a result of other drivers’ disruptive behaviour. The industry will need to consider carefully how to resolve inequalities like this, and how to produce fair and comparable reports.
This article appeared in the Q3 2019 issue of M:bility | Magazine. Follow this link to download the full issue.
