Waymo says it built a better benchmark for comparing robotaxis to humans | TechCrunch

Waymo Develops Updated Computer Model to Compare Autonomous Driving Systems With Human Behavior

Waymo has developed a new computer model intended to provide a more precise comparison between its autonomous driving software and human driver performance. The Alphabet-owned company created the model in collaboration with TU Delft and described the work in a research paper published in Nature Communications.

The model is based on a framework called active inference, which represents drivers as continuously generating predictions about possible future scenarios and selecting actions to achieve the safest and most predictable outcomes. Waymo stated that the updated model is expected to improve accuracy over the version used in prior years. It is designed to help analyze how human drivers respond in crash scenarios that autonomous vehicles encounter.

In a company blog post, Waymo compared the model to longstanding automotive safety practices: “For decades, the automotive industry has used physical and virtual crash dummies to evaluate a car’s safety features, including its hardware and structural integrity.” The new model, the company said, extends this approach by serving as a behavioral benchmark that represents “reasonable expectations on how a careful and competent human driver responds to traffic conflicts.”

Autonomous vehicle developers require accurate models of human driving behavior to assess robotaxi performance during crashes. Waymo is currently expanding operations to additional cities while subject to regulatory and public review.

In one documented incident in January in Santa Monica, California, a Waymo vehicle struck a child near a school. Using its earlier model, the company estimated that an attentive human driver would have struck the child at approximately 14 miles per hour. The Waymo vehicle made contact at 6 miles per hour after slowing from 17 miles per hour; the company reported that the child sustained minor injuries. The National Highway Traffic Safety Administration and the National Transportation Safety Board are continuing to investigate the event.

The primary distinction between the new model, referred to as the Reference Driver, and earlier versions is its ability to simulate driver behavior leading up to a collision. Previous models used by Waymo and other companies primarily replicated last-second reactive maneuvers. The Reference Driver incorporates simulation of a driver’s internal “surprise” during a conflict, according to Arkady Zgonnikov of TU Delft.

Waymo indicated that the model can be adjusted to represent a range of road user behaviors beyond collision avoidance and is structured for application to large sets of test scenarios. The company stated that the model supports evaluation of complex real-world crashes in virtual environments and identification of performance changes at greater scale.

Waymo has released the research code for the Reference Driver under an academic, non-commercial license. The license permits use for research, teaching, personal experimentation, and scientific publication.