Waymo Showers Us With More Information On Driving Simulation

Support CleanTechnica’s work through a Substack subscription or on Stripe.


Or support our Kickstarter campaign!

It’s been a busy week for Waymo. The California company, born out of Google many seasons ago, is at a critical point in its life and development. The company is on the verge of something, but that something is completely different depending on who you ask. This week, Waymo has raised $16 billion, has entered two new cities, and has had to defend itself and its use of remote fleet response agents in front of the US Congress.

The robotaxi company is on the verge of extending its leadership quickly and broadly as it scales up, or, if its technological approach is not competitive enough, before it eventually crashes and burns billions as it goes bankrupt. Personally, I think the company’s approach is convincing and responsible. I think it has been driven by long-term thinking and excellent planning. But I have been wrong about things in the past, so I try to always be open to the possibility I will be wrong again.

In any case, given everything that has been underway, Waymo apparently decided that now is the time to provide an extensive summary into how it uses simulation to improve its technology and scale up its operations more quickly.

“The Waymo Driver has traveled nearly 200 million fully autonomous miles, becoming a vital part of the urban fabric in major U.S. cities and improving road safety. What riders and local communities don’t see is our Driver navigating billions of miles in virtual worlds, mastering complex scenarios long before it encounters them on public roads. Today, we are excited to introduce the Waymo World Model, a frontier generative model that sets a new bar for large-scale, hyper-realistic autonomous driving simulation,” Waymo started off its article today.

“Simulation is a critical component of Waymo’s AI ecosystem and one of the three key pillars of our approach to demonstrably safe AI. The Waymo World Model, which we detail below, is the component that is responsible for generating hyper-realistic simulated environments.

“The Waymo World Model is built upon Genie 3—Google DeepMind’s most advanced general-purpose world model that generates photorealistic and interactive 3D environments—and is adapted for the rigors of the driving domain. By leveraging Genie’s immense world knowledge, it can simulate exceedingly rare events—from a tornado to a casual encounter with an elephant—that are almost impossible to capture at scale in reality. The model’s architecture offers high controllability, allowing our engineers to modify simulations with simple language prompts, driving inputs, and scene layouts. Notably, the Waymo World Model generates high-fidelity, multi-sensor outputs that include both camera and lidar data.”

That is convincing. Of course, many will claim that you simply cannot simulate adequately, and it’s only real-world training that actually matters. However, much can surely be simulated, especially using Genie 3. In fact, there is something proactive and extremely tied to the real world in Waymo’s system compared to others.

“Most simulation models in the autonomous driving industry are trained from scratch based on only the on-road data they collect. That approach means the system only learns from limited experience. Genie 3’s strong world knowledge, gained from its pre-training on an extremely large and diverse set of videos, allows us to explore situations that were never directly observed by our fleet.

“Through our specialized post-training, we are transferring that vast world knowledge from 2D video into 3D lidar outputs unique to Waymo’s hardware suite. While cameras excel at depicting visual details, lidar sensors provide valuable complementary signals like precise depth. The Waymo World Model can generate virtually any scene—from regular, day-to-day driving to rare, long-tail scenarios—across multiple sensor modalities.”

That all does make sense now. Waymo adds that this allows it to explore how Waymo robotaxis will respond and how they should in extreme weather, in natural disasters, and more.

For numerous visuals — mostly GIFs — as well as further explanation and elaboration in text, check out the full Waymo post.

Perhaps I am too easily fooled, but I find that every time Waymo shares more information and insight into how it does things, the more impressed and bullish I become about the company and its prospects. I do think we will watch Waymo scale to huge heights, and I hope we will also see it able to achieve profits one day with the approach it’s taking.