GAIA-4: Multimodal World Models Powering Closed-Loop Simulation for Safe and Scalable Autonomy

Safety & simulation in the world of AV2.0

Autonomous driving is shifting from modular stacks, i.e. separate perception and planning components, to end-to-end AI: driving models that map raw sensor inputs to driving decisions in a single network. This is AV2.0, and it’s what enables autonomy to generalize across cities, sensors, and vehicles in a way modular systems never could.

But the very thing that makes an end-to-end model powerful – that it is one learned system, not separable parts – breaks many of the tools the industry has used historically to prove safety, including simulation. The cost, time, and credibility of the safety case are critical, and real-world testing alone is not sufficient: as a model’s performance improves, the miles needed to validate safety sharply increase, and almost all are uneventful. Simulation bridges the gap, but the simulation stack needed to prove end-to-end AI is safe needs to be rebuilt.

Why driving models need a new kind of simulation

Simulation 1.0 was designed for modular systems. When perception and planning are separate components, you can validate each on its own terms, namely detection accuracy and trajectory error, each with its own metrics and its own tests. You can even mix open-loop perception replay with closed-loop planning. Separability is what makes that tractable.

In contrast, end-to-end driving models are not separable. There is no intermediate perception output to score and no planning module to test in isolation; behavior emerges from a single network. The only meaningful unit of evaluation is the whole system, judged by how it behaves. 

That is a fundamentally different technical approach, and it requires a different set of simulation tools. We call this Simulation 2.0: simulation purpose-built to evaluate a driving model as one behavioral system.

Two requirements follow. First, the simulator has to faithfully reproduce the full sensor experience the model actually drives on, across all sensor modalities, because a driving model consumes raw sensor data. Second, and more fundamentally: replay and open-loop evaluation are not enough. Showing a model a fixed recording tells you what it would predict, but its predictions never change what happens next, so the fixed recording has already decided the future. In order to know what the vehicle would actually do, you have to close the loop and let the model’s decisions change what it sees next.

Introducing GAIA-4

GAIA-4 is the world model at the core of Simulation 2.0, and the latest step in our GAIA line of world models. Critically, end-to-end learning applies to safety and simulation too, not just driving models. Just like driving, embodied AI safety improves faster with learned representations than with anything hand-engineered.

Over the past 8 years, Wayve has pioneered the development of world models  to change how autonomous vehicles are trained and evaluated across the industry, starting with the first world model for autonomous driving in 2018, extending to enable simulated validation through GAIA-1, GAIA-2, and GAIA-3.

Now with GAIA-4, we can close the loop, turning a recorded scene into the starting point for a new simulated run with the AI Driver back in the loop: as the model makes different driving decisions, the sensor inputs it receives change accordingly. If the AI Driver brakes, the simulated viewpoint slows; if it steers, the scene it sees next shifts with it. Rather than playing a fixed recording, the simulator generates the futures those decisions create, including futures that never occurred in the real world, while keeping the underlying scene anchored to what actually happened, providing a deterministic baseline to measure against.

Closed-loop simulation grounded in the real world

GAIA does not rely on HD maps, hand-built scene graphs, prompts or a separate annotation stack to construct the world. It generates the scene directly from recorded sensor data. Because GAIA is grounded in real recordings, we can keep the world faithful to what actually happened while letting only the ego vehicle change. We call this world-on-rails: the AI Driver’s actions drive the simulation forward — its position, its viewpoint, and the sensor stream it receives — but every other agent in the scene keeps the exact behavior it showed in the real-world log. The other road users do not react to the ego vehicle. A car that was going to enter the intersection still enters it; a cyclist crossing the street still crosses.

That constraint gives the simulation two properties essential for validation. First, other road users stay grounded in real behavior because they continue to do what they actually did in the recorded drive. Second, the evaluation stays conservative: no vehicle, pedestrian, or cyclist changes its behavior in response to the AI Driver, preventing the simulator from altering a safety-critical outcome. Under the world-on-rails constraint, we can assess the safety outcome for any scenario, including those where the vehicle safety operator intervened.

World-on-rails is a capability we can switch on, not a limitation of the world model. Relaxing this constraint gives us reactive agents, where other road users no longer simply replay the recording; they react to what the ego vehicle does, opening up scenarios that only real interaction can test, such as negotiated merges and give-and-take at junctions. The rest of the scene remains unchanged from the original recording. Reactive agents extends the world-on-rails capability. For safety validation, world-on-rails can provide a conservative, faithful recreation of the original scene. Reactive agents then extend the range of interactions we can evaluate by allowing the world to respond to the AI Driver.

Because the simulated video is grounded in real recordings and trained to hold the rest of the world-on-rails, GAIA can reconstruct a recorded scene with higher fidelity and consistency than a general-purpose world model. Training GAIA for this task improves how faithfully it preserves the recorded world by 2.5x, turning the generator into a stable system that can be used as a closed-loop simulator.

Counterfactuals: Understanding what the AI Driver would have done

Closed-loop simulation grounded in real driving is one of the ways we can introspect and debug the AI Driver. We can take any real moment, change one thing, and watch what the AI Driver would have done. This enables us to better understand the model’s behavior. 

We can debug and introspect the AI Driver with counterfactuals. Closing the loop lets us ask the questions that matter most for safety. What would the AI Driver have done if the vehicle safety operator had not intervened? How does a new model behave in exactly the situation that tripped up the last one? This is how we turn an on-road event into a repeatable diagnosis of the model’s behavior.

In the scenarios below, a vehicle safety operator intervened on the road to ensure safety. With GAIA-4’s closed loop capability, we can take that exact moment and ask what the AI Driver would have done if the world had continued to unfold as observed.

GAIA allows testing at scale and creates a flywheel. It runs many candidate driving models through a library of scenarios in closed loop, without re-driving. Because each predicted action feeds back into the generation, every run is a real test of what the model would do. Validation stops scaling with road miles and starts scaling with compute: efficient enough to run on every candidate model and fast enough to move deployment forward.

A production driving system relies on several sensing modalities, such as cameras, radar, and lidar, and simulation becomes far more valuable when those signals are generated coherently, as part of the same world, rather than assembled separately. To validate an AI Driver, simulation must produce every modality it relies on, coherently, as part of the same world—another key element of GAIA-4’s closed loop features.

Multimodality: bringing radar into the loop

Useful simulation for autonomy needs to go beyond camera realism. GAIA-4 extends this capability with multimodal generation, bringing radar into the simulation alongside cameras, an industry-first for autonomous driving simulators. Radar is particularly important for redundancy and safety-critical sensing, providing robustness in adverse conditions such as fog, spray, and low light, where visibility is poor. 

In the highway scene below, radar tracks the lead vehicle and an oncoming car, with color-coded Doppler measurements showing each object’s velocity relative to the ego vehicle. These detections are generated by the same world model that produces the video, rather than being added separately afterward.

A multimodal world model brings simulation closer to the signals that drive real decisions, and it strengthens the case that the simulator produces usable model inputs. Here we observe a GAIA simulation with and without radar across a situation in which the vehicle failed to slow down. This enables us to ask: how would the AI Driver handle this scenario with radar? In this case, the addition of radar measurably shifts the closed-loop trajectory, providing evidence that the generated radar carries decision-relevant information and ensures the vehicle decelerates appropriately.

Measuring the behavior fidelity of a simulator

For autonomy validation, the simulator should be judged on behavior and outcomes alongside visual fidelity. We measure the simulator at three levels: outcome fidelity, closed-loop fidelity, and component fidelity.

Outcome fidelity: Do the simulated runs reproduce the outcomes we observed in the real world? This is what ultimately matters: whether the simulation and the real-world results agree on what happened, and above all, on which scenarios are safety-critical. These outcomes are graded using the same metrics we use to evaluate real-world driving.

Closed-loop fidelity: Are the system outputs in simulation consistent with on-road driving? This confirms how the AI Driver arrived at the outcome and makes any divergence interpretable. To confirm end-to-end closed-loop fidelity, we test the consistency of the trajectories’ outputs. One way to visualize this is by comparing the trajectory the AI Driver plans in simulation alongside the trajectory it planned on the road, frame by frame. With the same model, the two should agree, as shown below as a tight “ribbon” of convergence.

Component fidelity: Are the simulator components valid? This focuses on testing each aspect of the simulator individually, focusing on different aspects of the system and the model like 3D realism and radar fidelity. The accumulation of these metrics gives us confidence in the simulator’s overall fidelity. 

3D realism is another important metric for visual fidelity testing each part of the generation on its own, like geometric consistency, assessing how faithfully we capture the fidelity of vulnerable road users (VRUs), traffic signs, speed limits, lane features, and other aspects of the scene. 

Because the original recording captured what was actually happening, we can geometrically project every element in the scene from the original viewpoint into the new viewpoint, and check whether GAIA’s generation aligns with it. The video below shows a scenario where we perturb the pose of the ego vehicle to highlight the other vehicles and lane markings remain stable as the ego vehicle navigates through the scene.

We also measure how far the ego vehicle can diverge from the original path while ensuring  cues such as traffic signs and roadside objects remain legible and correctly placed, providing independent evidence that the simulated world remains trustworthy. Traffic signs are particularly important for driving and are typically hard to recreate in simulation.

An important aspect of simulator validation is testing that, when we deviate from the original viewpoint or trajectory, GAIA’s generation maintains high fidelity and important aspects of the scene remain as expected, such as the positions of vehicles and pedestrians, the way radar points track objects, and the legibility of traffic signs.

One of the metrics for radar fidelity measures whether radar points continue to track important elements of the scene, like the front vehicle across different conditions. When we change the ego vehicle to accelerate and decelerate, GAIA generates in-distribution radar in response, with realistic changes in Doppler and coverage as we approach vehicles ahead.

Taken together, the combination of outcome fidelity, closed-loop fidelity and the various aspects of component fidelity, assessed across many scenarios, gives us confidence in the simulator’s overall accuracy.

The foundation for AI Driver validation

GAIA-4 paves the path to a new way to validate autonomy. The shift to end-to-end driving models is an industry-wide one: any team credibly building end-to-end autonomy inherits a system that can no longer be taken apart and tested in pieces, and the simulation tools built for modular stacks cannot validate it. 

GAIA-4, particularly its closed loop capability, turns an end-to-end AI Driver from a supposed “black box” into a system that can be examined, validated and trusted. Every intervention can be replayed and interrogated, every model change measured against the same real scenarios, and every sensor the vehicle depends on can be put under test. This introspection, repeatability, and evidence are the foundation of a rigorous safety case. Mapless, end-to-end AI is the most realistic path to autonomy at geographic scale, and closed-loop simulation grounded in the real world is what makes it possible to scale safely across millions of vehicles around the world with our automaker partners. 

GAIA is one building block of Simulation 2.0, our broader framework for evaluating Wayve AI Driver for commercial deployment. There is much more to come — stay tuned.

Acknowledgements

GAIA-4 is enabled by the key research, engineering and evaluation contributions from Sofía Dudas, Sofi Toteva, Seva Konjahhin, Qianli Xing, Zhenya Kashin, Pragya Kale, Holly Garvey, Xabier Romero Sáenz, Arvind de menezes pereira, Nathan Tseng, Yibo Chen, Jayson Salkey, Vassia Simaiaki