28 November 2022 | Engineering
Introducing Wayve Infinity Simulator
Simulation is a superpower critical to unlocking safe and adaptable autonomous driving. We are building Wayve Infinity Simulator as the most realistic and diverse simulator for AV2.0 end-to-end autonomous driving.
At the touch of a button, Infinity provides us with unbounded quantities of complex and challenging driving scenarios that allow us to train, understand, and validate our AI model’s driving intelligence. Access to these simulated scenarios gives us insights that would be vastly slower or even impossible to get from real-world testing. It provides us with noise-free, reproducible metrics to track and engineer against. It allows us to test “what-if” counterfactuals and quickly scale deployments to new vehicle platforms or geographies. Ultimately, it helps ensure that our AV technology is safe by putting our software through the world’s most challenging driving test before it ever hits the road, including scenarios that could not be safely tested in the real world.
Introducing Wayve Infinity Simulator
Why is simulation important for autonomous driving?
Wayve aims to be the first to deploy autonomous driving technology in 100 cities worldwide. This depends on a vast amount of real-world testing to demonstrate that our AVs can safely and reliably drive through challenging urban environments. While real-world testing is a critical part of our development process, it also comes with some significant limitations:
- It is impossible to recreate the same scenario twice in the real world. Even on the same stretch of road, many things change between one test and another. This increases the amount of noise in the metrics.
- It is expensive in both time and money. Operational realities, such as the size of our development fleet and the number of AV Safety Operators, restrict the quantity of real-world testing that can be done.
- Some “edge cases” are too rare to observe in the real world or too risky to conduct tests on public roads, so we can’t rely on real-world testing alone to stress-test our system.
Simulation offers a powerful complement to real-world testing, addressing the above drawbacks while providing additional benefits. In simulation, we can recreate the same scenario as often as we like. Simulation is software and can thus be scaled arbitrarily and efficiently in the cloud. We can generate challenging, adversarial, or risky scenarios in simulation that would be unsafe to test in the real world and ensure our system can drive safely throughout.
Beyond testing, simulation offers us other tools to accelerate our development. We can explore how our driving intelligence behaves as we vary aspects of the scene, ensuring that it drives correctly regardless of weather, season, or time of day. We can rewind time and test a scenario repeatedly while varying the behaviour of the other agents in the scene. We can also use simulation to augment our training data sets to reduce bias and to unlock reinforcement learning approaches which depend on trial-and-error driving that would be impossible to conduct directly on public streets.
Wayve Infinity Simulator
Wayve Infinity Simulator is an end-to-end simulator custom designed to meet the needs of Wayve’s end-to-end driving system. We emulate the entire driving system, including the roads, traffic lights, behaviours of vehicles, cyclists, and pedestrians, vehicle cameras and other sensors, world physics, vehicle dynamics, and more. As a result, an AI model can be tested rigorously in simulation before being driven in the real world minutes later.
Infinity is built across four main pillars to ensure a high-quality simulation representative of the real world that we need to train our AI and evaluate its capabilities: realism, diversity, controllability, and scale.
Realism is a critical component of any simulation. Visual realism of the sensor simulation is essential, but it is only a small part of the story. Realism also critically encompasses the dynamic response of the vehicle to actuation commands, the latency and timing of execution on the virtual car, and much more. We’ll return to this topic in more detail in a future blog post.
Realism alone is not particularly useful: imagine a perfectly realistic simulation of a single straight road devoid of other vehicles or pedestrians. This would not take us very far. So beyond realism, we must also have diversity that exposes our driving intelligence to as broad a range of driving scenarios as possible so it can learn to navigate the real world in all its complexity. This requires a system that can not only produce the full range of scenarios we see in the real world but also generate vast numbers of variations of each.
Trying to handcraft individual scenarios to cover this rich diversity would be impossible. Instead, Infinity’s procedural generation systems capture the characteristics of the real world as parameterised rules, allowing us to generate countless real world-like variations at the touch of a button. We can randomise every aspect of our simulated scenarios: the roads, building types, signals, weather, time of day, and the behaviour of other road users. Everything is parameterised to provide as much diversity as possible.
In the film above, you can see a subset of the huge number of features supported by Wayve Infinity. Our procedural world generation system ensures these features occur in many different configurations across countless worlds, either at random or to meet specific requirements. Our rule-based general driver can navigate all of these scenarios, providing near-limitless diverse training data and dynamic tests for our AI models.
At times we may want to create highly constrained scenarios to target specific areas of interest to our driving intelligence. These might be particularly challenging scenarios or safety-critical edge cases that would be impossible to engineer in the real world.
Infinity gives us complete control over the simulated environment. Every aspect of the simulation is configurable and can be set to a targeted value or randomised depending on our requirements. The simulator is fully scriptable, meaning a few lines of code can control any aspect of the simulation even while it is running. Further, we can easily access as many sources of ground truth labels, such as depth or semantic segmentation, as we need.
Wayve Infinity gives us complete control over the nature of each simulated scenario. In the video above, a world has been generated to have roundabouts at every junction simply by setting a few configuration parameters. We can test and train our driving model on events that are hard to engineer in the real world, such as wheelie bins spontaneously crossing the road. We can control any aspect of the sim, from agent behaviour to the position of the camera, and with our custom-built renderer, we can output any ground truth information we need, such as depth, object segmentation or unique index information.
The full power of this diversity can only be harnessed at scale. We need a platform that can cover the vast range of scenario combinations we can produce. With this in mind, we have worked with Microsoft Azure to realise a highly scalable cloud infrastructure that allows us to run countless simulations in parallel efficiently. This system will enable us to achieve the vast number of simulations required to push us towards AV2.0. Today our development AI models are tested in minutes across thousands of miles of driving in simulation before they are allowed out onto the road.
In the film above, you can see countless variants of the same initial scenario; all these can be run in parallel in Azure to provide huge numbers of training and test cases for our driving intelligence models.
We’re just getting started
As we continue to develop Infinity, we’re iteratively enhancing the system across all four pillars of realism, diversity, controllability, and scale to constantly improve the quality of the insights we get. That, in turn, helps accelerate the development of our AV2.0 driving intelligence. While there is plenty we are still working on, we already see enormous value from simulation propelling us forward at an accelerating pace. In particular simulated, off-the-road evaluation allows us to efficiently build the scaling curves we need to scale up our AI foundation models for autonomous driving.
As with many aspects of the AV space, simulation is a rapidly developing field. Despite all the benefits simulation provides us today, we have barely scratched the surface of the enormous benefits it can afford. With the fully controllable, scalable end-to-end simulator we have built, we have the perfect platform to iterate towards AV2.0 quickly, efficiently, and safely.