LA-Pose: What 10 million Driving Videos Taught Us About Camera Pose Estimation

Every serious approach to learning camera pose – knowing precisely where and how a vehicle moved through space – has relied on the same bottleneck: expensive, carefully labeled 3D ground truth. LiDAR rigs, reconstruction pipelines, simulation engines. These costly-to-produce datasets are small by necessity. The models trained on them inherit their limits.

We wanted a way out of that loop. Instead of asking the model to learn pose from labeled 3D data, we set the model free to predict the next frame using a small amount of information (a latent action). The magic result: this latent action encodes our cameras’ ego-pose, and improves as we scale unlabelled data.

Vehicles are, by definition, constantly moving through the world. Every sequence of frames already contains the signal we need: how the scene changed between one moment and the next. Those changes, whether the car is turning left, accelerating, or slowing to a stop, are encoded in the pixels themselves. We don’t need to label it.

LA-Pose uses an inverse dynamics model, trained on 10.2 million unlabelled driving clips, to learn what we call a latent action: a compact representation of how the world shifted between consecutive frames. The model is never told the vehicle’s speed or heading. It learns to infer them in a self-supervised way.

When we visualize these representations in T-SNE space, something incredible happens. The latent actions organize into distinct clusters that map cleanly onto real driving behaviors: straight, left, right and stopped. Structure emerges from scale, without a single pose label. That’s the pre-training stage. It teaches the model what motion looks like.

From motion to metric space

The second stage is where this becomes useful for autonomy. We freeze the pretrained inverse dynamics model and train a lightweight pose estimation head on top, using a small set of labeled driving data from Waymo, nuScenes, and Argoverse. The head learns to translate those latent motion representations into precise 3D estimates: translation, rotation, field-of-view, and metric scale in a single forward pass.

LA-Pose achieves over 10% higher pose accuracy than state-of-the-art feed-forward methods, using orders of magnitude less labeled data. On PandaSet, a dataset the model had never seen during training, it still outperforms all baselines. That zero-shot generalization is what matters most for deployment.

Why use a small latent action? One finding clearly showed this: a 50-dimensional latent bottleneck produces better downstream pose estimates than a 1,536-dimensional one, even though the larger space reconstructs video more accurately. The compression forces the model to discard appearance and retain motion. The constraint is what makes the representation useful.

As far as we know, this is the first time inverse-dynamics self-supervised learning has been used to drive pose estimation. We hope it encourages others to look at what else geometry perception can learn from scale.

Why this matters for how we build

Wayve vehicles have driven in hundreds of cities across vastly different conditions. That data exists. The question has been whether that experience can generalize to a new city or scenario without running another LiDAR rig or building another labeled dataset from scratch.

Most 3D perception systems can’t. They’re trained on curated datasets and calibrated hardware. That keeps them accurate in conditions they’ve seen and fragile in the ones they haven’t.

LA-Pose is a different bet: that unlabelled video at scale contains enough signal to build rich geometric priors, and that a small amount of targeted supervision can ground those priors in metric reality. That changes both the economics and the trajectory of improvement.

There’s still work to do. For example, LA-Pose currently degrades for reversing motion. We found that backing up is rare enough in the post-training data that the model hasn’t fully learned it. Scaling both the pre-training and post-training corpus is the natural next step, extending beyond driving to robot-collected and handheld video, allowing the inverse dynamics model to learn much richer motion and generalize better.

But the core idea holds. Motion is the signal. And it’s already everywhere.

The paper and demo are available at la-pose.github.io.