Unveiling LINGO-1's Show and Tell Capability with Referential Segmentation

But how do we know that LINGO-1 is paying attention to the right parts of the driving scene?

In the example above, how do we know which “pedestrians crossing the road” the AI model is focusing on? Another related question could be about understanding the road rules that are relevant to the scene. For instance, we might ask, “What does the box with the cycle image indicate?” It’s important for us to ascertain that LINGO-1 is paying attention to such elements alongside its textual response. 

Therefore, to go beyond text description of objects in the scene, we’ve developed a new “show and tell” capability that allows LINGO-1 to visually indicate its focus of attention using referential segmentation, which means marking the region in the image that is relevant to the textual response of LINGO-1. This new development establishes a stronger “grounding” (or connection) between language and vision tasks, increasing confidence in LINGO-1’s responses. 

In the video below, when we ask about the colour of the traffic light, LINGO-1 can accurately describe that “the traffic light is currently red” and simultaneously highlight the traffic light in the image frame, denoted by [A]. Furthermore, LINGO-1 can elaborate on the correct driving response to a red traffic light by stating, “We are waiting for it to turn green before proceeding.” As you can see, this new feature strengthens the link between language and vision tasks, improving our confidence in and ability to assess the accuracy and relevancy of LINGO-1’s responses.

Referential segmentation paves the way for multimodal explainability through cross-verification. By dynamically highlighting objects of interest, LINGO-1 demonstrates a holistic understanding of the scene. In the following example, LINGO-1 accurately identifies and associates elements within the scene to specific road rules outlined in the Highway Code. When we ask, “What do those squiggly lines indicate?” LINGO-1 correctly replies, “The zig-zag lines [A] you see are typically found near pedestrian crossings. They indicate that drivers should not park or overtake in this area.”

Here, we can ask LINGO-1 what it knows about safe stopping distances based on the weather. LINGO-1 accurately states, “The weather is a bit overcast, with some rain. It’s not too heavy, but it’s enough to make the road a bit slippery.” Given the weather, when we ask, “How many seconds of distance is recommended if the road is slippery?” LINGO-1 replies, “The recommended distance is at least 4 seconds. This allows for enough reaction time in case the motorcyclist [A] makes any sudden moves.” It also knows that “During normal conditions, I would maintain a 2-second drivable distance area [A] behind the motorcyclist [B]. This is the standard safe distance recommended by the UK Highway Code.”

In this example, we ask if driving over a bus lane is advisable to get around the vehicle turning right. LINGO-1 responds, “Yes, that’s a good observation. However, it’s important to note that the bus lane [A] is marked by a solid white line. According to the UK Highway Code, we’re not allowed to enter a lane marked by a solid white line unless it’s necessary and safe to do so. So, we will continue to follow our current lane.”

Here, we can see that LINGO-1 is “focusing on the driveable area [A] to ensure we stay within the lane. I’m also monitoring the traffic [B] in front of our lead car [C] to anticipate my next action.” It can also correctly identify that the broken white lines are lane dividers, which “indicate that drivers can change lanes when it’s safe to do so.”

This final example demonstrates how LINGO-1 reasons about potential hazards while driving at night. You can see the model first verifies that the scenario involves driving in the dark and acknowledges the need to be cautious of the following potential hazards: [A] a narrow drivable area, [B] a cyclist on the road, and [C] the presence of a car in front, which is challenging to spot in this image. It also correctly recognises that “the cyclist could make sudden moves or turns, so it’s important to keep a safe distance and be prepared to react accordingly.”

Driving Forward with this New Capability

The inclusion of “show and tell” opens up new possibilities for autonomous driving and human-vehicle interaction. Here are a few potential use cases: 

1. Enhanced training and adaptability: By linking language and vision, we can speed up the learning process by providing more efficient supervision to these models and more information about the scene. This learning ability is critical to address long-tail scenarios in driving where there are more limited examples.
2. Addressing hallucinations with LLMs: Hallucinations are a well-known problem in large language models; LINGO-1 is no exception. By developing the capability to ground language with vision, we introduce additional sources of supervision that would compel our model to respond only to objects that it can accurately identify in the scene, which can help mitigate the risk of hallucinations.
3. Improved safety communication: By visually highlighting potential hazards in the scene, LINGO-1 can effectively communicate critical information such as the position of pedestrians, cyclists or obstacles. This can play a vital role in building trust and confidence in the capabilities of our AV technology.