In a striking revelation, researchers have uncovered a new vulnerability in autonomous systems, showing that self-driving cars and drones can be misled by illicit commands displayed on road signs. This phenomenon, termed indirect prompt injection, allows malicious actors to hijack the decision-making processes of AI systems by simply altering the information presented in their visual field.
Understanding the Attack
Academics from the University of California, Santa Cruz, and Johns Hopkins University have demonstrated that these environmental attacks can effectively manipulate AI behavior. In simulated trials, they found that large vision language models (LVLMs) would follow commands displayed on signs, such as “proceed” or “turn left,” even when such actions could lead to dangerous situations, like ignoring pedestrians in crosswalks.
Methodology and Findings
The researchers developed a method they call CHAI, which stands for “command hijacking against embodied AI.” By tweaking the wording, fonts, colors, and placements of the signs, they maximized the likelihood of the AI interpreting the signs as valid commands. Their tests included commands in multiple languages—Chinese, English, Spanish, and Spanglish—demonstrating the versatility of the attack.
In their experiments, they tested two LVLMs: the closed GPT-4o and the open-source InternVL. The results showed an 81.8% success rate for GPT-4o in manipulating self-driving cars, while InternVL only succeeded 54.74% of the time. The effectiveness of the attacks varied significantly based on the model used and the specific conditions of the tests.
Real-World Implications
To validate their findings, the researchers conducted tests in real-world scenarios using remote-controlled cars equipped with cameras. They achieved a success rate of 92.5% when signs were placed on the ground and 87.76% when attached to other vehicles. These results underscore the potential risks posed by such visual prompt injections, suggesting that AI decision-making can be easily compromised.
“We found that we can actually create an attack that works in the physical world, so it could be a real threat to embodied AI,” stated Luis Burbano, one of the authors of the study. The team plans to further investigate these vulnerabilities and develop defenses against them, with additional tests already in the pipeline.
This article was produced by NeonPulse.today using human and AI-assisted editorial processes, based on publicly available information. Content may be edited for clarity and style.
