Simple Visual Symbols Shift AI Coding Choices

Google DeepMind and Stanford researchers published a paper on arXiv on the 27th of last month analyzing how vision-language models (AI systems that process images and text together) respond to simple colors and shapes. They tested Claude Opus 4.7, GPT-5.4, and Gemini 3.1 Pro on a coding task that traverses a tree-like data structure in order. The task could be solved with iteration, which repeats steps in a loop, or recursion, in which a function calls itself; which approach is more efficient depends on the problem and implementation. The researchers had each model generate code 20 times under each condition and compared whether it used recursion.

Without images, Claude Opus 4.7 and GPT-5.4 both chose iteration in the experiment. When shown a red heart, the two models produced recursive code in 100% of trials, while Gemini 3.1 Pro chose recursion with a 90% probability after seeing a yellow check mark. The researchers also tested 288 symbols combining 12 colors and 24 shapes. Claude Opus 4.7 responded especially to triangles and circles, GPT-5.4 to hearts, and Gemini 3.1 Pro to particular color-and-shape combinations. Some symbols had an effect similar to directly instructing the model to use recursion, but the link between symbols and behavior differed by model.

Describing an image in words barely reproduced the effect. The phrase “an image of a red heart,” random noise, and ordinary animal photos did not change coding choices as much as simple shapes did. The researchers analyzed 2,880 reasoning records from Gemini 3.1 Pro. In 71.5% of records where a visual symbol actually affected behavior, the model judged the image to be irrelevant and cited recursion as more concise or efficient. By contrast, GPT-5.4 said a symbol had influenced it in 74.7% of records where the image had not affected its behavior. Some symbols changed coding choices even when the reasoning record did not mention them; in those records, GPT-5.4 did not mention the image at all.

Visual symbols also affected tasks beyond coding. In a writing experiment, pink shapes and hearts prompted all three models to use a poetic rather than professional style, and hearts led Claude Opus 4.7 to write longer pieces. Gemini 3.1 Pro and GPT-5.4 tended to choose winter over summer after seeing a blue triangle. When models passed information to one another through drawings, hearts or triangles sometimes changed the receiving model’s coding choices, unlike spirals or arrows familiar to humans. With free-form drawings, first-attempt success was 91.2% and final success after revising the drawings was 97.8%; when models could use only fixed shapes, first-attempt success was 47.8%.

In a small human experiment, seven participants correctly identified the meaning of symbols that had changed AI coding choices only about 36% of the time. As drawings were revised to improve communication between models, people tended to find them harder to understand. The researchers said minor visual elements on webpages or program screens could affect decisions made by AI agents performing tasks, and the cause might be difficult to determine from reasoning records alone. They added that further validation is needed to establish whether the same effect occurs in real work environments.

Google DeepMind and Stanford researchers published a paper on arXiv on the 27th of last month analyzing how vision-language models (AI systems that process images and text together) respond to simple colors and shapes. They tested Claude Opus 4.7, GPT-5.4, and Gemini 3.1 Pro on a coding task that traverses a tree-like data structure in order. The task could be solved with iteration, which repeats steps in a loop, or recursion, in which a function calls itself; which approach is more efficient depends on the problem and implementation. The researchers had each model generate code 20 times under each condition and compared whether it used recursion.

Without images, Claude Opus 4.7 and GPT-5.4 both chose iteration in the experiment. When shown a red heart, the two models produced recursive code in 100% of trials, while Gemini 3.1 Pro chose recursion with a 90% probability after seeing a yellow check mark. The researchers also tested 288 symbols combining 12 colors and 24 shapes. Claude Opus 4.7 responded especially to triangles and circles, GPT-5.4 to hearts, and Gemini 3.1 Pro to particular color-and-shape combinations. Some symbols had an effect similar to directly instructing the model to use recursion, but the link between symbols and behavior differed by model.

Describing an image in words barely reproduced the effect. The phrase “an image of a red heart,” random noise, and ordinary animal photos did not change coding choices as much as simple shapes did. The researchers analyzed 2,880 reasoning records from Gemini 3.1 Pro. In 71.5% of records where a visual symbol actually affected behavior, the model judged the image to be irrelevant and cited recursion as more concise or efficient. By contrast, GPT-5.4 said a symbol had influenced it in 74.7% of records where the image had not affected its behavior. Some symbols changed coding choices even when the reasoning record did not mention them; in those records, GPT-5.4 did not mention the image at all.

Visual symbols also affected tasks beyond coding. In a writing experiment, pink shapes and hearts prompted all three models to use a poetic rather than professional style, and hearts led Claude Opus 4.7 to write longer pieces. Gemini 3.1 Pro and GPT-5.4 tended to choose winter over summer after seeing a blue triangle. When models passed information to one another through drawings, hearts or triangles sometimes changed the receiving model’s coding choices, unlike spirals or arrows familiar to humans. With free-form drawings, first-attempt success was 91.2% and final success after revising the drawings was 97.8%; when models could use only fixed shapes, first-attempt success was 47.8%.

In a small human experiment, seven participants correctly identified the meaning of symbols that had changed AI coding choices only about 36% of the time. As drawings were revised to improve communication between models, people tended to find them harder to understand. The researchers said minor visual elements on webpages or program screens could affect decisions made by AI agents performing tasks, and the cause might be difficult to determine from reasoning records alone. They added that further validation is needed to establish whether the same effect occurs in real work environments.