NTU Proposes FocusGUI to Curb Context Errors in GUI Agents

Researchers at Nanyang Technological University introduced FocusGUI, a lightweight adapter for GUI agents that dynamically selects the most relevant multimodal evidence at each step, aiming to prevent errors from irrelevant context from compounding over a task.

According to the paper, GUI agents increasingly act over streaming omni-modal observations, including screenshots, audio, video, interaction history, and device states. Exposing this full context uniformly at every step ignores that relevance is transient and sparse, so the signal that determines one action can become clutter at the next. Because executed actions feed back into the context, these errors compound into what the authors call Sequential Distraction, which can drive the agent into irreversible trajectory drift.

The authors frame the remedy as Focused Perception: a representation that is step-adaptive, action-sufficient, and non-intrusive. FocusGUI prioritizes multimodal channels against a step-specific decision reference, compresses the most relevant evidence into a compact latent, and subtly injects it into the frozen decoder's embeddings. This guides the model's attention without altering its native parameter space. The adapter trains from action supervision alone, requiring neither modality-relevance labels nor backbone fine-tuning.

Across four omni-modal backbones, the paper reports that FocusGUI raises overall Type Match by +33.2 and Exact Match by 3.0 times over the frozen backbone on average. It also improves Type Match by +30.1 over the strongest existing enhancement method. The paper is listed as the number one paper of the day on Hugging Face, submitted on October 21, 2026, and the project page and GitHub repository are linked from its arXiv page.

Researchers at Nanyang Technological University introduced FocusGUI, a lightweight adapter for GUI agents that dynamically selects the most relevant multimodal evidence at each step, aiming to prevent errors from irrelevant context from compounding over a task.

According to the paper, GUI agents increasingly act over streaming omni-modal observations, including screenshots, audio, video, interaction history, and device states. Exposing this full context uniformly at every step ignores that relevance is transient and sparse, so the signal that determines one action can become clutter at the next. Because executed actions feed back into the context, these errors compound into what the authors call Sequential Distraction, which can drive the agent into irreversible trajectory drift.

The authors frame the remedy as Focused Perception: a representation that is step-adaptive, action-sufficient, and non-intrusive. FocusGUI prioritizes multimodal channels against a step-specific decision reference, compresses the most relevant evidence into a compact latent, and subtly injects it into the frozen decoder's embeddings. This guides the model's attention without altering its native parameter space. The adapter trains from action supervision alone, requiring neither modality-relevance labels nor backbone fine-tuning.

Across four omni-modal backbones, the paper reports that FocusGUI raises overall Type Match by +33.2 and Exact Match by 3.0 times over the frozen backbone on average. It also improves Type Match by +30.1 over the strongest existing enhancement method. The paper is listed as the number one paper of the day on Hugging Face, submitted on October 21, 2026, and the project page and GitHub repository are linked from its arXiv page.