VisSearch-Bench Tests Multimodal Visual Search

A research paper published on November 3 introduces VisSearch-Bench, a benchmark for testing whether multimodal agents can manage visual information during extended web searches. The tasks require agents to connect details across images, objects and search steps, rather than use an image only to identify an initial entity before relying on text retrieval. They may need to compare candidates, follow visual clues and combine appearance, local details or spatial relationships with external information.

VisSearch-Bench contains 204 human-written tasks across 7 domains and 25 subdomains. Its five categories are Set-wise Visual Search, Sequential Visual Search, Hybrid Wide-Deep Search, Competitive Visual Disambiguation and Visual-Structure-Guided Search. Each task includes a reference answer, traceable evidence and scoring rules. Accuracy measures whether an agent completes the whole task; Process Score measures the correctness and completeness of individual results and subquestion answers.

The authors evaluated 16 multimodal models, and the best-performing system fully solved only 26.47% of tasks, even when it had search tools. The paper identifies linking visual evidence with search results to produce complete answers as a challenge. The Hugging Face listing says the paper was submitted by Wu Zhengxian on November 6 and names Ant Group; the paper lists ZhengXian Wu and 12 other authors.

A research paper published on November 3 introduces VisSearch-Bench, a benchmark for testing whether multimodal agents can manage visual information during extended web searches. The tasks require agents to connect details across images, objects and search steps, rather than use an image only to identify an initial entity before relying on text retrieval. They may need to compare candidates, follow visual clues and combine appearance, local details or spatial relationships with external information.

VisSearch-Bench contains 204 human-written tasks across 7 domains and 25 subdomains. Its five categories are Set-wise Visual Search, Sequential Visual Search, Hybrid Wide-Deep Search, Competitive Visual Disambiguation and Visual-Structure-Guided Search. Each task includes a reference answer, traceable evidence and scoring rules. Accuracy measures whether an agent completes the whole task; Process Score measures the correctness and completeness of individual results and subquestion answers.

The authors evaluated 16 multimodal models, and the best-performing system fully solved only 26.47% of tasks, even when it had search tools. The paper identifies linking visual evidence with search results to produce complete answers as a challenge. The Hugging Face listing says the paper was submitted by Wu Zhengxian on November 6 and names Ant Group; the paper lists ZhengXian Wu and 12 other authors.