MMEnterprise-Bench Tests Enterprise AI Agents on Conflicting Records

Researchers Seohyun Lee, Jongha Kim, Dohwan Ko, Ji Soo Lee, Seoung Choi, Jayun Ku and Hyunwoo J. Kim introduced MMEnterprise-Bench, a benchmark for testing whether multimodal AI agents can answer questions over workplace records that change through team collaboration, according to an arXiv paper published on October 20 and submitted to Hugging Face on October 21. In enterprise settings, information is shared, corrected and revised across documents, chats and figures, so multiple versions of the same fact accumulate in the workspace. The authors argue that reliable question answering therefore requires agents to reason over sufficient evidence, resolve conflicts across sources and over time, and recognize when the available evidence is insufficient to establish an answer.

To measure these abilities, the team built a benchmark of 1,613 questions grounded in evolving multimodal records from 30 simulated collaboration projects, covering 7,140 textual and visual records. The questions fall into three categories: Evidence Reasoning, Conflict Resolution and Unanswerability. The paper states that the benchmark tests evidence reasoning, cross-source and temporal conflict resolution, and recognition of unanswerable questions.

The team evaluated 10 proprietary and open-source multimodal LLMs. According to the paper, even the strongest models struggle when evidence contains cross-source or temporal conflicts or spans multiple images. The authors also found that increasing the retrieval budget or using a larger retriever improves evidence recall but does not consistently raise answer accuracy. The paper concludes that reasoning over evolving workplace records remains a persistent challenge and positions MMEnterprise-Bench as a testbed for developing more reliable enterprise agents. Code and data are available at the project page linked in the paper.

Researchers Seohyun Lee, Jongha Kim, Dohwan Ko, Ji Soo Lee, Seoung Choi, Jayun Ku and Hyunwoo J. Kim introduced MMEnterprise-Bench, a benchmark for testing whether multimodal AI agents can answer questions over workplace records that change through team collaboration, according to an arXiv paper published on October 20 and submitted to Hugging Face on October 21. In enterprise settings, information is shared, corrected and revised across documents, chats and figures, so multiple versions of the same fact accumulate in the workspace. The authors argue that reliable question answering therefore requires agents to reason over sufficient evidence, resolve conflicts across sources and over time, and recognize when the available evidence is insufficient to establish an answer.

To measure these abilities, the team built a benchmark of 1,613 questions grounded in evolving multimodal records from 30 simulated collaboration projects, covering 7,140 textual and visual records. The questions fall into three categories: Evidence Reasoning, Conflict Resolution and Unanswerability. The paper states that the benchmark tests evidence reasoning, cross-source and temporal conflict resolution, and recognition of unanswerable questions.

The team evaluated 10 proprietary and open-source multimodal LLMs. According to the paper, even the strongest models struggle when evidence contains cross-source or temporal conflicts or spans multiple images. The authors also found that increasing the retrieval budget or using a larger retriever improves evidence recall but does not consistently raise answer accuracy. The paper concludes that reasoning over evolving workplace records remains a persistent challenge and positions MMEnterprise-Bench as a testbed for developing more reliable enterprise agents. Code and data are available at the project page linked in the paper.