YANAMI Tests Multimodal Models Across Dialogue Turns

Researchers introduced YANAMI, a multimodal dialogue dataset and benchmark for testing whether models can track and act on changing information across conversation turns. The paper was published on November 3 and submitted to Hugging Face on November 4. Its authors say existing unified multimodal datasets mostly test single-turn tasks, while many multi-turn tasks can be answered without remembering earlier exchanges.

YANAMI-400K contains 400K multi-turn dialogues, more than 2M images and 2.6M dialogue turns. The dialogues cover four scenarios: combining information from multiple images, editing images, world modeling and visual reasoning. YANAMI-Bench contains 400 multi-turn dialogues and evaluates both dialogue-state understanding and visual realization (producing images that reflect the conversation). This comparison measures whether a model can understand changing dialogue instructions and carry them out in visual outputs.

In experiments, fine-tuning BAGEL on YANAMI-400K improved image editing by 10.3%, generation by 2.0% and the paper's third reported task by 13.23%, while preserving general visual-understanding capabilities. The abstract does not name that third task; a Hugging Face community post describes the 13.23-point gain as an improvement on the benchmark. Evaluations on YANAMI-Bench found that existing unified multimodal models still struggle to maintain and update dialogue states and reflect them accurately in images.

Researchers introduced YANAMI, a multimodal dialogue dataset and benchmark for testing whether models can track and act on changing information across conversation turns. The paper was published on November 3 and submitted to Hugging Face on November 4. Its authors say existing unified multimodal datasets mostly test single-turn tasks, while many multi-turn tasks can be answered without remembering earlier exchanges.

YANAMI-400K contains 400K multi-turn dialogues, more than 2M images and 2.6M dialogue turns. The dialogues cover four scenarios: combining information from multiple images, editing images, world modeling and visual reasoning. YANAMI-Bench contains 400 multi-turn dialogues and evaluates both dialogue-state understanding and visual realization (producing images that reflect the conversation). This comparison measures whether a model can understand changing dialogue instructions and carry them out in visual outputs.

In experiments, fine-tuning BAGEL on YANAMI-400K improved image editing by 10.3%, generation by 2.0% and the paper's third reported task by 13.23%, while preserving general visual-understanding capabilities. The abstract does not name that third task; a Hugging Face community post describes the 13.23-point gain as an improvement on the benchmark. Evaluations on YANAMI-Bench found that existing unified multimodal models still struggle to maintain and update dialogue states and reflect them accurately in images.