DialFork

Description

DialFork is a benchmark for procedural hallucination in conversational agents: cases where an LLM-based agent stays fluent and factually plausible but stops following the procedure (workflow) it was given. It contains 36,278 English task-oriented dialogues across 24 tasks in 12 domains: 2,134 conformant dialogues, in which the assistant follows its workflow, and 34,144 "forks" of them, in which the assistant drifts away from it. Each fork is created by replaying a conformant dialogue with the same random seed and injecting one of four types of instruction (workflow identity corruption, step selection policy corruption, goal replacement, conflicting system constraints) at one of four fork positions. Each dialogue is labelled, with no manual annotation, with how far it deviates (the fork position) and the turn where the deviation starts. All dialogues are synthetic: they were generated by two LLM agents (Gemma 4 31B) role-playing the tasks, workflows and instructions of the STAR corpus (MIT license); no original STAR dialogue is included. Each dialogue is one JSON file (sdialog format) with the conversation, the personas and generation settings, the full generation trace and the annotations.

Reference

If you use this dataset, please cite the following publication:

 @misc{burdisso2026dialfork,
  title         = {DialFork: Going Beyond Factuality to Evaluate Procedural Hallucination in Conversational Agents},
  author        = {Burdisso, Sergio and Villatoro-Tello, Esa{\'u} and S{\'a}nchez-Cort{\'e}s, Dairazalia and Labrak, Yanis and Luque, Jordi and Cernak, Milos and Motlicek, Petr},
  year          = {2026},
  eprint        = {2610.21561},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  doi           = {10.48550/arXiv.2610.21561},
  url           = {https://arxiv.org/abs/2610.21561}
}

Burdisso, S., Villatoro-Tello, E., Sánchez-Cortés, D., Labrak, Y., Luque, J., Cernak, M., & Motlicek, P. (2026). DialFork: Going Beyond Factuality to Evaluate Procedural Hallucination in Conversational Agents. arXiv:2610.21561. https://doi.org/10.48550/arXiv.2610.21561