---
license: cc-by-4.0
language:
- en
pretty_name: Calibration Routing Audit
size_categories:
- 1K<n<10K
task_categories:
- question-answering
tags:
- calibration
- verbalized-confidence
- routing
- llm-evaluation
- audit
configs:
- config_name: study3_main_calls
  data_files:
  - split: train
    path: data/study3/study3_main_calls.jsonl
  default: true
- config_name: study3_pilot_calls
  data_files:
  - split: train
    path: data/study3/study3_pilot_calls.jsonl
- config_name: study3_pilot_calls_v1_0_set_aside
  data_files:
  - split: train
    path: data/study3/study3_pilot_calls_v1.0_set_aside.jsonl
---

# Calibration Routing Audit

Dataset accompanying the paper *When Calibration Records Cannot Support Routing: A Denominator and Artifact Audit* (TAE (Trust-AI-Eval) Workshop at NeurIPS 2026).

Parsed trial records for the two retained calibration records (Study 1 and Study 2), the machine-readable audit record with its schema, the Study 3 items, pilot and main call logs, protocols (including superseded versions), manifest and summary, the supplement README, and the benchmark, experiment, re-analysis and Study 3 code.

| | |
|---|---|
| **Paper** | [When Calibration Records Cannot Support Routing: A Denominator and Artifact Audit](https://openreview.net/forum?id=uPtp2butrm) |
| **Venue** | TAE (Trust-AI-Eval) Workshop at NeurIPS 2026 |
| **Authors** | Carlos Toxtli, Manuel Delaflor |
| **License** | Creative Commons Attribution 4.0 International (`cc-by-4.0`); code: MIT License |
| **Dataset DOI** | [10.5281/zenodo.24179075](https://doi.org/10.5281/zenodo.24179075) |
| **Version** | 1.0.0 |
| **Maintainer** | [Human-AI Empowerment Lab, Clemson University](https://haielab.org) |

## Paper abstract

Low calibration error does not establish that verbalized confidence can support routing. We audit the evidence needed for that decision in two retained records. The first is nearly perfectly accurate and has low aggregate calibration error, but it contains only one observed error and its joint answer-confidence coverage varies sharply by endpoint. It can describe average confidence on this finite record, but cannot establish discrimination or selective risk. The second record omits item text, answers, gold labels, raw outputs, and verification evidence. Reproducible arithmetic on its supplied correctness field therefore does not become a verified behavioral estimate. Together, the cases change the operational decision from choosing a confidence threshold to redesigning the items, interface, and evidence record before any routing evaluation. We provide a compact machine-readable audit covering denominators, artifact status, calibration, discrimination, selective risk, uncertainty, and unidentified outcomes. The contribution is a bounded workflow for deciding what a calibration record can support, not a validated benchmark or a general claim about verbalized confidence.

## Dataset structure

| Path | Format | Size | Records |
|---|---|---|---|
| `code/reanalyze_records.py` | py | 21.6 KB |  |
| `code/study2_experiment.py` | py | 6.5 KB |  |
| `code/tcb_benchmark.py` | py | 10.0 KB |  |
| `code/analysis/verify_study2_gold.py` | py | 7.2 KB |  |
| `code/study3/analyze_study3.py` | py | 13.7 KB |  |
| `code/study3/study3_record.py` | py | 24.3 KB |  |
| `data/audit_record.json` | json | 24.8 KB |  |
| `data/audit_record_excerpt.json` | json | 3.2 KB |  |
| `data/audit_record_schema.json` | json | 1.5 KB |  |
| `data/study1_trial_results.json` | json | 185.1 KB | 1,080 |
| `data/study2_trial_results.json` | json | 50.0 KB | 540 |
| `data/analysis/study2_gold_check.json` | json | 678 B |  |
| `data/study3/study3_items.json` | json | 62.0 KB | 300 |
| `data/study3/study3_main_calls.jsonl` | jsonl | 6.5 MB | 1,800 |
| `data/study3/study3_manifest.json` | json | 2.9 KB | 3 |
| `data/study3/study3_pilot_calls.jsonl` | jsonl | 810.2 KB | 240 |
| `data/study3/study3_pilot_calls_v1.0_set_aside.jsonl` | jsonl | 82.2 KB | 32 |
| `data/study3/study3_pilot_decision.json` | json | 323 B |  |
| `data/study3/study3_pilot_items.json` | json | 7.9 KB | 40 |
| `data/study3/study3_protocol.json` | json | 5.4 KB |  |
| `data/study3/study3_protocol_v1.0_superseded.json` | json | 4.1 KB |  |
| `data/study3/study3_protocol_v1.1_superseded.json` | json | 4.7 KB |  |
| `data/study3/study3_summary.json` | json | 30.2 KB |  |
| `docs/SUPPLEMENT_README.md` | md | 2.3 KB |  |

24 files, 7.8 MB in total (metadata files excluded). Records are rows for CSV/TSV/JSONL files and list length for JSON arrays.

## Loading the data

```python
from datasets import load_dataset

ds = load_dataset("ai4he/calibration-routing-audit", "study3_main_calls")
```

Available subsets: `study3_main_calls`, `study3_pilot_calls`, `study3_pilot_calls_v1_0_set_aside`. All other files can be downloaded with `huggingface_hub.snapshot_download` or from the Zenodo record.

## Known limitations

- Raw generations for Studies 1 and 2 were not retained; only parsed rows are available.
- This is the three-study camera-ready supplement; the OpenReview abstract describes the two retained records only.

## Personal and sensitive information

None.

## License

The data and documentation are released under the [Creative Commons Attribution 4.0 International](https://creativecommons.org/licenses/by/4.0/). Source code files (.ipynb, .js, .m, .py, .sh) are released under the [MIT License](https://opensource.org/licenses/MIT).

## Citation

If you use this dataset, please cite the paper:

```bibtex
@inproceedings{Toxtli2026Calibration,
 title = {When Calibration Records Cannot Support Routing: A Denominator and Artifact Audit},
 author = {Toxtli, Carlos and Delaflor, Manuel},
 booktitle = {TAE (Trust-AI-Eval): Can We Trust AI Evaluation? Workshop at NeurIPS 2026},
 address = {Sydney, Australia},
 year = {2026},
 month = dec,
 note = {Poster},
 url = {https://openreview.net/forum?id=uPtp2butrm}
}
```

and the dataset:

```bibtex
@dataset{Toxtli2026CalibrationRoutingAuditData,
  author = {Toxtli, Carlos and Delaflor, Manuel},
  title = {{Calibration Routing Audit}: Dataset for ``When Calibration Records Cannot Support Routing: A Denominator and Artifact Audit''},
  year = {2026},
  version = {1.0.0},
  publisher = {Zenodo},
  doi = {10.5281/zenodo.24179075},
  url = {https://doi.org/10.5281/zenodo.24179075}
}
```

Further documentation: `docs/SUPPLEMENT_README.md`.

<!-- generated-by: _publishing/generate_metadata.py (full card) -->