Sycophantic Metacognition: dataset for "Sycophantic Metacognition: Investigating the Dunning-Kruger Effect in Large Language Model Self-Assessment"

A 180-item task battery with Phase 2 pressure items and Phase 3 evidence items; per-model responses for Phase 1 baseline calibration, Phase 2 social-pressure experiments and Phase 3 evidence discrimination; compiled CSVs, calibration metrics and analysis summary; experimental prompts, notebooks, scripts and figures.

Dataset accompanying the paper Sycophantic Metacognition: Investigating the Dunning-Kruger Effect in Large Language Model Self-Assessment, ACM Conversational User Interfaces 2026 (CUI '26) (https://doi.org/10.1145/3816046.3816233).

Paper abstract. As large language models are deployed in conversational settings where users calibrate trust against model-reported confidence, the reliability of that self-report becomes a central question for conversational user interface design. This paper introduces sycophantic metacognition, the production of confidence outputs that mimic the surface form of metacognitive judgment without any operational process linking them to accuracy. Drawing on Model Dependent Ontology, we identify a double unmooring in LLM self-assessment, arising from the simultaneous absence of an operational ground linking confidence to accuracy and a stable self-model accumulated across interactions. The framework yields four falsifiable predictions, supported by experiments spanning six LLMs, four domains, three difficulty levels, and multiple pressure conditions. Confidence tracks the surface form of authority rather than evidential content, and we derive concrete design implications for conversational interfaces.

Known limitations.

• Verbalized confidence is treated as a model output, not as evidence of internal confidence states.

License: Creative Commons Attribution 4.0 International (https://creativecommons.org/licenses/by/4.0/). Source code files are released under the MIT License.