---
license: cc-by-4.0
language:
- en
pretty_name: Conjecture Context Ablation
size_categories:
- 1K<n<10K
task_categories:
- text-generation
tags:
- mathematics
- conjecture-generation
- ablation-study
- arxiv
- llm-evaluation
---

# Supplementary Material: Which Mathematical Context Supports Target-Aligned Conjecture Generation?

This repository contains supplementary material for the MathNLP 2026 paper **“Which Mathematical Context Supports Target-Aligned Conjecture Generation? A Paired Ablation Study.”**

The study examines which parts of theorem-local mathematical context help a language model generate a conjecture aligned with a hidden source theorem. It uses **61 main theorem targets from 12 papers** spanning four arXiv mathematics domains (`math.AC`, `math.AG`, `math.CO`, and `math.ST`). Each theorem is evaluated under **13 context conditions**: one full-context condition, six leave-one-out ablations, and six single-component conditions, for a total of **793 generated conjectures**.

## Repository contents

```text
.
├── README.md
├── requirements.txt
├── data/
│   ├── ablation_data_analysis.xlsx
│   └── theorem_runs/
│       ├── math.AC/
│       ├── math.AG/
│       ├── math.CO/
│       └── math.ST/
├── evaluation/
│   ├── cross_model_agreement/
│   └── expert_validation/
├── prompts/
│   ├── classification_task.md
│   ├── redaction_task.md
│   ├── generation_task.md
│   └── codex_prompts.txt
├── code/
│   ├── eval_deepeval_alignment.py
│   └── eval_deepeval_quality.py
└── sources/
    └── README.md
```

### `data/`

- `ablation_data_analysis.xlsx` contains the aggregate ablation data and analysis workbook.
- `theorem_runs/` contains theorem-specific artifacts organized by mathematical domain and source-paper arXiv identifier. These materials include theorem maps, redacted seeds, generated conjectures, evaluation outputs, and run summaries.
- The originally nested `theorem_objects.zip`, `redacted_seeds.zip`, and `generated_conjectures.zip` archives have been expanded into ordinary directories so that their Markdown contents can be browsed and version-controlled directly on GitHub.

The condition-directory names inside each paper folder are preserved from the original experiment. `All Inputs` is the full-context condition; the `R...` directories contain the corresponding ablation-condition outputs.

### `evaluation/`

- `cross_model_agreement/` contains repeated-judge and cross-model materials used to assess evaluator reliability and agreement, including Gemma-4-31B, GPT-5-mini, and exploratory GLM-5.1-FP8 outputs.
- `expert_validation/` contains the independent expert-validation dataset and associated automated evaluation outputs.

### `prompts/`

This directory contains the task instructions used for:

1. main-theorem classification and theorem-specific object-map construction;
2. theorem-specific redaction; and
3. blind conjecture generation.

`codex_prompts.txt` records the orchestration instructions associated with running these stages and the evaluation workflow.

### `code/`

The evaluation scripts implement the two theorem-conditioned metrics used in the paper:

- **Expected-Theorem Alignment**
- **Theorem-Conditioned Conjecture Quality**

The scripts use DeepEval and the model endpoint used in the study. **No API credentials are included in this repository.** They read the credential from the `RCD_LLM_API_KEY` environment variable and are currently configured for the Clemson RCD endpoint and `gemma-4-31b`. Users reproducing the evaluation on different infrastructure should update the endpoint/model configuration as needed.

Install the reported DeepEval version with:

```bash
pip install -r requirements.txt
```

## Mathematical-context components

Each redacted theorem seed is represented using six context components:

1. **Ambient Setting**
2. **Core Definitions and Notation**
3. **Local Mathematical Objects**
4. **Structural Constraints and Conditions**
5. **Auxiliary Objects and Constructions**
6. **Potential Generalization Axes**

For every theorem target, the study compares the full six-component context with six leave-one-out conditions and six single-component conditions.

## Models and evaluation

The experimental pipeline used DeepSeek-V4-Pro for theorem identification, object-map construction, redaction, and conjecture generation. Gemma-4-31B served as the primary DeepEval judge. Additional materials in this repository support repeated-Gemma evaluation, cross-model comparison with GPT-5-mini, an exploratory GLM-5.1-FP8 judge analysis, and expert validation of theorem-conditioned conjecture quality.

The reported evaluation scores are **target-conditioned estimates**. They should not be interpreted as independent proofs of mathematical correctness, truth, or novelty.

## Source papers

The experiments use publicly available mathematics papers identified by arXiv ID. Copies of the third-party source PDFs and LaTeX files have been removed from this public package. See [`sources/README.md`](sources/README.md) for the exact arXiv versions and citations used in the study.

## Reproducibility notes

This repository is intended to make the experimental artifacts, prompts, generated outputs, evaluation data, and analysis materials inspectable. Some components of the original computational environment require access to the same or equivalent model-serving infrastructure.

For the evaluation scripts, `requirements.txt` pins the DeepEval version reported in the paper (`4.1.5`).

## Citation

If you use these materials, please cite the associated paper:

> **Which Mathematical Context Supports Target-Aligned Conjecture Generation? A Paired Ablation Study.**

Full bibliographic information can be added here after publication.

<!-- BEGIN GENERATED METADATA (_publishing/generate_metadata.py) -->

## Dataset card

| | |
|---|---|
| **Paper** | [Which Mathematical Context Supports Target-Aligned Conjecture Generation? A Paired Ablation Study](https://www.carlostoxtli.com/papers/mathnlp2026-which-mathematical-context.pdf) |
| **Venue** | 4th Workshop on Mathematical Natural Language Processing (MathNLP 2026), EMNLP 2026 |
| **Authors** | Austin LaHue, Michael Burr, Luis David Garcia Puente, Vinita Gangaram Jansari, Benjamin E. Nye, Carlos Toxtli |
| **License** | Creative Commons Attribution 4.0 International (`cc-by-4.0`); code: MIT License |
| **Dataset DOI** | [10.5281/zenodo.24179081](https://doi.org/10.5281/zenodo.24179081) |
| **Version** | 1.0.0 |
| **Code repository** | https://github.com/alahue/conjecture-ablation |
| **Maintainer** | [Human-AI Empowerment Lab, Clemson University](https://haielab.org) |

## Dataset structure

| Path | Format | Size | Records |
|---|---|---|---|
| `requirements.txt` | txt | 16 B |  |
| `code/` (2 files) | py | 22.9 KB |  |
| `data/` (1 files) | xlsx | 52.9 KB |  |
| `data/theorem_runs/` (1694 files) | json, md, xlsx | 9.8 MB | 1,553 |
| `evaluation/cross_model_agreement/` (42 files) | json, xlsx | 1.9 MB | 943 |
| `evaluation/expert_validation/` (4 files) | json, xlsx | 339.4 KB | 45 |
| `prompts/` (4 files) | md, txt | 26.0 KB |  |
| `sources/` (1 files) | md | 2.6 KB |  |

1,749 files, 12.2 MB in total (metadata files excluded). Records are rows for CSV/TSV/JSONL files and list length for JSON arrays.

## Known limitations

- Theorem maps and redacted seeds quote short excerpts of the 12 source arXiv papers; the source PDFs/LaTeX are not redistributed (see sources/README.md for citations).

## Personal and sensitive information

None.

## License

The data and documentation are released under the [Creative Commons Attribution 4.0 International](https://creativecommons.org/licenses/by/4.0/). Source code files (.ipynb, .js, .m, .py, .sh) are released under the [MIT License](https://opensource.org/licenses/MIT).

## Citation

If you use this dataset, please cite the paper:

```bibtex
@inproceedings{LaHue2026Which,
 title = {Which Mathematical Context Supports Target-Aligned Conjecture Generation? A Paired Ablation Study},
 author = {LaHue, Austin and Burr, Michael and Garcia Puente, Luis David and Jansari, Vinita Gangaram and Nye, Benjamin E. and Toxtli-Hernandez, Carlos},
 booktitle = {Proceedings of the 4th Workshop on Mathematical Natural Language Processing (MathNLP 2026)},
 address = {Budapest, Hungary},
 publisher = {Association for Computational Linguistics},
 year = {2026},
 month = October,
 note = {Forthcoming},
 url = {https://sites.google.com/view/mathnlp2026}
}
```

and the dataset:

```bibtex
@dataset{LaHue2026ConjectureContextAblatioData,
  author = {LaHue, Austin and Burr, Michael and Garcia Puente, Luis David and Gangaram Jansari, Vinita and Nye, Benjamin E. and Toxtli, Carlos},
  title = {{Conjecture Context Ablation}: Dataset for ``Which Mathematical Context Supports Target-Aligned Conjecture Generation? A Paired Ablation Study''},
  year = {2026},
  version = {1.0.0},
  publisher = {Zenodo},
  doi = {10.5281/zenodo.24179081},
  url = {https://doi.org/10.5281/zenodo.24179081}
}
```

<!-- END GENERATED METADATA -->