Reproducing the paper’s numerical results

This package accompanies Fine Stratification of Survey Experiments (2026) by Max Cytrynbaum. The package enables a replicator to rerun the paper’s main empirical simulations and computational comparisons, then compare the resulting tables, figures, and numerical claims with those in the paper.

The main run starts from the original study data supplied in data/source_inputs/. It cleans the data, repeats model selection, fits the nine empirical models, and constructs the simulation DGPs before running the simulations and computational comparisons. It verifies the source files and checks that each processing stage uses the prescribed data, seeds, and model specifications. Your replicated results go to outputs/local-reproduction/. Copies of the paper’s tables, figures, and reported values are included in reference_outputs/paper/ for automated checks and side-by-side display. The paper itself is the authoritative source for comparison.

What this package reproduces

Open outputs/local-reproduction/RESULTS.html to view the replicated empirical tables alongside the values used in the paper, together with the reproduced figures and numerical summaries. The page shows which parts of the replication finished and identifies any computations that remain unfinished.

The ZIP includes paper-comparison targets, not a precomputed replication run. The commands below generate your own results and RESULTS.html; the outputs/ directory is not included in the ZIP.

Instructions for the replicator

1. Extract the ZIP into a directory named replication and open a terminal there. Keep the supplied data/ and reference_outputs/ directories unchanged.
2. Install the software using the commands below.
3. Choose Option 1 for one workstation or Option 2 for a Slurm cluster. They are alternative ways to run the same calculations, not two required runs.
4. Open outputs/local-reproduction/RESULTS.html and compare its tables, figures, and text-claim rows with the paper.

Install the software

Run these commands from the extracted replication/ directory on x86-64 Linux or an Apple Silicon Mac. Intel Macs are not supported. Use Python 3.10 or 3.11. The package automatically invokes a C compiler to build its numerical helpers, without any manual compilation. Python 3.12 is not supported by this replication package. The Appendix A1 comparisons also require R and a C++/Fortran build toolchain. R 4.4.1 was used for the measured runs described below.

python3 -m venv .venv
unset PYTHONPATH EBPYTHONPREFIXES
export PYTHONNOUSERSITE=1
.venv/bin/python -m pip install -r requirements-lock.txt
.venv/bin/python -m pip install -e . --no-deps
.venv/bin/python -m pip install -e auxiliary/computational_comparison --no-deps
.venv/bin/python -m pip check
bash auxiliary/computational_comparison/slurm/install_pinned_r_packages.sh "$PWD/.r-library"
export R_LIBS_USER="$PWD/.r-library"

The R installer verifies the six required direct package versions. It downloads their dependencies from CRAN, so installation normally needs internet access. If your shell session changes, set R_LIBS_USER again before the local run. Keep the .venv and .r-library directories for any resumed run.

On Linux, the first model-building stage automatically downloads additional software for fitting the models, including a separate copy of Python and its numerical libraries. It installs this software under outputs/local-reproduction/inputs/numerics/, without changing your system software. On Macs, the first empirical simulation automatically downloads and compiles part of the random-forest software to keep calculations consistent across platforms. These steps require internet access but no manual compilation. Keep the inputs/numerics/ directory when resuming a run.

The direct Python dependencies and versions are in requirements-lock.txt. The R versions are nbpMatching 1.5.6, blockTools 0.6.6, Hmisc 5.2-6, MASS 7.3-61, dplyr 1.2.1, and tibble 3.3.1. Their source downloads are authenticated by the installer. Dependencies of these packages are resolved during installation, not fully version-locked. Stata, commercial optimization software, and a LaTeX installation are not required.

The run saves your software versions, R package list, operating system, processor description when available, and starting free disk space under outputs/local-reproduction/environment/. Retain these records with the results.

After installation, choose one of the two options below. Both run the same analyses.

Option 1: Run on one workstation

The complete run includes several large timing comparisons in Appendix A1. Its largest task has an estimated memory requirement of 98.25 GiB. The restricted run below reproduces all empirical results, the Sections 2–3 illustrations, and all match quality comparisons, but omits 27 large timing comparisons.

The complete Slurm run used 64 CPU cores and a 120 GiB task-memory budget and finished within a 43-hour-12-minute allocation. Rebuilding the models and finishing the empirical simulations took about 9 hours 43 minutes; the Appendix A1 tasks spanned about 33 hours 26 minutes. The 32-core, 40-GiB feasible command took about 25 hours 17 minutes, with the 27 large timing comparisons described below deferred. These measurements include automatic first-use numerical setup but exclude initial software installation and queue time. Hardware and measurement scopes are detailed under Computing resources and measured runs below; other machines can take longer.

Complete run reproducing all results

This command rebuilds the models from the study data, runs all simulations and computational comparisons, and creates the results page.

For this example, use a machine with at least 160 GiB of RAM. The command sets a 120 GiB memory budget for replication tasks, leaving 40 GiB for the operating system and other programs. Set --cpu-budget to the number of cores you want to use, up to the number available on your machine. See Computing resources and measured runs below for disk-space requirements.

.venv/bin/python -m local_randomization.portable_workflow all \
  --config config/definitive_run.json \
  --output-root outputs/local-reproduction \
  --mode complete --cpu-budget 64 --memory-budget-gib 120

After rebuilding and checking all nine models, the program runs 4,000 simulated replications for each of nine studies and seven sampling/assignment designs, giving 63 empirical computations. It also runs 1,377 computational comparisons for Appendix A1.

.venv/bin/python -m local_randomization.portable_workflow all \
  --config config/definitive_run.json \
  --output-root outputs/local-reproduction \
  --mode complete --cpu-budget 64 --memory-budget-gib 120 --dry-run

If you stop the run for any reason, run the same command again with the same output directory. The program checks and reuses finished results, restarts interrupted computations, and continues the remaining work.

If a computation fails, first resolve the problem. Then run the same command with --retry-terminal added to retry failed or timed-out computations. Finished results are still reused. The program checks the results before marking a run complete.

Restricted run reproducing empirical results and smaller computational tasks

For a machine with at least 48 GiB of RAM, this command sets a 40 GiB memory budget for replication tasks, leaving 8 GiB for the operating system and other programs. Adjust --cpu-budget to the number of cores available.

.venv/bin/python -m local_randomization.portable_workflow all \
  --config config/definitive_run.json \
  --output-root outputs/local-reproduction \
  --mode feasible --cpu-budget 32 --memory-budget-gib 40

At this memory budget, all empirical simulations, Sections 2–3 illustrations, match quality comparisons, and million-unit tasks fit individually. The command defers 27 Appendix A1 runtime and peak-memory comparisons across the nine applications: exact pairing at 30,000 units and spatial sorting with polishing at 100,000 units with group sizes 2 and 20. The results page shows finished sections and identifies deferred tasks. The empirical tables appear after all 63 empirical computations finish. Each computational summary appears after all tasks needed for that summary finish. Missing results are never filled with paper values or averages from an incomplete subset.

To run the remaining Appendix A1 comparisons, rerun the main command with sufficient available RAM and a higher memory budget, using the same output directory. The program checks and reuses valid finished results and runs only the remaining computations.

Option 2: Run on a Slurm cluster

The included single-node Slurm script runs the same complete workflow and writes the same RESULTS.html. Install Python and R as above on the cluster’s shared filesystem, then submit from the extracted replication/ directory:

sbatch slurm/run_full_single_node.sbatch

The script requests 64 CPUs, 160G of Slurm RAM, and 48 hours. It sets a 120 GiB memory budget for replication tasks. Adjust the sbatch request for your cluster, or set RESTUD_MEMORY_BUDGET_GIB to a smaller budget if needed. To run every computation, that budget must accommodate the largest individual task. Your cluster may also require a partition or account option. Slurm queue time and run time vary. If the job stops, submit it again against the same outputs/local-reproduction/ directory so finished tasks can be authenticated and reused.

Read and compare the results

The starting point is outputs/local-reproduction/RESULTS.html. It first appears when the empirical stage ends and is updated after the computational comparisons. You can read finished parts while other computations remain unfinished. In particular:

1. Open the ATE and SATE sections. Each design’s Replicated and Paper rows let you compare the displayed numbers directly. Then check those values against the tables in the paper.
2. Check the text-claim rows against Sections 7–8 and the SATE appendix. The coverage ranges, sampling rate, predictable treatment-effect variation, design improvements, and interval-length comparison appear directly on the page. The discretization bound has its own check.
3. Compare the paper and replicated figures displayed side by side. The “Sections 2–3: illustrations” section shows the assignment and matching-path images. The “Appendix A1: computational comparison” section shows the computational figure once all A1 computations finish. This figure is produced as a PDF and displayed inline by a PDF-capable browser. If Poppler’s pdftoppm is installed, it converts the PDF to a PNG for display on the page instead.
4. Check Appendix A1’s match quality claims. All its benchmark time and peak-memory claims appear on the page, with your machine’s measurements next to the paper’s measurements. Differences in time and memory use are normal. Historical hardware observations and external software limitations are identified separately, including any verification not performed by the run.
5. Check the literature-survey counts and, if desired, open the linked paper-by-paper report.

The results page also checks numerical descriptions of the original experiments in the empirical-application appendix against the datasets supplied with this package. For example, it counts the neighborhoods assigned to treatment in Baysan’s study.

Orange highlighting identifies a difference at the precision printed in the paper, or a computation that remains unfinished. Running times and peak memory are displayed for comparison but are not graded as exact matches. Unrounded calculated values remain available through the supporting calculation records.

Data sources

The package contains the 14 original-study files actually used by the nine applications, along with the authors’ documentation. They total approximately 367 MB before compression. Some are the original authors’ public analysis datasets rather than unprocessed survey responses. The package rebuilds the models from these inputs.

The data availability statement below lists the included files, their locations, and how to obtain separate copies. Dataset citations appear in References. The analysis-variable dictionary (Markdown version) describes the variables used by each application.

For questions about the package, contact Max Cytrynbaum at max.cytrynbaum@yale.edu.

Data availability statement

All data needed to run the package are included. No data download, repository login, institutional membership, payment, or data-access application is needed to reproduce the results using these files.

The nine empirical sources are public datasets from the AEA Data and Code Repository at ICPSR (formerly openICPSR). For each study below, the listed files are included in full, without changes to their contents. The versioned DOI identifies the release from which to obtain a separate copy. Full citations appear in References. All paths in the table are relative to the package root. The .dta files are readable with the supplied Python software. No Stata license is needed.

Obtaining separate copies of the empirical data. These public releases are available without charge and do not require affiliation with an ICPSR member institution or a restricted-data application. To download them:

1. Follow the study’s DOI above and select the specified version, not a different release in the version history.
2. Select Download on the study record. Sign in with an ICPSR Researcher Passport account, or create a free account and complete the email-verification prompts. ICPSR also offers linked institutional and external-account sign-in options. See its account guidance.
3. Review and accept the repository’s terms of use. Download the listed .dta files, or the study’s full project archive if the files are inside an archive, and extract it. No variable-selection request or custom extract is needed. The package uses the complete files listed above. File names and capitalization must match.
4. Place the required files at the package paths shown in the table. Archive directory layouts can differ from these package paths. If using a separate input directory, reproduce the layout beneath data/source_inputs/ there and pass that directory with --raw-root to the main command. The program checks the required file sizes and hashes in manifests/data_sources.csv before using them.

Only the listed public files are used. The Domurat application does not require confidential risk scores, nonpublic administrative extracts, or a Covered California data-use agreement. The Lowe application does not use IHDS. The Baysan application does not require separate downloads of V-Dem, terrorism, exchange-rate, or geographic databases mentioned for other analyses in its original README.

Variable documentation. The original .dta files retain their stored variable and coded-value labels. The variable-metadata report displays these labels for all 14 files. The analysis-variable dictionary (Markdown) describes the outcome, fitting contrast, analysis unit, covariates, and recodings for each application, including variables with incomplete source labels. The original authors’ documentation is linked in the table above.

Literature-survey data. The included records contain the classifications and evidence used for the survey of 200 NBER papers. No external download is needed to recalculate its counts. The paper-by-paper report links to the original papers and records the evidence supporting each classification. Copyrighted NBER PDFs are not bundled. These are author-created research records included in this package, not a separately deposited dataset.

This paper does not collect a new human-subject experiment or survey. The package reanalyzes the original studies’ public files and generates simulations. The original-study documentation is retained with the inputs.

Data licensing

I certify that the author of the manuscript has legitimate access to and permission to use the data used in this manuscript.

I certify that the author of the manuscript has documented permission to redistribute/publish the data contained within this replication package.

The original-study datasets retain their own licenses. The public ICPSR catalog records list Creative Commons Attribution 4.0 International (CC BY 4.0) for all nine sources below. The license allows sharing with attribution and a license link. Full dataset citations appear in the References section, and the original authors’ documentation is retained with the inputs. The included source files are not modified. The package’s cleaning and model-building code produces separate derived files.

License: CC BY 4.0 (legal terms). Source-specific exceptions in the original documentation still apply. The confidential Domurat risk-score data, Fink’s separately restricted DHS data, and Lowe’s separately accessed IHDS data are not included or used.

The package code is licensed under LICENSE-CODE.txt (MIT). That license does not apply to the original-study datasets or documentation. See LICENSE.txt for the license notices.

Programs and output locations

All paths below are relative to outputs/local-reproduction/ unless they name source code. The full command runs these steps automatically.

Randomness

The empirical and source-building seed is 54321 in config/definitive_run.json. src/local_randomization/seeds.py derives separate deterministic seeds for each stage, study, design, and replication. They do not depend on worker order or wall-clock time. The A1 configurations use base seed 20260917 in auxiliary/computational_comparison/config/empirical_dgp_benchmark.json and empirical_quality.json. The documented Domurat quality subset uses seed 20260903. Figure seeds and drawing settings are listed in manifests/manuscript_artifacts.json. CPU and memory budgets control scheduling, not these scientific settings.

Computing resources and measured runs

Both documented routes were tested from separate clean extractions and software installations on an x86-64 Linux server with an AMD EPYC 9575F processor, Python 3.10.8, R 4.4.1, and glibc 2.34. Each rebuilt all nine models from the included data and ran all 63 empirical cases with 4,000 replications each. The runs used a shared compute node, not dedicated hardware.

The complete route’s elapsed time is the Slurm allocation duration, including startup and exit, not a separately stopwatch-measured Python command time. Installation commands took about 10 minutes for the complete route and 11 minutes for the feasible route. These installation times, separate setup checks, and cluster queue time are excluded from the scientific-run measurements above. Each timing comparison uses one core and one thread even when independent comparisons run concurrently. Benchmark times and memory use depend on hardware, software, and competing load.

The source inputs occupy about 367 MB and the supplied fitted models about 18 MB, before compression. The completed full run saved about 8.4 GiB under outputs/local-reproduction/; its entire extracted directory, including installed software, occupied about 9.2 GiB. The feasible run saved about 6.5 GiB of outputs. Reserve at least 20 GiB of free disk space for extraction, software, generated data/models, and outputs. This allowance provides headroom above measured final usage; peak disk use during installation and execution was not measured. Resumed runs can retain additional model-build directories and need more space.

References

Abebe, Girum, A. Stefano Caria, and Esteban Ortiz-Ospina. 2021. Data and code for: The Selection of Talent: Experimental and Structural Evidence from Ethiopia. Version 1. American Economic Association. Distributed by Inter-university Consortium for Political and Social Research. doi:10.3886/E127401V1.

Baysan, Ceren. 2022. Data and Code for: Persistent Polarizing Effects of Persuasion: Experimental Evidence from Turkey. Version 1. American Economic Association. Distributed by Inter-university Consortium for Political and Social Research. doi:10.3886/E172061V1.

Bergquist, Lauren, and Michael Dinerstein. 2020. Data and Code for: Competition and Entry in Agricultural Markets: Experimental Evidence from Kenya. Version 1. American Economic Association. Distributed by Inter-university Consortium for Political and Social Research. doi:10.3886/E119743V1.

Casey, Katherine, Abou Bakarr Kamara, and Niccolo Meriggi. 2021. Data and Code for: An Experiment in Candidate Selection. Version 1. American Economic Association. Distributed by Inter-university Consortium for Political and Social Research. doi:10.3886/E124501V1.

DellaVigna, Stefano, John List, Ulrike Malmendier, and Gautam Rao. 2021. Data and code for: Estimating Social Preferences and Gift Exchange at Work. Version 1. American Economic Association. Distributed by Inter-university Consortium for Political and Social Research. doi:10.3886/E148481V1.

Domurat, Richard, Isaac Menashe, and Wesley Yin. 2021. Data and code for: The Role of Behavioral Frictions in Health Insurance Marketplace Enrollment and Risk: Evidence from a Field Experiment. Version 1. American Economic Association. Distributed by Inter-university Consortium for Political and Social Research. doi:10.3886/E125801V1.

Fink, Günther, B. Kelsey Jack, and Felix Masiye. 2020. Data and code for: Seasonal liquidity, rural labor markets and agricultural production. Version 1. American Economic Association. Distributed by Inter-university Consortium for Political and Social Research. doi:10.3886/E119649V1.

Hussam, Reshmaan, Erin Kelley, Gregory Lane, and Fatima Zahra. 2025. Data and Code for: The Psychosocial Value of Employment: Evidence from a Refugee Camp. Version 2. American Economic Association. Distributed by Inter-university Consortium for Political and Social Research. doi:10.3886/E174781V2.

Lowe, Matt. 2026. Data and Code for: Types of Contact: A Field Experiment on Collaborative and Adversarial Caste Integration. Version 2. American Economic Association. Distributed by Inter-university Consortium for Political and Social Research. doi:10.3886/E128201V2.