StatLLM: A Dataset for Evaluating the Performance of Large Language Models in Statistical Analysis – Scientific Data
Large language models are increasingly capable coders, and their promise in automating statistical analysis has captivated data science and machine learning. But before teams hand off critical analytics to AI, one question looms: How reliable is the code these models produce? A new open-source benchmark, StatLLM, aims to answer that by rigorously evaluating LLM-generated statistical code—anchored in real analysis tasks, real datasets, and human expert review.
Why this matters
Most coding benchmarks focus on general programming. Statistical analysis is different: correctness isn’t just about syntax or runtime—it’s about valid methodology, interpretable outputs, and reproducible results. Until now, the field lacked a purpose-built benchmark to judge LLMs on those criteria. StatLLM fills that gap, offering a standardized way to assess accuracy and utility in statistical workflows.
What’s inside StatLLM
The dataset centers on three components designed to capture the full lifecycle of statistical coding:
- Statistical analysis tasks: A diverse collection of tasks spanning multiple analyses and datasets. Each task includes a clear problem description, dataset details, and human-verified SAS code that serves as a reference solution.
- LLM-generated SAS code: Code produced by leading models—GPT-3.5, GPT-4, and Llama-3.1 70B—applied to the same tasks. This enables head-to-head comparisons across systems and versions.
- Human evaluation scores: Expert reviewers assess each LLM submission across five dimensions crucial for statistical work: correctness, effectiveness, readability, executability, and output accuracy.
A benchmark tailored to statistical practice
By grounding tasks in SAS—a mainstay in many regulated and enterprise analytics environments—StatLLM emphasizes real-world statistical practice. The inclusion of human-verified reference code provides an anchor for objective comparison, while expert scoring captures the nuances that automated metrics often miss, such as methodological soundness and interpretability.
How researchers and teams can use it
- Evaluate and improve NLP metrics: Use StatLLM as a testbed for developing metrics that correlate better with human judgment on statistical code, beyond generic code-quality measures.
- Assess and enhance LLM performance: Track where models stumble—wrong models, faulty assumptions, poor data handling—and fine-tune prompts or training data to close those gaps.
- Build next-generation statistical software: Prototype tools that integrate LLMs into analysis pipelines, and stress-test them against a benchmark that reflects real analytical rigor.
What sets StatLLM apart
- End-to-end focus: From task description to executable code to validated outputs, the dataset reflects the complete analytical workflow.
- Model diversity: Including GPT-3.5, GPT-4, and Llama-3.1 70B enables performance mapping across proprietary and open models.
- Human-centered evaluation: Expert scoring goes beyond “does it run?” to “does it produce the right results in the right way?”
Where this could lead
As LLMs move from coding assistants to analytical collaborators, benchmarks like StatLLM can help set standards for trust and accountability. Expect it to inform best practices for prompt design, model selection, and guardrailing in statistical contexts. For academia and industry alike, it offers a shared yardstick for progress—and a foundation for reproducible, transparent evaluation.
Early-access note
This coverage reflects an unedited version of the manuscript released for early access. The paper will undergo further editing before final publication, and errors may be present that could affect interpretation. All legal disclaimers apply. The authors also provide online supplementary materials.
The bottom line
StatLLM is a timely, open-source benchmark for measuring how well large language models handle the demands of statistical coding. By combining realistic tasks, LLM-generated SAS code, and rigorous human evaluations, it offers a practical path to comparing models, improving metrics, and building safer, smarter analytics tools.
