Understanding Reliability in LLM-based Human Behavior Simulation

📌Overview

Reliability is not a single score — it is a property of a layered simulation process.

Large language models (LLMs) are increasingly used to simulate human survey responses and behavioral reactions, yet unreliable simulations can mislead social science conclusions. Existing evaluations focus on end-to-end scores, leaving it unclear how different aspects of the simulation process interact to determine reliability.

We propose ReliMap, which decomposes LLM-based human behavior simulation into three structured layers and evaluates reliability at both the individual level (R1) and population level (R2) across three configuration dimensions: model capacity, profile completeness, and population coverage. Through experiments across four simulation tasks and eleven LLMs, we find that all models exhibit substantial distributional bias without profile conditioning, and that profile conditioning reduces this bias with diminishing returns. Critically, R1 gains do not reliably transfer to R2 — individual and population-level reliability can move in opposite directions. At the population layer, increasing coverage reduces variance but not systematic bias, with R2 stabilizing at around 50–100 individuals.

🧭The ReliMap Framework

Three layers × three configuration dimensions × two evaluation levels.

🔍Key Findings

Six findings from experiments across four tasks and eleven LLMs.

1. Richer profiles improve R1 — with diminishing returns. Individual accuracy grows with profile completeness, but the marginal benefit quickly shrinks: newly added attributes are often uninformative or redundant, and denser profiles demand stronger contextual reasoning from the model.
2. Profile scaling is strongly model-dependent. Larger and stronger models (e.g., GPT-4.1, DeepSeek-V3.2-Exp) keep gaining as profiles grow, while smaller models plateau early — and Thinking variants show no consistent edge over Instruct ones.
3. Predictive information is highly concentrated. A small subset of attributes carries a disproportionate share of the signal: the top 20% of attributes contribute ≈50–54% of predictive information across the Party, Immigration, and Religion tasks.
4. Attribute informativeness matters more than attribute count. Accuracy correlates more strongly with the cumulative informativeness of provided attributes than with their raw count (+0.14/+0.18 average correlation gain on Party/Immigration).
5. R1 gains do not reliably transfer to R2. Individual accuracy improvements can coincide with worsening population-level alignment — a fundamental decoupling between the two levels of reliability.
6. Population coverage reduces variance, not systematic bias. R2 improves quickly with coverage and stabilizes at around 50–100 individuals; beyond that, a persistent gap to the human sampling baseline remains.

📚Citation

If you find this work useful, please cite:

@inproceedings{wang2026understanding,
  title     = {Understanding Reliability in {LLM}-based Human Behavior Simulation},
  author    = {Wang, Pei and Wang, Lei and Li, Yuanzi and Chen, Xu},
  booktitle = {Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing},
  year      = {2026},
  publisher = {Association for Computational Linguistics}
}