Rethinking the Evaluation and Optimization of LLM-Based Social Simulation

Abstract

LLM-based social simulation is a promising complement to traditional social science methods such as surveys and behavioral experiments. A core question in this area is how to evaluate the fidelity of LLM-simulated human behavior and, in turn, how to optimize LLMs toward it. Prevailing practice evaluates by accuracy, checking whether the model selects the single response observed from a human, and accordingly trains the LLM to reproduce this one hard label. However, human behavior is inherently subjective: the same person in the same situation may reasonably act in different ways, so an observed response is only one draw from an underlying response distribution, which renders accuracy-based evaluation unreliable and hard-label training misleading.

To address these problems, we first introduce the subjectivity coefficient, an entropy-based quantity that distinguishes objective tasks such as coding from subjective tasks such as social simulation, and use it to systematically analyze how accuracy-based evaluation and hard-label training fail as subjectivity grows. Based on the subjectivity coefficient, we further propose Subjectivity-Adaptive soft-Label Training (SALT): it pools observed outputs from semantically nearby inputs into soft distributional labels, with an aggregation radius adapted to the estimated subjectivity of each input; in the near-objective limit the neighborhood shrinks, so SALT naturally falls back to standard single-label training. Moreover, since existing datasets record only single observed responses and thus cannot support distributional evaluation, we construct SubjSim, a benchmark of 19,300 contexts covering 193 annotators and 100 subjective questions. Extensive results on SubjSim demonstrate the advantages of our method.

−77.6%

KL divergence
vs. SFT

−45.9%

Jensen–Shannon divergence
vs. SFT

−31.2%

Total variation distance
vs. SFT

−52.7%

MMD
vs. SFT

Average over 2,800 held-out annotator–question pairs in SubjSim, Qwen3-8B backbone.

Why accuracy fails

A recorded human answer is only one random draw from an underlying response distribution. As a result, checking whether the model's top action matches that single draw reveals little about fidelity: we construct explicit pairs of models whose accuracy scores are identical (or even reversed) while their true distributional errors differ substantially. We quantify task subjectivity with the subjectivity coefficient — the normalized entropy of the response distribution — and prove that the unavoidable error of single-label training is lower-bounded by (roughly) the subjectivity of the context, while repeated annotation would require Ω(K^2s) observations per context.

SALT: Subjectivity-Adaptive Soft-Label Training

SALT replaces each one-hot training target with a soft label: the empirical action distribution of semantically nearby contexts that share the same candidate actions. The key design question is how large the neighborhood should be. Our aggregation–estimation tradeoff theorem shows that the optimal radius grows with the effective number of actions of the context — the more subjective the context, the more neighbors should be pooled.

Soft labels from neighbors. Contexts are grouped by action space and embedded; each context aggregates the hard answers within an adaptive radius into its own distributional target.
Subjectivity-adaptive radius. The radius follows the theoretical optimum r(x) ∝ (K̂_eff(x) / n_g)^1/(d+2), estimated from the model's own current distribution and refreshed during training.
Hard labels as a special case. For near-objective contexts the neighborhood contracts, and SALT recovers standard single-label training.

Main Results

Method	KL ↓	JSD ↓	TVD ↓	MMD ↓
Pretrained (Qwen3-8B)	4.2801	0.2851	0.5745	0.6350
SFT	1.2871	0.1524	0.3953	0.3194
DPO	6.8819	0.2690	0.5497	0.5774
PPO	4.1393	0.2588	0.5403	0.5578
DSA	3.3904	0.2343	0.4937	0.4418
SALT (Ours)	0.2880	0.0825	0.2720	0.1510

Average distributional error over all 2,800 test pairs (lower is better; underlined = second best). SALT improves over SFT by 77.6% KL, 45.9% JSD, 31.2% TVD, and 52.7% MMD, and all pairwise differences are significant under a paired bootstrap test (1000 resamples, see paper appendix).

Gains across subjectivity levels

SALT's advantage over SFT is largest on mid- and high-subjectivity questions, where one observed response reveals the least about the full distribution. Percentages give SALT's relative change over SFT.

(a) JSD error as a function of the ground-truth subjectivity coefficient for all methods. (b–e) SALT's error gap over SFT, DPO, PPO, and DSA widens monotonically with subjectivity.

Ablations

Subjectivity-adaptive neighborhoods beat fixed-size neighborhoods and top-N variants across metrics.

Performance is stable over a wide range of the radius-scale hyperparameter C.

The SubjSim Benchmark

Existing social-simulation datasets record only a single response per context and thus cannot support distributional evaluation. SubjSim contains 19,300 annotator–question pairs from 193 annotators answering 100 subjective survey questions drawn from established social surveys (WVS, ISSP, CGSS and others) across eight domains. Each pair carries a full response-propensity distribution elicited through probability-ball allocation: annotators distribute 100 probability balls over the candidate options. Models are trained on one derived hard action per pair only, and evaluated against the full held-out distributions.

Domain and subjectivity-coefficient distribution of SubjSim.

Annotator demographics (193 participants).

Citation

@article{wang2026rethinking,
  title   = {Rethinking the Evaluation and Optimization of LLM-Based Social Simulation},
  author  = {Wang, Pei and Chen, Xu and Wen, Ji-Rong},
  journal = {arXiv preprint},
  year    = {2026}
}

Rethinking the Evaluation and Optimization ofLLM-Based Social Simulation