RealUserSim

RealUserSim: Bridging the Reality Gap in Agent Benchmarking via Grounded User Simulation Behavioral user profiles and evaluation benchmark for realistic LLM-powered user simulation, derived from the WildChat dataset. Dataset Summary This release contains: **7,273 behavioral user profiles** extracted from real conversations, each containing demographics and executable linguistic style commands **600 evaluation test cases** (6 splits x 100) for measuring user simulation fidelity **LLM-as-a-judge prompt** for evaluating simulation realism across 5 behavioral dimensions Profile-grounded simulation achieves **45.3%** fidelity vs. **24.2%** for baseline across 600 conversations and 5 dimensions (+21.1 percentage points). Dataset Structure Behavioral Profiles Each profile captures a real user's communication style as executable instructions an LLM can follow to simulate that user. Profile Format Key Fields | Field | Description | |-------|-------------| | `demographics` | Inferred/extracted demographic attributes. Each has a value, source (`extracted` or `inferred`), and confidence score. | | `linguistic_profile` | Executable style commands describing writing patterns — capitalization, punctuation, formality, message length, multi-turn behavior — with examples. | | `profile_completeness` | Fraction of demographic fields with non-null values (0.0–1.0). | | `conversation_interests` | Domains and task types the user engaged with. | Using Profiles for User Simulation The profiles are designed as system prompt instructions for an LLM acting as a simulated user. Build the Simulated User System Prompt Run Multi-Turn Simulation Evaluating Simulation Fidelity The LLM-as-a-judge evaluates whether a simulated user behaves like the real user across 5 dimensions: 1. **Persona & Affective Traits** — demeanor, emotional state, patience 2. **Linguistic Style & Mechanics** — vocabulary, formality, typos, message length 3. **Technical Competency & Knowledge** — domain expertise, terminology