WikiProfile

WikiProfile WikiProfile is a factual knowledge benchmark for evaluating how well language models encode and recall factual knowledge. It comprises 2,150 facts, each paired with 10 questions, for a total of 21,500 question instances. Each fact is grounded in the first paragraph (summary) of an English Wikipedia page and is defined as a proposition between two entities, a subject and an object (e.g., "Oasis played their first gig at the Boardwalk club" → subject: Oasis, object: Boardwalk). The ten questions per fact span three formats: text completion (encoding), closed-book question answering (recall), and multiple-choice. Facts are extracted from 2,125 Wikipedia pages spanning 9 topic categories and 19 entity types. Dataset Details **Dataset Name:** WikiProfile **Format:** CSV **Size:** ~5.22 MB **License:** CC-BY-SA 4.0 (Required due to Wikipedia content source) **Paper:** Empty Shelves or Lost Keys? Recall Is the Bottleneck for Parametric Factuality Dataset Sources **Source Text:** English Wikipedia page summaries. **Popularity Metrics:** Wikimedia Analytics page view counts (aggregated over 2023–2025). **Generation Pipeline:** Questions, entity annotations, and multiple-choice distractors were generated using Gemini-2.5-Pro, followed by Google Search–grounded filtering and final manual validation (removing <2% of facts). Dataset Structure The dataset consists of a single CSV file. Each row represents a single fact (2,150 rows total) with 21 columns containing different question formats and metadata. Data Fields | Column | Type | Description | | :--- | :--- | :--- | | `index` | int | Row index (0-based) | | `fact_id` | string | Unique fact identifier (format: `{page_id}__{entity_index}`) | | `page_id` | string | Wikipedia page ID | | `page_title` | string | Wikipedia page title | | `item_id` | string | Wikidata item ID (Q-number) | | `gbc` | int | Page view count (popularity metric, aggregated over 2023–2025) | | `creation_date` | string | Wikipedia page creation