factnet-bench

FactNet Benchmarks This repository contains three benchmark datasets derived from FactNet: 1. Knowledge Graph Completion (KGC) The KGC benchmark evaluates a model's ability to complete missing links in a knowledge graph. **Format**: (subject, relation, object) triples **Splits**: Train/Dev/Test **Task**: Predict missing entity (either subject or object) **Construction**: Extracted from entity-valued synsets and projected to (S, P, O) triples with careful cross-split collision handling 2. Multilingual Knowledge Graph QA (MKQA) The MKQA benchmark evaluates knowledge graph question answering across multiple languages. **Languages**: Multiple (en, zh, de, fr, etc.) **Format**: Natural language questions with structured answers **Task**: Answer factoid questions using knowledge graph information **Construction**: Generated from FactSynsets with canonical mentions across languages 3. Multilingual Fact Checking (MFC) The MFC benchmark evaluates fact verification capabilities across languages. **Languages**: Multiple (en, zh, de, fr, etc.) **Labels**: SUPPORTED, REFUTED, NOT_ENOUGH_INFO **Format**: Claims with associated evidence units **Construction**: SUPPORTED claims generated from synsets with FactSenses REFUTED claims generated by value replacement NOT_ENOUGH_INFO claims generated with no matching synsets Each claim associated with gold evidence units with character spans Usage Construction Process FactNet and its benchmarks were constructed through a multi-phase pipeline: 1. **Data Extraction**: Parsing Wikidata to extract FactStatements and labels Extracting Wikipedia pages using WikiExtractor Parsing pagelinks and redirects from SQL dumps 2. **Elasticsearch Indexing**: Indexing Wikipedia pages, FactStatements, and entity labels Creating optimized indices for retrieval 3. **FactNet Construction**: Building FactSense instances by linking statements to text Aggregating FactStatements into FactSynsets Building inter-synset relation edges 4. **Benchmark Generation**: Con