multiloko

MultiLoKo: a multilingual local knowledge benchmark for LLMs MultiLoKo is a multilingual knowledge benchmark, covering 30 languages plus English. The questions are separately sourced for each language, with an annotation protocol designed to target locally relevant topics for the respective language. MultiLoKo contains the original data for each language, as well as both human and machine-authored translations of each non-English subset into English and vice versa, facilitating studies into a variety of research questions relating to multilinguality. More information about the benchmark design can be found in the release paper of the benchmark: Dieuwke Hupkes and Nikolay Bogoychev. MultiLoKo: a multilingual local knowledge benchmark for LLMs spanning 31 languages. @article{hupkes2025multiloko, title={MultiLoKo: a multilingual local knowledge benchmark for LLMs spanning 31 languages }, author = {Dieuwke Hupkes and Nikolay Bogoychev}, year = {2025}, journal = {CoRR}, volume = {abs/2504.10356}, eprinttype = {arXiv}, eprint = {2504.10356}, url = {https://doi.org/10.48550/arXiv.2504.10356}, } Data Each language in MultiLoKo has its own subdirectory, containing: A jsonl file dev.jsonl containing the 250 locally sourced questions for the respective language. A jsonl file knowledge_fewshot.jsonl, containing five fewshot examples for the language. A jsonl file dev_translated_human_english.jsonl with the human translations of the file into English (for English, instead there will be 30 files dev_translated_human_{language}.jsonl, for each of the respective languages). A jsonl file dev_translated_machine_english.jsonl with the machine translations of the file into English (for English, instead there will be 30 files dev_translated_machine_{language}.jsonl, for each of the respective languages). Multilingual prompts can be found in our our github repository, in the file examples/prompts.py Data format The benchmark data is stored in jsonl files, containing a separate json objec