Multi-IF

Dataset Summary We introduce Multi-IF, a new benchmark designed to assess LLMs' proficiency in following multi-turn and multilingual instructions. Multi-IF, which utilizes a hybrid framework combining LLM and human annotators, expands upon the IFEval by incorporating multi-turn sequences and translating the English prompts into another 7 languages, resulting in a dataset of 4501 multilingual conversations, where each has three turns. Our evaluation of 14 state-of-the-art LLMs on Multi-IF reveals that it presents a significantly more challenging task than existing benchmarks. All the models tested showed a higher rate of failure in executing instructions correctly with each additional turn. For example, o1-preview drops from 0.877 at the first turn to 0.707 at the third turn in terms of average accuracy over all languages. Moreover, languages with non-Latin scripts (Hindi, Russian, and Chinese) generally exhibit higher error rates, suggesting potential limitations in the models’ multilingual capabilities. Evaluation Script https://github.com/facebookresearch/Multi-IF Data Fields `turns`: Placehold for saving the history conversation in evaluation. `responses`: Placehold for saving the latest response in evaluation. `turn_1_prompt`: The user prompt at the first turn, which is the input for LLM generation. `turn_1_instruction_id_list`: The instructions of the user prompt at the first turn, which is needed in the evaluation script. `turn_1_kwargs`: The arguments of the first turn instructions, which is needed in the evaluation script. `turn_2_prompt`: The user prompt at the second turn, which is the input for LLM generation. `turn_2_instruction_id_list`: The instructions of the user prompt at the second turn, which is needed in the evaluation script. `turn_2_kwargs`: The arguments of the second turn instructions, which is needed in the evaluation script. `turn_3_prompt`: The user prompt at the third turn, which is the input for LLM generation. `turn_3_instruction_id_lis