miracl/nomiracl

Tasks

• text-classification

Citation text from the card

@inproceedings{thakur-etal-2024-knowing,
    title = "{``}Knowing When You Don{'}t Know{''}: A Multilingual Relevance Assessment Dataset for Robust Retrieval-Augmented Generation",
    author = "Thakur, Nandan  and
      Bonifacio, Luiz  and
      Zhang, Crystina  and
      Ogundepo, Odunayo  and
      Kamalloo, Ehsan  and
      Alfonso-Hermelo, David  and
      Li, Xiaoguang  and
      Liu, Qun  and
      Chen, Boxing  and
      Rezagholizadeh, Mehdi  and
      Lin, Jimmy",
    editor = "Al-Onaizan, Yaser  and
      Bansal, Mohit  and
      Chen, Yun-Nung",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2024",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.findings-emnlp.730",
    pages = "12508--12526",
    abstract = "Retrieval-Augmented Generation (RAG) grounds Large Language Model (LLM) output by leveraging external knowledge sources to reduce factual hallucinations. However, prior work lacks a comprehensive evaluation of different language families, making it challenging to evaluate LLM robustness against errors in external retrieved knowledge. To overcome this, we establish **NoMIRACL**, a human-annotated dataset for evaluating LLM robustness in RAG across 18 typologically diverse languages. NoMIRACL includes both a non-relevant and a relevant subset. Queries in the non-relevant subset contain passages judged

All datasets marked Permissive or share-alike | JSON