jon-tow/feasibility_qa

Citation text from the card

@inproceedings{gupta-etal-2023-john,
    title = "{``}John is 50 years old, can his son be 65?{''} Evaluating {NLP} Models{'} Understanding of Feasibility",
    author = "Gupta, Himanshu  and
      Varshney, Neeraj  and
      Mishra, Swaroop  and
      Pal, Kuntal Kumar  and
      Sawant, Saurabh Arjun  and
      Scaria, Kevin  and
      Goyal, Siddharth  and
      Baral, Chitta",
    editor = "Vlachos, Andreas  and
      Augenstein, Isabelle",
    booktitle = "Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics",
    month = may,
    year = "2023",
    address = "Dubrovnik, Croatia",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.eacl-main.30",
    doi = "10.18653/v1/2023.eacl-main.30",
    pages = "407--417",
    abstract = "In current NLP research, large-scale language models and their abilities are widely being discussed. Some recent works have also found notable failures of these models. Often these failure examples involve complex reasoning abilities. This work focuses on a simple commonsense ability, reasoning about when an action (or its effect) is feasible. To this end, we introduce FeasibilityQA, a question-answering dataset involving binary classification (BCQ) and multi-choice multi-correct questions (MCQ) that test understanding of feasibility. We show that even state-of-the-art models such as GPT-3, GPT-2, and T5 struggle to answer the feasibility quest

All datasets marked Permissive or share-alike | JSON