jonsnow73/This-is-not-a-dataset

Tasks

• text-classification

Citation text from the card

@inproceedings{garcia-ferrero-etal-2023-dataset,
    title = "This is not a Dataset: A Large Negation Benchmark to Challenge Large Language Models",
    author = "Garc{\'\i}a-Ferrero, Iker  and
      Altuna, Bego{\~n}a  and
      Alvez, Javier  and
      Gonzalez-Dios, Itziar  and
      Rigau, German",
    editor = "Bouamor, Houda  and
      Pino, Juan  and
      Bali, Kalika",
    booktitle = "Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing",
    month = dec,
    year = "2023",
    address = "Singapore",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.emnlp-main.531",
    doi = "10.18653/v1/2023.emnlp-main.531",
    pages = "8596--8615",
    abstract = "Although large language models (LLMs) have apparently acquired a certain level of grammatical knowledge and the ability to make generalizations, they fail to interpret negation, a crucial step in Natural Language Processing. We try to clarify the reasons for the sub-optimal performance of LLMs understanding negation. We introduce a large semi-automatically generated dataset of circa 400,000 descriptive sentences about commonsense knowledge that can be true or false in which negation is present in about 2/3 of the corpus in different forms. We have used our dataset with the largest available open LLMs in a zero-shot approach to grasp their generalization and inference capability and we have also fine-tuned some of the models to assess 

All datasets marked Permissive or share-alike | JSON