doriab/CareMedEval

Tasks

• question-answering
• visual-question-answering
• visual-document-retrieval
• summarization

Citation text from the card

@inproceedings{bonzi-etal-2026-caremedeval,
  title = {CareMedEval Dataset: Evaluating Critical Appraisal and Reasoning in the Biomedical Field},
  author = {Bonzi, Doria and Guiggi, Alexandre and Bechet, Frederic and Ramisch, Carlos and Favre, Benoit},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026)},
  month = {May},
  year = {2026},
  pages = {5169--5181},
  address = {Palma, Mallorca, Spain},
  publisher = {European Language Resources Association (ELRA)},
  editor = {Piperidis, Stelios and Bel, Núria and van den Heuvel, Henk and Ide, Nancy and Krek, Simon and Toral, Antonio},
  doi = {10.63317/2erqq5fuxpge},
  abstract = {Critical appraisal of scientific literature is an essential skill in the biomedical field. While large language models (LLMs) can offer promising support in this task, their reliability remains limited, particularly for critical reasoning in specialized domains. We introduce CareMedEval, an original dataset designed to evaluate LLMs on biomedical critical appraisal and reasoning tasks. Derived from authentic exams taken by French medical students, the dataset contains 534 questions based on 37 scientific articles. Unlike existing benchmarks, CareMedEval explicitly evaluates critical reading and reasoning grounded in scientific papers. Benchmarking state-of-the-art generalist and biomedical-specialized LLMs under various context conditions reveals the difficulty of the task: open and commercial models f

All datasets marked Permissive or share-alike | JSON