Renad10/Absher-Benchmark

Citation text from the card

```bibtex
@article{ALMONEF202625,
title = {From words to proverbs: Evaluating LLMs’ linguistic and cultural competence in Saudi dialects with Absher},
journal = {Alexandria Engineering Journal},
volume = {137},
pages = {25-41},
year = {2026},
issn = {1110-0168},
doi = {https://doi.org/10.1016/j.aej.2025.12.066},
url = {https://www.sciencedirect.com/science/article/pii/S1110016825012487},
author = {Renad Al-Monef and Hassan Alhuzali and Nora Alturayeif and Ashwag Alasmari},
keywords = {Large language models, Natural language processing, Cultural aware LLMs, Dialects understanding},
abstract = {As large language models (LLMs) become increasingly central to Arabic NLP applications, their effectiveness in linguistically diverse settings, particularly regions with rich dialectal variation such as Saudi Arabia, remains underexplored. Existing evaluation paradigms tend to prioritize high-resource languages or Modern Standard Arabic (MSA), overlooking regional linguistic and cultural specificities. This leads to performance limitations and cultural biases in real-world deployments. To address this gap, we introduce Absher, the first comprehensive and fine-grained benchmark designed to assess the understanding of LLMs regarding Saudi dialects and their embedded cultural nuances. Absher consists of over 18,000 multiple choice questions derived from a curated dataset of dialectal words, phrases, and proverbs sourced from five major Saudi regions. The benchmark spans six task categories:

All datasets marked Permissive or share-alike | JSON