All datasets / oscar-corpus

oscar-corpus/OSCAR-2109

61 downloads in 30 days42 likesLast modified 6 Aug 2025View on Hugging Face

Permissive or share-alike

The declared licence (cc0-1.0) is classified as permissive.

Declared licence: cc0-1.0, from the tagEvidence: README.md at 3f043b8
Licence files: No licence file at the top level of the repositoryEvidence: file list at 3f043b8
Declared sources: None listed in the card metadata
Access gated on Hugging Face: Yes, manual approval required

Each cell links to what was checked: a file, a file list or a Hub record.

Tasks

sequence-modeling

Citation text from the card

@inproceedings{AbadjiOrtizSuarezRomaryetal.2021,
  author    = {Julien Abadji and Pedro Javier Ortiz Su{\'a}rez and Laurent Romary and Beno{\^i}t Sagot},
  title     = {Ungoliant: An optimized pipeline for the generation of a very large-scale multilingual web corpus},
  series = {Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-9) 2021. Limerick, 12 July 2021 (Online-Event)},
  editor    = {Harald L{\"u}ngen and Marc Kupietz and Piotr Bański and Adrien Barbaresi and Simon Clematide and Ines Pisetta},
  publisher = {Leibniz-Institut f{\"u}r Deutsche Sprache},
  address   = {Mannheim},
  doi       = {10.14618/ids-pub-10468},
  url       = {https://nbn-resolving.org/urn:nbn:de:bsz:mh39-104688},
  pages     = {1 -- 9},
  year      = {2021},
  abstract  = {Since the introduction of large language models in Natural Language Processing, large raw corpora have played a crucial role in Computational Linguistics. However, most of these large raw corpora are either available only for English or not available to the general public due to copyright issues. Nevertheless, there are some examples of freely available multilingual corpora for training Deep Learning NLP models, such as the OSCAR and Paracrawl corpora. However, they have quality issues, especially for low-resource languages. Moreover, recreating or updating these corpora is very complex. In this work, we try to reproduce and improve the goclassy pipeline used to create the OSCAR corpus. We propo

All datasets marked Permissive or share-alike | JSON

The same facts for your agent

A free MCP server at mcp.datalicense.uk/mcp. Connect it from an MCP client such as Claude or Cursor.

dataset_licence({ "id": "oscar-corpus/OSCAR-2109" })
signal: "permissive"
stated_licence: "cc0-1.0"
sources: 0