All datasets / toksuite

toksuite/toksuite_general

112 downloads in 30 days0 likesLast modified 20 Jan 2026View on Hugging Face

Permissive or share-alike

The declared licence (mit) is classified as permissive.

Declared licence: mit, from the tagEvidence: README.md at ae34a64
Licence files: No licence file at the top level of the repositoryEvidence: file list at ae34a64
Declared sources: None listed in the card metadata
Access gated on Hugging Face: No

Each cell links to what was checked: a file, a file list or a Hub record.

Tasks

multiple-choice

Citation text from the card

If you use this dataset in your research, please cite the TokSuite paper:

@inproceedings{toksuite2026,
  title={TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior},
  author={Altıntaş, Gül Sena and Ehghaghi, Malikeh and Lester, Brian and Liu, Fengyuan and Zhao, Wanru and Ciccone, Marco and Raffel, Colin},
  booktitle={Preprint},
  year={2026},,
  arxiv={https://arxiv.org/abs/2512.20757},
  url={TBD}
}


**Paper**: [TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior](TBD)

All datasets marked Permissive or share-alike | JSON

The same facts for your agent

A free MCP server at mcp.datalicense.uk/mcp. Connect it from an MCP client such as Claude or Cursor.

dataset_licence({ "id": "toksuite/toksuite_general" })
signal: "permissive"
stated_licence: "mit"
sources: 0