C-EB/darija_translation_6.py

Created February 6, 2025 16:01

Star (0) You must be signed in to star a gist
Fork (0) You must be signed in to fork a gist

Learn more about clone URLs
Clone this repository at <script src="https://gist.github.com/C-EB/bfbe7acdbd28cd80cdc515dbaae9c070.js"></script>
Save C-EB/bfbe7acdbd28cd80cdc515dbaae9c070 to your computer and use it in GitHub Desktop.

Download ZIP

tokenization application

Raw

darija_translation_6.py

	tokenized_datasets = dataset.map(
	preprocess_function,
	batched=True,
	remove_columns=dataset['train'].column_names
	)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment