Skip to content

Instantly share code, notes, and snippets.

@abdulrahimq
Created April 30, 2021 13:13
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save abdulrahimq/c6de8c2b53e1201b876ec5c693ce23aa to your computer and use it in GitHub Desktop.
Save abdulrahimq/c6de8c2b53e1201b876ec5c693ce23aa to your computer and use it in GitHub Desktop.
Traceback (most recent call last):
File "/scratch/amq259/nlu/experiment-script.py", line 354, in <module>
tot = loader(dataset_name, tokenizer, args.cache_dir)
File "/scratch/amq259/nlu/experiment-script.py", line 150, in loader
tot.append(_preprocess_dataset(dataset_name, data, sentence_col, tokenizer))
File "/scratch/amq259/nlu/experiment-script.py", line 157, in _preprocess_dataset
data = data.map(
File "/scratch/amq259/envName/lib/python3.9/site-packages/datasets/dataset_dict.py", line 432, in map
{
File "/scratch/amq259/envName/lib/python3.9/site-packages/datasets/dataset_dict.py", line 433, in <dictcomp>
k: dataset.map(
File "/scratch/amq259/envName/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 1407, in map
update_data = does_function_return_dict(test_inputs, test_indices)
File "/scratch/amq259/envName/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 1378, in does_function_return_dict
function(*fn_args, indices, **fn_kwargs) if with_indices else function(*fn_args, **fn_kwargs)
File "/scratch/amq259/nlu/experiment-script.py", line 158, in <lambda>
lambda x: tokenizer(x["input_text"], padding="max_length", truncation=True),
File "/scratch/amq259/envName/lib/python3.9/site-packages/transformers/tokenization_utils_base.py", line 2210, in __call__
assert isinstance(text, str) or (
AssertionError: text input must of type `str` (single example), `List[str]` (batch or single pretokenized example) or `List[List[str]]` (batch of pretokenized examples).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment