Skip to content

Instantly share code, notes, and snippets.

@LysandreJik

LysandreJik/loading_data.py Secret

Created October 8, 2019 02:01
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save LysandreJik/1baae04d87034100e42c7c956c654287 to your computer and use it in GitHub Desktop.
Save LysandreJik/1baae04d87034100e42c7c956c654287 to your computer and use it in GitHub Desktop.
Download ZIP
Loading data using tensorflow datasets
Raw
loading_data.py
import tensorflow_datasets
from transformers import glue_convert_examples_to_features
data = tensorflow_datasets.load("glue/mrpc")
train_dataset = data["train"]
validation_dataset = data["validation"]
train_dataset = glue_convert_examples_to_features(train_dataset, bert_tokenizer, 128, 'mrpc')
validation_dataset = glue_convert_examples_to_features(validation_dataset, bert_tokenizer, 128, 'mrpc')
train_dataset = train_dataset.shuffle(100).batch(32).repeat(2)
validation_dataset = validation_dataset.batch(64)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment