This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Comparison of deepspeed and gluon-nlp in SQuAD1.1 | |
Dataset:SQuAD1.1 | |
GPUs:p3.8x with four teslaV100 | |
BatchSize: 3 | |
Epoch: 2 | |
Max_seq_len: 384 | |
Model: google_en_uncased_bert_wwm_large | |
Results: | |
deepspeed : 93.11/87.03 time cost: 0.71hours |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import time | |
import torch | |
from transformers import GPTNeoForCausalLM, AutoConfig, GPT2Tokenizer | |
import torch | |
import transformers | |
import collections | |
import os |