Skip to content

Instantly share code, notes, and snippets.

@rizar

rizar/megatron.csv

Last active January 2, 2022 21:07
Show Gist options
  • Save rizar/40c858273c717dbd88e725fb686df5a3 to your computer and use it in GitHub Desktop.
Save rizar/40c858273c717dbd88e725fb686df5a3 to your computer and use it in GitHub Desktop.
Download ZIP
MegaTron throughput
Raw
megatron.csv
system GPU count Training time Model size Tokens teraWFLOP/s
MegaTron + DeepSpeed 2240 60.1 5.3E+11 3932160 92.9
HyperCLOVA 1024 1157760 8.2E+10 1.5E+11 62.2
MegaTron LM GPT-3 Example 1024 32 1.75E+11 3145728 100.8
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment