This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import os | |
import time | |
import torch | |
from torch.nn import Linear, ReLU | |
from torch.profiler import profile, ProfilerActivity | |
from codeparrot.build_table import build_table | |
device = torch.device('cuda') |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
system | GPU count | Training time | Model size | Tokens | teraWFLOP/s | |
---|---|---|---|---|---|---|
MegaTron + DeepSpeed | 2240 | 60.1 | 5.3E+11 | 3932160 | 92.9 | |
HyperCLOVA | 1024 | 1157760 | 8.2E+10 | 1.5E+11 | 62.2 | |
MegaTron LM GPT-3 Example | 1024 | 32 | 1.75E+11 | 3145728 | 100.8 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
model | precision | throughput in teraWFLOPs | |
---|---|---|---|
matmul 8192x1600x6400 | bfloat16 | 237 | |
linear MLP B=8192; 1600 -> 6400 -> 1600 -> 1 | bfloat16 | 230 | |
linear MLP B=8192; 1600 -> 6400 -> 1600 -> 1 | mixed | 207.6 | |
residual ReLU MLP B=8192; 1600 -> 6400 -> 1600 -> 1 | mixed | 185.7 | |
HuggingFace GPT2 B=32 L=256; d=1600 | mixed | 68 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import numpy | |
import sys | |
players = [ | |
'Alex Lacoste', | |
'Karl Janelle', | |
'JF Marcil', | |
'Pat Steeves', | |
'Dzmitry Bahdanau', | |
'Philippe Guay', |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
> static@1.0.0 dev-build /home/dzmitry/Dist/arl-omnishop/static | |
> webpack --progress -d --config webpack.config.js | |
clean-webpack-plugin: /home/dzmitry/Dist/arl-omnishop/static/dist has been removed. | |
10% building modules 4/11 modules 7 active .../Dist/arl-omnishop/static/sass/site.cssℹ 「atl」: Using typescript@3.1.1 from typescript | |
ℹ 「atl」: Using tsconfig.json from /home/dzmitry/Dist/arl-omnishop/static/tsconfig.json | |
94% after sealℹ 「atl」: Checking started in a separate process... | |
✖ 「atl」: Checking finished with 47 errors | |
Hash: 248bef2f47830ed686dd | |
Version: webpack 4.20.2 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
╰─$ npm run dev-build 2 ↵ | |
> static@1.0.0 dev-build /home/dzmitry/Dist/arl-omnishop/static | |
> webpack --progress -d --config webpack.config.js | |
clean-webpack-plugin: /home/dzmitry/Dist/arl-omnishop/static/dist has been removed. | |
10% building modules 4/11 modules 7 active .../Dist/arl-omnishop/static/sass/site.cssℹ 「atl」: Using typescript@3.1.1 from typescript | |
ℹ 「atl」: Using tsconfig.json from /home/dzmitry/Dist/arl-omnishop/static/tsconfig.json | |
94% after sealℹ 「atl」: Checking started in a separate process... | |
✖ 「atl」: Checking finished with 26 errors |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
╰─$ npm run dev-build 2 ↵ | |
> static@1.0.0 dev-build /home/dzmitry/Dist/arl-omnishop/static | |
> webpack --progress -d --config webpack.config.js | |
clean-webpack-plugin: /home/dzmitry/Dist/arl-omnishop/static/dist has been removed. | |
10% building modules 4/11 modules 7 active .../Dist/arl-omnishop/static/sass/site.cssℹ 「atl」: Using typescript@3.1.1 from typescript | |
ℹ 「atl」: Using tsconfig.json from /home/dzmitry/Dist/arl-omnishop/static/tsconfig.json | |
94% after sealℹ 「atl」: Checking started in a separate process... | |
✖ 「atl」: Checking finished with 26 errors |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
def compare_rl_and_gail(metric, window, ylabel=None, xlim=None, fs=15, ncol=2): | |
df1 = df_rl[df_rl['config.rp_weight'] == 1.0] | |
df2 = df_rl[df_rl['config.rp_weight'] == 0.0] | |
df3 = df_gail[df_gail['config.rmt.top_k_percent'] == 0.] | |
df4 = df_gail[df_gail['config.rmt.top_k_percent'] == 0.25] | |
df5 = df_gail[df_gail['config.rmt.top_k_percent'] == 0.5] | |
df6 = df_gail[df_gail['config.rmt.top_k_percent'] == 0.75] | |
for i, df in enumerate([df1, df2, df3, df4, df5, df6]): | |
value = 100 * pandas.rolling_mean(average_value(df, metric), window) | |
print(value.iloc[-1]) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
def compare_rl_and_gail(metric, window, ylabel=None, xlim=None, fs=15, ncol=2): | |
df1 = df_rl[df_rl['config.rp_weight'] == 1.0] | |
df2 = df_rl[df_rl['config.rp_weight'] == 0.0] | |
df3 = df_gail[df_gail['config.rmt.top_k_percent'] == 0.] | |
df4 = df_gail[df_gail['config.rmt.top_k_percent'] == 0.25] | |
df5 = df_gail[df_gail['config.rmt.top_k_percent'] == 0.5] | |
df6 = df_gail[df_gail['config.rmt.top_k_percent'] == 0.75] | |
for i, df in enumerate([df1, df2, df3, df4, df5, df6]): | |
value = 100 * pandas.rolling_mean(average_value(df, metric), window) | |
print(value.iloc[-1]) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
def plot_key_alphas(data, softmax=True, fixed_axis=True): | |
f, axes = pyplot.subplots(1, 3) | |
f.set_size_inches((15, 2)) | |
for k in range(3): | |
arr = numpy.array(data[k]) | |
arr = arr[:, [4, 5, 7]] | |
if softmax: | |
arr = numpy.exp(arr) / numpy.exp(arr).sum(axis=1)[:, None] | |
else: | |
arr = arr / arr.sum(axis=1)[:, None] |
NewerOlder