exelents

[2021-05-17 22:17:54,087] [INFO] [stage2.py:130:__init__] Reduce bucket size 50000000
[2021-05-17 22:17:54,088] [INFO] [stage2.py:131:__init__] Allgather bucket size 200000000
[2021-05-17 22:17:54,088] [INFO] [stage2.py:132:__init__] CPU Offload: True
Traceback (most recent call last):
  File "../pretrain_gpt3.py", line 832, in <module>
    main()
  File "../pretrain_gpt3.py", line 788, in main
    model, optimizer, lr_scheduler = setup_model_and_optimizer(args)
  File "../pretrain_gpt3.py", line 190, in setup_model_and_optimizer
    dist_init_required=False