shaowei-su

from ray.job_submission import JobSubmissionClient

client = JobSubmissionClient("http://127.0.0.1:8265")

kick_off_pytorch_benchmark = (
    # Run the benchmark.
    "python3.8 ./run_clm_deepspeed_train.py --model_name_or_path EleutherAI/gpt-neox-20b --block_size 2048 --output_dir /nvme/out2 --num_train_epochs 3 --learning_rate 5e-5 --weight_decay 0. --num_workers 16 --upload_dir '[S3]' --per_device_train_batch_size 1 --per_device_eval_batch_size 1 --gradient_accumulation_steps 1 --train_file /tmp/gpt/train.csv --validation_file /tmp/gpt/val.csv --seed 42"
)

shaowei-su

#!/usr/bin/env python
# coding=utf-8
# Copyright 2022 The HuggingFace Inc. team. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#

shaowei-su

package main

import (
        "fmt"
        "github.com/aws/aws-sdk-go/aws/session"
        "github.com/aws/aws-sdk-go/service/s3/s3manager"
        "github.com/aws/aws-sdk-go/service/s3"
        "os"
        "strconv"
        "time"

shaowei-su

Environments

• CPU: Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
• GPU: NVIDIA V100
• Memory: 251GiB
• OS: Ubuntu 16.04.6 LTS (Xenial Xerus)

Docker Images:

• tensorflow/tensorflow:latest-gpu
• tensorflow/serving:latest-gpu

shaowei-su

cd /Library/Application\ Support/VMware\ Tools/

sudo ./vmware-resolutionSet 3440 1440

./vmware-resolutionSet 2560 1080

shaowei-su

Error codes reference

pycodestyle

Indentation

• E101: indentation contains mixed spaces and tabs
• E111: indentation is not a multiple of four
• E112: expected an indented block
• E113: unexpected indentation
• E114: indentation is not a multiple of four (comment)
• E115: expected an indented block (comment)

shaowei-su

Note: $ denotes the start of a command. Don't actually type this.

Steps to Install Anaconda on Windows Ubuntu Terminal

1. Install WSL (Ubuntu for Windows - can be found in Windows Store). I recommend the latest version (I'm using 18.04) because there are some bugs they worked out during 14/16 (microsoft/WSL#785)
2. Go to https://repo.continuum.io/archive to find the list of Anaconda releases
3. Select the release you want. I have a 64-bit computer, so I chose the latest release ending in x86_65.sh. If I had a 32-bit computer, I'd select the x86.sh version. If you accidentally try to install the wrong one, you'll get a warning in the terminal. I chose Anaconda3-5.2.0-Linux-x86_64.sh.
4. From the terminal run wget https://repo.continuum.io/archive/[YOUR VERSION]. Example: $ wget https://repo.continuum.io/archive/Anaconda3-5.2.0-Linux-x86_64.sh
5. Run the installation script: $ bash Anaconda[YOUR VERSION].sh ($ bash Anaconda3-5.2.0-Linux-x86_64.sh)
6. Read the license

shaowei-su

Gauges are a constant data type. They are not subject to averaging, and they don’t change unless you change them. That is, once you set a gauge value, it will be a flat line on the graph until you change it again.

Gauges are useful for things that are already averaged, or don’t need to reset periodically. System load, for example, could be graphed with a gauge. You might use incr to count the number of logins to a system, but a gauge to track how many active WebSocket connections you have.

shaowei-su

logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)