ryanavella/gpt2-windows10.md

## gpt2-windows10.md

      
    Raw
  

              gpt2-windows10.md
            
          
    GPT-2 on Windows 10

Installation from Archived Repository

Install miniconda3-4.5.4, then run the following:
git clone https://github.com/openai/gpt-2.git
cd gpt-2
C:\path\to\miniconda\python.exe -m venv .venv
.venv\Scripts\activate.bat
pip install --upgrade pip
pip install tensorflow-gpu==1.15.0
pip install -r requirements.txt
python download_model.py 124M
python download_model.py 355M
python download_model.py 774M
python download_model.py 1558M
Installation from TensorFlow 2 Fork

Install Python 3.7+.
Install the Visual C++ Build Tools.
Install CUDA.
Install Cudnn, then run the following:
git clone https://github.com/ryanavella/gpt-2.git
cd gpt-2
C:\path\to\python.exe -m venv .venv
.venv\Scripts\activate.bat
pip install --upgrade pip
pip install tensorflow==2.10.1
pip install tensorflow-gpu==2.10.1
pip install -r requirements.txt
python download_model.py 124M
python download_model.py 355M
python download_model.py 774M
python download_model.py 1558M
Installation from TensorFlow 2 Fork, with GPU Support (unverified)

Install Python 3.7+.
Install the Visual C++ Build Tools.
Install CUDA 11.2.2.
Install Cudnn 	8.1, then run the following:
git clone https://github.com/ryanavella/gpt-2.git
cd gpt-2
C:\path\to\python.exe -m venv .venv
.venv\Scripts\activate.bat
pip install --upgrade pip
pip install tensorflow==2.10.1
pip install tensorflow-gpu==2.10.1
pip install -r requirements.txt
python download_model.py 124M
python download_model.py 355M
python download_model.py 774M
python download_model.py 1558M
Unconditional Sample Generation

set PYTHONIOENCODING=UTF-8
python src/generate_unconditional_samples.py --nsamples=1
Conditional Sample Generation

set PYTHONIOENCODING=UTF-8
python src\interactive_conditional_samples.py --nsamples=1
To Do

Arch-Specific Optimizations

Determine which compiler flags to provide when compiling TensorFlow to enable SSE, SSE2, SSE3, SSE4.1, SSE4.2, AVX, AVX2, and FMA. Will probably need to compile from source with Bazel and provide the MSVC equivalent to -march=native?
Warnings

Unconditional sample generation in the TensorFlow 2 compatible fork still gives the following warnings:
W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cudart64_110.dll'; dlerror: cudart64_110.dll not found
I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cudart64_110.dll'; dlerror: cudart64_110.dll not found
W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cublas64_11.dll'; dlerror: cublas64_11.dll not found
W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cublasLt64_11.dll'; dlerror: cublasLt64_11.dll not found
W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cufft64_10.dll'; dlerror: cufft64_10.dll not found
W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'curand64_10.dll'; dlerror: curand64_10.dll not found
W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cusolver64_11.dll'; dlerror: cusolver64_11.dll not found
W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cusparse64_11.dll'; dlerror: cusparse64_11.dll not found
W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cudnn64_8.dll'; dlerror: cudnn64_8.dll not found
W tensorflow/core/common_runtime/gpu/gpu_device.cc:1934] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING:tensorflow:From C:\...\gpt-2\src\sample.py:79: calling while_loop_v2 (from tensorflow.python.ops.control_flow_ops) with back_prop=False is deprecated and will be removed in a future version.
Instructions for updating:
back_prop=False is deprecated. Consider using tf.stop_gradient instead.
Instead of:
results = tf.while_loop(c, b, vars, back_prop=False)
Use:
results = tf.nest.map_structure(tf.stop_gradient, tf.while_loop(c, b, vars))
I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:354] MLIR V1 optimization pass is not enabled

GPU Support

May need to follow instructions here:

https://docs.nvidia.com/deeplearning/cudnn/install-guide/index.html#installwindows