kinoc

# a simple Flask API to emulate OpenAI's using llama models and/or transformers
# runs on 3080

import sys
import time
import torch
import json
from peft import PeftModel

from flask import Flask, make_response, request, abort

mcarilli

# This isn't supposed to run as a bash script, i named it with ".sh" for syntax highlighting.

# https://developer.nvidia.com/nsight-systems
# https://docs.nvidia.com/nsight-systems/profiling/index.html

# My preferred nsys (command line executable used to create profiles) commands
#
# In your script, write
# torch.cuda.nvtx.range_push("region name")
# ...

mcarilli

Ordinary launch commands (no profiling):

Single-process:

python main_amp.py -a resnet50 --b 224 --deterministic --workers 4 --opt-level O1 ./bare_metal_train_val/

Multi-process:

python -m torch.distributed.launch  --nproc_per_node=2 main_amp.py -a resnet50 --b 224 --deterministic --workers 4 --opt-level O1 ./bare_metal_train_val/

apokalyptik

The Goal

Set up Kubernetes on 3 Debian Jessie virtual machines: One master. Two nodes. Additionally do this without any "magic" so that what is required to be running to make everything work is plain and obvious.

We will be using flannel for the inter-machine networking layer. Mainly because it is useful and it seems to be pretty popular.

The Setup