This tutorial is based on this AWS tutorial. In this tutorial, we will install Nvidia driver on AWS EC2 instance and compile and run llama.cpp on it.
Here we use g5.4xlarge
instance with Ubuntu 22.04 AMI, which use Nvidia A10G GPU.
#!/usr/bin/env python3 | |
import json | |
import logging | |
from openai import OpenAI | |
logging.basicConfig(format="[%(asctime)s] [%(levelname)s] [%(name)s] %(message)s") | |
logger = logging.getLogger(__name__) | |
logger.setLevel(logging.INFO) |
export WASI_VERSION=20
export WASI_VERSION_FULL=${WASI_VERSION}.0
wget https://github.com/WebAssembly/wasi-sdk/releases/download/wasi-sdk-${WASI_VERSION}/wasi-sdk-${WASI_VERSION_FULL}-linux.tar.gz
tar xvf wasi-sdk-${WASI_VERSION_FULL}-linux.tar.gz
export WASI_SDK_PATH=`pwd`/wasi-sdk-${WASI_VERSION_FULL}
I hereby claim:
To claim this, I am signing this object:
### Keybase proof | |
I hereby claim: | |
* I am dm4 on github. | |
* I am dm4 (https://keybase.io/dm4) on keybase. | |
* I have a public key whose fingerprint is CD91 D480 881A A445 46BA 8A25 2846 33CF 9C3D CF17 | |
To claim this, I am signing this object: |
{ | |
"100": "1f4af", | |
"1234": "1f522", | |
"interrobang": "2049", | |
"tm": "2122", | |
"information_source": "2139", | |
"left_right_arrow": "2194", | |
"arrow_up_down": "2195", | |
"arrow_upper_left": "2196", | |
"arrow_upper_right": "2197", |
I hereby claim:
To claim this, I am signing this object:
pf's note (too old)