# how to build customized kuberay:V1.2.1
# environment
docker buildx create --use --name multiarch-builder --driver docker-container
docker buildx inspect --bootstrap
docker buildx use multiarch-builder
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
apiVersion: orchestration.aibrix.ai/v1alpha1 | |
kind: RayClusterFleet | |
metadata: | |
labels: | |
app.kubernetes.io/name: aibrix | |
model.aibrix.ai/name: deepseek-r1-671b | |
model.aibrix.ai/port: "8000" | |
name: deepseek-r1-671b | |
spec: | |
replicas: 1 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
apiVersion: orchestration.aibrix.ai/v1alpha1 | |
kind: RayClusterFleet | |
metadata: | |
labels: | |
app.kubernetes.io/name: aibrix | |
model.aibrix.ai/name: deepseek-r1-671b | |
model.aibrix.ai/port: "8000" | |
name: deepseek-r1-671b | |
spec: | |
replicas: 1 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
``` | |
apiVersion: orchestration.aibrix.ai/v1alpha1 | |
kind: RayClusterFleet | |
metadata: | |
labels: | |
app.kubernetes.io/name: aibrix | |
model.aibrix.ai/name: deepseek-r1-671b | |
name: deepseek-r1-671b | |
spec: | |
replicas: 1 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
``` | |
python3 -m vllm.entrypoints.openai.api_server --host "0.0.0.0" --port "8000" --model deepseek-ai/deepseek-coder-6.7b-instruct --max-model-len 32768 | |
``` | |
``` | |
git clone https://github.com/vllm-project/vllm.git | |
``` | |
``` | |
python3 benchmark_serving.py --backend openai --base-url http://127.0.0.1:8000 --dataset-name=random --model deepseek-ai/deepseek-coder-6.7b-instruct --seed 12345 |
Note: While it definitely works with RayCluster, using RayJob is much easier.
FROM vllm/vllm-openai:v0.5.2
RUN apt update && apt install -y wget # important for future healthcheck
RUN pip3 install ray[default] # important for future healthcheck
We can make this file beautiful and searchable if this error is corrected: It looks like row 3 should actually have 1 column, instead of 3 in line 2.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# Format | |
# If line starts with a '#' it is considered a comment | |
# DCGM FIELD, Prometheus metric type, help message | |
# Clocks | |
DCGM_FI_DEV_SM_CLOCK, gauge, SM clock frequency (in MHz). | |
DCGM_FI_DEV_MEM_CLOCK, gauge, Memory clock frequency (in MHz). | |
# Temperature | |
DCGM_FI_DEV_MEMORY_TEMP, gauge, Memory temperature (in C). |
- Make sure you have
ray-system
namespace. if not, kubectl create ns ray-system - kubectl create -f ray.io_workspaces.yaml
- kubectl apply -f workspace-operator.yaml
- Create a jupyter notebook. kubectl apply -f ray.io_v1alpha1_workspace.yaml
- Use the nodeport or port-forward the service. Then open browser
nodeip:nodeport/kuberay/workspace
.
Note: operator image and jupyter image can be used directly. I upload to my personal dockerhub. I will try to finish OSS process soon.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import os | |
import ray | |
import raydp | |
HEAD_SERVICE_IP_ENV = "EXAMPLE_CLUSTER_RAY_HEAD_SERVICE_HOST" | |
head_service_ip = os.environ[HEAD_SERVICE_IP_ENV] | |
ray.init(address=f"{head_service_ip}:6379") |
NewerOlder