Skip to content

Instantly share code, notes, and snippets.

@john-adeojo

john-adeojo/llama_index_vllm.py

Created January 30, 2024 13:05
Show Gist options
  • Save john-adeojo/5587d94d444eab25a343c3bc2ca7260d to your computer and use it in GitHub Desktop.
Save john-adeojo/5587d94d444eab25a343c3bc2ca7260d to your computer and use it in GitHub Desktop.
Download ZIP
Raw
llama_index_vllm.py
from llama_index.llms import OpenAILike
llm = OpenAILike(
api_key="EMPTY",
api_base=vllm_endpoint,
model = model_path ,
max_tokens=250,
stream=True
)
print(llm.complete("What is AI?"))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment