Skip to content

Instantly share code, notes, and snippets.

@john-adeojo

john-adeojo/langchain_vllm.py

Created January 30, 2024 13:02
Show Gist options
  • Save john-adeojo/560b72de0a710426fc2b6fbbff56af99 to your computer and use it in GitHub Desktop.
Save john-adeojo/560b72de0a710426fc2b6fbbff56af99 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
langchain_vllm.py
from langchain_community.llms import VLLMOpenAI
llm = VLLMOpenAI(
openai_api_key="EMPTY",
openai_api_base=vllm_endpoint,
model_path=model_path,
max_tokens=250,
temperature=0.5,
model_kwargs={"stop": ["\n\nHuman"]},
)
print(llm("What is AI?"))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment