Skip to content

Instantly share code, notes, and snippets.

@Mistobaan

Mistobaan/langchain_llamacpp_natural_functions.ipynb

Last active March 19, 2024 22:08
Show Gist options
  • Save Mistobaan/e44df41cd574c2f1a1023311c2b9defd to your computer and use it in GitHub Desktop.
Save Mistobaan/e44df41cd574c2f1a1023311c2b9defd to your computer and use it in GitHub Desktop.
Download ZIP
langchain_llamacpp_natural_functions.ipynb
Display the source blob
Display the rendered blob
Raw
langchain_llamacpp_natural_functions.ipynb
{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"colab": {
"provenance": [],
"gpuType": "T4",
"authorship_tag": "ABX9TyPabYMUNrPUaY39hQ0DeaDe",
"include_colab_link": true
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
},
"language_info": {
"name": "python"
},
"accelerator": "GPU"
},
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "view-in-github",
"colab_type": "text"
},
"source": [
"<a href=\"https://colab.research.google.com/gist/Mistobaan/e44df41cd574c2f1a1023311c2b9defd/langchain_llamacpp_natural_functions.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
"cell_type": "markdown",
"source": [
"# Overview\n",
"\n",
"Example on how to use natural functions with [langchain](https://www.langchain.com/) and [LLamaCPP](https://github.com/ggerganov/llama.cpp).\n",
"\n",
"Model: [natural-functions](https://huggingface.co/cfahlgren1/natural-functions) by [caleb](https://twitter.com/calebfahlgren)\n",
"\n",
"Author: [@fabmilo](http://twitter.com/fabmilo)\n",
"\n",
"check script from X thread: https://twitter.com/SebastianMaki/status/1752471319612936385"
],
"metadata": {
"id": "glmAGbjM4eH9"
}
},
{
"cell_type": "code",
"source": [
"%pip install langchain_experimental hf_transfer huggingface-cli -q"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "xxe4X-Vy3rDU",
"outputId": "495d5982-4a4d-4209-c5c1-31caca329862"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m165.7/165.7 kB\u001b[0m \u001b[31m3.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m4.5/4.5 MB\u001b[0m \u001b[31m14.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m803.6/803.6 kB\u001b[0m \u001b[31m10.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m235.9/235.9 kB\u001b[0m \u001b[31m20.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.6/1.6 MB\u001b[0m \u001b[31m14.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m54.0/54.0 kB\u001b[0m \u001b[31m1.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m49.4/49.4 kB\u001b[0m \u001b[31m2.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25h"
]
}
]
},
{
"cell_type": "code",
"source": [
"import sys\n",
"! CMAKE_ARGS=\"-DLLAMA_CUBLAS=on\" FORCE_CMAKE=1 {sys.executable} -m pip install llama-cpp-python"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "KqSzVNnx7psk",
"outputId": "f7c2d1a9-3bd7-4adc-f8a1-15a31753fcf7"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Collecting llama-cpp-python\n",
" Downloading llama_cpp_python-0.2.37.tar.gz (10.8 MB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m10.8/10.8 MB\u001b[0m \u001b[31m27.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25h Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
" Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
" Installing backend dependencies ... \u001b[?25l\u001b[?25hdone\n",
" Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
"Requirement already satisfied: typing-extensions>=4.5.0 in /usr/local/lib/python3.10/dist-packages (from llama-cpp-python) (4.5.0)\n",
"Requirement already satisfied: numpy>=1.20.0 in /usr/local/lib/python3.10/dist-packages (from llama-cpp-python) (1.23.5)\n",
"Requirement already satisfied: diskcache>=5.6.1 in /usr/local/lib/python3.10/dist-packages (from llama-cpp-python) (5.6.3)\n",
"Requirement already satisfied: jinja2>=2.11.3 in /usr/local/lib/python3.10/dist-packages (from llama-cpp-python) (3.1.3)\n",
"Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2>=2.11.3->llama-cpp-python) (2.1.4)\n",
"Building wheels for collected packages: llama-cpp-python\n",
" Building wheel for llama-cpp-python (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
" Created wheel for llama-cpp-python: filename=llama_cpp_python-0.2.37-cp310-cp310-manylinux_2_35_x86_64.whl size=9564584 sha256=3bab01aa47bdfcebab9821ad0209d85c1633225e3a6dd65e9ca4018fb0d2a321\n",
" Stored in directory: /root/.cache/pip/wheels/5b/4f/36/1df917d3d21e5d04ce960616a6fb9e39c1548149620528a67c\n",
"Successfully built llama-cpp-python\n",
"Installing collected packages: llama-cpp-python\n",
"Successfully installed llama-cpp-python-0.2.37\n"
]
}
]
},
{
"cell_type": "code",
"source": [
"!nvidia-smi"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "zEG5JS5j4_-R",
"outputId": "d9c18b38-0762-4fed-c503-a8ebdc369203"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Wed Jan 31 01:45:47 2024 \n",
"+---------------------------------------------------------------------------------------+\n",
"| NVIDIA-SMI 535.104.05 Driver Version: 535.104.05 CUDA Version: 12.2 |\n",
"|-----------------------------------------+----------------------+----------------------+\n",
"| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |\n",
"| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |\n",
"| | | MIG M. |\n",
"|=========================================+======================+======================|\n",
"| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 |\n",
"| N/A 71C P0 31W / 70W | 2631MiB / 15360MiB | 0% Default |\n",
"| | | N/A |\n",
"+-----------------------------------------+----------------------+----------------------+\n",
" \n",
"+---------------------------------------------------------------------------------------+\n",
"| Processes: |\n",
"| GPU GI CI PID Type Process name GPU Memory |\n",
"| ID ID Usage |\n",
"|=======================================================================================|\n",
"+---------------------------------------------------------------------------------------+\n"
]
}
]
},
{
"cell_type": "code",
"source": [
"#@title download model\n",
"import os\n",
"model_path = '/content/natural-functions'\n",
"model_file = 'natural-functions.Q4_1.gguf'\n",
"if not os.path.exists(os.path.join(model_path, model_file)):\n",
" !HF_HUB_ENABLE_HF_TRANSFER=1 huggingface-cli download \\\n",
" \"cfahlgren1/natural-functions-GGUF\" \\\n",
" {model_file} \\\n",
" --local-dir {model_path} \\\n",
" --local-dir-use-symlinks False"
],
"metadata": {
"id": "S6caVcw63xzQ"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"from langchain.callbacks.manager import CallbackManager\n",
"from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler\n",
"from langchain_community.llms import LlamaCpp"
],
"metadata": {
"id": "dYwVvf_o3vMR"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# download JSON grammar\n",
"grammar_path='/content/json_grammar.gbnf'\n",
"if not os.path.exists(grammar_path):\n",
" import requests\n",
" json_grammar_text = requests.get(\"https://raw.githubusercontent.com/ggerganov/llama.cpp/master/grammars/json_arr.gbnf\").text\n",
" with open(grammar_path, 'w') as fd:\n",
" fd.write(json_grammar_text)"
],
"metadata": {
"id": "4Ebly6fECumC"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"import os\n",
"# Callbacks support token-wise streaming\n",
"callback_manager = CallbackManager([StreamingStdOutCallbackHandler()])\n",
"\n",
"llm = LlamaCpp(\n",
" model_path=os.path.join(model_path, model_file),\n",
" temperature=0.1,\n",
" max_tokens=2000,\n",
" n_gpu_layers=-1,\n",
" top_p=0.9,\n",
" callback_manager=callback_manager,\n",
" verbose=True, # Verbose is required to pass to the callback manager\n",
" grammar_path=grammar_path\n",
")"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "8I0ndMf75FTL",
"outputId": "801cb103-043f-4517-e5ba-bcce342b1f3e"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stderr",
"text": [
"AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | \n",
"Model metadata: {'tokenizer.ggml.add_eos_token': 'false', 'tokenizer.ggml.unknown_token_id': '0', 'tokenizer.ggml.eos_token_id': '2', 'general.architecture': 'llama', 'llama.rope.freq_base': '10000.000000', 'llama.context_length': '32768', 'general.name': '.', 'tokenizer.ggml.add_bos_token': 'true', 'llama.embedding_length': '4096', 'llama.feed_forward_length': '14336', 'llama.attention.layer_norm_rms_epsilon': '0.000010', 'llama.rope.dimension_count': '128', 'tokenizer.ggml.bos_token_id': '1', 'llama.attention.head_count': '32', 'llama.block_count': '32', 'llama.attention.head_count_kv': '8', 'general.quantization_version': '2', 'tokenizer.ggml.model': 'llama', 'general.file_type': '3'}\n",
"from_string grammar:\n",
"root ::= arr \n",
"arr ::= [[] [<U+000A>] ws arr_12 []] \n",
"value ::= object | array | string | number | value_7 ws \n",
"object ::= [{] ws object_16 [}] ws \n",
"array ::= [[] ws array_20 []] ws \n",
"string ::= [\"] string_23 [\"] ws \n",
"number ::= number_24 number_30 number_34 ws \n",
"value_7 ::= [t] [r] [u] [e] | [f] [a] [l] [s] [e] | [n] [u] [l] [l] \n",
"ws ::= ws_36 \n",
"arr_9 ::= value arr_11 \n",
"arr_10 ::= [,] [<U+000A>] ws value \n",
"arr_11 ::= arr_10 arr_11 | \n",
"arr_12 ::= arr_9 | \n",
"object_13 ::= string [:] ws value object_15 \n",
"object_14 ::= [,] ws string [:] ws value \n",
"object_15 ::= object_14 object_15 | \n",
"object_16 ::= object_13 | \n",
"array_17 ::= value array_19 \n",
"array_18 ::= [,] ws value \n",
"array_19 ::= array_18 array_19 | \n",
"array_20 ::= array_17 | \n",
"string_21 ::= [^\"\\] | [\\] string_22 \n",
"string_22 ::= [\"\\/bfnrt] | [u] [0-9a-fA-F] [0-9a-fA-F] [0-9a-fA-F] [0-9a-fA-F] \n",
"string_23 ::= string_21 string_23 | \n",
"number_24 ::= number_25 number_26 \n",
"number_25 ::= [-] | \n",
"number_26 ::= [0-9] | [1-9] number_27 \n",
"number_27 ::= [0-9] number_27 | \n",
"number_28 ::= [.] number_29 \n",
"number_29 ::= [0-9] number_29 | [0-9] \n",
"number_30 ::= number_28 | \n",
"number_31 ::= [eE] number_32 number_33 \n",
"number_32 ::= [-+] | \n",
"number_33 ::= [0-9] number_33 | [0-9] \n",
"number_34 ::= number_31 | \n",
"ws_35 ::= [ <U+0009><U+000A>] ws \n",
"ws_36 ::= ws_35 | \n",
"\n"
]
}
]
},
{
"cell_type": "code",
"source": [
"DEFAULT_SYSTEM_TEMPLATE = \"\"\"You are a helpful assistant with access to the following functions. Use them if required -\n",
"\n",
"{tools}\n",
"\n",
"You must always select one of the above tools and respond with only a JSON object matching the following schema:\n",
"\n",
"{{\n",
" \"tool\": <name of the selected tool>,\n",
" \"tool_input\": <parameters for the selected tool, matching the tool's JSON schema>\n",
"}}\n",
"\"\"\""
],
"metadata": {
"id": "sxkNKRBb_hhV"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"from langchain_core.prompts import ChatPromptTemplate, SystemMessagePromptTemplate, HumanMessagePromptTemplate\n",
"\n",
"system_message_template = SystemMessagePromptTemplate.from_template(DEFAULT_SYSTEM_TEMPLATE)\n",
"human_template = HumanMessagePromptTemplate.from_template('{question}')\n",
"prompt = ChatPromptTemplate.from_messages([system_message_template, human_template])\n",
"\n",
"chain = prompt | llm"
],
"metadata": {
"id": "IOpcKvPZIsEd"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "TY31PzC53led"
},
"outputs": [],
"source": [
"light_state = {\n",
" \"name\": \"set_light_state\",\n",
" \"description\": \"Turn a light on or off and sets it to a given color and brightness\",\n",
" \"parameters\": {\n",
" \"type\": \"object\",\n",
" \"properties\": {\n",
" \"device\": {\n",
" \"type\": \"string\",\n",
" \"description\": \"The name of the light\"\n",
" },\n",
" \"state\": {\n",
" \"type\": \"string\",\n",
" \"enum\": [\"on\", \"off\"]\n",
" },\n",
" \"brightness\": {\n",
" \"type\": \"string\",\n",
" \"enum\": [\"low\", \"medium\", \"high\"]\n",
" },\n",
" \"color\": {\n",
" \"type\": \"string\",\n",
" \"enum\": [\"red\", \"white\", \"blue\", \"green\", \"yellow\", \"purple\", \"orange\", \"pink\", \"cyan\", \"magenta\", \"lime\", \"indigo\", \"teal\", \"olive\", \"brown\", \"black\", \"grey\", \"silver\", \"gold\", \"bronze\", \"platinum\", \"rainbow\"]\n",
" }\n",
" },\n",
" \"required\": [\"device\"]\n",
" }\n",
"}\n",
"\n",
"create_event = {\n",
" \"name\": \"create_event\",\n",
" \"description\": \"Create a new event in the calendar\",\n",
" \"parameters\": {\n",
" \"type\": \"object\",\n",
" \"properties\": {\n",
" \"title\": {\n",
" \"type\": \"string\",\n",
" \"description\": \"The title of the event\"\n",
" },\n",
" \"date\": {\n",
" \"type\": \"string\",\n",
" \"format\": \"date\",\n",
" \"description\": \"The date of the event\"\n",
" },\n",
" \"time\": {\n",
" \"type\": \"string\",\n",
" \"format\": \"time\",\n",
" \"description\": \"The time of the event\"\n",
" },\n",
" \"location\": {\n",
" \"type\": \"string\",\n",
" \"description\": \"The location of the event (optional)\"\n",
" }\n",
" },\n",
" \"required\": [\n",
" \"title\",\n",
" \"date\",\n",
" \"time\"\n",
" ]\n",
" }\n",
"}"
]
},
{
"cell_type": "code",
"source": [
"result = chain.invoke({\"question\": \"can you create a dinner event for friday at 10?\", \"tools\": [light_state, create_event]})"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "tENKpWjb91WD",
"outputId": "cf087ba7-cffd-4577-ce4d-8c60367a6ad7"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"[\n",
"{\"name\": \"create_event\", \"arguments\": {\n",
" \"title\": \"Dinner Event\",\n",
" \"date\": \"2023-05-19\",\n",
" \"time\": \"10:00\"\n",
"}}]"
]
}
]
},
{
"cell_type": "code",
"source": [
"import json\n",
"values = json.loads(result)\n",
"values"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "ePZ5yQrdD0Ci",
"outputId": "221f0f2c-67a3-4c9c-a1bf-9a3473079533"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"[{'name': 'create_event',\n",
" 'arguments': {'title': 'Dinner Event',\n",
" 'date': '2023-05-19',\n",
" 'time': '10:00'}}]"
]
},
"metadata": {},
"execution_count": 16
}
]
},
{
"cell_type": "code",
"source": [],
"metadata": {
"id": "7jKRP1OkOjt_"
},
"execution_count": null,
"outputs": []
}
]
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment