Diffbot Graph Transformer with FalkorDB

diffbot_graphtransformer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7f0b0c06-ee70-468c-8bf5-b023f9e5e0a2",
   "metadata": {},
   "source": [
    "# Diffbot Graph Transformer\n",
    "\n",
    "## Use case\n",
    "\n",
    "Text data often contain rich relationships and insights that can be useful for various analytics, recommendation engines, or knowledge management applications.\n",
    "\n",
    "Diffbot's NLP API allows for the extraction of entities, relationships, and semantic meaning from unstructured text data.\n",
    "\n",
    "By coupling Diffbot's NLP API with FalkorDB, a graph database, you can create powerful, dynamic graph structures based on the information extracted from text. These graph structures are fully queryable and can be integrated into various applications.\n",
    "\n",
    "This combination allows for use cases such as:\n",
    "\n",
    "* Building knowledge graphs from textual documents, websites, or social media feeds.\n",
    "* Generating recommendations based on semantic relationships in the data.\n",
    "* Creating advanced search features that understand the relationships between entities.\n",
    "* Building analytics dashboards that allow users to explore the hidden relationships in data.\n",
    "\n",
    "## Overview\n",
    "\n",
    "LangChain provides tools to interact with Graph Databases:\n",
    "\n",
    "1. `Construct knowledge graphs from text` using graph transformer and store integrations \n",
    "2. `Query a graph database` using chains for query creation and execution\n",
    "3. `Interact with a graph database` using agents for robust and flexible querying \n",
    "\n",
    "## Quickstart\n",
    "\n",
    "First, get required packages and set environment variables:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "975648da-b24f-4164-a671-6772179e12df",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install --break-system-packages langchain langchain-experimental openai wikipedia redis"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77718977-629e-46c2-b091-f9191b9ec569",
   "metadata": {},
   "source": [
    "## Diffbot NLP Service\n",
    "\n",
    "Diffbot's NLP service is a tool for extracting entities, relationships, and semantic context from unstructured text data.\n",
    "This extracted information can be used to construct a knowledge graph.\n",
    "To use their service, you'll need to obtain an API key from [Diffbot](https://www.diffbot.com/products/natural-language/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "2cbf97d0-3682-439b-8750-b695ff726789",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_experimental.graph_transformers.diffbot import DiffbotGraphTransformer\n",
    "\n",
    "diffbot_api_key = \"DIFFBOT_API_KEY\"\n",
    "diffbot_nlp = DiffbotGraphTransformer(diffbot_api_key=diffbot_api_key)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e3b894a-e3ee-46c7-8116-f8377f8f0159",
   "metadata": {},
   "source": [
    "This code fetches Wikipedia articles about \"Warren Buffett\" and then uses `DiffbotGraphTransformer` to extract entities and relationships.\n",
    "The `DiffbotGraphTransformer` outputs a structured data `GraphDocument`, which can be used to populate a graph database.\n",
    "Note that text chunking is avoided due to Diffbot's [character limit per API request](https://docs.diffbot.com/reference/introduction-to-natural-language-api)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "53f8df86-47a1-44a1-9a0f-6725b90703bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.document_loaders import WikipediaLoader\n",
    "\n",
    "query = \"Warren Buffett\"\n",
    "raw_documents = WikipediaLoader(query=query).load()\n",
    "graph_documents = diffbot_nlp.convert_to_graph_documents(raw_documents)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31bb851a-aab4-4b97-a6b7-fce397d32b47",
   "metadata": {},
   "source": [
    "## Loading the data into a knowledge graph\n",
    "\n",
    "You will need to have a running FalkorDB instance. You can run a local docker container by running the executing the following script:\n",
    "\n",
    "```bash\n",
    "docker run -p 6379:6379 -p 7687:7687 falkordb/falkordb\n",
    "```    \n",
    "\n",
    "If you are using the docker container, you need to wait a couple of second for the database to start."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "0b2b6641-5a5d-467c-b148-e6aad5e4baa7",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.graphs import FalkorDBGraph\n",
    "\n",
    "graph = FalkorDBGraph(\n",
    "    \"falkordb\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b15e840-fe6f-45db-9193-1b4e2df5c12c",
   "metadata": {},
   "source": [
    "The `GraphDocuments` can be loaded into a knowledge graph using the `add_graph_documents` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "1a67c4a8-955c-42a2-9c5d-de3ac0e640ec",
   "metadata": {},
   "outputs": [],
   "source": [
    "graph.add_graph_documents(graph_documents)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed411e05-2b03-460d-997e-938482774f40",
   "metadata": {},
   "source": [
    "## Refresh graph schema information\n",
    "If the schema of database changes, you can refresh the schema information needed to generate Cypher statements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "904c9ee3-787c-403f-857d-459ce5ad5a1b",
   "metadata": {},
   "outputs": [],
   "source": [
    "graph.refresh_schema()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f19d1387-5899-4258-8c94-8ef5fa7db464",
   "metadata": {},
   "source": [
    "## Querying the graph\n",
    "We can now use the graph cypher QA chain to ask question of the graph. It is advisable to use **gpt-4** to construct Cypher queries to get the best experience."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "9393b732-67c8-45c1-9ec2-089f49c62448",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.chains import GraphCypherQAChain\n",
    "from langchain.chat_models import ChatOpenAI\n",
    "\n",
    "chain = GraphCypherQAChain.from_llm(\n",
    "    cypher_llm=ChatOpenAI(temperature=0, model_name=\"gpt-4\"),\n",
    "    qa_llm=ChatOpenAI(temperature=0, model_name=\"gpt-3.5-turbo\"),\n",
    "    graph=graph, verbose=True,\n",
    "    \n",
    ")\n",
    "     "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "1a9b3652-b436-404d-aa25-5fb576f23dc0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\u001b[1m> Entering new GraphCypherQAChain chain...\u001b[0m\n",
      "Generated Cypher:\n",
      "\u001b[32;1m\u001b[1;3mMATCH (p:Person {name: \"Warren Buffett\"})-[:EDUCATED_AT]->(o:Organization)\n",
      "RETURN o.name\u001b[0m\n",
      "Full Context:\n",
      "\u001b[32;1m\u001b[1;3m[['Woodrow Wilson High School'], ['Alice Deal Junior High School'], ['New York Institute of Finance']]\u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'Warren Buffett attended the Woodrow Wilson High School, Alice Deal Junior High School, and the New York Institute of Finance.'"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chain.run(\"Which university did Warren Buffett attend?\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "adc0ba0f-a62c-4875-89ce-da717f3ab148",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\u001b[1m> Entering new GraphCypherQAChain chain...\u001b[0m\n",
      "Generated Cypher:\n",
      "\u001b[32;1m\u001b[1;3mMATCH (p:Person)-[r:EMPLOYEE_OR_MEMBER_OF]->(o:Organization) WHERE o.name = 'Berkshire Hathaway' RETURN p.name\u001b[0m\n",
      "Full Context:\n",
      "\u001b[32;1m\u001b[1;3m[['Warren Buffett'], ['Charlie Munger'], ['Howard Buffett'], ['Susan Buffett'], ['Howard'], ['Oliver Chace']]\u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'Warren Buffett, Charlie Munger, Howard Buffett, Susan Buffett, Howard, and Oliver Chace are or were working at Berkshire Hathaway.'"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chain.run(\"Who is or was working at Berkshire Hathaway?\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}