Working with Google Custom Search Engine using Python

gcse.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Google Custom Search API\n",
    "\n",
    "https://developers.google.com/custom-search/\n",
    "\n",
    "https://developers.google.com/custom-search/v1/overview\n",
    "\n",
    "![](https://cdn-images-1.medium.com/max/1600/1*GljTlf_8C-3eZwwncRmlEw.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Create a project on Google Developers Console\n",
    "\n",
    "https://console.developers.google.com\n",
    "\n",
    "\n",
    "## 2. Enable Custom Search API and generate API key\n",
    "\n",
    "\n",
    "## 3. Create a search engine\n",
    "\n",
    "https://cse.google.com/cse/all\n",
    "\n",
    "\n",
    "## 4. Installation\n",
    "\n",
    "```\n",
    "pip install google-api-python-client\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "api_key = \"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from apiclient.discovery import build"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "resource = build(\"customsearch\", 'v1', developerKey=api_key).cse()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "result = resource.list(q='python', cx='009557628044748784875:5lejfe73wrw').execute()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'kind': 'customsearch#result',\n",
       " 'title': 'Internal working of Python - GeeksforGeeks',\n",
       " 'htmlTitle': 'Internal working of <b>Python</b> - GeeksforGeeks',\n",
       " 'link': 'https://cdncontribute.geeksforgeeks.org/wp-content/uploads/python_working.png',\n",
       " 'displayLink': 'www.geeksforgeeks.org',\n",
       " 'snippet': 'Internal working of Python - GeeksforGeeks',\n",
       " 'htmlSnippet': 'Internal working of <b>Python</b> - GeeksforGeeks',\n",
       " 'mime': 'image/png',\n",
       " 'image': {'contextLink': 'https://www.geeksforgeeks.org/internal-working-of-python/',\n",
       "  'height': 371,\n",
       "  'width': 846,\n",
       "  'byteSize': 12937,\n",
       "  'thumbnailLink': 'https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcTMR1g0292SKbY-iXUA6ijVFrKgp4yvSiBkOH3j8B5faYCBAjdIkCOL3RVk',\n",
       "  'thumbnailHeight': 64,\n",
       "  'thumbnailWidth': 145}}"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result['items'][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(result['items'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Python Programming Language - GeeksforGeeks https://www.geeksforgeeks.org/python-programming-language/\n",
      "Python Language Introduction - GeeksforGeeks https://www.geeksforgeeks.org/python-language-introduction/\n",
      "Taking multiple inputs from user in Python - GeeksforGeeks https://www.geeksforgeeks.org/taking-multiple-inputs-from-user-in-python/\n",
      "Taking input in Python - GeeksforGeeks https://www.geeksforgeeks.org/taking-input-in-python/\n",
      "Python program to add two numbers - GeeksforGeeks https://www.geeksforgeeks.org/python-program-to-add-two-numbers/\n",
      "Statement, Indentation and Comment in Python - GeeksforGeeks https://www.geeksforgeeks.org/statement-indentation-and-comment-in-python/\n",
      "Python Programming Examples - GeeksforGeeks https://www.geeksforgeeks.org/python-programming-examples/\n",
      "Polymorphism in Python - GeeksforGeeks https://www.geeksforgeeks.org/polymorphism-in-python/\n",
      "Namespaces and Scope in Python - GeeksforGeeks https://www.geeksforgeeks.org/namespaces-and-scope-in-python/\n",
      "Python 3 basics - GeeksforGeeks https://www.geeksforgeeks.org/python-3-basics/\n"
     ]
    }
   ],
   "source": [
    "for item in result['items']:\n",
    "    print(item['title'], item['link'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Image search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "result = resource.list(q='python', cx='009557628044748784875:5lejfe73wrw', searchType='image').execute()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Suresh Kumar - Quora https://qph.fs.quoracdn.net/main-raw-350993202-zvusdjmacabgcydlxszxcwhhtjrarrdk.jpeg\n",
      "What does the Python logo stand for? - Quora https://qph.fs.quoracdn.net/main-qimg-28cadbd02699c25a88e5c78d73c7babc\n",
      "What is Python primarily used for? - Quora https://qph.fs.quoracdn.net/main-qimg-ff06c339780f23112c1cbbbb284158ff\n",
      "Python Django | Google authentication and Fetching mails from ... https://cdncontribute.geeksforgeeks.org/wp-content/uploads/django-google.png\n",
      "Implementation of Dynamic Array in Python - GeeksforGeeks https://cdncontribute.geeksforgeeks.org/wp-content/uploads/daynamic-array.png\n",
      "Can we use the Python programming language on Arduino? - Quora https://qph.fs.quoracdn.net/main-qimg-3cf229eedc092549277e8859aad2fca5\n",
      "Any & All in Python - GeeksforGeeks https://www.geeksforgeeks.org/wp-content/uploads/All-any-in-python.png\n",
      "Short Circuiting Techniques in Python - GeeksforGeeks http://cdncontribute.geeksforgeeks.org/wp-content/uploads/bool.png\n",
      "Where is Python mostly used? - Quora https://qph.fs.quoracdn.net/main-qimg-0613adfdf47a4691e947539c31be0247\n",
      "What are the pros and cons of Python? - Quora https://qph.fs.quoracdn.net/main-qimg-0b31556e89087f379dac3d482675987a\n"
     ]
    }
   ],
   "source": [
    "for item in result['items']:\n",
    "    print(item['title'], item['link'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pagination"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "images = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(1, 60, 10):\n",
    "    result = resource.list(q='python', cx='009557628044748784875:5lejfe73wrw',\n",
    "                      searchType='image', start=i).execute()\n",
    "    images += result['items']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "60"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(images)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Suresh Kumar - Quora https://qph.fs.quoracdn.net/main-raw-350993202-zvusdjmacabgcydlxszxcwhhtjrarrdk.jpeg\n",
      "What does the Python logo stand for? - Quora https://qph.fs.quoracdn.net/main-qimg-28cadbd02699c25a88e5c78d73c7babc\n",
      "What is Python primarily used for? - Quora https://qph.fs.quoracdn.net/main-qimg-ff06c339780f23112c1cbbbb284158ff\n",
      "Python Django | Google authentication and Fetching mails from ... https://cdncontribute.geeksforgeeks.org/wp-content/uploads/django-google.png\n",
      "Implementation of Dynamic Array in Python - GeeksforGeeks https://cdncontribute.geeksforgeeks.org/wp-content/uploads/daynamic-array.png\n",
      "Can we use the Python programming language on Arduino? - Quora https://qph.fs.quoracdn.net/main-qimg-3cf229eedc092549277e8859aad2fca5\n",
      "Any & All in Python - GeeksforGeeks https://www.geeksforgeeks.org/wp-content/uploads/All-any-in-python.png\n",
      "Short Circuiting Techniques in Python - GeeksforGeeks http://cdncontribute.geeksforgeeks.org/wp-content/uploads/bool.png\n",
      "Where is Python mostly used? - Quora https://qph.fs.quoracdn.net/main-qimg-0613adfdf47a4691e947539c31be0247\n",
      "What are the pros and cons of Python? - Quora https://qph.fs.quoracdn.net/main-qimg-0b31556e89087f379dac3d482675987a\n",
      "Can I make an Android app with Python? - Quora https://qph.fs.quoracdn.net/main-qimg-522be9c947f2ddec8713e8211a85e3a6\n",
      "Python 3 basics - GeeksforGeeks https://cdncontribute.geeksforgeeks.org/wp-content/uploads/Python_console.png\n",
      "How to learn Python for scripting - Quora https://qph.fs.quoracdn.net/main-qimg-0044b853c6516c030c17a17f4ee85153\n",
      "What are the advantages of using Python over other languages like ... https://qph.fs.quoracdn.net/main-qimg-7e9eb6a83fb8110e14b9cc06cd651f5b\n",
      "What is the best book to learn Python for data science? - Quora https://qph.fs.quoracdn.net/main-qimg-ccc07bd0e779be4c3fc9a3226f8f8409\n",
      "Create and Access a Python Package - GeeksforGeeks https://cdncontribute.geeksforgeeks.org/wp-content/uploads/Python-Packages.jpg\n",
      "Can you suggest me some good Python training institutes Bengaluru ... https://qph.fs.quoracdn.net/main-qimg-0821b15bd96a296f6dd1ee62341bdf53\n",
      "How good is it to learn Python with Django? - Quora https://qph.fs.quoracdn.net/main-qimg-7b0167d398e33e6075f975bf11d80e14\n",
      "Is it possible to learn Java AND Python in two months by myself ... https://qph.fs.quoracdn.net/main-qimg-e849c739b4c7a3ff50a892c7ab3da974\n",
      "Python | Print list after removing element at given index ... https://cdncontribute.geeksforgeeks.org/wp-content/uploads/WhatsApp-Image-2017-12-24-at-12.09.55-AM.jpeg\n",
      "Python | Find missing and additional values in two lists ... https://cdncontribute.geeksforgeeks.org/wp-content/uploads/WhatsApp-Image-2018-01-03-at-12.46.34-PM.jpeg\n",
      "Logging in Python - GeeksforGeeks https://cdncontribute.geeksforgeeks.org/wp-content/uploads/Python-log-levels.png\n",
      "Multithreading in Python | Set 1 - GeeksforGeeks https://cdncontribute.geeksforgeeks.org/wp-content/uploads/multithreading-python-11.png\n",
      "Why should I learn Python over other programming languages? - Quora https://qph.fs.quoracdn.net/main-qimg-05c5032f8e16855f0915e1b4621f0dbc\n",
      "Are pythons poisonous? - Quora https://qph.fs.quoracdn.net/main-qimg-1abbd0df1bce918455f5c83007a0ea81\n",
      "Python Desktop News Notifier in 20 lines - GeeksforGeeks http://cdncontribute.geeksforgeeks.org/wp-content/uploads/Screenshot-from-2017-04-24-10_00_06.png\n",
      "Are pythons poisonous? - Quora https://qph.fs.quoracdn.net/main-qimg-ae429dc47e62c045bf799a267ad7d594-c\n",
      "Simple Chat Room using Python - GeeksforGeeks http://cdncontribute.geeksforgeeks.org/wp-content/uploads/Screenshot-from-2017-04-18-18-31-03.png\n",
      "Taking multiple inputs from user in Python - GeeksforGeeks https://cdncontribute.geeksforgeeks.org/wp-content/uploads/Capture8-1.png\n",
      "Implementing Web Scraping in Python with BeautifulSoup - GeeksforGeeks https://indianpythonista.files.wordpress.com/2016/11/11.png\n",
      "What can Python do? - Quora https://qph.fs.quoracdn.net/main-qimg-48370fbb8d060920a2c3e20edfd37385\n",
      "What are the 10 best features of Python? - Quora https://qph.fs.quoracdn.net/main-qimg-8b1812786f86b6028e6e139dfeda4e7d\n",
      "Python | Pandas Series - GeeksforGeeks https://media.geeksforgeeks.org/wp-content/uploads/dataSER-1.png\n",
      "What are the pros and cons of learning Python as a first ... https://qph.fs.quoracdn.net/main-qimg-6972f0e767994131bd6b363a7730f2b8\n",
      "Twitter Sentiment Analysis using Python - GeeksforGeeks https://www.geeksforgeeks.org/wp-content/uploads/Twitter-Sentiment-Analysis-using-Python.jpg\n",
      "Coroutine in Python - GeeksforGeeks http://cdncontribute.geeksforgeeks.org/wp-content/uploads/subroutine.png\n",
      "Union() function in Python - GeeksforGeeks https://www.geeksforgeeks.org/wp-content/uploads/Union-in-python.jpg\n",
      "Keywords in Python | Set 1 - GeeksforGeeks https://media.geeksforgeeks.org/wp-content/uploads/and1.png\n",
      "What is the scope of Python in India? - Quora https://qph.fs.quoracdn.net/main-qimg-0ec877591c43dcc862ecd91d9ffefcd8\n",
      "Python | Iterate over multiple lists simultaneously - GeeksforGeeks https://cdncontribute.geeksforgeeks.org/wp-content/uploads/WhatsApp-Image-2017-12-30-at-2.28.37-AM-1.jpeg\n",
      "Python | Pandas Dataframe/Series.head() method - GeeksforGeeks https://cdncontribute.geeksforgeeks.org/wp-content/uploads/nba-1-1.png\n",
      "For building a website, which programming language is easier to ... https://qph.fs.quoracdn.net/main-qimg-b18fd8d57290e40a045f114f231c7fcc\n",
      "Namespaces and Scope in Python - GeeksforGeeks https://cdncontribute.geeksforgeeks.org/wp-content/uploads/types_namespace-1.png\n",
      "What are the best free Python IDEs on PC for a beginner? - Quora https://qph.fs.quoracdn.net/main-qimg-a216add584bb8638388db9a9006b9fd4\n",
      "Python | Check if all the values in a list that are greater than a ... https://cdncontribute.geeksforgeeks.org/wp-content/uploads/WhatsApp-Image-2017-12-23-at-6.50.28-PM-232x300.jpeg\n",
      "Intersection() function Python - GeeksforGeeks https://www.geeksforgeeks.org/wp-content/uploads/intersection-in-python.jpg\n",
      "Junk File Organizer in Python - GeeksforGeeks http://cdncontribute.geeksforgeeks.org/wp-content/uploads/1112.png\n",
      "os.walk() in Python - GeeksforGeeks https://contribute.geeksforgeeks.org/wp-content/uploads/osWalk-1.jpg\n",
      "Python | Django Admin Interface - GeeksforGeeks https://cdncontribute.geeksforgeeks.org/wp-content/uploads/Screenshot-534-1.png\n",
      "Multi-Messenger : A python project, messaging via Terminal ... https://www.geeksforgeeks.org/wp-content/uploads/multi_messenger_python1.png\n",
      "Internal working of Python - GeeksforGeeks https://cdncontribute.geeksforgeeks.org/wp-content/uploads/python_working.png\n",
      "Python 3 basics - GeeksforGeeks http://cdncontribute.geeksforgeeks.org/wp-content/uploads/Screenshot-from-2017-07-20-18-47-54.png\n",
      "Python | Numbers in a list within a given range - GeeksforGeeks https://cdncontribute.geeksforgeeks.org/wp-content/uploads/WhatsApp-Image-2017-12-23-at-11.01.00-PM-300x200.jpeg\n",
      "Multithreading in Python | Set 1 - GeeksforGeeks https://cdncontribute.geeksforgeeks.org/wp-content/uploads/multithreading-python-21.png\n",
      "What qualifications are required to learn Python? - Quora https://qph.fs.quoracdn.net/main-qimg-c46d66564ba2dfcf09095a8130092803\n",
      "What is negative index in Python? - Quora https://qph.fs.quoracdn.net/main-qimg-a380b1bc159589df5e0b9842e5b56b6d\n",
      "Python range() function - GeeksforGeeks https://cdncontribute.geeksforgeeks.org/wp-content/uploads/PythonRange6.png\n",
      "Are pythons poisonous? - Quora https://qph.fs.quoracdn.net/main-qimg-7531b169489c7d442b5a6cc0c3aa68e6\n",
      "What are the features of Python? - Quora https://qph.fs.quoracdn.net/main-qimg-ed18fef07c0cd9ebebb74daf147a6275\n",
      "Python Slicing | Reverse an array in groups of given size ... https://cdncontribute.geeksforgeeks.org/wp-content/uploads/geeks-4.png\n"
     ]
    }
   ],
   "source": [
    "for item in images:\n",
    "    print(item['title'], item['link'])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}

Sir I tried this code for www.google.com. But its showing the result for only one page. What should i do to show the result for 5 pages?

thanks for this post! really useful!

Dear Nikhil, Thanks for the post. It helped me in big time. The video post explains all the details in crisp and clear. Look forward to more videos in the same way. Appreciate it.

I would like to get the results of a google search with some parts of the text on the each homepages. Is that possible?

thanks for this post its clear and helpful.
but i would like to get the ads link. is that possible?

Sir I tried this code for www.google.com. But its showing the result for only one page. What should i do to show the result for 5 pages?

https://stackoverflow.com/questions/17427837/googlecustom-api-number-of-results