kstubs
eliasdabbas
/ get_bot_ip_addresses.py
Last active
June 28, 2024 09:45
Get the most up-to-date list of IP addresses for crawler bots, belonging to Google and Bing.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import ipaddress | |
| import requests | |
| import pandas as pd | |
| def bot_ip_addresses(): | |
| bots_urls = { | |
| 'google': 'https://developers.google.com/search/apis/ipranges/googlebot.json', | |
| 'bing': 'https://www.bing.com/toolbox/bingbot.json' | |
| } |