Last active
September 29, 2022 15:42
-
-
Save ph00lt0/c4a662c7a502026bd6800ea3705d687e to your computer and use it in GitHub Desktop.
Robots disallow
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| User-agent: archive.org_bot | |
| Disallow: / | |
| User-agent: ia_archiver | |
| Disallow: / | |
| User-agent: web collector | |
| Disallow: / | |
| User-agent: Sniper | |
| Disallow: / | |
| User-agent: sonic | |
| Disallow: / | |
| User-agent: eMailReaper | |
| Disallow: / | |
| User-agent: Email Magnet | |
| Disallow: / | |
| User-agent: Platinum 3 | |
| Disallow: / | |
| User-agent: Extractor | |
| Disallow: / | |
| User-agent: WebWeasel | |
| Disallow: / | |
| User-agent: Web Mole | |
| Disallow: / | |
| User-agent: ia_archiver/1.6 | |
| Disallow: / | |
| User-agent: Mediapartners-Google | |
| Disallow: / | |
| User-agent: Applebot | |
| Disallow: / | |
| User-agent: baiduspider | |
| Disallow: / | |
| Baiduspider-video | |
| Disallow: / | |
| Baiduspider-image | |
| Disallow: / | |
| User-agent: msnbot | |
| Disallow: / | |
| User-agent: Naverbot | |
| Disallow: / | |
| User-agent: seznambot | |
| Disallow: / | |
| User-agent: Slurp | |
| Disallow: / | |
| User-agent: teoma | |
| Disallow: / | |
| User-agent: Twitterbot | |
| Disallow: / | |
| User-agent: Yandex | |
| Disallow: / | |
| User-agent: Yeti | |
| Disallow: / | |
| User-agent: Teleport | |
| Disallow: / | |
| User-agent: TeleportPro | |
| Disallow: / | |
| User-agent: twiceler | |
| Disallow: / | |
| User-agent: Teoma | |
| Disallow: / | |
| User-agent: TinEye | |
| Disallow: / | |
| User-agent: looksmart | |
| Disallow: / | |
| User-agent: b2w/0.1 | |
| Disallow: / | |
| User-agent: psbot | |
| Disallow: / | |
| User-agent: Python-urllib | |
| Disallow: / | |
| User-agent: NetMechanic | |
| Disallow: / | |
| User-agent: URL_Spider_Pro | |
| Disallow: / | |
| User-agent: CherryPicker | |
| Disallow: / | |
| User-agent: EmailCollector | |
| Disallow: / | |
| User-agent: EmailSiphon | |
| Disallow: / | |
| User-agent: WebBandit | |
| Disallow: / | |
| User-agent: EmailWolf | |
| Disallow: / | |
| User-agent: ExtractorPro | |
| Disallow: / | |
| User-agent: CopyRightCheck | |
| Disallow: / | |
| User-agent: ProWebWalker | |
| Disallow: / | |
| User-agent: CheeseBot | |
| Disallow: / | |
| User-agent: Robozilla | |
| Disallow: / | |
| User-agent: Nutch | |
| Disallow: / | |
| User-agent: MIIxpc | |
| Disallow: / | |
| User-agent: Telesoft | |
| Disallow: / | |
| User-agent: Website Quester | |
| Disallow: / | |
| User-agent: moget/2.1 | |
| Disallow: / | |
| User-agent: WebZip/4.0 | |
| Disallow: / | |
| User-agent: WebSauger | |
| Disallow: / | |
| User-agent: NetAnts | |
| Disallow: / | |
| User-agent: Mister PiX | |
| Disallow: / | |
| User-agent: WebAuto | |
| Disallow: / | |
| User-agent: TheNomad | |
| Disallow: / | |
| User-agent: WWW-Collector-E | |
| Disallow: / | |
| User-agent: RMA | |
| Disallow: / | |
| User-agent: libWeb/clsHTTP | |
| Disallow: / | |
| User-agent: asterias | |
| Disallow: / | |
| User-agent: httplib | |
| Disallow: / | |
| User-agent: turingos | |
| Disallow: / | |
| User-agent: spanner | |
| Disallow: / | |
| User-agent: InfoNaviRobot | |
| Disallow: / | |
| User-agent: Harvest/1.5 | |
| Disallow: / | |
| User-agent: Bullseye/1.0 | |
| Disallow: / | |
| User-agent: Mozilla/4.0 (compatible; BullsEye; Windows 95) | |
| Disallow: / | |
| User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 | |
| Disallow: / | |
| User-agent: CherryPickerSE/1.0 | |
| Disallow: / | |
| User-agent: CherryPickerElite/1.0 | |
| Disallow: / | |
| User-agent: WebBandit/3.50 | |
| Disallow: / | |
| User-agent: NICErsPRO | |
| Disallow: / | |
| User-agent: Microsoft URL Control � 5.01.4511 | |
| Disallow: / | |
| User-agent: DittoSpyder | |
| Disallow: / | |
| User-agent: Foobot | |
| Disallow: / | |
| User-agent: WebmasterWorldForumBot | |
| Disallow: / | |
| User-agent: SpankBot | |
| Disallow: / | |
| User-agent: BotALot | |
| Disallow: / | |
| User-agent: lwp-trivial/1.34 | |
| Disallow: / | |
| User-agent: lwp-trivial | |
| Disallow: / | |
| User-agent: BunnySlippers | |
| Disallow: / | |
| User-agent: Microsoft URL Control � 6.00.8169 | |
| Disallow: / | |
| User-agent: URLy Warning | |
| Disallow: / | |
| User-agent: Wget/1.5.3 | |
| Disallow: / | |
| User-agent: Wget | |
| Disallow: / | |
| User-agent: LinkWalker | |
| Disallow: / | |
| User-agent: cosmos | |
| Disallow: / | |
| User-agent: moget | |
| Disallow: / | |
| User-agent: hloader | |
| Disallow: / | |
| User-agent: humanlinks | |
| Disallow: / | |
| User-agent: LinkextractorPro | |
| Disallow: / | |
| User-agent: Mata Hari | |
| Disallow: / | |
| User-agent: grub-client | |
| Disallow: / | |
| User-agent: grub | |
| Disallow: / | |
| User-agent: looksmart | |
| Disallow: / | |
| User-agent: WebZip | |
| Disallow: / | |
| User-agent: larbin | |
| Disallow: / | |
| User-agent: b2w/0.1 | |
| Disallow: / | |
| User-agent: psbot | |
| Disallow: / | |
| User-agent: NetMechanic | |
| Disallow: / | |
| User-agent: URL_Spider_Pro | |
| Disallow: / | |
| User-agent: CherryPicker | |
| Disallow: / | |
| User-agent: EmailCollector | |
| Disallow: / | |
| User-agent: EmailSiphon | |
| Disallow: / | |
| User-agent: WebBandit | |
| Disallow: / | |
| User-agent: EmailWolf | |
| Disallow: / | |
| User-agent: ExtractorPro | |
| Disallow: / | |
| User-agent: CopyRightCheck | |
| Disallow: / | |
| User-agent: Crescent | |
| Disallow: / | |
| User-agent: SiteSnagger | |
| Disallow: / | |
| User-agent: ProWebWalker | |
| Disallow: / | |
| User-agent: CheeseBot | |
| Disallow: / | |
| User-agent: LNSpiderguy | |
| Disallow: / | |
| User-agent: Website Quester | |
| Disallow: / | |
| User-agent: moget/2.1 | |
| Disallow: / | |
| User-agent: WebAuto | |
| Disallow: / | |
| User-agent: WebZip/4.0 | |
| Disallow: / | |
| User-agent: WebStripper | |
| Disallow: / | |
| User-agent: WebSauger | |
| Disallow: / | |
| User-agent: WebCopier | |
| Disallow: / | |
| User-agent: NetAnts | |
| Disallow: / | |
| User-agent: Mister PiX | |
| Disallow: / | |
| User-agent: TheNomad | |
| Disallow: / | |
| User-agent: WWW-Collector-E | |
| Disallow: / | |
| User-agent: RMA | |
| Disallow: / | |
| User-agent: libWeb/clsHTTP | |
| Disallow: / | |
| User-agent: asterias | |
| Disallow: / | |
| User-agent: httplib | |
| Disallow: / | |
| User-agent: turingos | |
| Disallow: / | |
| User-agent: spanner | |
| Disallow: / | |
| User-agent: InfoNaviRobot | |
| Disallow: / | |
| User-agent: Harvest/1.5 | |
| Disallow: / | |
| User-agent: Bullseye/1.0 | |
| Disallow: / | |
| User-agent: Mozilla/4.0 (compatible; BullsEye; Windows 95) | |
| Disallow: / | |
| User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 | |
| Disallow: / | |
| User-agent: CherryPickerSE/1.0 | |
| Disallow: / | |
| User-agent: CherryPickerElite/1.0 | |
| Disallow: / | |
| User-agent: WebBandit/3.50 | |
| Disallow: / | |
| User-agent: NICErsPRO | |
| Disallow: / | |
| User-agent: Microsoft URL Control � 5.01.4511 | |
| Disallow: / | |
| User-agent: DittoSpyder | |
| Disallow: / | |
| User-agent: Foobot | |
| Disallow: / | |
| User-agent: WebmasterWorldForumBot | |
| Disallow: / | |
| User-agent: SpankBot | |
| Disallow: / | |
| User-agent: BotALot | |
| Disallow: / | |
| User-agent: lwp-trivial/1.34 | |
| Disallow: / | |
| User-agent: lwp-trivial | |
| Disallow: / | |
| User-agent: BunnySlippers | |
| Disallow: / | |
| User-agent: Microsoft URL Control � 6.00.8169 | |
| Disallow: / | |
| User-agent: URLy Warning | |
| Disallow: / | |
| User-agent: Wget/1.6 | |
| Disallow: / | |
| User-agent: Wget/1.5.3 | |
| Disallow: / | |
| User-agent: Wget | |
| Disallow: / | |
| User-agent: LinkWalker | |
| Disallow: / | |
| User-agent: cosmos | |
| Disallow: / | |
| User-agent: moget | |
| Disallow: / | |
| User-agent: hloader | |
| Disallow: / | |
| User-agent: humanlinks | |
| Disallow: / | |
| User-agent: LinkextractorPro | |
| Disallow: / | |
| User-agent: Offline Explorer | |
| Disallow: / | |
| User-agent: Mata Hari | |
| Disallow: / | |
| User-agent: BuiltBotTough | |
| Disallow: / | |
| User-agent: LexiBot | |
| Disallow: / | |
| User-agent: Web Image Collector | |
| Disallow: / | |
| User-agent: The Intraformant | |
| Disallow: / | |
| User-agent: True_Robot/1.0 | |
| Disallow: / | |
| User-agent: True_Robot | |
| Disallow: / | |
| User-agent: BlowFish/1.0 | |
| Disallow: / | |
| User-agent: JennyBot | |
| Disallow: / | |
| User-agent: MIIxpc/4.2 | |
| Disallow: / | |
| User-agent: ProPowerBot/2.14 | |
| Disallow: / | |
| User-agent: BackDoorBot/1.0 | |
| Disallow: / | |
| User-agent: toCrawl/UrlDispatcher | |
| Disallow: / | |
| User-agent: WebEnhancer | |
| Disallow: / | |
| User-agent: suzuran | |
| Disallow: / | |
| User-agent: VCI WebViewer VCI WebViewer Win32 | |
| Disallow: / | |
| User-agent: VCI | |
| Disallow: / | |
| User-agent: Szukacz/1.4 | |
| Disallow: / | |
| User-agent: QueryN Metasearch | |
| Disallow: / | |
| User-agent: Openfind data gathere | |
| Disallow: / | |
| User-agent: Openfind | |
| Disallow: / | |
| User-agent: Xenu�s Link Sleuth 1.1c | |
| Disallow: / | |
| User-agent: Xenu�s | |
| Disallow: / | |
| User-agent: Zeus | |
| Disallow: / | |
| User-agent: RepoMonkey Bait & Tackle/v1.01 | |
| Disallow: / | |
| User-agent: RepoMonkey | |
| Disallow: / | |
| User-agent: Microsoft URL Control | |
| Disallow: / | |
| User-agent: URL Control | |
| Disallow: / | |
| User-agent: Zeus Link Scout | |
| Disallow: / | |
| User-agent: Zeus 32297 Webster Pro V2.9 Win32 | |
| Disallow: / | |
| User-agent: EroCrawler | |
| Disallow: / | |
| User-agent: LinkScan/8.1a Unix | |
| Disallow: / | |
| User-agent: Keyword Density/0.9 | |
| Disallow: / | |
| User-agent: Kenjin Spider | |
| Disallow: / | |
| User-agent: Iron33/1.0.2 | |
| Disallow: / | |
| User-agent: Bookmark search tool | |
| Disallow: / | |
| User-agent: GetRight/4.2 | |
| Disallow: / | |
| User-agent: FairAd Client | |
| Disallow: / | |
| User-agent: Gaisbot | |
| Disallow: / | |
| User-agent: Aqua_Products | |
| Disallow: / | |
| User-agent: Radiation Retriever 1.1 | |
| Disallow: / | |
| User-agent: Webster Pro | |
| Disallow: / | |
| User-agent: Flaming AttackBot | |
| Disallow: / | |
| User-agent: Oracle Ultra Search | |
| Disallow: / | |
| User-agent: PerMan | |
| Disallow: / | |
| User-agent: searchpreview | |
| Disallow: / | |
| User-agent: LexiBot | |
| Disallow: / | |
| User-agent: Web Image Collector | |
| Disallow: / | |
| User-agent: The Intraformant | |
| Disallow: / | |
| User-agent: True_Robot/1.0 | |
| Disallow: / | |
| User-agent: True_Robot | |
| Disallow: / | |
| User-agent: BlowFish/1.0 | |
| Disallow: / | |
| User-agent: JennyBot | |
| Disallow: / | |
| User-agent: MIIxpc/4.2 | |
| Disallow: / | |
| User-agent: BuiltBotTough | |
| Disallow: / | |
| User-agent: ProPowerBot/2.14 | |
| Disallow: / | |
| User-agent: BackDoorBot/1.0 | |
| Disallow: / | |
| User-agent: toCrawl/UrlDispatcher | |
| Disallow: / | |
| User-agent: WebEnhancer | |
| Disallow: / | |
| User-agent: suzuran | |
| Disallow: / | |
| User-agent: VCI WebViewer VCI WebViewer Win32 | |
| Disallow: / | |
| User-agent: VCI | |
| Disallow: / | |
| User-agent: Szukacz/1.4 | |
| Disallow: / | |
| User-agent: QueryN Metasearch | |
| Disallow: / | |
| User-agent: Openfind data gathere | |
| Disallow: / | |
| User-agent: Openfind | |
| Disallow: / | |
| User-agent: Xenu�s Link Sleuth 1.1c | |
| Disallow: / | |
| User-agent: Xenu�s | |
| Disallow: / | |
| User-agent: RepoMonkey Bait & Tackle/v1.01 | |
| Disallow: / | |
| User-agent: RepoMonkey | |
| Disallow: / | |
| User-agent: Microsoft URL Control | |
| Disallow: / | |
| User-agent: Openbot | |
| Disallow: / | |
| User-agent: URL Control | |
| Disallow: / | |
| User-agent: searchpreview | |
| Disallow: / | |
| User-agent: Zeus Link Scout | |
| Disallow: / | |
| User-agent: Zeus 32297 Webster Pro V2.9 Win32 | |
| Disallow: / | |
| User-agent: Webster Pro | |
| Disallow: / | |
| User-agent: EroCrawler | |
| Disallow: / | |
| User-agent: LinkScan/8.1a Unix | |
| Disallow: / | |
| User-agent: Keyword Density/0.9 | |
| Disallow: / | |
| User-agent: Kenjin Spider | |
| Disallow: / | |
| User-agent: Iron33/1.0.2 | |
| Disallow: / | |
| User-agent: Bookmark search tool | |
| Disallow: / | |
| User-agent: GetRight/4.2 | |
| Disallow: / | |
| User-agent: FairAd Client | |
| Disallow: / | |
| User-agent: Gaisbot | |
| Disallow: / | |
| User-agent: Aqua_Products | |
| Disallow: / | |
| User-agent: Radiation Retriever 1.1 | |
| Disallow: / | |
| User-agent: Flaming AttackBot | |
| Disallow: / | |
| User-agent: Oracle Ultra Search | |
| Disallow: / | |
| User-agent: MSIECrawler | |
| Disallow: / | |
| User-agent: PerMan | |
| Disallow: / | |
| User-agent: IsraBot | |
| Disallow: | |
| User-agent: Orthogaffe | |
| Disallow: | |
| User-agent: UbiCrawler | |
| Disallow: / | |
| User-agent: DOC | |
| Disallow: / | |
| User-agent: Zao | |
| Disallow: / | |
| User-agent: sitecheck.internetseer.com | |
| Disallow: / | |
| User-agent: Zealbot | |
| Disallow: / | |
| User-agent: MSIECrawler | |
| Disallow: / | |
| User-agent: SiteSnagger | |
| Disallow: / | |
| User-agent: WebStripper | |
| Disallow: / | |
| User-agent: WebCopier | |
| Disallow: / | |
| User-agent: Fetch | |
| Disallow: / | |
| User-agent: Offline Explorer | |
| Disallow: / | |
| User-agent: Teleport | |
| Disallow: / | |
| User-agent: TeleportPro | |
| Disallow: / | |
| User-agent: WebZIP | |
| Disallow: / | |
| User-agent: linko | |
| Disallow: / | |
| User-agent: HTTrack | |
| Disallow: / | |
| User-agent: Microsoft.URL.Control | |
| Disallow: / | |
| User-agent: Xenu | |
| Disallow: / | |
| User-agent: larbin | |
| Disallow: / | |
| User-agent: libwww | |
| Disallow: / | |
| User-agent: ZyBORG | |
| Disallow: / | |
| User-agent: Download Ninja | |
| Disallow: / | |
| User-agent: ZoominfoBot | |
| Disallow: / | |
| User-agent: NextGenSearchBot | |
| Disallow: / | |
| User-agent: Zoombot | |
| Disallow: / | |
| User-agent: dotbot | |
| Disallow: / | |
| User-agent: rogerbot | |
| Disallow: / | |
| User-agent: ichiro | |
| Disallow: / | |
| User-agent: sogou spider | |
| Disallow: / | |
| User-agent: YoudaoBot | |
| Disallow: / | |
| User-agent: MJ12bot | |
| Disallow: / | |
| User-agent: DJEcoBot | |
| Disallow: / | |
| # User-agent: Googlebot-Image | |
| # Disallow: / | |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment