Last active
April 7, 2020 17:01
-
-
Save eksiscloud/e95dc3aa72ada44072dc88f0e1c7d755 to your computer and use it in GitHub Desktop.
Another robots.txt for good bots, spiders and crawlers
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| User-agent: AffiliateLabz | |
| User-agent: AhrefsBot | |
| User-agent: AspiegelBot | |
| User-agent: coccocbot | |
| User-agent: IAS crawler | |
| User-agent: MJ12bot | |
| User-agent: seewithkids.com | |
| User-agent: SemrushBot/6~bl | |
| User-agent: TTD-Content | |
| User-agent: YandexBot | |
| Disallow: / | |
| User-agent: * | |
| Disallow: /wp-admin/ | |
| Allow: /wp-admin/admin-ajax.php | |
| Disallow: /wp-login.php | |
| Disallow: /xmlrpc.php | |
| Sitemap: https://www.example.tld/sitemap_index.xml |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Just an example but working one.
There is no point what so ever even bother to try stop bad bots using robots.txt. The list will be too long and difficult to read. Actually, you should use robots.txt just to limit those bots who have access, like Google etc, and stop everybody else (good and bad ones) usein webserver or reverse proxy.