Skip to content

Instantly share code, notes, and snippets.

@Inkimar

Inkimar/gist:ab9ead7ead70e85ee285c0b159f17bca

Created January 17, 2018 12:49
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save Inkimar/ab9ead7ead70e85ee285c0b159f17bca to your computer and use it in GitHub Desktop.
Save Inkimar/ab9ead7ead70e85ee285c0b159f17bca to your computer and use it in GitHub Desktop.
Download ZIP
example robot.txt
Raw
gistfile1.txt
from 2018-01-17, the 'as'-machine in /home/admin/wildfly-8.1.0-0/wildfly/welcome-content
User-agent: *
Disallow: /
Allow: /naturalist/
Allow: /loan/
Allow: /inventory/
Allow: /dnakey/
Allow: /naturarv/
@Inkimar
Copy link
Author

Inkimar commented May 18, 2021

https://alan.ivey.dev/posts/2017/robots.txt-disallow-all-with-nginx/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment