navjeetc/robots.txt

Created November 6, 2019 04:49

Star () You must be signed in to star a gist
Fork () You must be signed in to fork a gist

Learn more about clone URLs
Clone this repository at <script src="https://gist.github.com/navjeetc/3a2fb6c68556b44b5cdf14bf90a48653.js"></script>
Save navjeetc/3a2fb6c68556b44b5cdf14bf90a48653 to your computer and use it in GitHub Desktop.

Download ZIP

Raw

	# disallow dotbot crawler
	User-agent: dotbot

	Disallow: /

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment