Skip to content

Instantly share code, notes, and snippets.

@dehamzah

dehamzah/readme.md

Created April 14, 2020 14:34
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save dehamzah/aeddf1adf548d20b033ea38351aa390d to your computer and use it in GitHub Desktop.
Save dehamzah/aeddf1adf548d20b033ea38351aa390d to your computer and use it in GitHub Desktop.
Download ZIP
Stop google bot crawling your web
Raw
readme.md

in robots.txt

User-agent: *
Disallow: /

this will not allow the crawler access your website, will work if your web haven't been indexed.

in your html head

<meta name="robots" content="noindex,nofollow">

it will order the crawler to not index and not following any links. crawler will remove your web if its already indexed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment