Skip to content

Instantly share code, notes, and snippets.

@arifulhb

arifulhb/Block Baidu Spider via robots.txt

Created December 15, 2016 06:12
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save arifulhb/1615dfb60402d3843cfa3f4bf4bd5563 to your computer and use it in GitHub Desktop.
Save arifulhb/1615dfb60402d3843cfa3f4bf4bd5563 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
Block Baidu Spider via robots.txt
# Block Baidu spider
User-agent: Baiduspider
User-agent: baiduspider
User-agent: Baiduspider-video
User-agent: Baiduspider-image
User-agent: Baiduspider+
Disallow: /
# BLOCK Yandex
User-agent: Yandex
Disallow: /
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment