Skip to content

Instantly share code, notes, and snippets.

Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save payingattention/cd8a86dee7581f75aa35 to your computer and use it in GitHub Desktop.
Save payingattention/cd8a86dee7581f75aa35 to your computer and use it in GitHub Desktop.
User-agent: *
Allow: /log/
[...]
/* THANKS */
Name: name or url
/* PAINKS */
[...]
/* TEAM */
Your title: Your name.
Site: email, link to a contact form, etc.
Twitter: your Twitter username.
Location: City, Country.
[...]
/* THANKS */
Name: name or url
[...]
/* SITE */
Last update: YYYY/MM/DD
Standards: HTML5, CSS3,..
Components: Modernizr, jQuery, etc.
Software: Software used for the development
User-agent: Baiduspider
Disallow: /baidu
Disallow: /s?
Disallow: /ulink?
Disallow: /link?
User-agent: Googlebot
Disallow: /baidu
Disallow: /s?
Disallow: /shifen/
Disallow: /homepage/
Disallow: /cpro
Disallow: /ulink?
Disallow: /link?
User-agent: MSNBot
Disallow: /baidu
Disallow: /s?
Disallow: /shifen/
Disallow: /homepage/
Disallow: /cpro
Disallow: /ulink?
Disallow: /link?
User-agent: Baiduspider-image
Disallow: /baidu
Disallow: /s?
Disallow: /shifen/
Disallow: /homepage/
Disallow: /cpro
Disallow: /ulink?
Disallow: /link?
User-agent: YoudaoBot
Disallow: /baidu
Disallow: /s?
Disallow: /shifen/
Disallow: /homepage/
Disallow: /cpro
Disallow: /ulink?
Disallow: /link?
User-agent: Sogou web spider
Disallow: /baidu
Disallow: /s?
Disallow: /shifen/
Disallow: /homepage/
Disallow: /cpro
Disallow: /ulink?
Disallow: /link?
User-agent: Sogou inst spider
Disallow: /baidu
Disallow: /s?
Disallow: /shifen/
Disallow: /homepage/
Disallow: /cpro
Disallow: /ulink?
Disallow: /link?
User-agent: Sogou spider2
Disallow: /baidu
Disallow: /s?
Disallow: /shifen/
Disallow: /homepage/
Disallow: /cpro
Disallow: /ulink?
Disallow: /link?
User-agent: Sogou blog
Disallow: /baidu
Disallow: /s?
Disallow: /shifen/
Disallow: /homepage/
Disallow: /cpro
Disallow: /ulink?
Disallow: /link?
User-agent: Sogou News Spider
Disallow: /baidu
Disallow: /s?
Disallow: /shifen/
Disallow: /homepage/
Disallow: /cpro
Disallow: /ulink?
Disallow: /link?
User-agent: Sogou Orion spider
Disallow: /baidu
Disallow: /s?
Disallow: /shifen/
Disallow: /homepage/
Disallow: /cpro
Disallow: /ulink?
Disallow: /link?
User-agent: JikeSpider
Disallow: /baidu
Disallow: /s?
Disallow: /shifen/
Disallow: /homepage/
Disallow: /cpro
Disallow: /ulink?
Disallow: /link?
User-agent: Sosospider
Disallow: /baidu
Disallow: /s?
Disallow: /shifen/
Disallow: /homepage/
Disallow: /cpro
Disallow: /ulink?
Disallow: /link?
User-agent: PangusoSpider
Disallow: /baidu
Disallow: /s?
Disallow: /shifen/
Disallow: /homepage/
Disallow: /cpro
Disallow: /ulink?
Disallow: /link?
User-agent: yisouspider
Disallow: /baidu
Disallow: /s?
Disallow: /shifen/
Disallow: /homepage/
Disallow: /cpro
Disallow: /ulink?
Disallow: /link?
User-agent: EasouSpider
Disallow: /baidu
Disallow: /s?
Disallow: /shifen/
Disallow: /homepage/
Disallow: /cpro
Disallow: /ulink?
Disallow: /link?
User-agent: *
Disallow: /
User-agent: *
Disallow: /search
Disallow: /sdch
Disallow: /groups
Disallow: /images
Disallow: /catalogs
Allow: /catalogs/about
Allow: /catalogs/p?
Disallow: /catalogues
Allow: /newsalerts
Disallow: /news
Allow: /news/directory
Disallow: /nwshp
Disallow: /setnewsprefs?
Disallow: /index.html?
Disallow: /?
Allow: /?hl=
Disallow: /?hl=*&
Allow: /?hl=*&gws_rd=ssl$
Disallow: /?hl=*&*&gws_rd=ssl
Allow: /?gws_rd=ssl$
Allow: /?pt1=true$
Disallow: /addurl/image?
Allow: /mail/help/
Disallow: /mail/
Disallow: /pagead/
Disallow: /relpage/
Disallow: /relcontent
Disallow: /imgres
Disallow: /imglanding
Disallow: /sbd
Disallow: /keyword/
Disallow: /u/
Disallow: /univ/
Disallow: /cobrand
Disallow: /custom
Disallow: /advanced_group_search
Disallow: /googlesite
Disallow: /preferences
Disallow: /setprefs
Disallow: /swr
Disallow: /url
Disallow: /default
Disallow: /m?
Disallow: /m/
Allow: /m/finance
Disallow: /wml?
Disallow: /wml/?
Disallow: /wml/search?
Disallow: /xhtml?
Disallow: /xhtml/?
Disallow: /xhtml/search?
Disallow: /xml?
Disallow: /imode?
Disallow: /imode/?
Disallow: /imode/search?
Disallow: /jsky?
Disallow: /jsky/?
Disallow: /jsky/search?
Disallow: /pda?
Disallow: /pda/?
Disallow: /pda/search?
Disallow: /sprint_xhtml
Disallow: /sprint_wml
Disallow: /pqa
Disallow: /palm
Disallow: /gwt/
Disallow: /purchases
Disallow: /hws
Disallow: /bsd?
Disallow: /linux?
Disallow: /mac?
Disallow: /microsoft?
Disallow: /unclesam?
Disallow: /answers/search?q=
Disallow: /local?
Disallow: /local_url
Disallow: /shihui?
Disallow: /shihui/
Disallow: /froogle?
Disallow: /products?
Disallow: /froogle_
Disallow: /product_
Disallow: /products_
Disallow: /products;
Disallow: /print
Disallow: /books/
Disallow: /bkshp?*q=*
Disallow: /books?*q=*
Disallow: /books?*output=*
Disallow: /books?*pg=*
Disallow: /books?*jtp=*
Disallow: /books?*jscmd=*
Disallow: /books?*buy=*
Disallow: /books?*zoom=*
Allow: /books?*q=related:*
Allow: /books?*q=editions:*
Allow: /books?*q=subject:*
Allow: /books/about
Allow: /booksrightsholders
Allow: /books?*zoom=1*
Allow: /books?*zoom=5*
Disallow: /ebooks/
Disallow: /ebooks?*q=*
Disallow: /ebooks?*output=*
Disallow: /ebooks?*pg=*
Disallow: /ebooks?*jscmd=*
Disallow: /ebooks?*buy=*
Disallow: /ebooks?*zoom=*
Allow: /ebooks?*q=related:*
Allow: /ebooks?*q=editions:*
Allow: /ebooks?*q=subject:*
Allow: /ebooks?*zoom=1*
Allow: /ebooks?*zoom=5*
Disallow: /patents?
Disallow: /patents/download/
Disallow: /patents/pdf/
Disallow: /patents/related/
Disallow: /scholar
Disallow: /citations?
Allow: /citations?user=
Disallow: /citations?*cstart=
Allow: /citations?view_op=new_profile
Allow: /citations?view_op=top_venues
Disallow: /complete
Disallow: /s?
Disallow: /sponsoredlinks
Disallow: /videosearch?
Disallow: /videopreview?
Disallow: /videoprograminfo?
Allow: /maps?*output=classic*
Allow: /maps/api/js?
Allow: /maps/d/
Disallow: /maps?
Disallow: /mapstt?
Disallow: /mapslt?
Disallow: /maps/stk/
Disallow: /maps/br?
Disallow: /mapabcpoi?
Disallow: /maphp?
Disallow: /mapprint?
Disallow: /maps/api/js/
Disallow: /maps/api/staticmap?
Disallow: /mld?
Disallow: /staticmap?
Disallow: /places/
Allow: /places/$
Disallow: /maps/preview
Disallow: /maps/place
Disallow: /help/maps/streetview/partners/welcome/
Disallow: /help/maps/indoormaps/partners/
Disallow: /lochp?
Disallow: /center
Disallow: /ie?
Disallow: /sms/demo?
Disallow: /katrina?
Disallow: /blogsearch?
Disallow: /blogsearch/
Disallow: /blogsearch_feeds
Disallow: /advanced_blog_search
Disallow: /uds/
Disallow: /chart?
Disallow: /transit?
Disallow: /mbd?
Disallow: /extern_js/
Disallow: /xjs/
Disallow: /calendar/feeds/
Disallow: /calendar/ical/
Disallow: /cl2/feeds/
Disallow: /cl2/ical/
Disallow: /coop/directory
Disallow: /coop/manage
Disallow: /trends?
Disallow: /trends/music?
Disallow: /trends/hottrends?
Disallow: /trends/viz?
Disallow: /trends/embed.js?
Disallow: /trends/fetchComponent?
Disallow: /notebook/search?
Disallow: /musica
Disallow: /musicad
Disallow: /musicas
Disallow: /musicl
Disallow: /musics
Disallow: /musicsearch
Disallow: /musicsp
Disallow: /musiclp
Disallow: /browsersync
Disallow: /call
Disallow: /archivesearch?
Disallow: /archivesearch/url
Disallow: /archivesearch/advanced_search
Disallow: /base/reportbadoffer
Disallow: /urchin_test/
Disallow: /movies?
Disallow: /codesearch?
Disallow: /codesearch/feeds/search?
Disallow: /wapsearch?
Disallow: /safebrowsing
Allow: /safebrowsing/diagnostic
Allow: /safebrowsing/report_badware/
Allow: /safebrowsing/report_error/
Allow: /safebrowsing/report_phish/
Disallow: /reviews/search?
Disallow: /orkut/albums
Allow: /jsapi
Disallow: /views?
Disallow: /c/
Disallow: /cbk
Allow: /cbk?output=tile&cb_client=maps_sv
Disallow: /recharge/dashboard/car
Disallow: /recharge/dashboard/static/
Disallow: /translate_a/
Disallow: /translate_c
Disallow: /translate_f
Disallow: /translate_static/
Disallow: /translate_suggestion
Disallow: /profiles/me
Allow: /profiles
Disallow: /s2/profiles/me
Allow: /s2/profiles
Allow: /s2/oz
Allow: /s2/photos
Allow: /s2/search/social
Allow: /s2/static
Disallow: /s2
Disallow: /transconsole/portal/
Disallow: /gcc/
Disallow: /aclk
Disallow: /cse?
Disallow: /cse/home
Disallow: /cse/panel
Disallow: /cse/manage
Disallow: /tbproxy/
Disallow: /imesync/
Disallow: /shenghuo/search?
Disallow: /support/forum/search?
Disallow: /reviews/polls/
Disallow: /hosted/images/
Disallow: /ppob/?
Disallow: /ppob?
Disallow: /adwordsresellers
Disallow: /accounts/ClientLogin
Disallow: /accounts/ClientAuth
Disallow: /accounts/o8
Allow: /accounts/o8/id
Disallow: /topicsearch?q=
Disallow: /xfx7/
Disallow: /squared/api
Disallow: /squared/search
Disallow: /squared/table
Disallow: /toolkit/
Allow: /toolkit/*.html
Disallow: /globalmarketfinder/
Allow: /globalmarketfinder/*.html
Disallow: /qnasearch?
Disallow: /app/updates
Disallow: /sidewiki/entry/
Disallow: /quality_form?
Disallow: /labs/popgadget/search
Disallow: /buzz/post
Disallow: /compressiontest/
Disallow: /analytics/reporting/
Disallow: /analytics/admin/
Disallow: /analytics/web/
Disallow: /analytics/feeds/
Disallow: /analytics/settings/
Allow: /alerts/manage
Allow: /alerts/remove
Disallow: /alerts/
Allow: /alerts/$
Disallow: /ads/search?
Disallow: /ads/plan/action_plan?
Disallow: /ads/plan/api/
Disallow: /phone/compare/?
Disallow: /travel/clk
Disallow: /hotelfinder/rpc
Disallow: /hotels/rpc
Disallow: /flights/rpc
Disallow: /commercesearch/services/
Disallow: /evaluation/
Disallow: /chrome/browser/mobile/tour
Disallow: /compare/*/apply*
Disallow: /forms/perks/
Disallow: /baraza/*/search
Disallow: /baraza/*/report
Disallow: /shopping/suppliers/search
Disallow: /ct/
Disallow: /edu/cs4hs/
Disallow: /trustedstores/s/
Disallow: /trustedstores/tm2
Disallow: /trustedstores/verify
Disallow: /adwords/proposal
Disallow: /shopping/product/
Disallow: /shopping/seller
Disallow: /shopping/reviewer
Disallow: /about/careers/apply/
Disallow: /about/careers/applications/
Disallow: /landing/signout.html
Allow: /gb/images
Allow: /gb/js
Disallow: /gallery/
Sitemap: http://www.gstatic.com/culturalinstitute/sitemaps/www_google_com_culturalinstitute/sitemap-index.xml
Sitemap: https://www.google.com/edu/sitemap.xml
Sitemap: https://www.google.com/work/sitemap.xml
Sitemap: https://www.google.com/intx/sitemap.xml
Sitemap: http://www.google.com/hostednews/sitemap_index.xml
Sitemap: http://www.google.com/maps/views/sitemap.xml
Sitemap: http://www.google.com/sitemaps_webmasters.xml
Sitemap: http://www.google.com/ventures/sitemap_ventures.xml
Sitemap: http://www.gstatic.com/dictionary/static/sitemaps/sitemap_index.xml
Sitemap: http://www.gstatic.com/earth/gallery/sitemaps/sitemap.xml
Sitemap: http://www.gstatic.com/s2/sitemaps/profiles-sitemap.xml
Sitemap: http://www.gstatic.com/trends/websites/sitemaps/sitemapindex.xml
Sitemap: http://www.google.com/adwords/sitemap.xml
Sitemap: http://www.google.com/drive/sitemap.xml
# If you operate a search engine and would like to crawl Quora, please
# email robotstxt@quora.com. Thanks.
User-agent: Googlebot
Disallow: /log/
Disallow: /ajax/
Disallow: /*/log$
Disallow: /*/rss$
Disallow: /rss$
Disallow: /*/questions_rss$
Disallow: /*/answers_rss$
Disallow: /*_POST$
Disallow: /*/about$
Disallow: /*/all_questions$
Disallow: /*/all_posts$
Disallow: /*/blogs$
Disallow: /*/comment/
Disallow: /comment/
Disallow: /*/comments$
Disallow: /*/followers$
Disallow: /*/following$
Disallow: /*/mobile_collapsed$
Disallow: /*/questions$
Disallow: /*/reviews$
Disallow: /*/topic-bio
Disallow: /*/topics$
Disallow: /home/global_feed$
Disallow: /*/manage$
Disallow: /search?q=
Disallow: /search/?q=
Disallow: /digest/
Disallow: /email_optout/
Disallow: /qemail/
Disallow: /invite/
Disallow: /widgets/content_iframe/
Disallow: /widgets/content_js/
Disallow: /_/
Disallow: /webnode2/
User-agent: bingbot
Disallow: /log/
Disallow: /ajax/
Disallow: /*/log$
Disallow: /*/rss$
Disallow: /rss$
Disallow: /*/questions_rss$
Disallow: /*/answers_rss$
Disallow: /*_POST$
Disallow: /*/about$
Disallow: /*/all_questions$
Disallow: /*/all_posts$
Disallow: /*/blogs$
Disallow: /*/comment/
Disallow: /comment/
Disallow: /*/comments$
Disallow: /*/followers$
Disallow: /*/following$
Disallow: /*/mobile_collapsed$
Disallow: /*/questions$
Disallow: /*/reviews$
Disallow: /*/topic-bio
Disallow: /*/topics$
Disallow: /home/global_feed$
Disallow: /*/manage$
Disallow: /search?q=
Disallow: /search/?q=
Disallow: /digest/
Disallow: /email_optout/
Disallow: /qemail/
Disallow: /invite/
Disallow: /widgets/content_iframe/
Disallow: /widgets/content_js/
Disallow: /_/
Disallow: /webnode2/
User-agent: msnbot
Disallow: /log/
Disallow: /ajax/
Disallow: /*/log$
Disallow: /*/rss$
Disallow: /rss$
Disallow: /*/questions_rss$
Disallow: /*/answers_rss$
Disallow: /*_POST$
Disallow: /*/about$
Disallow: /*/all_questions$
Disallow: /*/all_posts$
Disallow: /*/blogs$
Disallow: /*/comment/
Disallow: /comment/
Disallow: /*/comments$
Disallow: /*/followers$
Disallow: /*/following$
Disallow: /*/mobile_collapsed$
Disallow: /*/questions$
Disallow: /*/reviews$
Disallow: /*/topic-bio
Disallow: /*/topics$
Disallow: /home/global_feed$
Disallow: /*/manage$
Disallow: /search?q=
Disallow: /search/?q=
Disallow: /digest/
Disallow: /email_optout/
Disallow: /qemail/
Disallow: /invite/
Disallow: /widgets/content_iframe/
Disallow: /widgets/content_js/
Disallow: /_/
Disallow: /webnode2/
User-agent: Slurp
Disallow: /log/
Disallow: /ajax/
Disallow: /*/log$
Disallow: /*/rss$
Disallow: /rss$
Disallow: /*/questions_rss$
Disallow: /*/answers_rss$
Disallow: /*_POST$
Disallow: /*/about$
Disallow: /*/all_questions$
Disallow: /*/all_posts$
Disallow: /*/blogs$
Disallow: /*/comment/
Disallow: /comment/
Disallow: /*/comments$
Disallow: /*/followers$
Disallow: /*/following$
Disallow: /*/mobile_collapsed$
Disallow: /*/questions$
Disallow: /*/reviews$
Disallow: /*/topic-bio
Disallow: /*/topics$
Disallow: /home/global_feed$
Disallow: /*/manage$
Disallow: /search?q=
Disallow: /search/?q=
Disallow: /digest/
Disallow: /email_optout/
Disallow: /qemail/
Disallow: /invite/
Disallow: /widgets/content_iframe/
Disallow: /widgets/content_js/
Disallow: /_/
Disallow: /webnode2/
User-agent: Teoma
Disallow: /log/
Disallow: /ajax/
Disallow: /*/log$
Disallow: /*_POST$
Disallow: /*/about$
Disallow: /*/all_questions$
Disallow: /*/all_posts$
Disallow: /*/blogs$
Disallow: /*/comment/
Disallow: /comment/
Disallow: /*/comments$
Disallow: /*/followers$
Disallow: /*/following$
Disallow: /*/mobile_collapsed$
Disallow: /*/questions$
Disallow: /*/reviews$
Disallow: /*/topic-bio
Disallow: /*/topics$
Disallow: /home/global_feed$
Disallow: /*/manage$
Disallow: /search?q=
Disallow: /search/?q=
Disallow: /digest/
Disallow: /email_optout/
Disallow: /qemail/
Disallow: /invite/
Disallow: /widgets/content_iframe/
Disallow: /widgets/content_js/
Disallow: /_/
Disallow: /webnode2/
User-agent: ScoutJet
Disallow: /log/
Disallow: /ajax/
Disallow: /*/log$
Disallow: /*/rss$
Disallow: /rss$
Disallow: /*/questions_rss$
Disallow: /*/answers_rss$
Disallow: /*_POST$
Disallow: /*/about$
Disallow: /*/all_questions$
Disallow: /*/all_posts$
Disallow: /*/blogs$
Disallow: /*/comment/
Disallow: /comment/
Disallow: /*/comments$
Disallow: /*/followers$
Disallow: /*/following$
Disallow: /*/mobile_collapsed$
Disallow: /*/questions$
Disallow: /*/reviews$
Disallow: /*/topic-bio
Disallow: /*/topics$
Disallow: /home/global_feed$
Disallow: /*/manage$
Disallow: /search?q=
Disallow: /search/?q=
Disallow: /digest/
Disallow: /email_optout/
Disallow: /qemail/
Disallow: /invite/
Disallow: /widgets/content_iframe/
Disallow: /widgets/content_js/
Disallow: /_/
Disallow: /webnode2/
User-agent: DuckDuckBot
Disallow: /log/
Disallow: /ajax/
Disallow: /*/log$
Disallow: /*/rss$
Disallow: /rss$
Disallow: /*/questions_rss$
Disallow: /*/answers_rss$
Disallow: /*_POST$
Disallow: /*/about$
Disallow: /*/all_questions$
Disallow: /*/all_posts$
Disallow: /*/blogs$
Disallow: /*/comment/
Disallow: /comment/
Disallow: /*/comments$
Disallow: /*/followers$
Disallow: /*/following$
Disallow: /*/mobile_collapsed$
Disallow: /*/questions$
Disallow: /*/reviews$
Disallow: /*/topic-bio
Disallow: /*/topics$
Disallow: /home/global_feed$
Disallow: /*/manage$
Disallow: /search?q=
Disallow: /search/?q=
Disallow: /digest/
Disallow: /email_optout/
Disallow: /qemail/
Disallow: /invite/
Disallow: /widgets/content_iframe/
Disallow: /widgets/content_js/
Disallow: /_/
Disallow: /webnode2/
User-agent: Twitterbot
Disallow: /log/
Disallow: /ajax/
Disallow: /*/log$
Disallow: /*/rss$
Disallow: /rss$
Disallow: /*/questions_rss$
Disallow: /*/answers_rss$
Disallow: /*_POST$
Disallow: /*/about$
Disallow: /*/all_questions$
Disallow: /*/all_posts$
Disallow: /*/blogs$
Disallow: /*/comment/
Disallow: /comment/
Disallow: /*/comments$
Disallow: /*/followers$
Disallow: /*/following$
Disallow: /*/mobile_collapsed$
Disallow: /*/questions$
Disallow: /*/reviews$
Disallow: /*/topic-bio
Disallow: /*/topics$
Disallow: /home/global_feed$
Disallow: /*/manage$
Disallow: /search?q=
Disallow: /search/?q=
Disallow: /digest/
Disallow: /email_optout/
Disallow: /qemail/
Disallow: /invite/
Disallow: /widgets/content_iframe/
Disallow: /widgets/content_js/
Disallow: /_/
Disallow: /webnode2/
User-agent: Yandex
Disallow: /log/
Disallow: /ajax/
Disallow: /*/log$
Disallow: /*/rss$
Disallow: /rss$
Disallow: /*/questions_rss$
Disallow: /*/answers_rss$
Disallow: /*_POST$
Disallow: /*/about$
Disallow: /*/all_questions$
Disallow: /*/all_posts$
Disallow: /*/blogs$
Disallow: /*/comment/
Disallow: /comment/
Disallow: /*/comments$
Disallow: /*/followers$
Disallow: /*/following$
Disallow: /*/mobile_collapsed$
Disallow: /*/questions$
Disallow: /*/reviews$
Disallow: /*/topic-bio
Disallow: /*/topics$
Disallow: /home/global_feed$
Disallow: /*/manage$
Disallow: /search?q=
Disallow: /search/?q=
Disallow: /digest/
Disallow: /email_optout/
Disallow: /qemail/
Disallow: /invite/
Disallow: /widgets/content_iframe/
Disallow: /widgets/content_js/
Disallow: /_/
Disallow: /webnode2/
User-agent: Relcybot
Disallow: /log/
Disallow: /ajax/
Disallow: /*/log$
Disallow: /*/rss$
Disallow: /rss$
Disallow: /*/questions_rss$
Disallow: /*/answers_rss$
Disallow: /*_POST$
Disallow: /*/about$
Disallow: /*/all_questions$
Disallow: /*/all_posts$
Disallow: /*/blogs$
Disallow: /*/comment/
Disallow: /comment/
Disallow: /*/comments$
Disallow: /*/followers$
Disallow: /*/following$
Disallow: /*/mobile_collapsed$
Disallow: /*/questions$
Disallow: /*/reviews$
Disallow: /*/topic-bio
Disallow: /*/topics$
Disallow: /home/global_feed$
Disallow: /*/manage$
Disallow: /search?q=
Disallow: /search/?q=
Disallow: /digest/
Disallow: /email_optout/
Disallow: /qemail/
Disallow: /invite/
Disallow: /widgets/content_iframe/
Disallow: /widgets/content_js/
Disallow: /_/
Disallow: /webnode2/
User-agent: PGBot
Disallow: /log/
Disallow: /ajax/
Disallow: /*/log$
Disallow: /*/rss$
Disallow: /rss$
Disallow: /*/questions_rss$
Disallow: /*/answers_rss$
Disallow: /*_POST$
Disallow: /*/about$
Disallow: /*/all_questions$
Disallow: /*/all_posts$
Disallow: /*/blogs$
Disallow: /*/comment/
Disallow: /comment/
Disallow: /*/comments$
Disallow: /*/followers$
Disallow: /*/following$
Disallow: /*/mobile_collapsed$
Disallow: /*/questions$
Disallow: /*/reviews$
Disallow: /*/topic-bio
Disallow: /*/topics$
Disallow: /home/global_feed$
Disallow: /*/manage$
Disallow: /search?q=
Disallow: /search/?q=
Disallow: /digest/
Disallow: /email_optout/
Disallow: /qemail/
Disallow: /invite/
Disallow: /widgets/content_iframe/
Disallow: /widgets/content_js/
Disallow: /_/
Disallow: /webnode2/
# People share a lot of sensitive material on Quora - controversial political
# views, workplace gossip and compensation, and negative opinions held of
# companies. Over many years, as they change jobs or change their views, it is
# important that they can delete or anonymize their previously-written answers.
#
# We opt out of the wayback machine because inclusion would allow people to
# discover the identity of authors who had written sensitive answers publicly and
# later had made them anonymous, and because it would prevent authors from being
# able to remove their content from the internet if they change their mind about
# publishing it. As far as we can tell, there is no way for sites to selectively
# programmatically remove content from the archive and so this is the only way
# for us to protect writers. If they open up an API where we can remove content
# from the archive when authors remove it from Quora, but leave the rest of the
# content archived, we would be happy to opt back in. See the page here:
#
# https://archive.org/about/exclude.php
#
# Meanwhile, if you are looking for an older version of any content on Quora, we
# have full edit history tracked and accessible in product (with the exception of
# content that has been removed by the author). You can generally access this by
# clicking on timestamps, or by appending "/log" to the URL of any content page.
#
# For any questions or feedback about this please email robotstxt@quora.com.
User-agent: ia_archiver
Disallow: /
User-agent: *
Allow: /$
Allow: /about$
Allow: /about/
Allow: /jobs$
Allow: /challenges$
Allow: /press$
Allow: /login/
Allow: /signup$
Disallow: /
Disallow: /log/
Disallow: /ajax/
Disallow: /*/log$
Disallow: /*/rss$
Disallow: /rss$
Disallow: /*/questions_rss$
Disallow: /*/answers_rss$
Disallow: /*_POST$
Disallow: /*/about$
Disallow: /*/all_questions$
Disallow: /*/all_posts$
Disallow: /*/blogs$
Disallow: /*/comment/
Disallow: /comment/
Disallow: /*/comments$
Disallow: /*/followers$
Disallow: /*/following$
Disallow: /*/mobile_collapsed$
Disallow: /*/questions$
Disallow: /*/reviews$
Disallow: /*/topic-bio
Disallow: /*/topics$
Disallow: /home/global_feed$
Disallow: /*/manage$
Disallow: /search?q=
Disallow: /search/?q=
Disallow: /digest/
Disallow: /email_optout/
Disallow: /qemail/
Disallow: /invite/
Disallow: /widgets/content_iframe/
Disallow: /widgets/content_js/
Disallow: /_/
Disallow: /webnode2/
Sitemap: http://www.quora.com/sitemap
#
# robots.txt for http://www.wikipedia.org/ and friends
#
# Please note: There are a lot of pages on this site, and there are
# some misbehaved spiders out there that go _way_ too fast. If you're
# irresponsible, your access to the site may be blocked.
#
# advertising-related bots:
User-agent: Mediapartners-Google*
Disallow: /
# Wikipedia work bots:
User-agent: IsraBot
Disallow:
User-agent: Orthogaffe
Disallow:
# Crawlers that are kind enough to obey, but which we'd rather not have
# unless they're feeding search engines.
User-agent: UbiCrawler
Disallow: /
User-agent: DOC
Disallow: /
User-agent: Zao
Disallow: /
# Some bots are known to be trouble, particularly those designed to copy
# entire sites. Please obey robots.txt.
User-agent: sitecheck.internetseer.com
Disallow: /
User-agent: Zealbot
Disallow: /
User-agent: MSIECrawler
Disallow: /
User-agent: SiteSnagger
Disallow: /
User-agent: WebStripper
Disallow: /
User-agent: WebCopier
Disallow: /
User-agent: Fetch
Disallow: /
User-agent: Offline Explorer
Disallow: /
User-agent: Teleport
Disallow: /
User-agent: TeleportPro
Disallow: /
User-agent: WebZIP
Disallow: /
User-agent: linko
Disallow: /
User-agent: HTTrack
Disallow: /
User-agent: Microsoft.URL.Control
Disallow: /
User-agent: Xenu
Disallow: /
User-agent: larbin
Disallow: /
User-agent: libwww
Disallow: /
User-agent: ZyBORG
Disallow: /
User-agent: Download Ninja
Disallow: /
# Misbehaving: requests much too fast:
User-agent: fast
Disallow: /
#
# Sorry, wget in its recursive mode is a frequent problem.
# Please read the man page and use it properly; there is a
# --wait option you can use to set the delay between hits,
# for instance.
#
User-agent: wget
Disallow: /
#
# The 'grub' distributed client has been *very* poorly behaved.
#
User-agent: grub-client
Disallow: /
#
# Doesn't follow robots.txt anyway, but...
#
User-agent: k2spider
Disallow: /
#
# Hits many times per second, not acceptable
# http://www.nameprotect.com/botinfo.html
User-agent: NPBot
Disallow: /
# A capture bot, downloads gazillions of pages with no public benefit
# http://www.webreaper.net/
User-agent: WebReaper
Disallow: /
# Don't allow the wayback-maschine to index user-pages
#User-agent: ia_archiver
#Disallow: /wiki/User
#Disallow: /wiki/Benutzer
#
# Friendly, low-speed bots are welcome viewing article pages, but not
# dynamically-generated pages please.
#
# Inktomi's "Slurp" can read a minimum delay between hits; if your
# bot supports such a thing using the 'Crawl-delay' or another
# instruction, please let us know.
#
# There is a special exception for API mobileview to allow dynamic
# mobile web & app views to load section content.
# These views aren't HTTP-cached but use parser cache aggressively
# and don't expose special: pages etc.
#
User-agent: *
Allow: /w/api.php?action=mobileview&
Disallow: /w/
Disallow: /trap/
Disallow: /wiki/Especial:Search
Disallow: /wiki/Especial%3ASearch
Disallow: /wiki/Special:Collection
Disallow: /wiki/Spezial:Sammlung
Disallow: /wiki/Special:Random
Disallow: /wiki/Special%3ARandom
Disallow: /wiki/Special:Search
Disallow: /wiki/Special%3ASearch
Disallow: /wiki/Spesial:Search
Disallow: /wiki/Spesial%3ASearch
Disallow: /wiki/Spezial:Search
Disallow: /wiki/Spezial%3ASearch
Disallow: /wiki/Specjalna:Search
Disallow: /wiki/Specjalna%3ASearch
Disallow: /wiki/Speciaal:Search
Disallow: /wiki/Speciaal%3ASearch
Disallow: /wiki/Speciaal:Random
Disallow: /wiki/Speciaal%3ARandom
Disallow: /wiki/Speciel:Search
Disallow: /wiki/Speciel%3ASearch
Disallow: /wiki/Speciale:Search
Disallow: /wiki/Speciale%3ASearch
Disallow: /wiki/Istimewa:Search
Disallow: /wiki/Istimewa%3ASearch
Disallow: /wiki/Toiminnot:Search
Disallow: /wiki/Toiminnot%3ASearch
#
# ar:
Disallow: /wiki/%D8%AE%D8%A7%D8%B5:Search
Disallow: /wiki/%D8%AE%D8%A7%D8%B5%3ASearch
#
# de:
# http://bugzilla.wikimedia.org/show_bug.cgi?id=4937
# sensible deletion and meta user discussion pages:
Disallow: /wiki/Wikipedia:L%C3%B6schkandidaten/
Disallow: /wiki/Wikipedia:Löschkandidaten/
Disallow: /wiki/Wikipedia:Vandalensperrung/
Disallow: /wiki/Wikipedia:Benutzersperrung/
Disallow: /wiki/Wikipedia:Vermittlungsausschuss/
Disallow: /wiki/Wikipedia:Administratoren/Probleme/
Disallow: /wiki/Wikipedia:Adminkandidaturen/
Disallow: /wiki/Wikipedia:Qualitätssicherung/
Disallow: /wiki/Wikipedia:Qualit%C3%A4tssicherung/
# Search- and random-page
Disallow: /wiki/Spezial:Suche
Disallow: /wiki/Special:Suche
Disallow: /wiki/Spezial:Zufällige_Seite
Disallow: /wiki/Spezial:Zuf%C3%A4llige_Seite
Disallow: /wiki/Special:Zufällige_Seite
Disallow: /wiki/Special:Zuf%C3%A4llige_Seite
# 4937#5
Disallow: /wiki/Wikipedia:Vandalismusmeldung/
Disallow: /wiki/Wikipedia:Gesperrte_Lemmata/
Disallow: /wiki/Wikipedia:Löschprüfung/
Disallow: /wiki/Wikipedia:L%C3%B6schprüfung/
Disallow: /wiki/Wikipedia:Administratoren/Notizen/
Disallow: /wiki/Wikipedia:Schiedsgericht/Anfragen/
Disallow: /wiki/Wikipedia:L%C3%B6schpr%C3%BCfung/
# http://bugzilla.wikimedia.org/show_bug.cgi?id=12111
Disallow: /wiki/Wikipedia:Checkuser/
Disallow: /wiki/Wikipedia_Diskussion:Checkuser/
Disallow: /wiki/Wikipedia_Diskussion:Adminkandidaturen/
# https://bugzilla.wikimedia.org/show_bug.cgi?id=13961
Disallow: /wiki/Wikipedia:Spam-Blacklist-Log
Disallow: /wiki/Wikipedia%3ASpam-Blacklist-Log
Disallow: /wiki/Wikipedia_Diskussion:Spam-Blacklist-Log
Disallow: /wiki/Wikipedia_Diskussion%3ASpam-Blacklist-Log
#
# Folks get annoyed when VfD discussions end up the number 1 google hit for
# their name. See bugzilla bug #4776
# en:
Disallow: /wiki/Wikipedia:Articles_for_deletion/
Disallow: /wiki/Wikipedia%3AArticles_for_deletion/
Disallow: /wiki/Wikipedia:Votes_for_deletion/
Disallow: /wiki/Wikipedia%3AVotes_for_deletion/
Disallow: /wiki/Wikipedia:Pages_for_deletion/
Disallow: /wiki/Wikipedia%3APages_for_deletion/
Disallow: /wiki/Wikipedia:Miscellany_for_deletion/
Disallow: /wiki/Wikipedia%3AMiscellany_for_deletion/
Disallow: /wiki/Wikipedia:Miscellaneous_deletion/
Disallow: /wiki/Wikipedia%3AMiscellaneous_deletion/
Disallow: /wiki/Wikipedia:Copyright_problems
Disallow: /wiki/Wikipedia%3ACopyright_problems
Disallow: /wiki/Wikipedia:Protected_titles/
Disallow: /wiki/Wikipedia%3AProtected_titles/
# https://bugzilla.wikimedia.org/show_bug.cgi?id=13398
Disallow: /wiki/Wikipedia:WikiProject_Spam/
Disallow: /wiki/Wikipedia%3AWikiProject_Spam/
# https://bugzilla.wikimedia.org/show_bug.cgi?id=14075
Disallow: /wiki/MediaWiki:Spam-blacklist
Disallow: /wiki/MediaWiki%3ASpam-blacklist
Disallow: /wiki/MediaWiki_talk:Spam-blacklist
Disallow: /wiki/MediaWiki_talk%3ASpam-blacklist
#
# fi:
# http://bugzilla.wikimedia.org/show_bug.cgi?id=8695
Disallow: /wiki/Wikipedia:Poistettavat_sivut
Disallow: /wiki/K%C3%A4ytt%C3%A4j%C3%A4:
Disallow: /wiki/Käyttäjä:
Disallow: /wiki/Keskustelu_k%C3%A4ytt%C3%A4j%C3%A4st%C3%A4:
Disallow: /wiki/Keskustelu_käyttäjästä:
Disallow: /wiki/Wikipedia:Yll%C3%A4pit%C3%A4j%C3%A4t/
Disallow: /wiki/Wikipedia:Ylläpitäjät/
#
# fr:
Disallow: /wiki/Wikip%C3%A9dia:Pages_%C3%A0_supprimer/
Disallow: /wiki/Wikip%C3%A9dia:Pages_soup%C3%A7onn%C3%A9es_de_violation_de_copyright/
#
# he:
Disallow: /wiki/%D7%9E%D7%99%D7%95%D7%97%D7%93:Search
Disallow: /wiki/%D7%9E%D7%99%D7%95%D7%97%D7%93%3ASearch
#
# hu:
Disallow: /wiki/Speci%C3%A1lis:Search
Disallow: /wiki/Speci%C3%A1lis%3ASearch
#
# ja:
Disallow: /wiki/%E7%89%B9%E5%88%A5:Search
Disallow: /wiki/%E7%89%B9%E5%88%A5%3ASearch
#
# ro:
# http://bugzilla.wikimedia.org/show_bug.cgi?id=12546
Disallow: /wiki/Wikipedia:Pagini_de_%C5%9Fters
Disallow: /wiki/Wikipedia%3APagini_de_%C5%9Fters
Disallow: /wiki/Discu%C5%A3ie_Wikipedia:Pagini_de_%C5%9Fters
Disallow: /wiki/Discu%C5%A3ie_Wikipedia%3APagini_de_%C5%9Fters
#
# ru:
Disallow: /wiki/%D0%A1%D0%BF%D0%B5%D1%86%D0%B8%D0%B0%D0%BB%D1%8C%D0%BD%D1%8B%D0%B5:Search
Disallow: /wiki/%D0%A1%D0%BF%D0%B5%D1%86%D0%B8%D0%B0%D0%BB%D1%8C%D0%BD%D1%8B%D0%B5%3ASearch
#
# ja:
# https://bugzilla.wikimedia.org/show_bug.cgi?id=5239
Disallow: /wiki/Wikipedia:%E5%89%8A%E9%99%A4%E4%BE%9D%E9%A0%BC/
Disallow: /wiki/Wikipedia%3A%E5%89%8A%E9%99%A4%E4%BE%9D%E9%A0%BC/
Disallow: /wiki/Wikipedia:%E5%88%A9%E7%94%A8%E8%80%85%E3%83%9A%E3%83%BC%E3%82%B8%E3%81%AE%E5%89%8A%E9%99%A4%E4%BE%9D%E9%A0%BC
Disallow: /wiki/Wikipedia%3A%E5%88%A9%E7%94%A8%E8%80%85%E3%83%9A%E3%83%BC%E3%82%B8%E3%81%AE%E5%89%8A%E9%99%A4%E4%BE%9D%E9%A0%BC
#
# pt:
# https://bugzilla.wikimedia.org/show_bug.cgi?id=5394
Disallow: /wiki/Wikipedia:Páginas_para_eliminar/
Disallow: /wiki/Wikipedia:P%C3%A1ginas_para_eliminar/
Disallow: /wiki/Wikipedia%3AP%C3%A1ginas_para_eliminar/
Disallow: /wiki/Wikipedia_Discussão:Páginas_para_eliminar/
Disallow: /wiki/Wikipedia_Discuss%C3%A3o:P%C3%A1ginas_para_eliminar/
Disallow: /wiki/Wikipedia_Discuss%C3%A3o%3AP%C3%A1ginas_para_eliminar/
#
# zh:
# https://bugzilla.wikimedia.org/show_bug.cgi?id=5104
Disallow: /wiki/Wikipedia:删除投票/侵权
Disallow: /wiki/Wikipedia:%E5%88%A0%E9%99%A4%E6%8A%95%E7%A5%A8/%E4%BE%B5%E6%9D%83
Disallow: /wiki/Wikipedia:删除投票和请求
Disallow: /wiki/Wikipedia:%E5%88%A0%E9%99%A4%E6%8A%95%E7%A5%A8%E5%92%8C%E8%AF%B7%E6%B1%82
Disallow: /wiki/Category:快速删除候选
Disallow: /wiki/Category:%E5%BF%AB%E9%80%9F%E5%88%A0%E9%99%A4%E5%80%99%E9%80%89
Disallow: /wiki/Category:维基百科需要翻译的文章
Disallow: /wiki/Category:%E7%BB%B4%E5%9F%BA%E7%99%BE%E7%A7%91%E9%9C%80%E8%A6%81%E7%BF%BB%E8%AF%91%E7%9A%84%E6%96%87%E7%AB%A0
#
# it: - http://bugzilla.wikimedia.org/show_bug.cgi?id=5545
Disallow: /wiki/Wikipedia:Pagine_da_cancellare
Disallow: /wiki/Wikipedia%3APagine_da_cancellare
Disallow: /wiki/Wikipedia:Utenti_problematici
Disallow: /wiki/Wikipedia%3AUtenti_problematici
Disallow: /wiki/Wikipedia:Vandalismi_in_corso
Disallow: /wiki/Wikipedia%3AVandalismi_in_corso
Disallow: /wiki/Wikipedia:Amministratori
Disallow: /wiki/Wikipedia%3AAmministratori
Disallow: /wiki/Wikipedia:Proposte_di_cancellazione_semplificata
Disallow: /wiki/Wikipedia%3AProposte_di_cancellazione_semplificata
Disallow: /wiki/Categoria:Da_cancellare_subito
Disallow: /wiki/Categoria%3ADa_cancellare_subito
Disallow: /wiki/Wikipedia:Sospette_violazioni_di_copyright
Disallow: /wiki/Wikipedia%3ASospette_violazioni_di_copyright
Disallow: /wiki/Categoria:Da_controllare_per_copyright
Disallow: /wiki/Categoria%3ADa_controllare_per_copyright
# added 2007-01-12
Disallow: /wiki/Progetto:Rimozione_contributi_sospetti
Disallow: /wiki/Progetto%3ARimozione_contributi_sospetti
Disallow: /wiki/Categoria:Da_cancellare_subito_per_violazione_integrale_copyright
Disallow: /wiki/Categoria%3ADa_cancellare_subito_per_violazione_integrale_copyright
Disallow: /wiki/Progetto:Cococo
Disallow: /wiki/Progetto%3ACococo
Disallow: /wiki/Discussioni_progetto:Cococo
Disallow: /wiki/Discussioni_progetto%3ACococo
#
# pl.wikipedia.org
# http://bugzilla.wikimedia.org/show_bug.cgi?id=8067
Disallow: /wiki/Wikipedia:Strony_do_usuni%C4%99cia
Disallow: /wiki/Wikipedia%3AStrony_do_usuni%C4%99cia
Disallow: /wiki/Wikipedia:Do_usuni%C4%99cia
Disallow: /wiki/Wikipedia%3ADo_usuni%C4%99cia
Disallow: /wiki/Wikipedia:SDU/
Disallow: /wiki/Wikipedia%3ASDU/
Disallow: /wiki/Wikipedia:Strony_podejrzane_o_naruszenie_praw_autorskich
Disallow: /wiki/Wikipedia%3AStrony_podejrzane_o_naruszenie_praw_autorskich
#
# en.wikinews:
# https://bugzilla.wikimedia.org/show_bug.cgi?id=5340
Disallow: /wiki/Portal:Prepared_stories/
Disallow: /wiki/Portal%3APrepared_stories/
#
# it.wikinews, http://bugzilla.wikimedia.org/show_bug.cgi?id=9138
Disallow: /wiki/Wikinotizie:Richieste_di_cancellazione
Disallow: /wiki/Wikinotizie:Sospette_violazioni_di_copyright
Disallow: /wiki/Categoria:Da_cancellare_subito
Disallow: /wiki/Categoria:Da_cancellare_subito_per_violazione_integrale_copyright
Disallow: /wiki/Wikinotizie:Storie_in_preparazione
#
# he.wikipedia, http://bugzilla.wikimedia.org/show_bug.cgi?id=9517
Disallow: /wiki/ויקיפדיה:רשימת_מועמדים_למחיקה/
Disallow: /wiki/ויקיפדיה%3Aרשימת_מועמדים_למחיקה/
Disallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94:%D7%A8%D7%A9%D7%99%D7%9E%D7%AA_%D7%9E%D7%95%D7%A2%D7%9E%D7%93%D7%99%D7%9D_%D7%9C%D7%9E%D7%97%D7%99%D7%A7%D7%94/
Disallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94%3A%D7%A8%D7%A9%D7%99%D7%9E%D7%AA_%D7%9E%D7%95%D7%A2%D7%9E%D7%93%D7%99%D7%9D_%D7%9C%D7%9E%D7%97%D7%99%D7%A7%D7%94/
#
Disallow: /wiki/ויקיפדיה:ערכים_לא_קיימים_ומוגנים
Disallow: /wiki/ויקיפדיה%3Aערכים_לא_קיימים_ומוגנים
Disallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94:%D7%A2%D7%A8%D7%9B%D7%99%D7%9D_%D7%9C%D7%90_%D7%A7%D7%99%D7%99%D7%9E%D7%99%D7%9D_%D7%95%D7%9E%D7%95%D7%92%D7%A0%D7%99%D7%9D
Disallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94%3A%D7%A2%D7%A8%D7%9B%D7%99%D7%9D_%D7%9C%D7%90_%D7%A7%D7%99%D7%99%D7%9E%D7%99%D7%9D_%D7%95%D7%9E%D7%95%D7%92%D7%A0%D7%99%D7%9D
Disallow: /wiki/ויקיפדיה:דפים_לא_קיימים_ומוגנים
Disallow: /wiki/ויקיפדיה%3Aדפים_לא_קיימים_ומוגנים
Disallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94:%D7%93%D7%A4%D7%99%D7%9D_%D7%9C%D7%90_%D7%A7%D7%99%D7%99%D7%9E%D7%99%D7%9D_%D7%95%D7%9E%D7%95%D7%92%D7%A0%D7%99%D7%9D
Disallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94%3A%D7%93%D7%A4%D7%99%D7%9D_%D7%9C%D7%90_%D7%A7%D7%99%D7%99%D7%9E%D7%99%D7%9D_%D7%95%D7%9E%D7%95%D7%92%D7%A0%D7%99%D7%9D
#
# sv.wikipedia, #10229
Disallow: /wiki/Wikipedia%3ASidor_f%C3%B6reslagna_f%C3%B6r_radering
Disallow: /wiki/Wikipedia:Sidor_f%C3%B6reslagna_f%C3%B6r_radering
Disallow: /wiki/Wikipedia:Sidor_föreslagna_för_radering
#
Disallow: /wiki/Användare
Disallow: /wiki/Anv%C3%A4ndare
#
Disallow: /wiki/Användardiskussion
Disallow: /wiki/Anv%C3%A4ndardiskussion
#
Disallow: /wiki/Wikipedia:Skyddade_sidnamn
Disallow: /wiki/Wikipedia%3ASkyddade_sidnamn
#
## *at least* 1 second please. preferably more :D
## we're disabling this experimentally 11-09-2006
#Crawl-delay: 1
#
Disallow: /wiki/Wikibooks:Votes_for_deletion
#
# 11291
Disallow: /wiki/Wikipedia:Sidor_som_bör_raderas
Disallow: /wiki/Wikipedia:Sidor_som_b%C3%B6r_raderas
Disallow: /wiki/Wikipedia%3ASidor_som_b%C3%B6r_raderas
#
# 11261
Disallow: /wiki/Wikipedia:Requests_for_arbitration/
Disallow: /wiki/Wikipedia%3ARequests_for_arbitration/
Disallow: /wiki/Wikipedia:Requests_for_comment/
Disallow: /wiki/Wikipedia%3ARequests_for_comment/
Disallow: /wiki/Wikipedia:Requests_for_adminship/
Disallow: /wiki/Wikipedia%3ARequests_for_adminship/
#
# 10288
Disallow: /wiki/Wikipedia_talk:Articles_for_deletion/
Disallow: /wiki/Wikipedia_talk%3AArticles_for_deletion/
Disallow: /wiki/Wikipedia_talk:Votes_for_deletion/
Disallow: /wiki/Wikipedia_talk%3AVotes_for_deletion/
Disallow: /wiki/Wikipedia_talk:Pages_for_deletion/
Disallow: /wiki/Wikipedia_talk%3APages_for_deletion/
Disallow: /wiki/Wikipedia_talk:Miscellany_for_deletion/
Disallow: /wiki/Wikipedia_talk%3AMiscellany_for_deletion/
Disallow: /wiki/Wikipedia_talk:Miscellaneous_deletion/
Disallow: /wiki/Wikipedia_talk%3AMiscellaneous_deletion/
#
# 6746
Disallow: /wiki/Wikipedia:Consultas_de_borrado/
Disallow: /wiki/Wikipedia%3AConsultas_de_borrado/
#
# 11432
Disallow: /wiki/Bruker:
Disallow: /wiki/Bruker%3A
Disallow: /wiki/Brukerdiskusjon
Disallow: /wiki/Wikipedia:Administratorer
Disallow: /wiki/Wikipedia%3AAdministratorer
Disallow: /wiki/Wikipedia-diskusjon:Administratorer
Disallow: /wiki/Wikipedia-diskusjon%3AAdministratorer
Disallow: /wiki/Wikipedia:Sletting
Disallow: /wiki/Wikipedia%3ASletting
Disallow: /wiki/Wikipedia-diskusjon:Sletting
Disallow: /wiki/Wikipedia-diskusjon%3ASletting
Disallow: /wiki/Spesial:
Disallow: /wiki/Spesial%3A
#
# working...
Disallow: /wiki/Fundraising_2007/comments
#
# 14793
Disallow: /wiki/Wikipedia:Changing_username
Disallow: /wiki/Wikipedia%3AChanging_username
Disallow: /wiki/Wikipedia:Changing_username/
Disallow: /wiki/Wikipedia%3AChanging_username/
Disallow: /wiki/Wikipedia_talk:Changing_username
Disallow: /wiki/Wikipedia_talk%3AChanging_username
Disallow: /wiki/Wikipedia_talk:Changing_username/
Disallow: /wiki/Wikipedia_talk%3AChanging_username/
#
# 15095
Disallow: /wiki/Wikiquote:Votes_for_deletion/
Disallow: /wiki/Wikiquote%3AVotes_for_deletion/
Disallow: /wiki/Wikiquote_talk:Votes_for_deletion/
Disallow: /wiki/Wikiquote_talk%3AVotes_for_deletion/
Disallow: /wiki/Wikiquote:Votes_for_deletion_archive/
Disallow: /wiki/Wikiquote%3AVotes_for_deletion_archive/
Disallow: /wiki/Wikiquote_talk:Votes_for_deletion_archive/
Disallow: /wiki/Wikiquote_talk%3AVotes_for_deletion_archive/
Disallow: /wiki/Special:Maintenance
# Do not show banner content or record hits
Disallow: /wiki/Special:BannerLoader
Disallow: /wiki/Special:RecordImpression
#
#
#----------------------------------------------------------#
#
#
#
# <!-- Please do not remove the space at the start of this line, it breaks the rendering. http://www.robotstxt.org/orig.html says spaces before comments are OK. --><syntaxhighlight lang="robots">
#
# Localisable part of robots.txt for en.wikipedia.org
#
# Edit at https://en.wikipedia.org/w/index.php?title=MediaWiki:Robots.txt&action=edit
# Don't add newlines here. All rules set here are active for every user-agent.
#
# Please check any changes using a syntax validator such as http://tool.motoricerca.info/robots-checker.phtml
# Enter https://en.wikipedia.org/robots.txt as the URL to check.
#
# https://bugzilla.wikimedia.org/show_bug.cgi?id=14075
Disallow: /wiki/MediaWiki:Spam-blacklist
Disallow: /wiki/MediaWiki%3ASpam-blacklist
Disallow: /wiki/MediaWiki_talk:Spam-blacklist
Disallow: /wiki/MediaWiki_talk%3ASpam-blacklist
Disallow: /wiki/Wikipedia:WikiProject_Spam
Disallow: /wiki/Wikipedia_talk:WikiProject_Spam
#
# Folks get annoyed when XfD discussions end up the number 1 google hit for
# their name.
# http://bugzilla.wikimedia.org/show_bug.cgi?id=4776
Disallow: /wiki/Wikipedia:Articles_for_deletion/
Disallow: /wiki/Wikipedia%3AArticles_for_deletion/
Disallow: /wiki/Wikipedia:Votes_for_deletion/
Disallow: /wiki/Wikipedia%3AVotes_for_deletion/
Disallow: /wiki/Wikipedia:Pages_for_deletion/
Disallow: /wiki/Wikipedia%3APages_for_deletion/
Disallow: /wiki/Wikipedia:Miscellany_for_deletion/
Disallow: /wiki/Wikipedia%3AMiscellany_for_deletion/
Disallow: /wiki/Wikipedia:Miscellaneous_deletion/
Disallow: /wiki/Wikipedia%3AMiscellaneous_deletion/
Disallow: /wiki/Wikipedia:Categories_for_discussion/
Disallow: /wiki/Wikipedia%3ACategories_for_discussion/
Disallow: /wiki/Wikipedia:Templates_for_deletion/
Disallow: /wiki/Wikipedia%3ATemplates_for_deletion/
Disallow: /wiki/Wikipedia:Redirects_for_discussion/
Disallow: /wiki/Wikipedia%3ARedirects_for_discussion/
Disallow: /wiki/Wikipedia:Deletion_review/
Disallow: /wiki/Wikipedia%3ADeletion_review/
Disallow: /wiki/Wikipedia:WikiProject_Deletion_sorting/
Disallow: /wiki/Wikipedia%3AWikiProject_Deletion_sorting/
Disallow: /wiki/Wikipedia:Files_for_deletion/
Disallow: /wiki/Wikipedia%3AFiles_for_deletion/
Disallow: /wiki/Wikipedia:Possibly_unfree_files/
Disallow: /wiki/Wikipedia%3APossibly_unfree_files/
#
# http://bugzilla.wikimedia.org/show_bug.cgi?id=10288
Disallow: /wiki/Wikipedia_talk:Articles_for_deletion/
Disallow: /wiki/Wikipedia_talk%3AArticles_for_deletion/
Disallow: /wiki/Wikipedia_talk:Votes_for_deletion/
Disallow: /wiki/Wikipedia_talk%3AVotes_for_deletion/
Disallow: /wiki/Wikipedia_talk:Pages_for_deletion/
Disallow: /wiki/Wikipedia_talk%3APages_for_deletion/
Disallow: /wiki/Wikipedia_talk:Miscellany_for_deletion/
Disallow: /wiki/Wikipedia_talk%3AMiscellany_for_deletion/
Disallow: /wiki/Wikipedia_talk:Miscellaneous_deletion/
Disallow: /wiki/Wikipedia_talk%3AMiscellaneous_deletion/
Disallow: /wiki/Wikipedia_talk:Templates_for_deletion/
Disallow: /wiki/Wikipedia_talk%3ATemplates_for_deletion/
Disallow: /wiki/Wikipedia_talk:Categories_for_discussion/
Disallow: /wiki/Wikipedia_talk%3ACategories_for_discussion/
Disallow: /wiki/Wikipedia_talk:Deletion_review/
Disallow: /wiki/Wikipedia_talk%3ADeletion_review/
Disallow: /wiki/Wikipedia_talk:WikiProject_Deletion_sorting/
Disallow: /wiki/Wikipedia_talk%3AWikiProject_Deletion_sorting/
Disallow: /wiki/Wikipedia_talk:Files_for_deletion/
Disallow: /wiki/Wikipedia_talk%3AFiles_for_deletion/
Disallow: /wiki/Wikipedia_talk:Possibly_unfree_files/
Disallow: /wiki/Wikipedia_talk%3APossibly_unfree_files/
#
Disallow: /wiki/Wikipedia:Copyright_problems
Disallow: /wiki/Wikipedia%3ACopyright_problems
Disallow: /wiki/Wikipedia_talk:Copyright_problems
Disallow: /wiki/Wikipedia_talk%3ACopyright_problems
Disallow: /wiki/Wikipedia:Suspected_copyright_violations
Disallow: /wiki/Wikipedia%3ASuspected_copyright_violations
Disallow: /wiki/Wikipedia_talk:Suspected_copyright_violations
Disallow: /wiki/Wikipedia_talk%3ASuspected_copyright_violations
Disallow: /wiki/Wikipedia:Contributor_copyright_investigations
Disallow: /wiki/Wikipedia%3AContributor_copyright_investigations
Disallow: /wiki/Wikipedia:Contributor_copyright_investigations/
Disallow: /wiki/Wikipedia%3AContributor_copyright_investigations/
Disallow: /wiki/Wikipedia_talk:Contributor_copyright_investigations
Disallow: /wiki/Wikipedia_talk%3AContributor_copyright_investigations
Disallow: /wiki/Wikipedia_talk:Contributor_copyright_investigations/
Disallow: /wiki/Wikipedia_talk%3AContributor_copyright_investigations/
Disallow: /wiki/Wikipedia:Protected_titles/
Disallow: /wiki/Wikipedia%3AProtected_titles/
Disallow: /wiki/Wikipedia_talk:Protected_titles/
Disallow: /wiki/Wikipedia_talk%3AProtected_titles/
Disallow: /wiki/Wikipedia:Articles_for_creation/
Disallow: /wiki/Wikipedia%3AArticles_for_creation/
Disallow: /wiki/Wikipedia_talk:Articles_for_creation/
Disallow: /wiki/Wikipedia_talk%3AArticles_for_creation/
Disallow: /wiki/Wikipedia_talk:Article_wizard/
Disallow: /wiki/Wikipedia_talk%3AArticle_wizard/
#
# http://bugzilla.wikimedia.org/show_bug.cgi?id=11261
Disallow: /wiki/Wikipedia:Requests_for_arbitration/
Disallow: /wiki/Wikipedia%3ARequests_for_arbitration/
Disallow: /wiki/Wikipedia_talk:Requests_for_arbitration/
Disallow: /wiki/Wikipedia_talk%3ARequests_for_arbitration/
Disallow: /wiki/Wikipedia:Requests_for_comment/
Disallow: /wiki/Wikipedia%3ARequests_for_comment/
Disallow: /wiki/Wikipedia_talk:Requests_for_comment/
Disallow: /wiki/Wikipedia_talk%3ARequests_for_comment/
Disallow: /wiki/Wikipedia:Requests_for_adminship/
Disallow: /wiki/Wikipedia%3ARequests_for_adminship/
Disallow: /wiki/Wikipedia_talk:Requests_for_adminship/
Disallow: /wiki/Wikipedia_talk%3ARequests_for_adminship/
#
# http://bugzilla.wikimedia.org/show_bug.cgi?id=12111
Disallow: /wiki/Wikipedia:Requests_for_checkuser/
Disallow: /wiki/Wikipedia%3ARequests_for_checkuser/
Disallow: /wiki/Wikipedia_talk:Requests_for_checkuser/
Disallow: /wiki/Wikipedia_talk%3ARequests_for_checkuser/
#
# https://bugzilla.wikimedia.org/show_bug.cgi?id=13398
Disallow: /wiki/Wikipedia:WikiProject_Spam/
Disallow: /wiki/Wikipedia%3AWikiProject_Spam/
#
# http://bugzilla.wikimedia.org/show_bug.cgi?id=14793
Disallow: /wiki/Wikipedia:Changing_username
Disallow: /wiki/Wikipedia%3AChanging_username
Disallow: /wiki/Wikipedia:Changing_username/
Disallow: /wiki/Wikipedia%3AChanging_username/
Disallow: /wiki/Wikipedia_talk:Changing_username
Disallow: /wiki/Wikipedia_talk%3AChanging_username
Disallow: /wiki/Wikipedia_talk:Changing_username/
Disallow: /wiki/Wikipedia_talk%3AChanging_username/
#
Disallow: /wiki/Wikipedia:Administrators%27_noticeboard/
Disallow: /wiki/Wikipedia%3AAdministrators%27_noticeboard/
Disallow: /wiki/Wikipedia_talk:Administrators%27_noticeboard/
Disallow: /wiki/Wikipedia_talk%3AAdministrators%27_noticeboard/
Disallow: /wiki/Wikipedia:Community_sanction_noticeboard/
Disallow: /wiki/Wikipedia%3ACommunity_sanction_noticeboard/
Disallow: /wiki/Wikipedia_talk:Community_sanction_noticeboard/
Disallow: /wiki/Wikipedia_talk%3ACommunity_sanction_noticeboard/
Disallow: /wiki/Wikipedia:Bureaucrats%27_noticeboard/
Disallow: /wiki/Wikipedia%3ABureaucrats%27_noticeboard/
Disallow: /wiki/Wikipedia_talk:Bureaucrats%27_noticeboard/
Disallow: /wiki/Wikipedia_talk%3ABureaucrats%27_noticeboard/
#
Disallow: /wiki/Wikipedia:Sockpuppet_investigations/
Disallow: /wiki/Wikipedia%3ASockpuppet_investigations/
Disallow: /wiki/Wikipedia_talk:Sockpuppet_investigations/
Disallow: /wiki/Wikipedia_talk%3ASockpuppet_investigations/
#
Disallow: /wiki/Wikipedia:Neutral_point_of_view/Noticeboard/
Disallow: /wiki/Wikipedia%3ANeutral_point_of_view/Noticeboard/
Disallow: /wiki/Wikipedia_talk:Neutral_point_of_view/Noticeboard/
Disallow: /wiki/Wikipedia_talk%3ANeutral_point_of_view/Noticeboard/
#
Disallow: /wiki/Wikipedia:No_original_research/noticeboard/
Disallow: /wiki/Wikipedia%3ANo_original_research/noticeboard/
Disallow: /wiki/Wikipedia_talk:No_original_research/noticeboard/
Disallow: /wiki/Wikipedia_talk%3ANo_original_research/noticeboard/
#
Disallow: /wiki/Wikipedia:Fringe_theories/Noticeboard/
Disallow: /wiki/Wikipedia%3AFringe_theories/Noticeboard/
Disallow: /wiki/Wikipedia_talk:Fringe_theories/Noticeboard/
Disallow: /wiki/Wikipedia_talk%3AFringe_theories/Noticeboard/
#
Disallow: /wiki/Wikipedia:Conflict_of_interest/Noticeboard/
Disallow: /wiki/Wikipedia%3AConflict_of_interest/Noticeboard/
Disallow: /wiki/Wikipedia_talk:Conflict_of_interest/Noticeboard/
Disallow: /wiki/Wikipedia_talk%3AConflict_of_interest/Noticeboard/
#
Disallow: /wiki/Wikipedia:Long_term_abuse/
Disallow: /wiki/Wikipedia%3ALong_term_abuse/
Disallow: /wiki/Wikipedia_talk:Long_term_abuse/
Disallow: /wiki/Wikipedia_talk%3ALong_term_abuse/
#
Disallow: /wiki/Wikipedia:Abuse_reports/
Disallow: /wiki/Wikipedia%3AAbuse_reports/
Disallow: /wiki/Wikipedia_talk:Abuse_reports/
Disallow: /wiki/Wikipedia_talk%3AAbuse_reports/
#
Disallow: /wiki/Wikipedia:Reliable_sources/Noticeboard/
Disallow: /wiki/Wikipedia%3AReliable_sources/Noticeboard/
Disallow: /wiki/Wikipedia_talk:Reliable_sources/Noticeboard/
Disallow: /wiki/Wikipedia_talk%3AReliable_sources/Noticeboard/
#
Disallow: /wiki/Wikipedia:Suspected_sock_puppets/
Disallow: /wiki/Wikipedia%3ASuspected_sock_puppets/
Disallow: /wiki/Wikipedia_talk:Suspected_sock_puppets/
Disallow: /wiki/Wikipedia_talk%3ASuspected_sock_puppets/
#
Disallow: /wiki/Wikipedia:Biographies_of_living_persons/Noticeboard/
Disallow: /wiki/Wikipedia%3ABiographies_of_living_persons/Noticeboard/
Disallow: /wiki/Wikipedia_talk:Biographies_of_living_persons/Noticeboard/
Disallow: /wiki/Wikipedia_talk%3ABiographies_of_living_persons/Noticeboard/
#
Disallow: /wiki/Wikipedia:Content_noticeboard/
Disallow: /wiki/Wikipedia%3AContent_noticeboard/
Disallow: /wiki/Wikipedia_talk:Content_noticeboard/
Disallow: /wiki/Wikipedia_talk%3AContent_noticeboard/
#
Disallow: /wiki/Template:Editnotices/
Disallow: /wiki/Template%3AEditnotices/
#
Disallow: /wiki/Wikipedia:Arbitration/
Disallow: /wiki/Wikipedia%3AArbitration/
Disallow: /wiki/Wikipedia_talk:Arbitration/
Disallow: /wiki/Wikipedia_talk%3AArbitration/
#
Disallow: /wiki/Wikipedia:Arbitration_Committee/
Disallow: /wiki/Wikipedia%3AArbitration_Committee/
Disallow: /wiki/Wikipedia_talk:Arbitration_Committee/
Disallow: /wiki/Wikipedia_talk%3AArbitration_Committee/
#
Disallow: /wiki/Wikipedia:Arbitration_Committee_Elections
Disallow: /wiki/Wikipedia%3AArbitration_Committee_Elections
Disallow: /wiki/Wikipedia_talk:Arbitration_Committee_Elections
Disallow: /wiki/Wikipedia_talk%3AArbitration_Committee_Elections
#
Disallow: /wiki/Wikipedia:Mediation_Committee/
Disallow: /wiki/Wikipedia%3AMediation_Committee/
Disallow: /wiki/Wikipedia_talk:Mediation_Committee/
Disallow: /wiki/Wikipedia_talk%3AMediation_Committee/
#
Disallow: /wiki/Wikipedia:Mediation_Cabal/Cases/
Disallow: /wiki/Wikipedia%3AMediation_Cabal/Cases/
#
Disallow: /wiki/Wikipedia:Requests_for_bureaucratship/
Disallow: /wiki/Wikipedia%3ARequests_for_bureaucratship/
Disallow: /wiki/Wikipedia_talk:Requests_for_bureaucratship/
Disallow: /wiki/Wikipedia_talk%3ARequests_for_bureaucratship/
#
Disallow: /wiki/Wikipedia:Administrator_review/
Disallow: /wiki/Wikipedia%3AAdministrator_review/
Disallow: /wiki/Wikipedia_talk:Administrator_review/
Disallow: /wiki/Wikipedia_talk%3AAdministrator_review/
#
Disallow: /wiki/Wikipedia:Editor_review/
Disallow: /wiki/Wikipedia%3AEditor_review/
Disallow: /wiki/Wikipedia_talk:Editor_review/
Disallow: /wiki/Wikipedia_talk%3AEditor_review/
#
Disallow: /wiki/Wikipedia:Article_Incubator/
Disallow: /wiki/Wikipedia%3AArticle_Incubator/
Disallow: /wiki/Wikipedia_talk:Article_Incubator/
Disallow: /wiki/Wikipedia_talk%3AArticle_Incubator/
#
Disallow: /wiki/Category:Noindexed_pages
Disallow: /wiki/Category%3ANoindexed_pages
#
# </syntaxhighlight>
Google is built by a large team of engineers, designers, researchers, robots, and others in many different sites across the globe. It is updated continuously, and built with more tools and technologies than we can shake a stick at. If you'd like to help us out, see google.com/careers.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment