Skip to content

Instantly share code, notes, and snippets.

@ilyahoilik
Last active May 1, 2021 21:44
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save ilyahoilik/c6d70d828f42aae429e3e84283fb85f7 to your computer and use it in GitHub Desktop.
Save ilyahoilik/c6d70d828f42aae429e3e84283fb85f7 to your computer and use it in GitHub Desktop.
# This robots.txt file is created by Ilya Hoilik.
# You can contribute by visiting the link bellow.
# https://gist.github.com/ilyahoilik/c6d70d828f42aae429e3e84283fb85f7
User-agent: *
# Folders and files
Disallow: /admin
Disallow: /api
Disallow: /applications
Disallow: /datastore
Disallow: /dev
Disallow: /oauth
Disallow: /plugins
Disallow: /system
Disallow: /uploads
Disallow: /vendor
Disallow: /404error.php
Disallow: /Credits.txt
Disallow: /error.php
# Authentication
Disallow: /login
Disallow: /logout
Disallow: /register
Disallow: /lostpassword
# Legal information
Disallow: /privacy
Disallow: /cookies
Disallow: /guidelines
Disallow: /terms
# Staff and users
Disallow: /online
Disallow: /staff
# Contact page
Disallow: /contact
# Announcements
Disallow: /announcement*
# Content discovery
Disallow: /search
Disallow: /discover*
Disallow: /rss*
Disallow: /tags*
Disallow: /activity
Disallow: /new-content
# Content promotion
Disallow: /promote*
Disallow: /ourpicks
# Leaderboard
Disallow: /leaderboard
Disallow: /pastleaders
Disallow: /topmembers
# With extra parameters
Disallow: /*do=*
Disallow: /*sort=*
Disallow: /*sortby=*
Disallow: /*csrf=*
Disallow: /*csrfKey=*
Disallow: */?tab=*
Disallow: */?_fromLogin=*
Disallow: */?_fromLogout=*
# Create and edit pages
Disallow: */submit
Disallow: */create
Disallow: */edit
# Allow attachments in the uploads folder
Allow: /uploads/monthly_*_*/*
Host: https://example.com
Sitemap: https://example.com/sitemap.php
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment