Skip to content
Create a gist now

Instantly share code, notes, and snippets.

@Stanback /.htaccess
Last active

Example Apache configuration for serving pre-rendered HTML from Javascript pages/apps using the Prerender Service at https://github.com/collectiveip/prerender
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /prerender-test/
# Virtual/pushState URIs (requests not matching an existing file get forwarded to index.html)
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule .* index.html [L]
<IfModule mod_proxy_http.c>
# Enable prerendering for .html and directory index files
RewriteCond %{HTTP_USER_AGENT} googlebot|yahoo|bingbot|baiduspider [NC,OR]
RewriteCond %{QUERY_STRING} _escaped_fragment_|prerender=1
RewriteCond %{HTTP_USER_AGENT} !^Prerender
RewriteRule ^(.*\.html)?$ http://prerender.herokuapp.com/http://www.yourdomain.com/prerender-test/$1 [P,L]
</IfModule>
</IfModule>
# Optionally add a ProxyPassReverse directive to ensure that 301/302 redirects
# issued by the prerender service are correctly forwarded to the client.
# Note that this must be located in your Apache config rather than .htaccess
# <IfModule mod_proxy_http.c>
# ProxyPassReverse /prerender-test/ http://prerender.herokuapp.com/http://www.yourdomain.com/prerender-test/
# </IfModule>
@homerjam

Here is a more extensive list of some of the latest bot user agents sourced from http://user-agent-string.info/list-of-ua/bots

Googlebot|bingbot|Googlebot-Mobile|Baiduspider|Yahoo|YahooSeeker|DoCoMo|Twitterbot|TweetmemeBot|Twikle|Netseer|Daumoa|SeznamBot|Ezooms|MSNBot|Exabot|MJ12bot|sogou\sspider|YandexBot|bitlybot|ia_archiver|proximic|spbot|ChangeDetection|NaverBot|MetaJobBot|magpie-crawler|Genieo\sWeb\sfilter|Qualidator.com\sBot|Woko|Vagabondo|360Spider|ExB\sLanguage\sCrawler|AddThis.com|aiHitBot|Spinn3r|BingPreview|GrapeshotCrawler|CareerBot|ZumBot|ShopWiki|bixocrawler|uMBot|sistrix|linkdexbot|AhrefsBot|archive.org_bot|SeoCheckBot|TurnitinBot|VoilaBot|SearchmetricsBot|Butterfly|Yahoo!|Plukkie|yacybot|trendictionbot|UASlinkChecker|Blekkobot|Wotbox|YioopBot|meanpathbot|TinEye|LuminateBot|FyberSpider|Infohelfer|linkdex.com|Curious\sGeorge|Fetch-Guess|ichiro|MojeekBot|SBSearch|WebThumbnail|socialbm_bot|SemrushBot|Vedma|alexa\ssite\saudit|SEOkicks-Robot|Browsershots|BLEXBot|woriobot|AMZNKAssocBot|Speedy|oBot|HostTracker|OpenWebSpider|WBSearchBot|FacebookExternalHit

@Stanback
Owner

Thanks for the list @homerjam!

@abobwhite

Hey guys, this isn't working for me. The request hits prerender but everything gets loaded into prerender as index.html and returns a blank html page: <html><head></head><body></body></html> (I am using User-Agent Switcher as a chrome extension to test behavior).

Before I switched to the pushState method, prerender was working with the "#!" method and I didn't need the following:

RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule .* index.html [L]

Thoughts?

@abobwhite

Hey again. Just wanted to let you guys know that I got things figured out but the fallback URL condition I listed above needed to be below the prerender proxy rewrite rule or else the page falls back to index.html before there is a chance to send the url to prerender.

@Stanback
Owner

FYI:
Googlebot is starting to render Javascript so you may want to remove it from the user agents list (after testing in webmaster tools)

See:
http://googlewebmastercentral.blogspot.com/2014/05/rendering-pages-with-fetch-as-google.html

@yagobski

How to blacklist a single url path or multiple url paths. Compares using regex, so be specific when possible. If a blacklist is supplied, all url's will be prerendered except ones containing a blacklist path.

Do you have any solution with htaccess . This exist for other plugins

@wichon

Hi @yagobski !, did you find a solution for blacklisting paths using htaccess?, i have the same problem :/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.