# Change YOUR_TOKEN to your prerender token | |
# Change http://example.com (at the end of the last RewriteRule) to your website url | |
<IfModule mod_headers.c> | |
RequestHeader set X-Prerender-Token "YOUR_TOKEN" | |
</IfModule> | |
<IfModule mod_rewrite.c> | |
RewriteEngine On | |
<IfModule mod_proxy_http.c> | |
RewriteCond %{HTTP_USER_AGENT} googlebot|bingbot|yandex|baiduspider|facebookexternalhit|twitterbot|rogerbot|linkedinbot|embedly|quora\ link\ preview|showyoubot|outbrain|pinterest\/0\.|pinterestbot|slackbot|vkShare|W3C_Validator [NC,OR] | |
RewriteCond %{QUERY_STRING} _escaped_fragment_ | |
# Only proxy the request to Prerender if it's a request for HTML | |
RewriteRule ^(?!.*?(\.js|\.css|\.xml|\.less|\.png|\.jpg|\.jpeg|\.gif|\.pdf|\.doc|\.txt|\.ico|\.rss|\.zip|\.mp3|\.rar|\.exe|\.wmv|\.doc|\.avi|\.ppt|\.mpg|\.mpeg|\.tif|\.wav|\.mov|\.psd|\.ai|\.xls|\.mp4|\.m4a|\.swf|\.dat|\.dmg|\.iso|\.flv|\.m4v|\.torrent|\.ttf|\.woff|\.svg))(.*) http://service.prerender.io/http://example.com/$2 [P,L] | |
</IfModule> | |
</IfModule> |
This comment has been minimized.
This comment has been minimized.
Thanks @benceg! |
This comment has been minimized.
This comment has been minimized.
For me, I had to change the proxy target to remove an extra "/", so http://service.prerender.io/http://example.com$2 This is because the pattern being matched has a leading '/' already. |
This comment has been minimized.
This comment has been minimized.
There is an issue where the service is caching the / home page for all urls. Any idea what might be the issue Thanks |
This comment has been minimized.
This comment has been minimized.
@baki250 just saw your comment. Following up with you via email. |
This comment has been minimized.
This comment has been minimized.
@baki250 I had a similar issue except it was always caching /index.html. The cause was that I had another rewrite for pushstate that always returned index.html: RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule .* index.html [L] The problem was that if this rule is carried out first, then the second rule receives /index.html as its URI. So prerender is always being asked for example.com/index.html. The solution is to switch the order around. |
This comment has been minimized.
This comment has been minimized.
Thanks @iolo-matchbook! |
This comment has been minimized.
This comment has been minimized.
your missing "visionutils" and "Facebot" for facebook user agents |
This comment has been minimized.
This comment has been minimized.
|
This comment has been minimized.
This comment has been minimized.
Thanks @ianmstew! |
This comment has been minimized.
This comment has been minimized.
Hey @thoop, I ran into an issue with this .htaccess that had me stumped for days, until I realized it was the issue proxying the server root specifically using mod_rewrite described here and here. Both solutions recommended using the ProxyPass directive instead, which does not work in our case requiring special rewrite conditions. I discovered a workaround, however. This solves the server root case where, by the time mod_rewrite rules are evaluated, the incoming blank ("/") root request has already been converted to "index.html" by Apache. I would love to find a more elegant solution, but for now this has solved my problem. RewriteCond %{HTTP_USER_AGENT} baiduspider|facebookexternalhit|twitterbot|rogerbot|linkedinbot|embedly|quora\ link\ preview|showyoubot|outbrain|pinterest [NC,OR]
RewriteCond %{QUERY_STRING} _escaped_fragment_
RewriteCond %{REQUEST_URI} ^/index\.html$
# Proxy the server root
RewriteRule .* http://service.prerender.io/http://example.com/ [P,L]
RewriteCond %{HTTP_USER_AGENT} baiduspider|facebookexternalhit|twitterbot|rogerbot|linkedinbot|embedly|quora\ link\ preview|showyoubot|outbrain|pinterest [NC,OR]
RewriteCond %{QUERY_STRING} _escaped_fragment_
# Only proxy the request to Prerender if it's a request for HTML
RewriteRule ^(?!.*?(\.js|\.css|\.xml|\.less|\.png|\.jpg|\.jpeg|\.gif|\.pdf|\.doc|\.txt|\.ico|\.rss|\.zip|\.mp3|\.rar|\.exe|\.wmv|\.doc|\.avi|\.ppt|\.mpg|\.mpeg|\.tif|\.wav|\.mov|\.psd|\.ai|\.xls|\.mp4|\.m4a|\.swf|\.dat|\.dmg|\.iso|\.flv|\.m4v|\.torrent))(.*) http://service.prerender.io/http://example.com/$2 [P,L] |
This comment has been minimized.
This comment has been minimized.
Hmm, interesting. Is this just because you have your server behind a path and not the root? |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
I couldn't get it working the first time round, it had to do with multiple things, the instructions on most sites were for html5 push state, while we used #! etc. So I pretty much tested every setting and configuration and it resulted in the following overview and solutions for both #! and html5 push state. SEO / Prerender.io (Angular) etc.How the solution works in general:
Instructions depending on your routing setup:Option 1: # Hash routingExample:
Problem: Solution: $locationProvider.hashPrefix('!');
$locationProvider.html5Mode(false); In html remove the following meta header if there: <head>
<!-- REMOVE --> <meta name="fragment" content="!"> <!-- /REMOVE -->
</head> Everytime a search engine finds a URI like this:
It will send a request like this:
Configure Apache:
Option 2: HTML5 push state routingExample:
Problem: Solution: In angular: $locationProvider.html5Mode(true); In html, add this meta header: <head>
<meta name="fragment" content="!">
</head> Everytime a search engine finds a URI like this:
It will send a request like this:
Configure Apache:
If submitted to prerender.io, how it sends requests to your server again
Other resources
|
This comment has been minimized.
This comment has been minimized.
You may want to omit |
This comment has been minimized.
This comment has been minimized.
Having trouble getting Google Bot (maybe others) cache my pages. I'm using EmberJS and Prerender.io. Here is my Apache .htaccess AddOutputFilterByType DEFLATE text/plain
And here is my index file head: ...I have included the sitemap.xml file that Prerender generates, and have cached all the main pages of this site. Any ideas on why this isn't working? Thanks for your help ahead of time. |
This comment has been minimized.
This comment has been minimized.
Send me an email at support@prerender.io if anyone has any issues. I don't get notified when someone comments on this gist. |
This comment has been minimized.
This comment has been minimized.
Why is google bot not included? |
This comment has been minimized.
This comment has been minimized.
Because I'm using CodeIgniter, I needed the URL that's passed to prerender.io to not include the hashbang, and include the In other words, instead of pages looking like the following in "Cached Pages"
I needed them to look like this
I used this:
For reference, my full .htaccess for a site with CodeIgniter + AngularJS is as follows:
I have my location provider set as:
|
This comment has been minimized.
This comment has been minimized.
im currently working on a project and this is my first time using angularjs and i think its pretty powerful but there's a problem. Right now im hosting my own Prerender Service. but whenever i request for a url after a #! http-:// -->> http:// ------ because i have limited links http-://prerender.host.com/http-://host.com/#!/ something it always renders the http-://host.com/ route and not the http-://host.com/#!/something. please help me!! i dont really know whats happening. i also tried html5mode with a fragment meta but still the same. SETUP
ROUTES i do have hashprefix and explicitly turning off html5mode knowing its off by default. --- i thought it was the problem :))
APACHE im using the settings from prerender apache guide.
Do i need to do something else or is there anything wrong with my setup?? i've figuring out this for 3 days and now i really need help. any suggestions?? i really need to finish this for our defense. |
This comment has been minimized.
This comment has been minimized.
Googlebot, bingbot, and many others are not included in the config because they support the |
This comment has been minimized.
This comment has been minimized.
Google Snippet (for sharing on Google+) is missing. Add this: |
This comment has been minimized.
This comment has been minimized.
Sorry if this question seems stupid Can anyone tell what to write in RedirectRule in .htaccess for localhost deployment? |
This comment has been minimized.
This comment has been minimized.
Just something i learned today, while trying to make this work with an angular. Apache by default tries to redirect your request to /index.php so you will need to set DirectoryIndex index.html before the request is proxied, hope this saves time to others. This is how we have it now: DirectoryIndex index.html
|
This comment has been minimized.
This comment has been minimized.
You might want to add applebot to the list like here |
This comment has been minimized.
This comment has been minimized.
Hey Guys, I have access to httpd.conf (/etc/httpd/conf/httpd.conf). Can I add those lines in this file directly? If yes, then do i need to change some other configuration also? like AllowOverride etc? i have other conf files with virtualhost configuration: ProxyPass / http://localhost:8080/ retry=0 LogFormat "%h (%{X-Forwarded-For}i) %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"" PS: it's an elastic beanstalk machine with tomcat stack and httpd server (httpd server is just there to forward request on port 80 to port 8080 of tomcat) |
This comment has been minimized.
This comment has been minimized.
Hi guys im new, i need help, i did an angular website but when i try crawler tools it give me the same problem everytime {{ description }} for example when i changed my .htaccess and changed the token on it gaves me the same problem. how i can solve that problem? |
This comment has been minimized.
This comment has been minimized.
Wanted to let anyone know. I had a problem where this wasn't working on a fresh server. The fix was to enable the mod_proxy_http module in apache. |
This comment has been minimized.
This comment has been minimized.
I can't seem to get my apache config working i go to prerender io and it tells me to instal token anyone can help? I have this code RewriteEngine On If requested resource exists as a file or directory(REQUEST_FILENAME is only relative in virtualhost context, so not usable)
If non existent
Handle Prerender.io
If non existent
|
This comment has been minimized.
This comment has been minimized.
@thelakshya @thoop |
This comment has been minimized.
This comment has been minimized.
thanks @vinceMeens . Your solution worked great for me! |
This comment has been minimized.
This comment has been minimized.
This SEO prerender.io is such a mess. I have Laravel 5.1 and Angularjs. So how should my .htaccess configuration look like? I'm using .htaccess that comes with Laravel. I have the hashBang #! set in my angular routes; meta fragment in my layout.blade.php like so: ?escaped_fragment= ... so whats next? My prerender.io key is in my .env - I'm totally confused. |
This comment has been minimized.
This comment has been minimized.
.svg and .svgz should be added to the excluded file extensions |
This comment has been minimized.
This comment has been minimized.
@fabiowitt In fact we changed the order of scripts, because when we used the order of @vinceMeens, our home did not render even using escape_frament. We changed If requested resource exists as a file or directory#(REQUEST_FILENAME is only relative in virtualhost context, so not usable) down of RewriteCond %{HTTP_USER_AGENT} baiduspider|facebookexternalhit|twitterbot|rogerbot|linkedinbot|embedly|quora\ link\ preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator [NC,OR] Proxy the requestRewriteRule ^(.*)$ http://service.prerender.io/http://%{HTTP_HOST}/$1 [P,L] in this way my home was found in prerender, is there any problem with this change @ vinceMeens? |
This comment has been minimized.
This comment has been minimized.
Thanks @vinceMeens |
This comment has been minimized.
This comment has been minimized.
The escaped_fragment aspect of this config only gets applied if your routes utilize parameter strings. If you use "pretty" urls like www.mysite.com/foo/bar, then that rewrite condition will not trigger for Google, Bing and others unless you add them yourself |
This comment has been minimized.
This comment has been minimized.
I am new with .htaccess and I am having some trouble with it
I am using Laravel 5 as backend and AngularJs for frontend.
In index.php I have:
|
This comment has been minimized.
This comment has been minimized.
Trying to implement vinceMeens solution with Angular but prerender seems to be receiving request for things minus a / somewhere and ends up with a 504. Prerenders crawlstats look like: When evidently it should be querying songsaboutanimals.co.uk/album or /assets/book/1.jpg htaccess is setup as Vince suggests, except I've tried moving the bit that replaces the / to after the prerender handling in hope it might make a difference rewrite portion looks like this:
Then in my Angular route controller: And in index.html: Going to songaboutanimals.co.uk/stories/ succesffully routes to songaboutanimals.co.uk/stories (minus slash) Site is live and viewable at songsaboutanimals.co.uk |
This comment has been minimized.
This comment has been minimized.
Solution appears to work once I force an additional slash into the line RewriteRule ^(.*)$ http://service.prerender.io/http://%{HTTP_HOST}{ADDITIONAL / HERE}$1 [P,L] |
This comment has been minimized.
This comment has been minimized.
Only thing I cant get now is for facebook etc to read my dynamically generated meta tags Fine when sharing the base url because that defaults set, but for any other URL, angular replaces the meta per page using ... but nothing is returned for facebook from the prerender shot Definatey coming through though... |
This comment has been minimized.
This comment has been minimized.
@vinceMeens I tried your solution. But its not working. Can anyone help me figure out this problem. (Drupal 7 + Angular JS) ` RewriteEngine On If requested resource exists as a file or directory(REQUEST_FILENAME is only relative in virtualhost context, so not usable)RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} -f [OR] Go to it as isRewriteRule ^ - [L] If non existentIf path ends with / and is not just a single /, redirect to without the trailing /RewriteCond %{REQUEST_URI} ^./$ Handle Prerender.ioDirectoryIndex /qld/qld_angular/index.html RewriteCond %{HTTP_USER_AGENT} baiduspider|facebookexternalhit|twitterbot|rogerbot|linkedinbot|embedly|quora\ link\ preview|showyoubot|outbrain|pinterest [NC,OR] Proxy the requestRewriteRule ^(.*)$ http://service.prerender.io/http://%{HTTP_HOST}$1 [P,L] If non existentAccept everything on index.htmlRewriteRule ^ /qld/qld_angular/index.html` |
This comment has been minimized.
This comment has been minimized.
Hello all, I am new in angular js. I have created a site using angularjs and for rendering I have used prerender.io services. How can I do it. |
This comment has been minimized.
This comment has been minimized.
@jsoncontrust Good observation. Is there any way to make the |
This comment has been minimized.
This comment has been minimized.
I have a Problem that Prerender cant cache links correct. Does this have something todo with : $urlRouterProvider.otherwise('/'); ? Also have: and in the index: Thanks alot |
This comment has been minimized.
This comment has been minimized.
I was having problem to get Prerender working and I want to share my solution. This is my
|
This comment has been minimized.
This comment has been minimized.
@toioski thanks it worked. But not working image and css with render. |
This comment has been minimized.
This comment has been minimized.
@toioski your solution works for me! Thanks! The only issue I still have is that when using prerender with this htaccess solution is that facebook crawler sees my Canonical URL as "http://service.prerender.io/https://www.domain.com" and this is also shown below a facebook post. Anyone know a solution for this problem? |
This comment has been minimized.
This comment has been minimized.
|
This comment has been minimized.
This comment has been minimized.
RewriteCond %{HTTP_USER_AGENT} (facebookexternalhit/[0-9]|Twitterbot|Pinterest|Slackbot\s[0-9.]|Slackbot-LinkExpanding\s[0-9.]|Slack-ImgProxy\s[0-9.]|Google.*snippet) RewriteRule ^(?!.?(.js|.css|.xml|.less|.png|.jpg|.jpeg|.gif|.pdf|.doc|.txt|.ico|.rss|.zip|.mp3|.rar|.exe|.wmv|.doc|.avi|.ppt|.mpg|.mpeg|.tif|.wav|.mov|.psd|.ai|.xls|.mp4|.m4a|.swf|.dat|.dmg|.iso|.flv|.m4v|.torrent|.ttf|.woff))(.) http://service.prerender.io/http://example.com/$2 [P,L] This is the code I have placed in my .htaccess file, but its working only for facebook and its not working for other socialbots(facebook, LinkedIn, Slack, etc..). Thanks, |
This comment has been minimized.
This comment has been minimized.
@simonpeters I'm having the same problem with the canonical URL appearing below a facebook post. |
This comment has been minimized.
This comment has been minimized.
With google having deprecated the escaped_fragment syntax (https://webmasters.googleblog.com/2015/10/deprecating-our-ajax-crawling-scheme.html), anecdotally in my apache logs I see a lot of requests from Google coming without the escaped_fragment parameter. I added |googlebot| to my .htaccess for redirection to pre-render and my cache hits went way up. Anyone else have similar experience? Is there any particular reason NOT to have googlebot explicitly listed? |
This comment has been minimized.
This comment has been minimized.
Hi I have an application developed with spring boot + angularjs I tried every solution that is suggested here and on web but could not succeed to make it work. Do you have any suggestions for me? Thanks. |
This comment has been minimized.
This comment has been minimized.
I'm having some issues with my .htaccess file for a site I've been working on - basically, it's a WordPress site with an Angular theme, hence the need for Prerender.io. What's happening is that the site is throwing a 404 every time the page is accessed, yet the page still displays and can be refreshed. Since it's throwing a 404, Prerender.io isn't picking it up for caching. This is only in HTML5 'pretty URLs' mode - it doesn't throw a 404 if you access the page with the hashbang. Here's my .htaccess file - any suggestions would be very much appreciated.
|
This comment has been minimized.
This comment has been minimized.
Thanks everyone for posting on this forum. I'm on AngularJS and apache, #! URLs. I noticed today that google was not crawling my site. I made the following changes: Per other comments, converted the #! RewriteCond %{QUERY_STRING} escaped_fragment=(%2F|/)(.). I also added the google bots to the list being the test searches did not work till I added the bots. Finally, I'm https so the url definition at the end of the file has to be HTTPS! Good luck everyone DirectoryIndex index.html
|
This comment has been minimized.
This comment has been minimized.
I don't get notified when someone comments on this gist. Send us an email at support@prerender.io if anyone has any questions or issues. |
This comment has been minimized.
This comment has been minimized.
THX @benceg this was what i was looking for for two days! Anyone suffering with Angular/Worpress/prerender.io/.htaccess combos look at his comment :) |
This comment has been minimized.
This comment has been minimized.
.htaccess
in index.html But robots can't see my site. Could you please help me. |
This comment has been minimized.
This comment has been minimized.
guys need help here, Where do I need to place .htaccess file ? I am using hashbang(#!) URL methodology and the steps I followed are as follows. Please suggest if there needs to be any changes
.htaccess file RequestHeader set X-Prerender-Token "XXXXXXXXXXXXXXXXX" RewriteEngine On
Now, the question is where do I need to place the .htaccess file and what other changes required in .htaccess configuartion. open for any suggestions. Thank you |
This comment has been minimized.
This comment has been minimized.
This forum helps me in my web design china base site. It still need some work. |
This comment has been minimized.
This comment has been minimized.
I am using zf2 and have apache installed on server. I have added the following lines in my .htaccess file. RewriteEngine On
But still it is not redirecting my url to corresponding prerender url. Even I have changed the User Agent to google bot and added the escaped_fragment as query paramter. Am I testing correctly or there is something wrong with this htaccess code ? |
This comment has been minimized.
This comment has been minimized.
How to pass username and password in htaccess if I am using basic auth |
This comment has been minimized.
This comment has been minimized.
Is there a way to have the page cached when a user visits a page? or should this be automatic and that I'm doing it wrong? |
This comment has been minimized.
This comment has been minimized.
It doesn't work when sharing in whatsapp even when adding the Whatsapp user agent. <IfModule mod_headers.c>
RequestHeader set X-Prerender-Token "%MYTOKEN%"
</IfModule>
<IfModule mod_rewrite.c>
RewriteEngine On
# If requested resource exists as a file or directory
# (REQUEST_FILENAME is only relative in virtualhost context, so not usable)
RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} -f [OR]
RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} -d
# Only exception is /index.htm, /index.html
RewriteCond %{REQUEST_URI} !/index\.html?
# Go to it as is
RewriteRule ^ - [L]
# If non existent
# If path ends with / and is not just a single /, redirect to without the trailing /
RewriteCond %{REQUEST_URI} ^.*/$
RewriteCond %{REQUEST_URI} !^/$
RewriteRule ^(.*)/$ $1 [R,QSA,L]
# If path that is not empty or / or /index.htm or /index.html, redirect to /#!/path
RewriteCond %{REQUEST_URI} !(/index\.html?|/|)$
RewriteRule ^(.*)$ /#!$1 [R,QSA,NE,L]
# If not /, redirect to it.
RewriteCond %{REQUEST_URI} !^/$
RewriteRule ^ / [R,QSA,L]
# Handle Prerender.io
RequestHeader set X-Prerender-Token "%MYTOKEN%"
RewriteCond %{HTTP_USER_AGENT} baiduspider|facebookexternalhit|twitterbot|Whatsapp|rogerbot|linkedinbot|embedly|quora\ link\ preview|showyoubot|outbrain|pinterest [NC,OR]
RewriteCond %{QUERY_STRING} _escaped_fragment_
RewriteCond %{QUERY_STRING} _escaped_fragment_=([^&]*)
# Proxy the request
RewriteRule ^ http://service.prerender.io/http://%{HTTP_HOST}%?_escaped_fragment_=%1 [P,L]
</IfModule> |
This comment has been minimized.
This comment has been minimized.
interesting note I found re: iMessage User Agent. Hope this helps someone! |
This comment has been minimized.
This comment has been minimized.
I don't get notified when someone comments on this gist. Send us an email at support@prerender.io if anyone has any questions or issues. |
This comment has been minimized.
This comment has been minimized.
it doesn't work on my site
please help me to check config |
This comment has been minimized.
This comment has been minimized.
If you have prerender working and running for all pages except the root(home) page.
I added (index.php or index.shtml)? to capture that out if found and changed the $2 to $3. Don't forget to change the http://www.example.com to your domain. |
This comment has been minimized.
This comment has been minimized.
Hi i have used prerender for my angular project i did everything for activate prerender however not success My web site url is:http://ugurdogrusoy.com/preparing/ firstly addred Then added prerender token inside .htaccess you can see rule and token below RewriteEngine On # If requested resource exists as a file or directory # (REQUEST_FILENAME is only relative in virtualhost context, so not usable) RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} -f [OR] RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} -d # Go to it as is RewriteRule ^ - [L]
i can't escape We haven't seen a request with your prerender token yet where have i done wrong you can help me ? Thanks |
This comment has been minimized.
This comment has been minimized.
Any Idea how to integrate that with tomcat? seems P flag in Tomcat doesn't work. |
This comment has been minimized.
This comment has been minimized.
If anybody else is having a problem with always returning index file if it's not a crawler, so your SPA router can catch 404, I created a small gist - |
This comment has been minimized.
This comment has been minimized.
A warning to anyone using vector files. You will need to add a |
This comment has been minimized.
This comment has been minimized.
If anyone needs to redirect .html files only (skipping extensions .xhtml,.shtml,.phtml,.jp,.cgi,.ppl,.pl,.perl,.php) |
This comment has been minimized.
This comment has been minimized.
@Dizzyspb Did you fix it? |
This comment has been minimized.
This comment has been minimized.
Hey all, I'm running into a problem when trying to Rewrite part of the URL being cached in prerender.io. I've tried removing that part of the URL using:
Doing this on the live server results in |
This comment has been minimized.
This comment has been minimized.
I have had the same issue, our site is using react, and we even tried to shift the config file further up to the root. Still the same issue. Any luck with this ? |
This comment has been minimized.
This comment has been minimized.
you are awesome!! i almost wept when i finally saw the token not used message go away. thanks!! |
This comment has been minimized.
This comment has been minimized.
Just adding 2 cents:
note the seemingly missing OR on the first line On the other hand, my second cent is that last WhatsApp entry - turns out WhatsApp does the scraping directly from inside the app (at least it did on my android phone today ;) So my full line is now (And my full htaccess
|
This comment has been minimized.
A warning to anyone using CodeIgniter (or any PHP script that warrants a RewriteRule on index.php): you will need to add a capture group to the proxy rule, for index.php.
The line will read as follows: