Create a gist now

Instantly share code, notes, and snippets.

@thoop /nginx.conf
Last active Jul 18, 2018

Embed
What would you like to do?
Official prerender.io nginx.conf for nginx
# Change YOUR_TOKEN to your prerender token and uncomment that line if you want to cache urls and view crawl stats
# Change example.com (server_name) to your website url
# Change /path/to/your/root to the correct value
server {
listen 80;
server_name example.com;
root /path/to/your/root;
index index.html;
location / {
try_files $uri @prerender;
}
location @prerender {
#proxy_set_header X-Prerender-Token YOUR_TOKEN;
set $prerender 0;
if ($http_user_agent ~* "googlebot|bingbot|yandex|baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator") {
set $prerender 1;
}
if ($args ~ "_escaped_fragment_") {
set $prerender 1;
}
if ($http_user_agent ~ "Prerender") {
set $prerender 0;
}
if ($uri ~* "\.(js|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent|ttf|woff|svg|eot)") {
set $prerender 0;
}
#resolve using Google's DNS server to force DNS resolution and prevent caching of IPs
resolver 8.8.8.8;
if ($prerender = 1) {
#setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
set $prerender "service.prerender.io";
rewrite .* /$scheme://$host$request_uri? break;
proxy_pass http://$prerender;
}
if ($prerender = 0) {
rewrite .* /index.html break;
}
}
}
@pvolyntsev

This comment has been minimized.

Show comment
Hide comment
@pvolyntsev

pvolyntsev Feb 15, 2014

condition

    if ($uri ~ ".js|.css|.xml|.less|.png|.jpg|.jpeg|.gif|.pdf|.doc|.txt|.ico|.rss|.zip|.mp3|.rar|.exe|.wmv|.doc|.avi|.ppt|.mpg|.mpeg|.tif|.wav|.mov|.psd|.ai|.xls|.mp4|.m4a|.swf|.dat|.dmg|.iso|.flv|.m4v|.torrent") {

incorrect because

    $uri ~ ".ico"

mean "string consist of one any character THEN 'ico'" that matched the URL /icon/ and it is wrong

I suggest this one:

    if ($uri ~ "\.(js|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent)$") {

that means "string is finished dot and one of strings 'js' OR 'css' etc"

condition

    if ($uri ~ ".js|.css|.xml|.less|.png|.jpg|.jpeg|.gif|.pdf|.doc|.txt|.ico|.rss|.zip|.mp3|.rar|.exe|.wmv|.doc|.avi|.ppt|.mpg|.mpeg|.tif|.wav|.mov|.psd|.ai|.xls|.mp4|.m4a|.swf|.dat|.dmg|.iso|.flv|.m4v|.torrent") {

incorrect because

    $uri ~ ".ico"

mean "string consist of one any character THEN 'ico'" that matched the URL /icon/ and it is wrong

I suggest this one:

    if ($uri ~ "\.(js|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent)$") {

that means "string is finished dot and one of strings 'js' OR 'css' etc"

@thoop

This comment has been minimized.

Show comment
Hide comment
@thoop

thoop Feb 27, 2014

You're right, thanks. I'll update that.

Owner

thoop commented Feb 27, 2014

You're right, thanks. I'll update that.

@AttilaSATAN

This comment has been minimized.

Show comment
Hide comment
@AttilaSATAN

AttilaSATAN May 4, 2014

How about html5mode of angularjs? Is prerender supports html5mode, would it be enough to add googlebot to the if statement?

        if ($http_user_agent ~* "baiduspider|twitterbot|facebookexternalhit|googlebot") {
            set $prerender 1;
        }

How about html5mode of angularjs? Is prerender supports html5mode, would it be enough to add googlebot to the if statement?

        if ($http_user_agent ~* "baiduspider|twitterbot|facebookexternalhit|googlebot") {
            set $prerender 1;
        }
@thoop

This comment has been minimized.

Show comment
Hide comment
@thoop

thoop May 26, 2014

Google's recommended way is to use escaped_fragment. In theory, yes we should add it to the user agent check, but in reality we wouldn't want Google to request the same URL using another user agent and then think the user is cloaking because the content is different. So we try to stay on the safe side.

Owner

thoop commented May 26, 2014

Google's recommended way is to use escaped_fragment. In theory, yes we should add it to the user agent check, but in reality we wouldn't want Google to request the same URL using another user agent and then think the user is cloaking because the content is different. So we try to stay on the safe side.

@sansmischevia

This comment has been minimized.

Show comment
Hide comment
@sansmischevia

sansmischevia Jul 2, 2014

Confused... why isn't googlebot listed?

Confused... why isn't googlebot listed?

@Michal-sk

This comment has been minimized.

Show comment
Hide comment
@Michal-sk

Michal-sk Jul 14, 2014

@sansmischevia : because googlebot uses the escaped_fragment, which is listed.

@sansmischevia : because googlebot uses the escaped_fragment, which is listed.

@jamiel

This comment has been minimized.

Show comment
Hide comment
@jamiel

jamiel Aug 26, 2014

If users are using HTML5 pushState, surely Google will request the URL's without escaped_fragment ?

jamiel commented Aug 26, 2014

If users are using HTML5 pushState, surely Google will request the URL's without escaped_fragment ?

@toamitkumar

This comment has been minimized.

Show comment
Hide comment
@toamitkumar

toamitkumar Sep 4, 2014

Search bots look for --> <meta name="fragment" content="!" /> in the head tag. Read https://developers.google.com/webmasters/ajax-crawling/docs/specification
Pages without hash fragments It may be impossible or undesirable for some pages....

Search bots look for --> <meta name="fragment" content="!" /> in the head tag. Read https://developers.google.com/webmasters/ajax-crawling/docs/specification
Pages without hash fragments It may be impossible or undesirable for some pages....

@sentient

This comment has been minimized.

Show comment
Hide comment
@sentient

sentient Sep 18, 2014

I'm not too sure on

  rewrite .* /$scheme://example.com$request_uri? break;

Am I only replacing the example.com ? Could we not use the $server_name variable for this?

  rewrite .* /$scheme://$server_name$request_uri? break;

I'm not too sure on

  rewrite .* /$scheme://example.com$request_uri? break;

Am I only replacing the example.com ? Could we not use the $server_name variable for this?

  rewrite .* /$scheme://$server_name$request_uri? break;
@akoumjian

This comment has been minimized.

Show comment
Hide comment
@akoumjian

akoumjian Sep 24, 2014

If our "location /" block is where we typically specify a reverse proxy with proxy_pass, should I assume we would essentially add that to the "if (@prerender = 0)" section?

If our "location /" block is where we typically specify a reverse proxy with proxy_pass, should I assume we would essentially add that to the "if (@prerender = 0)" section?

@thoop

This comment has been minimized.

Show comment
Hide comment
@thoop

thoop Sep 25, 2014

@sentient good idea :)

@akoumjian yes, in that case you would do your own proxy_pass "if @prerender = 0"

Owner

thoop commented Sep 25, 2014

@sentient good idea :)

@akoumjian yes, in that case you would do your own proxy_pass "if @prerender = 0"

@thoop

This comment has been minimized.

Show comment
Hide comment
@thoop

thoop Sep 25, 2014

I added this since we were seeing issues where nginx was caching IPs and hitting servers that might have been taken out of our load balancer rotation:

#resolve using Google's DNS server
resolver 8.8.8.8;

#setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
set $prerender "service.prerender.io";
rewrite .* /$scheme://$server_name$request_uri? break;
proxy_pass http://$prerender;
Owner

thoop commented Sep 25, 2014

I added this since we were seeing issues where nginx was caching IPs and hitting servers that might have been taken out of our load balancer rotation:

#resolve using Google's DNS server
resolver 8.8.8.8;

#setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
set $prerender "service.prerender.io";
rewrite .* /$scheme://$server_name$request_uri? break;
proxy_pass http://$prerender;
@patrickng

This comment has been minimized.

Show comment
Hide comment
@patrickng

patrickng Oct 1, 2014

With nginx 1.4.6 (ubuntu) I get an error when I try to restart nginx service or reload the configuration.

nginx: [emerg] "resolver" directive is not allowed here in /etc/nginx/sites-enabled/nginx.conf

I had to move resolver out of the conditional for it to work but this isn't ideal. Any ideas?

With nginx 1.4.6 (ubuntu) I get an error when I try to restart nginx service or reload the configuration.

nginx: [emerg] "resolver" directive is not allowed here in /etc/nginx/sites-enabled/nginx.conf

I had to move resolver out of the conditional for it to work but this isn't ideal. Any ideas?

@fergusg

This comment has been minimized.

Show comment
Hide comment
@fergusg

fergusg Oct 4, 2014

I'm wondering if $host is not a better choice than $server_name in

rewrite .* /$scheme://$server_name$request_uri? break;

There can be multiple $server_name values in a single server{...} but $host is the one in the HTTP requests (and falls back to $server_name anyway)

fergusg commented Oct 4, 2014

I'm wondering if $host is not a better choice than $server_name in

rewrite .* /$scheme://$server_name$request_uri? break;

There can be multiple $server_name values in a single server{...} but $host is the one in the HTTP requests (and falls back to $server_name anyway)

@cjroebuck

This comment has been minimized.

Show comment
Hide comment
@cjroebuck

cjroebuck Oct 15, 2014

@thoop @patrickng I too have the same issue ('resolver' directive is not allowed here)

@thoop @patrickng I too have the same issue ('resolver' directive is not allowed here)

@thoop

This comment has been minimized.

Show comment
Hide comment
@thoop

thoop Oct 17, 2014

Sorry, I wish github would send notifications for comments on gists :(

@fergusg we used to have it set to $host but recently changed it to $server_name.

I'll look into a better solution for the resolver in the if-conditional.

Owner

thoop commented Oct 17, 2014

Sorry, I wish github would send notifications for comments on gists :(

@fergusg we used to have it set to $host but recently changed it to $server_name.

I'll look into a better solution for the resolver in the if-conditional.

@evityuk

This comment has been minimized.

Show comment
Hide comment
@evityuk

evityuk Oct 21, 2014

Vkontakte social network uses this - "Mozilla/5.0 (compatible; vkShare; +http://vk.com/dev/Share)"
UserAgent for sharing functionality. Therefore you need to add 'vkshare' to detect list

pinterest|vkshare

See embeding docs. Sharing uses the same userAgent

evityuk commented Oct 21, 2014

Vkontakte social network uses this - "Mozilla/5.0 (compatible; vkShare; +http://vk.com/dev/Share)"
UserAgent for sharing functionality. Therefore you need to add 'vkshare' to detect list

pinterest|vkshare

See embeding docs. Sharing uses the same userAgent

@evandhoffman

This comment has been minimized.

Show comment
Hide comment
@evandhoffman

evandhoffman Oct 23, 2014

If behind a load balancer, the $scheme var may not be set right - if the LB is doing SSL termination, the scheme on the machine behind the box may be http. Prerender service would then try to access http://, but get bounced to https://, which in my case did not work. I had to hard-code https:// in there.

If behind a load balancer, the $scheme var may not be set right - if the LB is doing SSL termination, the scheme on the machine behind the box may be http. Prerender service would then try to access http://, but get bounced to https://, which in my case did not work. I had to hard-code https:// in there.

@leorue

This comment has been minimized.

Show comment
Hide comment
@leorue

leorue Nov 3, 2014

Any update on @patrickng resolver issue? I took out the conditional as well.

leorue commented Nov 3, 2014

Any update on @patrickng resolver issue? I took out the conditional as well.

@thoop

This comment has been minimized.

Show comment
Hide comment
@thoop

thoop Nov 5, 2014

@leorue can you try the new nginx config? I just updated it to move the resolver outside of the "if" statement.

Owner

thoop commented Nov 5, 2014

@leorue can you try the new nginx config? I just updated it to move the resolver outside of the "if" statement.

@thoop

This comment has been minimized.

Show comment
Hide comment
@thoop

thoop Nov 5, 2014

I just changed $server_name back to $host. Hopefully that clears up any issues with the server name not being the actual url of your site.

Owner

thoop commented Nov 5, 2014

I just changed $server_name back to $host. Hopefully that clears up any issues with the server name not being the actual url of your site.

@shirokoweb

This comment has been minimized.

Show comment
Hide comment
@shirokoweb

shirokoweb Nov 30, 2014

Hi, do I have to install something on my webserver running nginx as frontend HTTP server, or do I simply need to add this snippet to my vhost .conf ?

Hi, do I have to install something on my webserver running nginx as frontend HTTP server, or do I simply need to add this snippet to my vhost .conf ?

@thoop

This comment has been minimized.

Show comment
Hide comment
@thoop

thoop Dec 2, 2014

You should just be able to add this snippet to your .conf file. Email me at todd@prerender.io if you're having any problems with it. Github doesn't send notifications on gists so I'll be able to help you more quickly over email.

Owner

thoop commented Dec 2, 2014

You should just be able to add this snippet to your .conf file. Email me at todd@prerender.io if you're having any problems with it. Github doesn't send notifications on gists so I'll be able to help you more quickly over email.

@varuzhnikov

This comment has been minimized.

Show comment
Hide comment
@varuzhnikov

varuzhnikov Dec 5, 2014

http://wiki.nginx.org/IfIsEvil

Directive if has problems when used in location context, in some cases it doesn't do what you expect but something completely different instead. In some cases it even segfaults. It's generally a good idea to avoid it if possible.

The only 100% safe things which may be done inside if in location context are:

return ...;
rewrite ... last;
Anything else may possibly cause unpredictable behaviour, including potential SIGSEGV.

It is important to note that the behaviour of if is not inconsistent, given two identical requests it will not randomly fail on one and work on the other, with proper testing and understanding ifs can be used. The advice to use other directives where available still very much apply, though.

http://wiki.nginx.org/IfIsEvil

Directive if has problems when used in location context, in some cases it doesn't do what you expect but something completely different instead. In some cases it even segfaults. It's generally a good idea to avoid it if possible.

The only 100% safe things which may be done inside if in location context are:

return ...;
rewrite ... last;
Anything else may possibly cause unpredictable behaviour, including potential SIGSEGV.

It is important to note that the behaviour of if is not inconsistent, given two identical requests it will not randomly fail on one and work on the other, with proper testing and understanding ifs can be used. The advice to use other directives where available still very much apply, though.

@intellix

This comment has been minimized.

Show comment
Hide comment
@intellix

intellix Dec 8, 2014

Google doesn't use ?escaped_fragment= for all of it's services. It might do for it's indexer but for instance when I use it from the Webmaster tools "Fetch as Google", it correctly renders it but the HTML it received was before rendering:

66.249.75.21 - - [08/Dec/2014:12:11:51 +0000] "GET /owner/ HTTP/1.1" 200 6292 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.75.37 - - [08/Dec/2014:12:11:51 +0000] "GET /owner/?_escaped_fragment_= HTTP/1.1" 200 9401 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

In fact, I am ALWAYS seeing two requests from Googlebot - One with and without the escaped fragment.
I'm seeing other services without the escaped fragment as well: Google Page Speed Insights, Google-StructuredDataTestingTool, Google Web Preview.

A few that I added myself: (tumblr|slackbot|xml-sitemaps|google-structureddatatestingtool). Definitely need tubmlr :) It's a shame we have to go with a whitelist approach, because there are so many services where my site is just invisible.

intellix commented Dec 8, 2014

Google doesn't use ?escaped_fragment= for all of it's services. It might do for it's indexer but for instance when I use it from the Webmaster tools "Fetch as Google", it correctly renders it but the HTML it received was before rendering:

66.249.75.21 - - [08/Dec/2014:12:11:51 +0000] "GET /owner/ HTTP/1.1" 200 6292 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.75.37 - - [08/Dec/2014:12:11:51 +0000] "GET /owner/?_escaped_fragment_= HTTP/1.1" 200 9401 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

In fact, I am ALWAYS seeing two requests from Googlebot - One with and without the escaped fragment.
I'm seeing other services without the escaped fragment as well: Google Page Speed Insights, Google-StructuredDataTestingTool, Google Web Preview.

A few that I added myself: (tumblr|slackbot|xml-sitemaps|google-structureddatatestingtool). Definitely need tubmlr :) It's a shame we have to go with a whitelist approach, because there are so many services where my site is just invisible.

@csbenjamin

This comment has been minimized.

Show comment
Hide comment
@csbenjamin

csbenjamin Dec 8, 2014

@intellix I guess that in the first request google find the and then do a second request with escaped_fragment. Before the first request it did not know that it needs to use escaped_fragment

@intellix I guess that in the first request google find the and then do a second request with escaped_fragment. Before the first request it did not know that it needs to use escaped_fragment

@thoop

This comment has been minimized.

Show comment
Hide comment
@thoop

thoop Dec 9, 2014

@varuzhnikov if we could refactor to remove if statements, that would be ideal. Any idea on the best way to do that? We haven't had any problems with this configuration yet.

@intellix @csbenjamin is correct. The first request see's your tag and re-requests the page with the escaped fragment parameter. "Fetch as Google" does not follow the escaped fragment, and that is a known bug. If you are seeing services that do not send two requests (one with escaped fragment), then we should add them to the whitelist of user agents.

Owner

thoop commented Dec 9, 2014

@varuzhnikov if we could refactor to remove if statements, that would be ideal. Any idea on the best way to do that? We haven't had any problems with this configuration yet.

@intellix @csbenjamin is correct. The first request see's your tag and re-requests the page with the escaped fragment parameter. "Fetch as Google" does not follow the escaped fragment, and that is a known bug. If you are seeing services that do not send two requests (one with escaped fragment), then we should add them to the whitelist of user agents.

@bastijan

This comment has been minimized.

Show comment
Hide comment
@bastijan

bastijan Dec 14, 2014

I use self hosted prerender and nginx server/angularjs html application but when try share contentn on facebook or linked I always got root page (index.html) /

nginx conf
set $prerender "85.10.211.83:4000";
rewrite .* /$scheme://$host$request_uri? break;
proxy_pass http://$prerender;

just for review:
http://85.10.211.83:4000/http://novi.bktvnews.com:3030/#!/
and single pages is same
http://85.10.211.83:4000/http://novi.bktvnews.com:3030/#!/grafika-iz-dalarne-na-putu-vas

in app.js
I have

$locationProvider.hashPrefix('!');

and in index.html template

I use self hosted prerender and nginx server/angularjs html application but when try share contentn on facebook or linked I always got root page (index.html) /

nginx conf
set $prerender "85.10.211.83:4000";
rewrite .* /$scheme://$host$request_uri? break;
proxy_pass http://$prerender;

just for review:
http://85.10.211.83:4000/http://novi.bktvnews.com:3030/#!/
and single pages is same
http://85.10.211.83:4000/http://novi.bktvnews.com:3030/#!/grafika-iz-dalarne-na-putu-vas

in app.js
I have

$locationProvider.hashPrefix('!');

and in index.html template

@lethaldose

This comment has been minimized.

Show comment
Hide comment
@lethaldose

lethaldose Dec 22, 2014

we had a AWS nginx proxy setup and were having timeout issues - service.prerender.io could not be resolved (60: Operation timed out). We were able to get around this by using nginx upstream:

upstream pre-render {
   server service.prerender.io;
}

and the if in location block changed to:

if ($prerender = 1) {
   rewrite ^(.*)$ /https://$server_name..... break;
   proxy_pass http://pre-render;
}

we had a AWS nginx proxy setup and were having timeout issues - service.prerender.io could not be resolved (60: Operation timed out). We were able to get around this by using nginx upstream:

upstream pre-render {
   server service.prerender.io;
}

and the if in location block changed to:

if ($prerender = 1) {
   rewrite ^(.*)$ /https://$server_name..... break;
   proxy_pass http://pre-render;
}
@creatorkuang

This comment has been minimized.

Show comment
Hide comment
@creatorkuang

creatorkuang Dec 26, 2014

I use the official server :"service.prerender.io" and it works, but when i try to use my own server ,it didn't. I have test it with http://myserver.com/http://www.google.com and it work with my server which i think it mean the server work. But when i replace "service.prerender.io" with "myserver.com" , it didn't work and got an error with 502 Bad Gateway . Anyone know why?

I use the official server :"service.prerender.io" and it works, but when i try to use my own server ,it didn't. I have test it with http://myserver.com/http://www.google.com and it work with my server which i think it mean the server work. But when i replace "service.prerender.io" with "myserver.com" , it didn't work and got an error with 502 Bad Gateway . Anyone know why?

@geolart

This comment has been minimized.

Show comment
Hide comment
@geolart

geolart Jan 17, 2015

I use nginx like a reverse proxy and i would like to use prerender. I don't know how to adapt the nginx.conf especially this section:

location / {
try_files $uri @prerender;
}

I can't use the "try_files" instruction as i use "proxy_pass" instruction.

Do you know how to do ? Thank you

geolart commented Jan 17, 2015

I use nginx like a reverse proxy and i would like to use prerender. I don't know how to adapt the nginx.conf especially this section:

location / {
try_files $uri @prerender;
}

I can't use the "try_files" instruction as i use "proxy_pass" instruction.

Do you know how to do ? Thank you

@rkulla

This comment has been minimized.

Show comment
Hide comment
@rkulla

rkulla Jan 17, 2015

@creatorkuang it'd be hard to know without seeing your configuration. myserver.com needs to be running the open-source prerender node server and on the right port (by default it's 3000), and your nginx prerender middleware has to be configured to proxy there if the prerender variable = 1. E.g. proxy_pass http://myserver.com; But I've only tried the open-source server from localhost:3000.

rkulla commented Jan 17, 2015

@creatorkuang it'd be hard to know without seeing your configuration. myserver.com needs to be running the open-source prerender node server and on the right port (by default it's 3000), and your nginx prerender middleware has to be configured to proxy there if the prerender variable = 1. E.g. proxy_pass http://myserver.com; But I've only tried the open-source server from localhost:3000.

@rkulla

This comment has been minimized.

Show comment
Hide comment
@rkulla

rkulla Jan 17, 2015

@geolart I think some people above commented that you can do that by moving your proxy_pass line into the location @prerender block if the prerender variable = 0 (instead of rewrite .* /index.html break;). Let me know if it doesn't work for you though.

rkulla commented Jan 17, 2015

@geolart I think some people above commented that you can do that by moving your proxy_pass line into the location @prerender block if the prerender variable = 0 (instead of rewrite .* /index.html break;). Let me know if it doesn't work for you though.

@geolart

This comment has been minimized.

Show comment
Hide comment
@geolart

geolart Jan 18, 2015

@rkulla Thanks to you it's work for me !
Thank you Rhulla !

geolart commented Jan 18, 2015

@rkulla Thanks to you it's work for me !
Thank you Rhulla !

@rkulla

This comment has been minimized.

Show comment
Hide comment
@rkulla

rkulla Jan 18, 2015

When also using nginx as a reverse proxy via proxy_pass http://myupstream, I had an issue where if I do proxy_set_header X-Prerender-Token XXX in the location @prerender block, it reset my other proxy_set_header lines, causing 'http://myupstream' to be treated as a literal URL. However, things work fine if I redefine my proxy_set_header 'Host', etc in the same block as X-Prerender-Token -- either all in the location / or all in the location @prerender, but not divided.

rkulla commented Jan 18, 2015

When also using nginx as a reverse proxy via proxy_pass http://myupstream, I had an issue where if I do proxy_set_header X-Prerender-Token XXX in the location @prerender block, it reset my other proxy_set_header lines, causing 'http://myupstream' to be treated as a literal URL. However, things work fine if I redefine my proxy_set_header 'Host', etc in the same block as X-Prerender-Token -- either all in the location / or all in the location @prerender, but not divided.

@thoop

This comment has been minimized.

Show comment
Hide comment
@thoop

thoop Feb 18, 2015

Send me an email at support@prerender.io if anyone has any issues. I don't get notified when someone comments on this gist.

Owner

thoop commented Feb 18, 2015

Send me an email at support@prerender.io if anyone has any issues. I don't get notified when someone comments on this gist.

@BenjaminPrice

This comment has been minimized.

Show comment
Hide comment
@BenjaminPrice

BenjaminPrice Apr 2, 2015

As per Facebook official documentation, you should also add the user agent 'facebot' to $http_user_agent

https://developers.facebook.com/docs/sharing/best-practices#crawl

As per Facebook official documentation, you should also add the user agent 'facebot' to $http_user_agent

https://developers.facebook.com/docs/sharing/best-practices#crawl

@colthreepv

This comment has been minimized.

Show comment
Hide comment
@colthreepv

colthreepv Apr 8, 2015

I have monitored many requests that come with Googlebot and AdsBot-Google-Mobile as User-Agent, so I added it to my list.

There is an exhaustive of those User-Agents somewhere?

I have monitored many requests that come with Googlebot and AdsBot-Google-Mobile as User-Agent, so I added it to my list.

There is an exhaustive of those User-Agents somewhere?

@jstoiko

This comment has been minimized.

Show comment
Hide comment
@jstoiko

jstoiko Apr 28, 2015

You might want to add 'svg' to the list of extensions not prerendered.

jstoiko commented Apr 28, 2015

You might want to add 'svg' to the list of extensions not prerendered.

@ermakovich

This comment has been minimized.

Show comment
Hide comment
@ermakovich

ermakovich May 15, 2015

@evandhoffman we have the same issue when hosting on Heroku, which in turn seems to be using Amazon ELB. $scheme is always HTTP. As an alternative to hardcoding we decided to use $http_x_forwarded_proto instead of $scheme.

@evandhoffman we have the same issue when hosting on Heroku, which in turn seems to be using Amazon ELB. $scheme is always HTTP. As an alternative to hardcoding we decided to use $http_x_forwarded_proto instead of $scheme.

@thoop

This comment has been minimized.

Show comment
Hide comment
@thoop

thoop May 26, 2015

@MrGamer you don't want to add Googlebot (or any other crawlers that support the escaped fragment protocol) to the user agent list. You could get penalized for cloaking. You want Google to continue using the escaped fragment protocol

@ermakovich great idea!

Send me an email at support@prerender.io if anyone has any issues. I don't get notified when someone comments on this gist.

Owner

thoop commented May 26, 2015

@MrGamer you don't want to add Googlebot (or any other crawlers that support the escaped fragment protocol) to the user agent list. You could get penalized for cloaking. You want Google to continue using the escaped fragment protocol

@ermakovich great idea!

Send me an email at support@prerender.io if anyone has any issues. I don't get notified when someone comments on this gist.

@ashishgupta2

This comment has been minimized.

Show comment
Hide comment
@ashishgupta2

ashishgupta2 Jun 25, 2015

For nginx proxy_pass users: do two things.

  1. Comment out: following two lines in server block

    root /path/to/your/root;

    index index.html;

  2. In the last if block ("if ($prerender = 0) {....."):

replace #rewrite .* /index.html break; with your proxy_pass as shown below (my application is running on port 3009).

proxy_pass http://127.0.0.1:3009;

For nginx proxy_pass users: do two things.

  1. Comment out: following two lines in server block

    root /path/to/your/root;

    index index.html;

  2. In the last if block ("if ($prerender = 0) {....."):

replace #rewrite .* /index.html break; with your proxy_pass as shown below (my application is running on port 3009).

proxy_pass http://127.0.0.1:3009;

@matfin

This comment has been minimized.

Show comment
Hide comment
@matfin

matfin Aug 31, 2015

Does anyone have an example of something like this working for Meteor ?

matfin commented Aug 31, 2015

Does anyone have an example of something like this working for Meteor ?

@blaind

This comment has been minimized.

Show comment
Hide comment
@blaind

blaind Sep 13, 2015

@matfin see https://gist.github.com/blaind/12e53d5d9aa77c9fd841 for a meteor config. Had to use global proxy configs for both (can't put them in if's), but prerender.io seems to accept upgraded HTTP 1.1 too.

blaind commented Sep 13, 2015

@matfin see https://gist.github.com/blaind/12e53d5d9aa77c9fd841 for a meteor config. Had to use global proxy configs for both (can't put them in if's), but prerender.io seems to accept upgraded HTTP 1.1 too.

@maxklenk

This comment has been minimized.

Show comment
Hide comment
@maxklenk

maxklenk Sep 15, 2015

Please add redditbot to the list of user agents.

Please add redditbot to the list of user agents.

@alphastorm

This comment has been minimized.

Show comment
Hide comment
@alphastorm

alphastorm Oct 1, 2015

There's also trailing whitespace in 6 lines, might want to remove that.

There's also trailing whitespace in 6 lines, might want to remove that.

@donny08

This comment has been minimized.

Show comment
Hide comment
@donny08

donny08 Oct 16, 2015

@thoop I'm configuring prerender using nginx with nodejs backened server.Below is my nginx.conf file:-

server {
listen 80;
server_name _;

    location / {        
    try_files $uri @prerender;         
  }

   location @prerender {
    proxy_set_header X-Prerender-Token your_token;

    set $prerender 0;        

    if ($http_user_agent ~* "baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator") {
        set $prerender 1;
    }
    if ($args ~ "_escaped_fragment_") {           
        set $prerender 1;
    }
    if ($http_user_agent ~ "Prerender") {
        set $prerender 0;
    }
    if ($uri ~ "\.(js|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent|ttf|woff)") {
        set $prerender 0;
    }       

    #resolve using Google's DNS server to force DNS resolution and prevent caching of IPs
    resolver 8.8.8.8;
    if ($prerender = 1) {            
        #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
        set $prerender "localhost:8000";
        rewrite .* /$scheme://$host$request_uri? break;
        proxy_pass http://localhost:8000;
    }    

    if ($prerender = 0) {
        proxy_pass http://localhost:3000;

    }
}

    #error_page  404              /404.html;

    # redirect server error pages to the static page /50x.html
    #
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   html;
    }


}

But I am seeing JavaScript and not the static HTML.I had the checked Common Problem section,It has mentioned Try moving your Prerender middleware configuration higher in your file,I want to know what is
configuration code for nginx that i need to move to higher section of file.

donny08 commented Oct 16, 2015

@thoop I'm configuring prerender using nginx with nodejs backened server.Below is my nginx.conf file:-

server {
listen 80;
server_name _;

    location / {        
    try_files $uri @prerender;         
  }

   location @prerender {
    proxy_set_header X-Prerender-Token your_token;

    set $prerender 0;        

    if ($http_user_agent ~* "baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator") {
        set $prerender 1;
    }
    if ($args ~ "_escaped_fragment_") {           
        set $prerender 1;
    }
    if ($http_user_agent ~ "Prerender") {
        set $prerender 0;
    }
    if ($uri ~ "\.(js|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent|ttf|woff)") {
        set $prerender 0;
    }       

    #resolve using Google's DNS server to force DNS resolution and prevent caching of IPs
    resolver 8.8.8.8;
    if ($prerender = 1) {            
        #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
        set $prerender "localhost:8000";
        rewrite .* /$scheme://$host$request_uri? break;
        proxy_pass http://localhost:8000;
    }    

    if ($prerender = 0) {
        proxy_pass http://localhost:3000;

    }
}

    #error_page  404              /404.html;

    # redirect server error pages to the static page /50x.html
    #
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   html;
    }


}

But I am seeing JavaScript and not the static HTML.I had the checked Common Problem section,It has mentioned Try moving your Prerender middleware configuration higher in your file,I want to know what is
configuration code for nginx that i need to move to higher section of file.

@thabofletcher

This comment has been minimized.

Show comment
Hide comment
@thabofletcher

thabofletcher Oct 16, 2015

I've posted a small project that demonstrates an implementation of how to use this without violating the "If Is Evil" principle:
https://github.com/thabofletcher/prerender.io-nginx

This allows any configuration for your app, including rewrites, try_files, and proxy_path, all outside if blocks. I believe it can be consolidated into a single gist/config, but I prefer the use of includes for readability.

I've posted a small project that demonstrates an implementation of how to use this without violating the "If Is Evil" principle:
https://github.com/thabofletcher/prerender.io-nginx

This allows any configuration for your app, including rewrites, try_files, and proxy_path, all outside if blocks. I believe it can be consolidated into a single gist/config, but I prefer the use of includes for readability.

@donny08

This comment has been minimized.

Show comment
Hide comment
@donny08

donny08 Oct 27, 2015

This is my url :http://example.com/customer/#/index,I have already done prerender configuration.How do i access the prerender page of http://example.com/customer/#/index, what should be my url format

donny08 commented Oct 27, 2015

This is my url :http://example.com/customer/#/index,I have already done prerender configuration.How do i access the prerender page of http://example.com/customer/#/index, what should be my url format

@asquel

This comment has been minimized.

Show comment
Hide comment
@asquel

asquel Nov 27, 2015

+1 for the question from @thiagofesta ! (what do you think @thoop ?)

asquel commented Nov 27, 2015

+1 for the question from @thiagofesta ! (what do you think @thoop ?)

@TheProdigyFilippo

This comment has been minimized.

Show comment
Hide comment
@TheProdigyFilippo

TheProdigyFilippo Nov 29, 2015

@thiagofesta @asquel don't believe in this. It's not as nice as google tells. We shot ourselves in feets when we changed our static title and meta description into dynamic using AngularJS <meta name="description" content="{{ pageDescription }}" />. Now we have { pageDescription }} as page description in google results and Alexa ranks raised up half milion points :) So we want to start to use prerender.io soon.

Correction:
I have to appologize google and have to admit to our mistake. We used <meta name="fragment" content="!"> and in this case google expects that you provide snapshot. After removing <meta name="fragment" content="!"> site and meta description were rendered properly. Other thing is that prerendering is good for other services and crawlers.

So when you depend on google rendering only, you should remember to delete <meta name="fragment" content="!"> meta tag!

@thiagofesta @asquel don't believe in this. It's not as nice as google tells. We shot ourselves in feets when we changed our static title and meta description into dynamic using AngularJS <meta name="description" content="{{ pageDescription }}" />. Now we have { pageDescription }} as page description in google results and Alexa ranks raised up half milion points :) So we want to start to use prerender.io soon.

Correction:
I have to appologize google and have to admit to our mistake. We used <meta name="fragment" content="!"> and in this case google expects that you provide snapshot. After removing <meta name="fragment" content="!"> site and meta description were rendered properly. Other thing is that prerendering is good for other services and crawlers.

So when you depend on google rendering only, you should remember to delete <meta name="fragment" content="!"> meta tag!

@antoineco

This comment has been minimized.

Show comment
Hide comment
@antoineco

antoineco Nov 29, 2015

@thabofletcher I used a similar configuration but when I got rid of

if ($prerender = 0) {
  rewrite .* /index.html break;
}

and replaced it by a single try_files directive, Prerender started getting 404 errors while I got a 202 cURLing the same URL. I wrongly assumed this would work as long as $prerender != 1, but it doesn't when proxied through Prerender. Have you ever faced this issue with your configuration?

edit: the explanation is here, IfIsEvil (nginx docs)

@thabofletcher I used a similar configuration but when I got rid of

if ($prerender = 0) {
  rewrite .* /index.html break;
}

and replaced it by a single try_files directive, Prerender started getting 404 errors while I got a 202 cURLing the same URL. I wrongly assumed this would work as long as $prerender != 1, but it doesn't when proxied through Prerender. Have you ever faced this issue with your configuration?

edit: the explanation is here, IfIsEvil (nginx docs)

@rogierslag

This comment has been minimized.

Show comment
Hide comment
@rogierslag

rogierslag Jan 21, 2016

I've made a fork and updated it with the following user agents baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest|slackbot|vkShare|Slack-ImgProxy|Slackbot-LinkExpanding|Site Analyzer|SiteAnalyzerBot|Viber|Whatsapp|Telegram|W3C_Validator.

These now also include stuff as Viber, WhatsApp, and Telegram which all try to parse the og tags

I've made a fork and updated it with the following user agents baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest|slackbot|vkShare|Slack-ImgProxy|Slackbot-LinkExpanding|Site Analyzer|SiteAnalyzerBot|Viber|Whatsapp|Telegram|W3C_Validator.

These now also include stuff as Viber, WhatsApp, and Telegram which all try to parse the og tags

@charlesdg

This comment has been minimized.

Show comment
Hide comment
@charlesdg

charlesdg Jan 22, 2016

@ermakovich thanks for the hack for heroku!

If you are using the Heroku Buildpack: NGINX (https://elements.heroku.com/buildpacks/ryandotsmith/nginx-buildpack), just replace

rewrite .* /$scheme://$host$request_uri? break;

by

rewrite .* /$http_x_forwarded_proto://$host$request_uri? break;

Hope that will be useful !

@ermakovich thanks for the hack for heroku!

If you are using the Heroku Buildpack: NGINX (https://elements.heroku.com/buildpacks/ryandotsmith/nginx-buildpack), just replace

rewrite .* /$scheme://$host$request_uri? break;

by

rewrite .* /$http_x_forwarded_proto://$host$request_uri? break;

Hope that will be useful !

@kascote

This comment has been minimized.

Show comment
Hide comment
@kascote

kascote Jan 26, 2016

I'm using this setup with an Angular application and works ok except to for the root (/) path that never goes throw prerender.
I think the problem is with try_files that found the index.html and serve it without going to the @prerender section.

someone have this issue ? how to resolve it ?

kascote commented Jan 26, 2016

I'm using this setup with an Angular application and works ok except to for the root (/) path that never goes throw prerender.
I think the problem is with try_files that found the index.html and serve it without going to the @prerender section.

someone have this issue ? how to resolve it ?

@intellix

This comment has been minimized.

Show comment
Hide comment
@intellix

intellix Jan 31, 2016

Been having issues with prerender dying and having my entire site disappear from Google when everything returns 504 errors (my fault for not having server alerts of any kind). I think in such a case where prerender just dies, we should have a fallback so google at least gets served your HTML/JS (they just won't for API calls to resolve, as prerender is there for).

Example error that'll bring your prerender service down

2015-12-10T17:00:31.669Z started phantom
events.js:85
      throw er; // Unhandled 'error' event
            ^
Error: channel closed
    at ChildProcess.target.send (child_process.js:414:26)
    at sendHelper (cluster.js:676:8)
    at RoundRobinHandle.handoff (cluster.js:198:3)
    at RoundRobinHandle.distribute (cluster.js:184:20)

Been playing around with how to get error_page working with a proxy and this is what works.:

server {
    # Add this line, to redirect 50x errors back to index.html
    error_page 500 502 503 504 =200 /index.html;

    location / {
        index     index.html;
        try_files $uri @prerender;
    }

    location @prerender {
        # Add this line inside @prerender location
        proxy_intercept_errors on;
    }
}

Been having issues with prerender dying and having my entire site disappear from Google when everything returns 504 errors (my fault for not having server alerts of any kind). I think in such a case where prerender just dies, we should have a fallback so google at least gets served your HTML/JS (they just won't for API calls to resolve, as prerender is there for).

Example error that'll bring your prerender service down

2015-12-10T17:00:31.669Z started phantom
events.js:85
      throw er; // Unhandled 'error' event
            ^
Error: channel closed
    at ChildProcess.target.send (child_process.js:414:26)
    at sendHelper (cluster.js:676:8)
    at RoundRobinHandle.handoff (cluster.js:198:3)
    at RoundRobinHandle.distribute (cluster.js:184:20)

Been playing around with how to get error_page working with a proxy and this is what works.:

server {
    # Add this line, to redirect 50x errors back to index.html
    error_page 500 502 503 504 =200 /index.html;

    location / {
        index     index.html;
        try_files $uri @prerender;
    }

    location @prerender {
        # Add this line inside @prerender location
        proxy_intercept_errors on;
    }
}
@thedug

This comment has been minimized.

Show comment
Hide comment
@thedug

thedug Feb 4, 2016

Why not just simply do this vs. the rewrite?

proxy_pass http://service.prerender.io/$scheme://$host$request_uri?;

thedug commented Feb 4, 2016

Why not just simply do this vs. the rewrite?

proxy_pass http://service.prerender.io/$scheme://$host$request_uri?;

@marcelpanse

This comment has been minimized.

Show comment
Hide comment
@marcelpanse

marcelpanse Feb 22, 2016

Google has deprecated the escaped fragment scheme:
https://googlewebmastercentral.blogspot.nl/2015/10/deprecating-our-ajax-crawling-scheme.html

I think googlebot should be added to the whitelist, the escaped fragment scheme will probably still used by other bots like bing.

Google has deprecated the escaped fragment scheme:
https://googlewebmastercentral.blogspot.nl/2015/10/deprecating-our-ajax-crawling-scheme.html

I think googlebot should be added to the whitelist, the escaped fragment scheme will probably still used by other bots like bing.

@thoop

This comment has been minimized.

Show comment
Hide comment
@thoop

thoop Feb 26, 2016

@marcelpanse Google is still not doing a good enough job when crawling javascript, which they even admitted in their last Google Webmaster Hangout. Google still follows the escaped fragment crawling protocol like they did before so Prerendering pages for Google is still the recommended way of doing things. Adding googlebot to the whitelist could get your website penalized for cloaking so it's not recommended. You should support the escaped fragment crawling protocol for Googlebot.

Owner

thoop commented Feb 26, 2016

@marcelpanse Google is still not doing a good enough job when crawling javascript, which they even admitted in their last Google Webmaster Hangout. Google still follows the escaped fragment crawling protocol like they did before so Prerendering pages for Google is still the recommended way of doing things. Adding googlebot to the whitelist could get your website penalized for cloaking so it's not recommended. You should support the escaped fragment crawling protocol for Googlebot.

@thoop

This comment has been minimized.

Show comment
Hide comment
@thoop

thoop Feb 26, 2016

@thedug The rewrite is the way that it is because nginx caches IP addresses. If an IP address changes on our end, then your prerendering could stop working. The way we have it with the variable will prevent nginx from caching IP addresses to prevent this problem.

Owner

thoop commented Feb 26, 2016

@thedug The rewrite is the way that it is because nginx caches IP addresses. If an IP address changes on our end, then your prerendering could stop working. The way we have it with the variable will prevent nginx from caching IP addresses to prevent this problem.

@thoop

This comment has been minimized.

Show comment
Hide comment
@thoop

thoop Feb 26, 2016

I don't get any notifications when people comment on this thread so please email me at todd@prerender.io if anyone has any questions.

Owner

thoop commented Feb 26, 2016

I don't get any notifications when people comment on this thread so please email me at todd@prerender.io if anyone has any questions.

@ray65536

This comment has been minimized.

Show comment
Hide comment
@ray65536

ray65536 Mar 11, 2016

You should move block of If -- set -- out of location to server definition level. Or you'll get a lot of problems with "if is Evil" in Nginx.

Like this:

server {
    listen 80;
    server_name example.com;

    root   /path/to/your/root;
    index  index.html;

        set $prerender 0;
        if ($http_user_agent ~* "baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator") {
            set $prerender 1;
        }
        if ($args ~ "_escaped_fragment_") {
            set $prerender 1;
        }
        if ($http_user_agent ~ "Prerender") {
            set $prerender 0;
        }
        if ($uri ~ "\.(js|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent|ttf|woff)") {
            set $prerender 0;
        }

    location / {
        try_files $uri @prerender;
    }

    location @prerender {
        #proxy_set_header X-Prerender-Token YOUR_TOKEN;

        #resolve using Google's DNS server to force DNS resolution and prevent caching of IPs
        resolver 8.8.8.8;

        if ($prerender = 1) {

            #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
            set $prerender "service.prerender.io";
            rewrite .* /$scheme://$host$request_uri? break;
            proxy_pass http://$prerender;
        }
        if ($prerender = 0) {
            rewrite .* /index.html break;
        }
    }
}

You should move block of If -- set -- out of location to server definition level. Or you'll get a lot of problems with "if is Evil" in Nginx.

Like this:

server {
    listen 80;
    server_name example.com;

    root   /path/to/your/root;
    index  index.html;

        set $prerender 0;
        if ($http_user_agent ~* "baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator") {
            set $prerender 1;
        }
        if ($args ~ "_escaped_fragment_") {
            set $prerender 1;
        }
        if ($http_user_agent ~ "Prerender") {
            set $prerender 0;
        }
        if ($uri ~ "\.(js|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent|ttf|woff)") {
            set $prerender 0;
        }

    location / {
        try_files $uri @prerender;
    }

    location @prerender {
        #proxy_set_header X-Prerender-Token YOUR_TOKEN;

        #resolve using Google's DNS server to force DNS resolution and prevent caching of IPs
        resolver 8.8.8.8;

        if ($prerender = 1) {

            #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
            set $prerender "service.prerender.io";
            rewrite .* /$scheme://$host$request_uri? break;
            proxy_pass http://$prerender;
        }
        if ($prerender = 0) {
            rewrite .* /index.html break;
        }
    }
}
@SiddharthTerse

This comment has been minimized.

Show comment
Hide comment
@SiddharthTerse

SiddharthTerse Mar 26, 2016

Hi,

If i am browsing any webpage so 1st time with ?_escaped_fragment redirecting 301 & from next time 200 OK . this is happening for all pages....

My nginx config file for prerender is...

location @prerender {
#proxy_set_header X-Prerender-Token YOUR_TOKEN;

    set $prerender 0;
    if ($http_user_agent ~* "baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator") {
        set $prerender 1;
    }
    if ($args ~ "_escaped_fragment_") {
        set $prerender 1;
    }
    if ($http_user_agent ~ "Prerender") {
        set $prerender 0;
    }
    if ($uri ~ "\.(js|ico|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent|ttf|woff)") {
        set $prerender 0;
    }

    #resolve using Google's DNS server to force DNS resolution and prevent caching of IPs
    resolver 8.8.8.8;

    if ($prerender = 1) {

        #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
        set $prerender "192.168.152.191:3000";
             rewrite .* /$scheme://$server_name$request_uri? break;
        proxy_pass http://$prerender;

    }
    if ($prerender = 0) {
        rewrite .* /index.html break;
    }
}

    location ~* \.(gif|jpg|png|ico)$ {
            expires 30d;
    }

}

Hi,

If i am browsing any webpage so 1st time with ?_escaped_fragment redirecting 301 & from next time 200 OK . this is happening for all pages....

My nginx config file for prerender is...

location @prerender {
#proxy_set_header X-Prerender-Token YOUR_TOKEN;

    set $prerender 0;
    if ($http_user_agent ~* "baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator") {
        set $prerender 1;
    }
    if ($args ~ "_escaped_fragment_") {
        set $prerender 1;
    }
    if ($http_user_agent ~ "Prerender") {
        set $prerender 0;
    }
    if ($uri ~ "\.(js|ico|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent|ttf|woff)") {
        set $prerender 0;
    }

    #resolve using Google's DNS server to force DNS resolution and prevent caching of IPs
    resolver 8.8.8.8;

    if ($prerender = 1) {

        #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
        set $prerender "192.168.152.191:3000";
             rewrite .* /$scheme://$server_name$request_uri? break;
        proxy_pass http://$prerender;

    }
    if ($prerender = 0) {
        rewrite .* /index.html break;
    }
}

    location ~* \.(gif|jpg|png|ico)$ {
            expires 30d;
    }

}

@SiddharthTerse

This comment has been minimized.

Show comment
Hide comment
@SiddharthTerse

SiddharthTerse Mar 28, 2016

@thoop, i found article where google disclosed they have stopped to support official for escaped_fragment (http://searchengineland.com/google-has-deprecated-their-ajax-crawling-scheme-233402) . So now what is the alternative for google crowling now ?

@thoop, i found article where google disclosed they have stopped to support official for escaped_fragment (http://searchengineland.com/google-has-deprecated-their-ajax-crawling-scheme-233402) . So now what is the alternative for google crowling now ?

@wellington1993

This comment has been minimized.

Show comment
Hide comment
@wellington1993

wellington1993 Mar 30, 2016

Thanks in a lot!

Thanks in a lot!

@bwoodlt

This comment has been minimized.

Show comment
Hide comment
@bwoodlt

bwoodlt Mar 30, 2016

Hi Guys,

I'm trying to use this config, but keep returning 404 error. I'll appreciate any help in having a quick look.

server {
listen 443;
server_name localhost;

location / {
    root   /usr/share/nginx/html;
    index  index.html index.htm;
    try_files $uri @prerender;
}
location @prerender {
        proxy_intercept_errors on;
        proxy_set_header X-Prerender-Token our_token;

        set $prerender 0;
        if ($http_user_agent ~* "baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link     preview|showyoubot|outbrain|pinterest|slackbot|vkShare|Slack-ImgProxy|Slackbot-LinkExpanding|Site Analyzer|SiteAnalyzerBot|Viber|Whatsapp|Telegram|W3C_Validator") {
            set $prerender 1;
        }
        if ($args ~ "_escaped_fragment_") {
            set $prerender 1;
        }
        if ($http_user_agent ~ "Prerender") {
            set $prerender 0;
        }
        if ($uri ~ "\.(js|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent|ttf|woff)") {
            set $prerender 0;
        }

        #resolve using Google's DNS server to force DNS resolution and prevent caching of IPs
        resolver 8.8.8.8;

        #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
        set $prerender "service.prerender.io";
        rewrite .* /$scheme://$host$request_uri? break;
        proxy_pass http://$prerender;

        if ($prerender = 1) {

            #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
            set $prerender "service.prerender.io";
            rewrite .* /$scheme://$host$request_uri? break;
            proxy_pass http://$prerender;
        }
        if ($prerender = 0) {
            rewrite .* /index.html break;
        }
    }

}

bwoodlt commented Mar 30, 2016

Hi Guys,

I'm trying to use this config, but keep returning 404 error. I'll appreciate any help in having a quick look.

server {
listen 443;
server_name localhost;

location / {
    root   /usr/share/nginx/html;
    index  index.html index.htm;
    try_files $uri @prerender;
}
location @prerender {
        proxy_intercept_errors on;
        proxy_set_header X-Prerender-Token our_token;

        set $prerender 0;
        if ($http_user_agent ~* "baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link     preview|showyoubot|outbrain|pinterest|slackbot|vkShare|Slack-ImgProxy|Slackbot-LinkExpanding|Site Analyzer|SiteAnalyzerBot|Viber|Whatsapp|Telegram|W3C_Validator") {
            set $prerender 1;
        }
        if ($args ~ "_escaped_fragment_") {
            set $prerender 1;
        }
        if ($http_user_agent ~ "Prerender") {
            set $prerender 0;
        }
        if ($uri ~ "\.(js|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent|ttf|woff)") {
            set $prerender 0;
        }

        #resolve using Google's DNS server to force DNS resolution and prevent caching of IPs
        resolver 8.8.8.8;

        #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
        set $prerender "service.prerender.io";
        rewrite .* /$scheme://$host$request_uri? break;
        proxy_pass http://$prerender;

        if ($prerender = 1) {

            #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
            set $prerender "service.prerender.io";
            rewrite .* /$scheme://$host$request_uri? break;
            proxy_pass http://$prerender;
        }
        if ($prerender = 0) {
            rewrite .* /index.html break;
        }
    }

}

@rickyk586

This comment has been minimized.

Show comment
Hide comment
@rickyk586

rickyk586 Apr 6, 2016

lethaldose's solution above worked for me

lethaldose's solution above worked for me

@Gbeschbacher

This comment has been minimized.

Show comment
Hide comment
@Gbeschbacher

Gbeschbacher Apr 12, 2016

Hey there,

i am trying to use prerender with nginx and our angular.js app.
The setting is as follows: i have docker running on the server. Nginx is used as a reverse proxy to forward the requests to the docker-container. Inside the container is another Nginx running, which serves the web app.
Now i do not know exactly where i should put the prerender config. Following i post both nginx-configs for clarity and maybe some more convenience:

This is the nginx-config which proxies to the docker-container

server {
        listen                  80;
        return 301              https://$host$request_uri;
}

### DEVELOP ###
server {
        listen                  443 ssl;
        ssl_certificate         /path/to/file
        ssl_certificate_key     /path/to/file
        server_name             develop.vidatio.mediacube.at;

        location / {
                proxy_pass http://localhost:6000;
        }

        location /.well-known {
                root /var/www/do.not.delete.vidatio.com;
        }
}

and this is the currently working nginx-config inside the container:

worker_processes  1;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;

    server {
        listen       80;
        server_name  localhost;

        location / {
            root /var/www/build;
            try_files $uri /statics/master.html;
        }

        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
}

i tried to get it to work like this but it doesnt:

http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;

    server {
        listen       80;
        server_name  localhost;

        root /var/www/build;

        location / {
            try_files $uri @prerender /statics/master.html;
        }

        location @prerender {
            proxy_set_header X-Prerender-Token --TOKEN--

            set $prerender 0;
            if ($http_user_agent ~* "baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator") {
                set $prerender 1;
            }
            if ($args ~ "_escaped_fragment_") {
                set $prerender 1;
            }
            if ($http_user_agent ~ "Prerender") {
                set $prerender 0;
            }
            if ($uri ~ "\.(js|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent|ttf|woff)") {
                set $prerender 0;
            }

            #resolve using Google's DNS server to force DNS resolution and prevent caching of IPs
            resolver 8.8.8.8;

            if ($prerender = 1) {

                #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
                set $prerender "service.prerender.io";
                rewrite .* /$scheme://$host$request_uri? break;
                proxy_pass http://$prerender;
            }
            if ($prerender = 0) {
                rewrite .* /statics/master.html break;
            }
        }

        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
}

Thanks in advance!

Hey there,

i am trying to use prerender with nginx and our angular.js app.
The setting is as follows: i have docker running on the server. Nginx is used as a reverse proxy to forward the requests to the docker-container. Inside the container is another Nginx running, which serves the web app.
Now i do not know exactly where i should put the prerender config. Following i post both nginx-configs for clarity and maybe some more convenience:

This is the nginx-config which proxies to the docker-container

server {
        listen                  80;
        return 301              https://$host$request_uri;
}

### DEVELOP ###
server {
        listen                  443 ssl;
        ssl_certificate         /path/to/file
        ssl_certificate_key     /path/to/file
        server_name             develop.vidatio.mediacube.at;

        location / {
                proxy_pass http://localhost:6000;
        }

        location /.well-known {
                root /var/www/do.not.delete.vidatio.com;
        }
}

and this is the currently working nginx-config inside the container:

worker_processes  1;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;

    server {
        listen       80;
        server_name  localhost;

        location / {
            root /var/www/build;
            try_files $uri /statics/master.html;
        }

        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
}

i tried to get it to work like this but it doesnt:

http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;

    server {
        listen       80;
        server_name  localhost;

        root /var/www/build;

        location / {
            try_files $uri @prerender /statics/master.html;
        }

        location @prerender {
            proxy_set_header X-Prerender-Token --TOKEN--

            set $prerender 0;
            if ($http_user_agent ~* "baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator") {
                set $prerender 1;
            }
            if ($args ~ "_escaped_fragment_") {
                set $prerender 1;
            }
            if ($http_user_agent ~ "Prerender") {
                set $prerender 0;
            }
            if ($uri ~ "\.(js|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent|ttf|woff)") {
                set $prerender 0;
            }

            #resolve using Google's DNS server to force DNS resolution and prevent caching of IPs
            resolver 8.8.8.8;

            if ($prerender = 1) {

                #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
                set $prerender "service.prerender.io";
                rewrite .* /$scheme://$host$request_uri? break;
                proxy_pass http://$prerender;
            }
            if ($prerender = 0) {
                rewrite .* /statics/master.html break;
            }
        }

        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
}

Thanks in advance!

@vic3lord

This comment has been minimized.

Show comment
Hide comment
@vic3lord

vic3lord Apr 25, 2016

if other people like me tried to proxy_pass if $prerender = 0 then just remove the if condition as it acts like else
You won't get any nginx config errors

Hope it helps

if other people like me tried to proxy_pass if $prerender = 0 then just remove the if condition as it acts like else
You won't get any nginx config errors

Hope it helps

@AntonAL

This comment has been minimized.

Show comment
Hide comment
@AntonAL

AntonAL Apr 26, 2016

Hello, guys!

I've started prerender server listening port 3000, in my production machine, used this config file.
So, I've replaced this part to localhost:3000:

if ($prerender = 1) {
    #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
    set $prerender "service.prerender.io";
    rewrite .* /$scheme://$host$request_uri? break;
    proxy_pass http://$prerender;
}
if ($prerender = 1) {            
    #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
    set $prerender "localhost:3000";
    rewrite .* /$scheme://$host$request_uri? break;
    proxy_pass http://$prerender;
}

When I requesting home page from external client host, I'm always getting 504 and no rendering:
Fragment from Nginx (tail -f /var/log/nginx/access.log):

213.21.36.130 - - [26/Apr/2016:08:08:46 +0000] "GET / HTTP/1.1" 504 0 "-" "twitterbot"

When I requesting home page directly from production server, everything is working:
curl -A 'twitterbot' http://localhost:3000/http://my-website.com

When I using set $prerender "service.prerender.io";, everyting is also working.

It seems, like I have problem with local prerenderer…

Also, I don't understand, how this configuration passes requests to meteor in case of non-bot.
As far, as I understood this fragment:

if ($prerender = 0) {
    rewrite .* /index.html break;
}

It loads static index.html…but we are using Meteor…

My initial configuration of nginx without prerender is taken from guide How To Deploy a Meteor.js Application on Ubuntu 14.04 with Nginx

Anyone having the same issue?
Thanks in advance!

AntonAL commented Apr 26, 2016

Hello, guys!

I've started prerender server listening port 3000, in my production machine, used this config file.
So, I've replaced this part to localhost:3000:

if ($prerender = 1) {
    #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
    set $prerender "service.prerender.io";
    rewrite .* /$scheme://$host$request_uri? break;
    proxy_pass http://$prerender;
}
if ($prerender = 1) {            
    #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
    set $prerender "localhost:3000";
    rewrite .* /$scheme://$host$request_uri? break;
    proxy_pass http://$prerender;
}

When I requesting home page from external client host, I'm always getting 504 and no rendering:
Fragment from Nginx (tail -f /var/log/nginx/access.log):

213.21.36.130 - - [26/Apr/2016:08:08:46 +0000] "GET / HTTP/1.1" 504 0 "-" "twitterbot"

When I requesting home page directly from production server, everything is working:
curl -A 'twitterbot' http://localhost:3000/http://my-website.com

When I using set $prerender "service.prerender.io";, everyting is also working.

It seems, like I have problem with local prerenderer…

Also, I don't understand, how this configuration passes requests to meteor in case of non-bot.
As far, as I understood this fragment:

if ($prerender = 0) {
    rewrite .* /index.html break;
}

It loads static index.html…but we are using Meteor…

My initial configuration of nginx without prerender is taken from guide How To Deploy a Meteor.js Application on Ubuntu 14.04 with Nginx

Anyone having the same issue?
Thanks in advance!

@aaronbuchanan

This comment has been minimized.

Show comment
Hide comment
@aaronbuchanan

aaronbuchanan May 12, 2016

@intellix can you shed some light on prerender dying on your site? Know what the root cause was? Find a resolution you're happy with?

@intellix can you shed some light on prerender dying on your site? Know what the root cause was? Find a resolution you're happy with?

@tpoxa

This comment has been minimized.

Show comment
Hide comment
@tpoxa

tpoxa Jun 3, 2016

What about php-fpm?

tpoxa commented Jun 3, 2016

What about php-fpm?

@partounian

This comment has been minimized.

Show comment
Hide comment
@partounian

partounian Jul 2, 2016

Yes, how could we make this play nicely with PHP?

Yes, how could we make this play nicely with PHP?

@cbbaaa1989

This comment has been minimized.

Show comment
Hide comment
@cbbaaa1989

cbbaaa1989 Jul 4, 2016

prerender get response of 304,how can i fix it?

prerender get response of 304,how can i fix it?

@apaatsio

This comment has been minimized.

Show comment
Hide comment
@apaatsio

apaatsio Sep 2, 2016

I hit a problem with a redirect loop. This can happen if you have a redirect from http to https. To detect this you can curl your site curl -A "facebookexternalhit" -v https://example.com/. If it returns 302 with Location: https://example.com/ then you have a redirect loop.

To fix this you need to update this row
rewrite .* /$scheme://$host$request_uri? break;
to
rewrite .* /https://$host$request_uri? break;

This situation can occur when you have a setup where you have one server/proxy handling the SSL termination and http->https redirect and a separate web server that is just an http server running nginx with prerender configuration.

apaatsio commented Sep 2, 2016

I hit a problem with a redirect loop. This can happen if you have a redirect from http to https. To detect this you can curl your site curl -A "facebookexternalhit" -v https://example.com/. If it returns 302 with Location: https://example.com/ then you have a redirect loop.

To fix this you need to update this row
rewrite .* /$scheme://$host$request_uri? break;
to
rewrite .* /https://$host$request_uri? break;

This situation can occur when you have a setup where you have one server/proxy handling the SSL termination and http->https redirect and a separate web server that is just an http server running nginx with prerender configuration.

@yogeshsajanikar

This comment has been minimized.

Show comment
Hide comment
@yogeshsajanikar

yogeshsajanikar Sep 14, 2016

How can this be mixed with open_file_cache?

How can this be mixed with open_file_cache?

@jsonmaur

This comment has been minimized.

Show comment
Hide comment
@jsonmaur

jsonmaur Oct 2, 2016

Confused about why we need a long blacklist of extensions. Wouldn't we just want a whitelist instead? Such as .html, .htm, and no extension at all?

jsonmaur commented Oct 2, 2016

Confused about why we need a long blacklist of extensions. Wouldn't we just want a whitelist instead? Such as .html, .htm, and no extension at all?

@jdkealy

This comment has been minimized.

Show comment
Hide comment
@jdkealy

jdkealy Oct 5, 2016

how would this work if your nginx has an upstream proxy?
e.g.

upstream bfa_clj {
   server 0.0.0.0:3000;
}
server {
    listen 80 default_server;
    listen [::]:80 default_server ipv6only=on;

    root /usr/share/nginx/html;
    index index.html index.htm;

    # Make site accessible from http://localhost/
    server_name localhost;

    location / {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_redirect off;
        proxy_pass http://bfa_clj;
        break;
    }
}

jdkealy commented Oct 5, 2016

how would this work if your nginx has an upstream proxy?
e.g.

upstream bfa_clj {
   server 0.0.0.0:3000;
}
server {
    listen 80 default_server;
    listen [::]:80 default_server ipv6only=on;

    root /usr/share/nginx/html;
    index index.html index.htm;

    # Make site accessible from http://localhost/
    server_name localhost;

    location / {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_redirect off;
        proxy_pass http://bfa_clj;
        break;
    }
}
@thoop

This comment has been minimized.

Show comment
Hide comment
@thoop

thoop Oct 5, 2016

I don't get any notifications when people comment on this thread so please email me at support@prerender.io if anyone has any questions. We're happy to help you get set up.

Owner

thoop commented Oct 5, 2016

I don't get any notifications when people comment on this thread so please email me at support@prerender.io if anyone has any questions. We're happy to help you get set up.

@chatoo2412

This comment has been minimized.

Show comment
Hide comment
@chatoo2412

chatoo2412 Nov 16, 2016

I wonder why bingbot is not included in the list not like Stanback's one.
I tested with Fetch as Bingbot tool.
I also sent an e-mail, Thanks.

chatoo2412 commented Nov 16, 2016

I wonder why bingbot is not included in the list not like Stanback's one.
I tested with Fetch as Bingbot tool.
I also sent an e-mail, Thanks.

@scne

This comment has been minimized.

Show comment
Hide comment
@scne

scne Nov 26, 2016

If chain of http_user_agent match but my url contains #! can I replace this with ?_escaped_fragment_=?

scne commented Nov 26, 2016

If chain of http_user_agent match but my url contains #! can I replace this with ?_escaped_fragment_=?

@scne

This comment has been minimized.

Show comment
Hide comment
@scne

scne Nov 27, 2016

Please add WhatsApp to user agent chain

scne commented Nov 27, 2016

Please add WhatsApp to user agent chain

@simonepri

This comment has been minimized.

Show comment
Hide comment
@simonepri

simonepri Dec 17, 2016

@thoop What do you think about the improvment propsed by @rogierslag ?
baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest|slackbot|vkShare|Slack-ImgProxy|Slackbot-LinkExpanding|Site Analyzer|SiteAnalyzerBot|Viber|Whatsapp|Telegram|W3C_Validator

Also i think that these:
Validator.nu|W3C-checklink|W3C-mobileOK|W3C_I18n-Checker|NING|FeedValidator|Jigsaw|W3C_Unicorn
should be added as well.
https://validator.w3.org/services#list

@thoop What do you think about the improvment propsed by @rogierslag ?
baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest|slackbot|vkShare|Slack-ImgProxy|Slackbot-LinkExpanding|Site Analyzer|SiteAnalyzerBot|Viber|Whatsapp|Telegram|W3C_Validator

Also i think that these:
Validator.nu|W3C-checklink|W3C-mobileOK|W3C_I18n-Checker|NING|FeedValidator|Jigsaw|W3C_Unicorn
should be added as well.
https://validator.w3.org/services#list

@zwolej

This comment has been minimized.

Show comment
Hide comment
@zwolej

zwolej Jan 6, 2017

for people that trying to use this config on own prerender servers, you must just replace:

set $prerender "service.prerender.io";
to
set $prerender "127.0.0.1:3000";// one wird thing localhost didn't worked for me so it must be 127.0.0.1:<YOUR_PRERENDER_PORT>

zwolej commented Jan 6, 2017

for people that trying to use this config on own prerender servers, you must just replace:

set $prerender "service.prerender.io";
to
set $prerender "127.0.0.1:3000";// one wird thing localhost didn't worked for me so it must be 127.0.0.1:<YOUR_PRERENDER_PORT>

@Sureshkumars

This comment has been minimized.

Show comment
Hide comment
@Sureshkumars

Sureshkumars Jan 28, 2017

@thoop
server {
listen 80;
listen [::]:80 default_server ipv6only=on;
return 301 https://$host$request_uri;
}

server {
    listen 443;
    server_name example.me;

    ssl on;
    # Use certificate and key provided by Let's Encrypt:
    ssl_certificate /etc/letsencrypt/live/example.me/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/example.me/privkey.pem;
    ssl_session_timeout 5m;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_prefer_server_ciphers on;
    ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';

    # Pass requests for / to localhost:8080:
    location / {
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-NginX-Proxy true;
            proxy_pass http://localhost:8080/;
            proxy_ssl_session_reuse off;
            proxy_set_header Host $http_host;
            proxy_cache_bypass $http_upgrade;
            proxy_redirect off;
    }

}
The above is my Nginx config file am using proxy with https. So how can I configure the above prerender and it's MEAN stack application. What should I give in the path to your root section either my app folder or public folder. Thanks

Sureshkumars commented Jan 28, 2017

@thoop
server {
listen 80;
listen [::]:80 default_server ipv6only=on;
return 301 https://$host$request_uri;
}

server {
    listen 443;
    server_name example.me;

    ssl on;
    # Use certificate and key provided by Let's Encrypt:
    ssl_certificate /etc/letsencrypt/live/example.me/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/example.me/privkey.pem;
    ssl_session_timeout 5m;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_prefer_server_ciphers on;
    ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';

    # Pass requests for / to localhost:8080:
    location / {
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-NginX-Proxy true;
            proxy_pass http://localhost:8080/;
            proxy_ssl_session_reuse off;
            proxy_set_header Host $http_host;
            proxy_cache_bypass $http_upgrade;
            proxy_redirect off;
    }

}
The above is my Nginx config file am using proxy with https. So how can I configure the above prerender and it's MEAN stack application. What should I give in the path to your root section either my app folder or public folder. Thanks

@bdombro

This comment has been minimized.

Show comment
Hide comment
@bdombro

bdombro Feb 10, 2017

I had to add googlebot for google's webmaster tool's "Fetch as Google" to work.

https://www.google.com/webmasters/tools/googlebot-fetch-details

bdombro commented Feb 10, 2017

I had to add googlebot for google's webmaster tool's "Fetch as Google" to work.

https://www.google.com/webmasters/tools/googlebot-fetch-details

@hetdev

This comment has been minimized.

Show comment
Hide comment
@hetdev

hetdev Feb 21, 2017

Hi
Anyone knows how to do it on nginx on docker and AWS?

hetdev commented Feb 21, 2017

Hi
Anyone knows how to do it on nginx on docker and AWS?

@jeerbl

This comment has been minimized.

Show comment
Hide comment
@jeerbl

jeerbl Mar 1, 2017

This does not include Google +. As stated in the Google+ docs, the user agent contains:

Google (+https://developers.google.com/+/web/snippet/)

which should be added to the list of possible user agents.

jeerbl commented Mar 1, 2017

This does not include Google +. As stated in the Google+ docs, the user agent contains:

Google (+https://developers.google.com/+/web/snippet/)

which should be added to the list of possible user agents.

@ietuday

This comment has been minimized.

Show comment
Hide comment
@ietuday

ietuday Apr 19, 2017

Hi Guys,

I was running four node servers using

Configuring multiple applications on same port using nginx reverse proxy based on request urls

its working but the bundles files cannot loaded. I am stuck. Please Help

on Browser Console

Failed to load resource: the server responded with a status of 404 (Not Found)
http://localhost/inline.bundle.js Failed to load resource: the server responded with a status of 404 (Not Found)
http://localhost/scripts.bundle.js Failed to load resource: the server responded with a status of 404 (Not Found)
http://localhost/vendor.bundle.js Failed to load resource: the server responded with a status of 404 (Not Found)
http://localhost/main.bundle.js Failed to load resource: the server responded with a status of 404 (Not Found)
http://localhost/styles.bundle.css Failed to load resource: the server responded with a status of 404 (Not Found)

ietuday commented Apr 19, 2017

Hi Guys,

I was running four node servers using

Configuring multiple applications on same port using nginx reverse proxy based on request urls

its working but the bundles files cannot loaded. I am stuck. Please Help

on Browser Console

Failed to load resource: the server responded with a status of 404 (Not Found)
http://localhost/inline.bundle.js Failed to load resource: the server responded with a status of 404 (Not Found)
http://localhost/scripts.bundle.js Failed to load resource: the server responded with a status of 404 (Not Found)
http://localhost/vendor.bundle.js Failed to load resource: the server responded with a status of 404 (Not Found)
http://localhost/main.bundle.js Failed to load resource: the server responded with a status of 404 (Not Found)
http://localhost/styles.bundle.css Failed to load resource: the server responded with a status of 404 (Not Found)

@thoop

This comment has been minimized.

Show comment
Hide comment
@thoop

thoop Apr 26, 2017

I don't get any notifications when people comment on this thread so please email me at support@prerender.io if anyone has any questions. We're happy to help you get set up.

Owner

thoop commented Apr 26, 2017

I don't get any notifications when people comment on this thread so please email me at support@prerender.io if anyone has any questions. We're happy to help you get set up.

@regilero

This comment has been minimized.

Show comment
Hide comment
@regilero

regilero May 18, 2017

Some modifications, especially if removal with map, available on this fork : https://gist.github.com/regilero/dbddb029381dd3b781bc2906a5533620

Some modifications, especially if removal with map, available on this fork : https://gist.github.com/regilero/dbddb029381dd3b781bc2906a5533620

@fedealconada

This comment has been minimized.

Show comment
Hide comment
@fedealconada

fedealconada Jun 28, 2017

is it possible to avoid prerendering some specific files? as I have a bower dependency that, when is prerendered, is adding an html tag that is causing the bot to take an incorrect Title name of the page...(ngnix config)

fedealconada commented Jun 28, 2017

is it possible to avoid prerendering some specific files? as I have a bower dependency that, when is prerendered, is adding an html tag that is causing the bot to take an incorrect Title name of the page...(ngnix config)

@azulkipli

This comment has been minimized.

Show comment
Hide comment
@azulkipli

azulkipli Jul 28, 2017

Hi, i have 504 issue.
already set nginx as written above & add metatag
screen shot 2017-07-28 at 17 48 44

azulkipli commented Jul 28, 2017

Hi, i have 504 issue.
already set nginx as written above & add metatag
screen shot 2017-07-28 at 17 48 44

@jeerbl

This comment has been minimized.

Show comment
Hide comment
@jeerbl

jeerbl Aug 4, 2017

As stated in the Yahoo! Help, the Yahoo bot identifies itself as:

Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Therefore adding Slurp to the list would handle this bot correctly.

jeerbl commented Aug 4, 2017

As stated in the Yahoo! Help, the Yahoo bot identifies itself as:

Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Therefore adding Slurp to the list would handle this bot correctly.

@fernandosanchezjr

This comment has been minimized.

Show comment
Hide comment
@fernandosanchezjr

fernandosanchezjr Aug 10, 2017

How would one use uWSGI with this config? We have a django site with angular front-end, and we would like to cache static files.

How would one use uWSGI with this config? We have a django site with angular front-end, and we would like to cache static files.

@scne

This comment has been minimized.

Show comment
Hide comment
@scne

scne Aug 24, 2017

I've trouble with whatsapp, seems that not full url is catch: I see log of nginx and when whatsapp's crawler explore my url this match only / instead of full path

scne commented Aug 24, 2017

I've trouble with whatsapp, seems that not full url is catch: I see log of nginx and when whatsapp's crawler explore my url this match only / instead of full path

@Ingeskhaled

This comment has been minimized.

Show comment
Hide comment
@Ingeskhaled

Ingeskhaled Sep 24, 2017

Hi everyone,

I have an issue with my config, google crowl only my home page, even i suggest the other pages it's shows the home page content.

Also can i put a config for node.js and another in nginx in the same time ?

Many thanks in advance !

  server {
    listen 443;

server_name www.test.com test.com;
server_tokens off;


ssl on;
ssl_certificate /etc/nginx/ssl/test.com.chained.crt;
ssl_certificate_key /etc/nginx/ssl/test.com.key;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:30m;
ssl_session_timeout 1d;
ssl_session_tickets off;
ssl_dhparam /etc/nginx/cert/dhparam.pem;
ssl_ciphers 'ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS:EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA RC4 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS !RC4';

ssl_stapling on;
ssl_stapling_verify on;
ssl_trusted_certificate /etc/nginx/ssl/trustchain.crt;
resolver 8.8.8.8 8.8.4.4;

location / {
try_files $uri @prerender;
}

 location @prerender {
 proxy_set_header X-Prerender-Token xxxxxxxxxxxxxxxxxx;
 proxy_set_header X-Real-IP $remote_addr;
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 proxy_set_header X-Forwarded-Proto $scheme;
 add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
 proxy_http_version 1.1;
 proxy_set_header Upgrade $http_upgrade;
 proxy_set_header Connection "upgrade";
 #proxy_set_header Host $host;
 server_tokens off;

set $prerender 0;
 if ($http_user_agent ~* "googlebot|baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link 
 preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator") {
 set $prerender 1;
 }

 if ($args ~ "_escaped_fragment_") {
  set $prerender 1;
 }
 if ($http_user_agent ~ "Prerender") {
   set $prerender 0;
 }
 if ($uri ~ "\.
      (js|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|
       swf|dat|dmg|iso|flv|m4v|torrent|ttf|woff|svg|eot)") {
   set $prerender 0;
 }
   #resolve using Google's DNS server to force DNS resolution and prevent caching of IPs
    resolver 8.8.8.8;

   if ($prerender = 1) {
  #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
  set $prerender "service.prerender.io";
   rewrite .* /https://$host$request_uri? break;
   proxy_pass http://$prerender;
   }
   if ($prerender = 0) {
     proxy_pass http://127.0.0.1:3000;

   }
 }

 }

 server {
   listen  80;
   server_name    www.test.com test.com;
   return         301 https://www.test.cmo$request_uri;
   if ($host = 'test.com') {
   return 301 https://www.test.com$request_uri;
   }

}

Hi everyone,

I have an issue with my config, google crowl only my home page, even i suggest the other pages it's shows the home page content.

Also can i put a config for node.js and another in nginx in the same time ?

Many thanks in advance !

  server {
    listen 443;

server_name www.test.com test.com;
server_tokens off;


ssl on;
ssl_certificate /etc/nginx/ssl/test.com.chained.crt;
ssl_certificate_key /etc/nginx/ssl/test.com.key;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:30m;
ssl_session_timeout 1d;
ssl_session_tickets off;
ssl_dhparam /etc/nginx/cert/dhparam.pem;
ssl_ciphers 'ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS:EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA RC4 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS !RC4';

ssl_stapling on;
ssl_stapling_verify on;
ssl_trusted_certificate /etc/nginx/ssl/trustchain.crt;
resolver 8.8.8.8 8.8.4.4;

location / {
try_files $uri @prerender;
}

 location @prerender {
 proxy_set_header X-Prerender-Token xxxxxxxxxxxxxxxxxx;
 proxy_set_header X-Real-IP $remote_addr;
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 proxy_set_header X-Forwarded-Proto $scheme;
 add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
 proxy_http_version 1.1;
 proxy_set_header Upgrade $http_upgrade;
 proxy_set_header Connection "upgrade";
 #proxy_set_header Host $host;
 server_tokens off;

set $prerender 0;
 if ($http_user_agent ~* "googlebot|baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link 
 preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator") {
 set $prerender 1;
 }

 if ($args ~ "_escaped_fragment_") {
  set $prerender 1;
 }
 if ($http_user_agent ~ "Prerender") {
   set $prerender 0;
 }
 if ($uri ~ "\.
      (js|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|
       swf|dat|dmg|iso|flv|m4v|torrent|ttf|woff|svg|eot)") {
   set $prerender 0;
 }
   #resolve using Google's DNS server to force DNS resolution and prevent caching of IPs
    resolver 8.8.8.8;

   if ($prerender = 1) {
  #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
  set $prerender "service.prerender.io";
   rewrite .* /https://$host$request_uri? break;
   proxy_pass http://$prerender;
   }
   if ($prerender = 0) {
     proxy_pass http://127.0.0.1:3000;

   }
 }

 }

 server {
   listen  80;
   server_name    www.test.com test.com;
   return         301 https://www.test.cmo$request_uri;
   if ($host = 'test.com') {
   return 301 https://www.test.com$request_uri;
   }

}

@Startouf

This comment has been minimized.

Show comment
Hide comment
@Startouf

Startouf Nov 29, 2017

Seems like this guy has a working approach using map instead of IFs
https://gist.github.com/regilero/dbddb029381dd3b781bc2906a5533620

Seems like this guy has a working approach using map instead of IFs
https://gist.github.com/regilero/dbddb029381dd3b781bc2906a5533620

@blackandred

This comment has been minimized.

Show comment
Hide comment
@blackandred

blackandred Dec 9, 2017

Code review failed, one time you put boolean in a variable, other time a string 😂

Code review failed, one time you put boolean in a variable, other time a string 😂

@sam-untapt

This comment has been minimized.

Show comment
Hide comment
@sam-untapt

sam-untapt Dec 15, 2017

Sorry, n00b question.

If I have configured my nginx.conf file correctly, when I hit https://service.prerender.io/https://myurl (which displays correctly), will this request show up on https://prerender.io? Currently I see the message "We haven't seen a request with your Prerender token yet."

Thanks.

Sorry, n00b question.

If I have configured my nginx.conf file correctly, when I hit https://service.prerender.io/https://myurl (which displays correctly), will this request show up on https://prerender.io? Currently I see the message "We haven't seen a request with your Prerender token yet."

Thanks.

@thucnd

This comment has been minimized.

Show comment
Hide comment
@thucnd

thucnd Jan 29, 2018

Hi All,
It doesn't work on my server: Centos 6.5
Below are my config on nginx: it returns 200 status

server {
  listen       80;
  server_name  {{ frontend_server_name }};

  charset UTF-8;
  access_log  /var/log/nginx/skill-test.frontend.access.log;
  error_log  /var/log/nginx/skill-test.frontend.error.log;
  root   {{ remote_stage_dir }}/frontend;
  index  index.html;

  location / {
    try_files $uri @prerender;
  }

  location @prerender {
      #proxy_set_header X-Prerender-Token YOUR_TOKEN;

      set $prerender 0;
      if ($http_user_agent ~* "baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator") {
          set $prerender 1;
      }
      if ($args ~ "_escaped_fragment_") {
          set $prerender 1;
      }
      if ($http_user_agent ~ "Prerender") {
          set $prerender 0;
      }
      if ($uri ~* "\.(js|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent|ttf|woff|svg|eot)") {
          set $prerender 0;
      }

      #resolve using Google's DNS server to force DNS resolution and prevent caching of IPs
      resolver 8.8.8.8;

      if ($prerender = 1) {
          #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
          set $prerender "127.0.0.1:3000";
          rewrite .* /$scheme://$host$request_uri? break;
          proxy_pass http://$prerender;
      }
      if ($prerender = 0) {
          rewrite .* /index.html break;
      }
  }

thucnd commented Jan 29, 2018

Hi All,
It doesn't work on my server: Centos 6.5
Below are my config on nginx: it returns 200 status

server {
  listen       80;
  server_name  {{ frontend_server_name }};

  charset UTF-8;
  access_log  /var/log/nginx/skill-test.frontend.access.log;
  error_log  /var/log/nginx/skill-test.frontend.error.log;
  root   {{ remote_stage_dir }}/frontend;
  index  index.html;

  location / {
    try_files $uri @prerender;
  }

  location @prerender {
      #proxy_set_header X-Prerender-Token YOUR_TOKEN;

      set $prerender 0;
      if ($http_user_agent ~* "baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator") {
          set $prerender 1;
      }
      if ($args ~ "_escaped_fragment_") {
          set $prerender 1;
      }
      if ($http_user_agent ~ "Prerender") {
          set $prerender 0;
      }
      if ($uri ~* "\.(js|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent|ttf|woff|svg|eot)") {
          set $prerender 0;
      }

      #resolve using Google's DNS server to force DNS resolution and prevent caching of IPs
      resolver 8.8.8.8;

      if ($prerender = 1) {
          #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
          set $prerender "127.0.0.1:3000";
          rewrite .* /$scheme://$host$request_uri? break;
          proxy_pass http://$prerender;
      }
      if ($prerender = 0) {
          rewrite .* /index.html break;
      }
  }
@mdeolmos

This comment has been minimized.

Show comment
Hide comment
@mdeolmos

mdeolmos Feb 5, 2018

Is there a way to set YOUR_TOKEN as an environment variable?

mdeolmos commented Feb 5, 2018

Is there a way to set YOUR_TOKEN as an environment variable?

@IuryAlves

This comment has been minimized.

Show comment
Hide comment
@IuryAlves

IuryAlves Feb 6, 2018

@mdeolmos

You can do that using the lua-nginx-module

@mdeolmos

You can do that using the lua-nginx-module

@closca

This comment has been minimized.

Show comment
Hide comment
@closca

closca Mar 9, 2018

Hello, i see that SkypeUriPreview and other new crawlers are missing (eg: WhatsApp)
I found a decent list here: https://github.com/monperrus/crawler-user-agents/blob/master/crawler-user-agents.json

closca commented Mar 9, 2018

Hello, i see that SkypeUriPreview and other new crawlers are missing (eg: WhatsApp)
I found a decent list here: https://github.com/monperrus/crawler-user-agents/blob/master/crawler-user-agents.json

@courtneyschwartz

This comment has been minimized.

Show comment
Hide comment
@courtneyschwartz

courtneyschwartz Mar 23, 2018

@sam-untapt Change your browser's user agent string (there are extensions for this) to look like a bot, then try the request. The pre-render shouldn't be hit if you're browsing normally.

@sam-untapt Change your browser's user agent string (there are extensions for this) to look like a bot, then try the request. The pre-render shouldn't be hit if you're browsing normally.

@courtneyschwartz

This comment has been minimized.

Show comment
Hide comment
@courtneyschwartz

courtneyschwartz Mar 23, 2018

Working w/ a hosting provider, we couldn't use the above nginx.conf as is. (Restricted to what we can add within a single location block.) Below is the alternative config for inside a location{} block -- not server{}.
Note Our case is not for an SPA, and therefore we removed the if around _escaped_fragment.

proxy_set_header X-Prerender-Token YOUR_TOKEN;  
# Using prerender or not as a variable forces DNS resolution since nginx caches IPs and doesn't play well with load balancing
set $prerender 0;

# Prerender if it's a search bot that doesn't support JavaScript AND the URL is SPA/JavsScript-rendered
if ($http_user_agent ~* "baiduspider|facebookexternalhit|googlebot|twitterbot|rogerbot|linkedinbot|embedly|bingbot|Baiduspider|Yahoo|YahooSeeker|quora link preview|showyoubot|outbrain|pinterest|applebot|slackbot|vkShare|W3C_Validator") {
  # Old Google-recommended URL parameter to flag SPA
  if ($args ~ "_escaped_fragment_") {
	set $prerender 1;
  }
  # A specific URL that you know is JS-rendered
  if ($uri ~* "^/api-reference") {
	set $prerender 1;
  }
}

# Don't prerender for the prerender bot's own user agent though
if ($http_user_agent ~ "Prerender") {
  set $prerender 0;
}
# Don't prerender media files (they don't contain JavaScript)
if ($uri ~ "\.(js|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent|ttf|woff)") {
  set $prerender 0;
}

set $prerenderprotocol "http";
set $prerenderservice "http://service.prerender.io";

if ($http_x_forwarded_proto = 'https') {
  set $prerenderprotocol "https";
  set $prerenderservice "https://service.prerender.io";
}

# Resolve using Google's DNS server to force DNS resolution and prevent caching of IPs
resolver 8.8.8.8;

# Do the proxy if the request met the tests
if ($prerender = 1) {
  rewrite .* /$prerenderprotocol://$host$request_uri? break;
  proxy_pass $prerenderservice;
}

courtneyschwartz commented Mar 23, 2018

Working w/ a hosting provider, we couldn't use the above nginx.conf as is. (Restricted to what we can add within a single location block.) Below is the alternative config for inside a location{} block -- not server{}.
Note Our case is not for an SPA, and therefore we removed the if around _escaped_fragment.

proxy_set_header X-Prerender-Token YOUR_TOKEN;  
# Using prerender or not as a variable forces DNS resolution since nginx caches IPs and doesn't play well with load balancing
set $prerender 0;

# Prerender if it's a search bot that doesn't support JavaScript AND the URL is SPA/JavsScript-rendered
if ($http_user_agent ~* "baiduspider|facebookexternalhit|googlebot|twitterbot|rogerbot|linkedinbot|embedly|bingbot|Baiduspider|Yahoo|YahooSeeker|quora link preview|showyoubot|outbrain|pinterest|applebot|slackbot|vkShare|W3C_Validator") {
  # Old Google-recommended URL parameter to flag SPA
  if ($args ~ "_escaped_fragment_") {
	set $prerender 1;
  }
  # A specific URL that you know is JS-rendered
  if ($uri ~* "^/api-reference") {
	set $prerender 1;
  }
}

# Don't prerender for the prerender bot's own user agent though
if ($http_user_agent ~ "Prerender") {
  set $prerender 0;
}
# Don't prerender media files (they don't contain JavaScript)
if ($uri ~ "\.(js|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent|ttf|woff)") {
  set $prerender 0;
}

set $prerenderprotocol "http";
set $prerenderservice "http://service.prerender.io";

if ($http_x_forwarded_proto = 'https') {
  set $prerenderprotocol "https";
  set $prerenderservice "https://service.prerender.io";
}

# Resolve using Google's DNS server to force DNS resolution and prevent caching of IPs
resolver 8.8.8.8;

# Do the proxy if the request met the tests
if ($prerender = 1) {
  rewrite .* /$prerenderprotocol://$host$request_uri? break;
  proxy_pass $prerenderservice;
}
@jordanandres1

This comment has been minimized.

Show comment
Hide comment
@jordanandres1

jordanandres1 Apr 7, 2018

I tried with the follow configuration and it didn't work.
Prerender.io page stays with the message "We haven't seen a request with your Prerender token yet."
Obviously I replaced YOUR_TOKEN with mine
Someone can help me ?

server {
    listen       80;
    server_name  localhost;

root /usr/share/nginx/html;
index index.html index.htm;

location / {
   try_files $uri $uri/ /index.html =404;
}

location @prerender {
    proxy_set_header X-Prerender-Token YOUR_TOKEN;
    
    set $prerender 0;
    if ($http_user_agent ~* "baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator") {
        set $prerender 1;
    }
    if ($args ~ "_escaped_fragment_") {
        set $prerender 1;
    }
    if ($http_user_agent ~ "Prerender") {
        set $prerender 0;
    }
    if ($uri ~* "\.(js|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent|ttf|woff|svg|eot)") {
        set $prerender 0;
    }
    
    #resolve using Google's DNS server to force DNS resolution and prevent caching of IPs
    resolver 8.8.8.8;

    if ($prerender = 1) {
        
        #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
        set $prerender "service.prerender.io";
        rewrite .* /$scheme://$host$request_uri? break;
        proxy_pass http://$prerender;
    }
    if ($prerender = 0) {
        rewrite .* /index.html break;
    }
}	
}

I tried with the follow configuration and it didn't work.
Prerender.io page stays with the message "We haven't seen a request with your Prerender token yet."
Obviously I replaced YOUR_TOKEN with mine
Someone can help me ?

server {
    listen       80;
    server_name  localhost;

root /usr/share/nginx/html;
index index.html index.htm;

location / {
   try_files $uri $uri/ /index.html =404;
}

location @prerender {
    proxy_set_header X-Prerender-Token YOUR_TOKEN;
    
    set $prerender 0;
    if ($http_user_agent ~* "baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator") {
        set $prerender 1;
    }
    if ($args ~ "_escaped_fragment_") {
        set $prerender 1;
    }
    if ($http_user_agent ~ "Prerender") {
        set $prerender 0;
    }
    if ($uri ~* "\.(js|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent|ttf|woff|svg|eot)") {
        set $prerender 0;
    }
    
    #resolve using Google's DNS server to force DNS resolution and prevent caching of IPs
    resolver 8.8.8.8;

    if ($prerender = 1) {
        
        #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
        set $prerender "service.prerender.io";
        rewrite .* /$scheme://$host$request_uri? break;
        proxy_pass http://$prerender;
    }
    if ($prerender = 0) {
        rewrite .* /index.html break;
    }
}	
}
@RicoFactset

This comment has been minimized.

Show comment
Hide comment
@RicoFactset

RicoFactset May 8, 2018

Hey, I just read ajax-crawling and the usage of escaped_fragment is deprecated: https://developers.google.com/webmasters/ajax-crawling/docs/specification

I guess
if ($args ~ "escaped_fragment") {
set $prerender 1;
}
could be removed from the gist then!?

Hey, I just read ajax-crawling and the usage of escaped_fragment is deprecated: https://developers.google.com/webmasters/ajax-crawling/docs/specification

I guess
if ($args ~ "escaped_fragment") {
set $prerender 1;
}
could be removed from the gist then!?

@celiovasconcelos

This comment has been minimized.

Show comment
Hide comment
@celiovasconcelos

celiovasconcelos May 13, 2018

Why try_files?

Using try_files the location @prerender will never be fired!
I was looking the following behaviour:

If any page is accessed from the browser, nginx serve it statically/locally. If that page is accessed from some crawler, it's served by the proxy.

The official snippet seems not to do that.

Am I crazy?

celiovasconcelos commented May 13, 2018

Why try_files?

Using try_files the location @prerender will never be fired!
I was looking the following behaviour:

If any page is accessed from the browser, nginx serve it statically/locally. If that page is accessed from some crawler, it's served by the proxy.

The official snippet seems not to do that.

Am I crazy?

@jindrichbartek

This comment has been minimized.

Show comment
Hide comment
@jindrichbartek

jindrichbartek Jul 7, 2018

From "Deprecating our AJAX crawling scheme" (https://webmasters.googleblog.com/2017/12/rendering-ajax-crawling-pages.html)

Given these advances, in the second quarter of 2018, we'll be switching to rendering these pages on Google's side, rather than on requiring that sites do this themselves. In short, we'll no longer be using the AJAX crawling scheme.

So what do you think? Did you guys do some tests with "escaped_fragment" condition?

jindrichbartek commented Jul 7, 2018

From "Deprecating our AJAX crawling scheme" (https://webmasters.googleblog.com/2017/12/rendering-ajax-crawling-pages.html)

Given these advances, in the second quarter of 2018, we'll be switching to rendering these pages on Google's side, rather than on requiring that sites do this themselves. In short, we'll no longer be using the AJAX crawling scheme.

So what do you think? Did you guys do some tests with "escaped_fragment" condition?

@mindtonic

This comment has been minimized.

Show comment
Hide comment
@mindtonic

mindtonic Jul 11, 2018

+1 to @jeerbl for pointing out that this config file is missing Google Plus. The Google+ Snippet Fetcher User Agent is:

Google (+https://developers.google.com/+/web/snippet/)

For my part I added it to my config file UserAgent regex as developers\.google\.com

Also, if you want to use the Structured Data Testing Tool, you will want to add Google-Structured-Data-Testing-Tool to the regex as well.

Final example:

if ($http_user_agent ~* "baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator|developers\.google\.com|Google-Structured-Data-Testing-Tool") {
  set $prerender 1;
 }

mindtonic commented Jul 11, 2018

+1 to @jeerbl for pointing out that this config file is missing Google Plus. The Google+ Snippet Fetcher User Agent is:

Google (+https://developers.google.com/+/web/snippet/)

For my part I added it to my config file UserAgent regex as developers\.google\.com

Also, if you want to use the Structured Data Testing Tool, you will want to add Google-Structured-Data-Testing-Tool to the regex as well.

Final example:

if ($http_user_agent ~* "baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator|developers\.google\.com|Google-Structured-Data-Testing-Tool") {
  set $prerender 1;
 }
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment