Skip to content
Create a gist now

Instantly share code, notes, and snippets.

@thoop /nginx.conf
Last active May 24, 2016

Official prerender.io nginx.conf for nginx
# Change YOUR_TOKEN to your prerender token and uncomment that line if you want to cache urls and view crawl stats
# Change example.com (server_name) to your website url
# Change /path/to/your/root to the correct value
server {
listen 80;
server_name example.com;
root /path/to/your/root;
index index.html;
location / {
try_files $uri @prerender;
}
location @prerender {
#proxy_set_header X-Prerender-Token YOUR_TOKEN;
set $prerender 0;
if ($http_user_agent ~* "baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator") {
set $prerender 1;
}
if ($args ~ "_escaped_fragment_") {
set $prerender 1;
}
if ($http_user_agent ~ "Prerender") {
set $prerender 0;
}
if ($uri ~ "\.(js|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent|ttf|woff)") {
set $prerender 0;
}
#resolve using Google's DNS server to force DNS resolution and prevent caching of IPs
resolver 8.8.8.8;
if ($prerender = 1) {
#setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
set $prerender "service.prerender.io";
rewrite .* /$scheme://$host$request_uri? break;
proxy_pass http://$prerender;
}
if ($prerender = 0) {
rewrite .* /index.html break;
}
}
}
@pvolyntsev

condition

    if ($uri ~ ".js|.css|.xml|.less|.png|.jpg|.jpeg|.gif|.pdf|.doc|.txt|.ico|.rss|.zip|.mp3|.rar|.exe|.wmv|.doc|.avi|.ppt|.mpg|.mpeg|.tif|.wav|.mov|.psd|.ai|.xls|.mp4|.m4a|.swf|.dat|.dmg|.iso|.flv|.m4v|.torrent") {

incorrect because

    $uri ~ ".ico"

mean "string consist of one any character THEN 'ico'" that matched the URL /icon/ and it is wrong

I suggest this one:

    if ($uri ~ "\.(js|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent)$") {

that means "string is finished dot and one of strings 'js' OR 'css' etc"

@thoop
Owner
thoop commented Feb 27, 2014

You're right, thanks. I'll update that.

@AttilaSATAN

How about html5mode of angularjs? Is prerender supports html5mode, would it be enough to add googlebot to the if statement?

        if ($http_user_agent ~* "baiduspider|twitterbot|facebookexternalhit|googlebot") {
            set $prerender 1;
        }
@thoop
Owner
thoop commented May 26, 2014

Google's recommended way is to use escaped_fragment. In theory, yes we should add it to the user agent check, but in reality we wouldn't want Google to request the same URL using another user agent and then think the user is cloaking because the content is different. So we try to stay on the safe side.

@sansmischevia

Confused... why isn't googlebot listed?

@Michal-sk

@sansmischevia : because googlebot uses the escaped_fragment, which is listed.

@jamiel
jamiel commented Aug 26, 2014

If users are using HTML5 pushState, surely Google will request the URL's without escaped_fragment ?

@toamitkumar

Search bots look for --> <meta name="fragment" content="!" /> in the head tag. Read https://developers.google.com/webmasters/ajax-crawling/docs/specification
Pages without hash fragments
It may be impossible or undesirable for some pages....

@sentient

I'm not too sure on

  rewrite .* /$scheme://example.com$request_uri? break;

Am I only replacing the example.com ? Could we not use the $server_name variable for this?

  rewrite .* /$scheme://$server_name$request_uri? break;
@akoumjian

If our "location /" block is where we typically specify a reverse proxy with proxy_pass, should I assume we would essentially add that to the "if (@prerender = 0)" section?

@thoop
Owner
thoop commented Sep 25, 2014

@sentient good idea :)

@akoumjian yes, in that case you would do your own proxy_pass "if @prerender = 0"

@thoop
Owner
thoop commented Sep 25, 2014

I added this since we were seeing issues where nginx was caching IPs and hitting servers that might have been taken out of our load balancer rotation:

#resolve using Google's DNS server
resolver 8.8.8.8;

#setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
set $prerender "service.prerender.io";
rewrite .* /$scheme://$server_name$request_uri? break;
proxy_pass http://$prerender;
@patrickng

With nginx 1.4.6 (ubuntu) I get an error when I try to restart nginx service or reload the configuration.

nginx: [emerg] "resolver" directive is not allowed here in /etc/nginx/sites-enabled/nginx.conf

I had to move resolver out of the conditional for it to work but this isn't ideal. Any ideas?

@fergusg
fergusg commented Oct 4, 2014

I'm wondering if $host is not a better choice than $server_name in

rewrite .* /$scheme://$server_name$request_uri? break;

There can be multiple $server_name values in a single server{...} but $host is the one in the HTTP requests (and falls back to $server_name anyway)

@cjroebuck

@thoop @patrickng I too have the same issue ('resolver' directive is not allowed here)

@thoop
Owner
thoop commented Oct 17, 2014

Sorry, I wish github would send notifications for comments on gists :(

@fergusg we used to have it set to $host but recently changed it to $server_name.

I'll look into a better solution for the resolver in the if-conditional.

@evityuk
evityuk commented Oct 21, 2014

Vkontakte social network uses this - "Mozilla/5.0 (compatible; vkShare; +http://vk.com/dev/Share)"
UserAgent for sharing functionality. Therefore you need to add 'vkshare' to detect list

pinterest|vkshare

See embeding docs. Sharing uses the same userAgent

@evandhoffman

If behind a load balancer, the $scheme var may not be set right - if the LB is doing SSL termination, the scheme on the machine behind the box may be http. Prerender service would then try to access http://, but get bounced to https://, which in my case did not work. I had to hard-code https:// in there.

@leorue
leorue commented Nov 3, 2014

Any update on @patrickng resolver issue? I took out the conditional as well.

@thoop
Owner
thoop commented Nov 5, 2014

@leorue can you try the new nginx config? I just updated it to move the resolver outside of the "if" statement.

@thoop
Owner
thoop commented Nov 5, 2014

I just changed $server_name back to $host. Hopefully that clears up any issues with the server name not being the actual url of your site.

@shirokoweb

Hi, do I have to install something on my webserver running nginx as frontend HTTP server, or do I simply need to add this snippet to my vhost .conf ?

@thoop
Owner
thoop commented Dec 2, 2014

You should just be able to add this snippet to your .conf file. Email me at todd@prerender.io if you're having any problems with it. Github doesn't send notifications on gists so I'll be able to help you more quickly over email.

@varuzhnikov

http://wiki.nginx.org/IfIsEvil

Directive if has problems when used in location context, in some cases it doesn't do what you expect but something completely different instead. In some cases it even segfaults. It's generally a good idea to avoid it if possible.

The only 100% safe things which may be done inside if in location context are:

return ...;
rewrite ... last;
Anything else may possibly cause unpredictable behaviour, including potential SIGSEGV.

It is important to note that the behaviour of if is not inconsistent, given two identical requests it will not randomly fail on one and work on the other, with proper testing and understanding ifs can be used. The advice to use other directives where available still very much apply, though.

@intellix
intellix commented Dec 8, 2014

Google doesn't use ?escaped_fragment= for all of it's services. It might do for it's indexer but for instance when I use it from the Webmaster tools "Fetch as Google", it correctly renders it but the HTML it received was before rendering:

66.249.75.21 - - [08/Dec/2014:12:11:51 +0000] "GET /owner/ HTTP/1.1" 200 6292 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.75.37 - - [08/Dec/2014:12:11:51 +0000] "GET /owner/?_escaped_fragment_= HTTP/1.1" 200 9401 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

In fact, I am ALWAYS seeing two requests from Googlebot - One with and without the escaped fragment.
I'm seeing other services without the escaped fragment as well: Google Page Speed Insights, Google-StructuredDataTestingTool, Google Web Preview.

A few that I added myself: (tumblr|slackbot|xml-sitemaps|google-structureddatatestingtool). Definitely need tubmlr :) It's a shame we have to go with a whitelist approach, because there are so many services where my site is just invisible.

@csbenjamin

@intellix I guess that in the first request google find the and then do a second request with escaped_fragment. Before the first request it did not know that it needs to use escaped_fragment

@thoop
Owner
thoop commented Dec 9, 2014

@varuzhnikov if we could refactor to remove if statements, that would be ideal. Any idea on the best way to do that? We haven't had any problems with this configuration yet.

@intellix @csbenjamin is correct. The first request see's your tag and re-requests the page with the escaped fragment parameter. "Fetch as Google" does not follow the escaped fragment, and that is a known bug. If you are seeing services that do not send two requests (one with escaped fragment), then we should add them to the whitelist of user agents.

@bastijan

I use self hosted prerender and nginx server/angularjs html application but when try share contentn on facebook or linked I always got root page (index.html) /

nginx conf
set $prerender "85.10.211.83:4000";

rewrite .* /$scheme://$host$request_uri? break;
proxy_pass http://$prerender;

just for review:
http://85.10.211.83:4000/http://novi.bktvnews.com:3030/#!/
and single pages is same
http://85.10.211.83:4000/http://novi.bktvnews.com:3030/#!/grafika-iz-dalarne-na-putu-vas

in app.js
I have

$locationProvider.hashPrefix('!');

and in index.html template

@lethaldose

we had a AWS nginx proxy setup and were having timeout issues - service.prerender.io could not be resolved (60: Operation timed out). We were able to get around this by using nginx upstream:

upstream pre-render {
   server service.prerender.io;
}

and the if in location block changed to:

if ($prerender = 1) {
   rewrite ^(.*)$ /https://$server_name..... break;
   proxy_pass http://pre-render;
}
@creatorkuang

I use the official server :"service.prerender.io" and it works, but when i try to use my own server ,it didn't. I have test it with http://myserver.com/http://www.google.com and it work with my server which i think it mean the server work. But when i replace "service.prerender.io" with "myserver.com" , it didn't work and got an error with 502 Bad Gateway . Anyone know why?

@geolart
geolart commented Jan 17, 2015

I use nginx like a reverse proxy and i would like to use prerender. I don't know how to adapt the nginx.conf especially this section:

location / {
try_files $uri @prerender;
}

I can't use the "try_files" instruction as i use "proxy_pass" instruction.

Do you know how to do ? Thank you

@rkulla
rkulla commented Jan 17, 2015

@creatorkuang it'd be hard to know without seeing your configuration. myserver.com needs to be running the open-source prerender node server and on the right port (by default it's 3000), and your nginx prerender middleware has to be configured to proxy there if the prerender variable = 1. E.g. proxy_pass http://myserver.com; But I've only tried the open-source server from localhost:3000.

@rkulla
rkulla commented Jan 17, 2015

@geolart I think some people above commented that you can do that by moving your proxy_pass line into the location @prerender block if the prerender variable = 0 (instead of rewrite .* /index.html break;). Let me know if it doesn't work for you though.

@geolart
geolart commented Jan 18, 2015

@rkulla Thanks to you it's work for me !
Thank you Rhulla !

@rkulla
rkulla commented Jan 18, 2015

When also using nginx as a reverse proxy via proxy_pass http://myupstream, I had an issue where if I do proxy_set_header X-Prerender-Token XXX in the location @prerender block, it reset my other proxy_set_header lines, causing 'http://myupstream' to be treated as a literal URL. However, things work fine if I redefine my proxy_set_header 'Host', etc in the same block as X-Prerender-Token -- either all in the location / or all in the location @prerender, but not divided.

@thoop
Owner
thoop commented Feb 18, 2015

Send me an email at support@prerender.io if anyone has any issues. I don't get notified when someone comments on this gist.

@BenjaminPrice

As per Facebook official documentation, you should also add the user agent 'facebot' to $http_user_agent

https://developers.facebook.com/docs/sharing/best-practices#crawl

@colthreepv

I have monitored many requests that come with Googlebot and AdsBot-Google-Mobile as User-Agent, so I added it to my list.

There is an exhaustive of those User-Agents somewhere?

@jstoiko
jstoiko commented Apr 28, 2015

You might want to add 'svg' to the list of extensions not prerendered.

@ermakovich

@evandhoffman we have the same issue when hosting on Heroku, which in turn seems to be using Amazon ELB. $scheme is always HTTP. As an alternative to hardcoding we decided to use $http_x_forwarded_proto instead of $scheme.

@thoop
Owner
thoop commented May 26, 2015

@mrgamer you don't want to add Googlebot (or any other crawlers that support the escaped fragment protocol) to the user agent list. You could get penalized for cloaking. You want Google to continue using the escaped fragment protocol

@ermakovich great idea!

Send me an email at support@prerender.io if anyone has any issues. I don't get notified when someone comments on this gist.

@ashishgupta2

For nginx proxy_pass users: do two things.

  1. Comment out: following two lines in server block

    #root /path/to/your/root;
    #index index.html;

  2. In the last if block ("if ($prerender = 0) {....."):

replace #rewrite .* /index.html break; with your proxy_pass as shown below (my application is running on port 3009).

proxy_pass http://127.0.0.1:3009;

@matfin
matfin commented Aug 31, 2015

Does anyone have an example of something like this working for Meteor ?

@blaind
blaind commented Sep 13, 2015

@matfin see https://gist.github.com/blaind/12e53d5d9aa77c9fd841 for a meteor config. Had to use global proxy configs for both (can't put them in if's), but prerender.io seems to accept upgraded HTTP 1.1 too.

@maxklenk

Please add redditbot to the list of user agents.

@alphastorm

There's also trailing whitespace in 6 lines, might want to remove that.

@donny08
donny08 commented Oct 16, 2015

@thoop I'm configuring prerender using nginx with nodejs backened server.Below is my nginx.conf file:-

server {
listen 80;
server_name _;

    location / {        
    try_files $uri @prerender;         
  }

   location @prerender {
    proxy_set_header X-Prerender-Token your_token;

    set $prerender 0;        

    if ($http_user_agent ~* "baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator") {
        set $prerender 1;
    }
    if ($args ~ "_escaped_fragment_") {           
        set $prerender 1;
    }
    if ($http_user_agent ~ "Prerender") {
        set $prerender 0;
    }
    if ($uri ~ "\.(js|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent|ttf|woff)") {
        set $prerender 0;
    }       

    #resolve using Google's DNS server to force DNS resolution and prevent caching of IPs
    resolver 8.8.8.8;
    if ($prerender = 1) {            
        #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
        set $prerender "localhost:8000";
        rewrite .* /$scheme://$host$request_uri? break;
        proxy_pass http://localhost:8000;
    }    

    if ($prerender = 0) {
        proxy_pass http://localhost:3000;

    }
}

    #error_page  404              /404.html;

    # redirect server error pages to the static page /50x.html
    #
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   html;
    }


}

But I am seeing JavaScript and not the static HTML.I had the checked Common Problem section,It has mentioned Try moving your Prerender middleware configuration higher in your file,I want to know what is
configuration code for nginx that i need to move to higher section of file.

@thabofletcher

I've posted a small project that demonstrates an implementation of how to use this without violating the "If Is Evil" principle:
https://github.com/thabofletcher/prerender.io-nginx

This allows any configuration for your app, including rewrites, try_files, and proxy_path, all outside if blocks. I believe it can be consolidated into a single gist/config, but I prefer the use of includes for readability.

@donny08
donny08 commented Oct 27, 2015

This is my url :http://example.com/customer/#/index,I have already done prerender configuration.How do i access the prerender page of http://example.com/customer/#/index, what should be my url format

@asquel
asquel commented Nov 27, 2015

+1 for the question from @thiagofesta ! (what do you think @thoop ?)

@TheProdigyFilippo

@thiagofesta @asquel don't believe in this. It's not as nice as google tells. We shot ourselves in feets when we changed our static title and meta description into dynamic using AngularJS <meta name="description" content="{{ pageDescription }}" />. Now we have { pageDescription }} as page description in google results and Alexa ranks raised up half milion points :) So we want to start to use prerender.io soon.

Correction:
I have to appologize google and have to admit to our mistake. We used <meta name="fragment" content="!"> and in this case google expects that you provide snapshot. After removing <meta name="fragment" content="!"> site and meta description were rendered properly. Other thing is that prerendering is good for other services and crawlers.

So when you depend on google rendering only, you should remember to delete <meta name="fragment" content="!"> meta tag!

@antoineco

@thabofletcher I used a similar configuration but when I got rid of

if ($prerender = 0) {
  rewrite .* /index.html break;
}

and replaced it by a single try_files directive, Prerender started getting 404 errors while I got a 202 cURLing the same URL. I wrongly assumed this would work as long as $prerender != 1, but it doesn't when proxied through Prerender. Have you ever faced this issue with your configuration?

edit: the explanation is here, IfIsEvil (nginx docs)

@rogierslag

I've made a fork and updated it with the following user agents baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest|slackbot|vkShare|Slack-ImgProxy|Slackbot-LinkExpanding|Site Analyzer|SiteAnalyzerBot|Viber|Whatsapp|Telegram|W3C_Validator.

These now also include stuff as Viber, WhatsApp, and Telegram which all try to parse the og tags

@charlesdg

@ermakovich thanks for the hack for heroku!

If you are using the Heroku Buildpack: NGINX (https://elements.heroku.com/buildpacks/ryandotsmith/nginx-buildpack), just replace

rewrite .* /$scheme://$host$request_uri? break;

by

rewrite .* /$http_x_forwarded_proto://$host$request_uri? break;

Hope that will be useful !

@kascote
kascote commented Jan 26, 2016

I'm using this setup with an Angular application and works ok except to for the root (/) path that never goes throw prerender.
I think the problem is with try_files that found the index.html and serve it without going to the @prerender section.

someone have this issue ? how to resolve it ?

@intellix

Been having issues with prerender dying and having my entire site disappear from Google when everything returns 504 errors (my fault for not having server alerts of any kind). I think in such a case where prerender just dies, we should have a fallback so google at least gets served your HTML/JS (they just won't for API calls to resolve, as prerender is there for).

Example error that'll bring your prerender service down

2015-12-10T17:00:31.669Z started phantom
events.js:85
      throw er; // Unhandled 'error' event
            ^
Error: channel closed
    at ChildProcess.target.send (child_process.js:414:26)
    at sendHelper (cluster.js:676:8)
    at RoundRobinHandle.handoff (cluster.js:198:3)
    at RoundRobinHandle.distribute (cluster.js:184:20)

Been playing around with how to get error_page working with a proxy and this is what works.:

server {
    # Add this line, to redirect 50x errors back to index.html
    error_page 500 502 503 504 =200 /index.html;

    location / {
        index     index.html;
        try_files $uri @prerender;
    }

    location @prerender {
        # Add this line inside @prerender location
        proxy_intercept_errors on;
    }
}
@thedug
thedug commented Feb 4, 2016

Why not just simply do this vs. the rewrite?

proxy_pass http://service.prerender.io/$scheme://$host$request_uri?;

@marcelpanse

Google has deprecated the escaped fragment scheme:
https://googlewebmastercentral.blogspot.nl/2015/10/deprecating-our-ajax-crawling-scheme.html

I think googlebot should be added to the whitelist, the escaped fragment scheme will probably still used by other bots like bing.

@thoop
Owner
thoop commented Feb 26, 2016

@marcelpanse Google is still not doing a good enough job when crawling javascript, which they even admitted in their last Google Webmaster Hangout. Google still follows the escaped fragment crawling protocol like they did before so Prerendering pages for Google is still the recommended way of doing things. Adding googlebot to the whitelist could get your website penalized for cloaking so it's not recommended. You should support the escaped fragment crawling protocol for Googlebot.

@thoop
Owner
thoop commented Feb 26, 2016

@thedug The rewrite is the way that it is because nginx caches IP addresses. If an IP address changes on our end, then your prerendering could stop working. The way we have it with the variable will prevent nginx from caching IP addresses to prevent this problem.

@thoop
Owner
thoop commented Feb 26, 2016

I don't get any notifications when people comment on this thread so please email me at todd@prerender.io if anyone has any questions.

@ray65536

You should move block of If -- set -- out of location to server definition level. Or you'll get a lot of problems with "if is Evil" in Nginx.

Like this:

server {
    listen 80;
    server_name example.com;

    root   /path/to/your/root;
    index  index.html;

        set $prerender 0;
        if ($http_user_agent ~* "baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator") {
            set $prerender 1;
        }
        if ($args ~ "_escaped_fragment_") {
            set $prerender 1;
        }
        if ($http_user_agent ~ "Prerender") {
            set $prerender 0;
        }
        if ($uri ~ "\.(js|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent|ttf|woff)") {
            set $prerender 0;
        }

    location / {
        try_files $uri @prerender;
    }

    location @prerender {
        #proxy_set_header X-Prerender-Token YOUR_TOKEN;

        #resolve using Google's DNS server to force DNS resolution and prevent caching of IPs
        resolver 8.8.8.8;

        if ($prerender = 1) {

            #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
            set $prerender "service.prerender.io";
            rewrite .* /$scheme://$host$request_uri? break;
            proxy_pass http://$prerender;
        }
        if ($prerender = 0) {
            rewrite .* /index.html break;
        }
    }
}
@SiddharthTerse

Hi,

If i am browsing any webpage so 1st time with ?__escaped_fragment_ redirecting 301 & from next time 200 OK . this is happening for all pages....

My nginx config file for prerender is...

location @prerender {
#proxy_set_header X-Prerender-Token YOUR_TOKEN;

    set $prerender 0;
    if ($http_user_agent ~* "baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator") {
        set $prerender 1;
    }
    if ($args ~ "_escaped_fragment_") {
        set $prerender 1;
    }
    if ($http_user_agent ~ "Prerender") {
        set $prerender 0;
    }
    if ($uri ~ "\.(js|ico|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent|ttf|woff)") {
        set $prerender 0;
    }

    #resolve using Google's DNS server to force DNS resolution and prevent caching of IPs
    resolver 8.8.8.8;

    if ($prerender = 1) {

        #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
        set $prerender "192.168.152.191:3000";
             rewrite .* /$scheme://$server_name$request_uri? break;
        proxy_pass http://$prerender;

    }
    if ($prerender = 0) {
        rewrite .* /index.html break;
    }
}

    location ~* \.(gif|jpg|png|ico)$ {
            expires 30d;
    }

}

@SiddharthTerse

@thoop, i found article where google disclosed they have stopped to support official for escaped_fragment (http://searchengineland.com/google-has-deprecated-their-ajax-crawling-scheme-233402) . So now what is the alternative for google crowling now ?

@wellington1993

Thanks in a lot!

@bwoodlt
bwoodlt commented Mar 30, 2016

Hi Guys,

I'm trying to use this config, but keep returning 404 error. I'll appreciate any help in having a quick look.

server {
listen 443;
server_name localhost;

location / {
    root   /usr/share/nginx/html;
    index  index.html index.htm;
    try_files $uri @prerender;
}
location @prerender {
        proxy_intercept_errors on;
        proxy_set_header X-Prerender-Token our_token;

        set $prerender 0;
        if ($http_user_agent ~* "baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link     preview|showyoubot|outbrain|pinterest|slackbot|vkShare|Slack-ImgProxy|Slackbot-LinkExpanding|Site Analyzer|SiteAnalyzerBot|Viber|Whatsapp|Telegram|W3C_Validator") {
            set $prerender 1;
        }
        if ($args ~ "_escaped_fragment_") {
            set $prerender 1;
        }
        if ($http_user_agent ~ "Prerender") {
            set $prerender 0;
        }
        if ($uri ~ "\.(js|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent|ttf|woff)") {
            set $prerender 0;
        }

        #resolve using Google's DNS server to force DNS resolution and prevent caching of IPs
        resolver 8.8.8.8;

        #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
        set $prerender "service.prerender.io";
        rewrite .* /$scheme://$host$request_uri? break;
        proxy_pass http://$prerender;

        if ($prerender = 1) {

            #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
            set $prerender "service.prerender.io";
            rewrite .* /$scheme://$host$request_uri? break;
            proxy_pass http://$prerender;
        }
        if ($prerender = 0) {
            rewrite .* /index.html break;
        }
    }

}

@rickyk586

lethaldose's solution above worked for me

@Gbeschbacher

Hey there,

i am trying to use prerender with nginx and our angular.js app.
The setting is as follows: i have docker running on the server. Nginx is used as a reverse proxy to forward the requests to the docker-container. Inside the container is another Nginx running, which serves the web app.
Now i do not know exactly where i should put the prerender config. Following i post both nginx-configs for clarity and maybe some more convenience:

This is the nginx-config which proxies to the docker-container

server {
        listen                  80;
        return 301              https://$host$request_uri;
}

### DEVELOP ###
server {
        listen                  443 ssl;
        ssl_certificate         /path/to/file
        ssl_certificate_key     /path/to/file
        server_name             develop.vidatio.mediacube.at;

        location / {
                proxy_pass http://localhost:6000;
        }

        location /.well-known {
                root /var/www/do.not.delete.vidatio.com;
        }
}

and this is the currently working nginx-config inside the container:

worker_processes  1;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;

    server {
        listen       80;
        server_name  localhost;

        location / {
            root /var/www/build;
            try_files $uri /statics/master.html;
        }

        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
}

i tried to get it to work like this but it doesnt:

http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;

    server {
        listen       80;
        server_name  localhost;

        root /var/www/build;

        location / {
            try_files $uri @prerender /statics/master.html;
        }

        location @prerender {
            proxy_set_header X-Prerender-Token --TOKEN--

            set $prerender 0;
            if ($http_user_agent ~* "baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator") {
                set $prerender 1;
            }
            if ($args ~ "_escaped_fragment_") {
                set $prerender 1;
            }
            if ($http_user_agent ~ "Prerender") {
                set $prerender 0;
            }
            if ($uri ~ "\.(js|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent|ttf|woff)") {
                set $prerender 0;
            }

            #resolve using Google's DNS server to force DNS resolution and prevent caching of IPs
            resolver 8.8.8.8;

            if ($prerender = 1) {

                #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
                set $prerender "service.prerender.io";
                rewrite .* /$scheme://$host$request_uri? break;
                proxy_pass http://$prerender;
            }
            if ($prerender = 0) {
                rewrite .* /statics/master.html break;
            }
        }

        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
}

Thanks in advance!

@vic3lord

if other people like me tried to proxy_pass if $prerender = 0 then just remove the if condition as it acts like else
You won't get any nginx config errors

Hope it helps

@AntonAL
AntonAL commented Apr 26, 2016 edited

Hello, guys!

I've started prerender server listening port 3000, in my production machine, used this config file.
So, I've replaced this part to localhost:3000:

if ($prerender = 1) {
    #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
    set $prerender "service.prerender.io";
    rewrite .* /$scheme://$host$request_uri? break;
    proxy_pass http://$prerender;
}
if ($prerender = 1) {            
    #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
    set $prerender "localhost:3000";
    rewrite .* /$scheme://$host$request_uri? break;
    proxy_pass http://$prerender;
}

When I requesting home page from external client host, I'm always getting 504 and no rendering:
Fragment from Nginx (tail -f /var/log/nginx/access.log):

213.21.36.130 - - [26/Apr/2016:08:08:46 +0000] "GET / HTTP/1.1" 504 0 "-" "twitterbot"

When I requesting home page directly from production server, everything is working:
curl -A 'twitterbot' http://localhost:3000/http://my-website.com

When I using set $prerender "service.prerender.io";, everyting is also working.

It seems, like I have problem with local prerenderer…

Also, I don't understand, how this configuration passes requests to meteor in case of non-bot.
As far, as I understood this fragment:

if ($prerender = 0) {
    rewrite .* /index.html break;
}

It loads static index.html…but we are using Meteor…

My initial configuration of nginx without prerender is taken from guide How To Deploy a Meteor.js Application on Ubuntu 14.04 with Nginx

Anyone having the same issue?
Thanks in advance!

@aaronbuchanan

@intellix can you shed some light on prerender dying on your site? Know what the root cause was? Find a resolution you're happy with?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.