Official prerender.io nginx.conf for nginx

nginx.conf

# Change YOUR_TOKEN to your prerender token and uncomment that line if you want to cache urls and view crawl stats
# Change example.com (server_name) to your website url
# Change /path/to/your/root to the correct value

server {
    listen 80;
    server_name example.com;
 
    root   /path/to/your/root;
    index  index.html;

    location / {
        try_files $uri @prerender;
    }
 
    location @prerender {
        #proxy_set_header X-Prerender-Token YOUR_TOKEN;
        
        set $prerender 0;
        if ($http_user_agent ~* "baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator") {
            set $prerender 1;
        }
        if ($args ~ "_escaped_fragment_") {
            set $prerender 1;
        }
        if ($http_user_agent ~ "Prerender") {
            set $prerender 0;
        }
        if ($uri ~* "\.(js|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent|ttf|woff|svg|eot)") {
            set $prerender 0;
        }
        
        #resolve using Google's DNS server to force DNS resolution and prevent caching of IPs
        resolver 8.8.8.8;
 
        if ($prerender = 1) {
            
            #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
            set $prerender "service.prerender.io";
            rewrite .* /$scheme://$host$request_uri? break;
            proxy_pass http://$prerender;
        }
        if ($prerender = 0) {
            rewrite .* /index.html break;
        }
    }
}

condition

    if ($uri ~ ".js|.css|.xml|.less|.png|.jpg|.jpeg|.gif|.pdf|.doc|.txt|.ico|.rss|.zip|.mp3|.rar|.exe|.wmv|.doc|.avi|.ppt|.mpg|.mpeg|.tif|.wav|.mov|.psd|.ai|.xls|.mp4|.m4a|.swf|.dat|.dmg|.iso|.flv|.m4v|.torrent") {

incorrect because

    $uri ~ ".ico"

mean "string consist of one any character THEN 'ico'" that matched the URL /icon/ and it is wrong

I suggest this one:

    if ($uri ~ "\.(js|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent)$") {

that means "string is finished dot and one of strings 'js' OR 'css' etc"

You're right, thanks. I'll update that.

How about html5mode of angularjs? Is prerender supports html5mode, would it be enough to add googlebot to the if statement?

        if ($http_user_agent ~* "baiduspider|twitterbot|facebookexternalhit|googlebot") {
            set $prerender 1;
        }

Google's recommended way is to use escaped_fragment. In theory, yes we should add it to the user agent check, but in reality we wouldn't want Google to request the same URL using another user agent and then think the user is cloaking because the content is different. So we try to stay on the safe side.

Confused... why isn't googlebot listed?

@sansmischevia : because googlebot uses the escaped_fragment, which is listed.

If users are using HTML5 pushState, surely Google will request the URL's without escaped_fragment ?

Search bots look for --> <meta name="fragment" content="!" /> in the head tag. Read https://developers.google.com/webmasters/ajax-crawling/docs/specification
Pages without hash fragments It may be impossible or undesirable for some pages....

I'm not too sure on

  rewrite .* /$scheme://example.com$request_uri? break;

Am I only replacing the example.com ? Could we not use the $server_name variable for this?

  rewrite .* /$scheme://$server_name$request_uri? break;

If our "location /" block is where we typically specify a reverse proxy with proxy_pass, should I assume we would essentially add that to the "if (@prerender = 0)" section?

@sentient good idea :)

@akoumjian yes, in that case you would do your own proxy_pass "if @prerender = 0"

I added this since we were seeing issues where nginx was caching IPs and hitting servers that might have been taken out of our load balancer rotation:

#resolve using Google's DNS server
resolver 8.8.8.8;

#setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
set $prerender "service.prerender.io";
rewrite .* /$scheme://$server_name$request_uri? break;
proxy_pass http://$prerender;

With nginx 1.4.6 (ubuntu) I get an error when I try to restart nginx service or reload the configuration.

nginx: [emerg] "resolver" directive is not allowed here in /etc/nginx/sites-enabled/nginx.conf

I had to move resolver out of the conditional for it to work but this isn't ideal. Any ideas?

I'm wondering if $host is not a better choice than $server_name in

rewrite .* /$scheme://$server_name$request_uri? break;

There can be multiple $server_name values in a single server{...} but $host is the one in the HTTP requests (and falls back to $server_name anyway)

@thoop @patrickng I too have the same issue ('resolver' directive is not allowed here)

Sorry, I wish github would send notifications for comments on gists :(

@fergusg we used to have it set to $host but recently changed it to $server_name.

I'll look into a better solution for the resolver in the if-conditional.

Vkontakte social network uses this - "Mozilla/5.0 (compatible; vkShare; +http://vk.com/dev/Share)"
UserAgent for sharing functionality. Therefore you need to add 'vkshare' to detect list

pinterest|vkshare

See embeding docs. Sharing uses the same userAgent

If behind a load balancer, the $scheme var may not be set right - if the LB is doing SSL termination, the scheme on the machine behind the box may be http. Prerender service would then try to access http://, but get bounced to https://, which in my case did not work. I had to hard-code https:// in there.

Any update on @patrickng resolver issue? I took out the conditional as well.

@leorue can you try the new nginx config? I just updated it to move the resolver outside of the "if" statement.

I just changed $server_name back to $host. Hopefully that clears up any issues with the server name not being the actual url of your site.

Hi, do I have to install something on my webserver running nginx as frontend HTTP server, or do I simply need to add this snippet to my vhost .conf ?

You should just be able to add this snippet to your .conf file. Email me at todd@prerender.io if you're having any problems with it. Github doesn't send notifications on gists so I'll be able to help you more quickly over email.

http://wiki.nginx.org/IfIsEvil

Directive if has problems when used in location context, in some cases it doesn't do what you expect but something completely different instead. In some cases it even segfaults. It's generally a good idea to avoid it if possible.

The only 100% safe things which may be done inside if in location context are:

return ...;
rewrite ... last;
Anything else may possibly cause unpredictable behaviour, including potential SIGSEGV.

It is important to note that the behaviour of if is not inconsistent, given two identical requests it will not randomly fail on one and work on the other, with proper testing and understanding ifs can be used. The advice to use other directives where available still very much apply, though.

Google doesn't use ?escaped_fragment= for all of it's services. It might do for it's indexer but for instance when I use it from the Webmaster tools "Fetch as Google", it correctly renders it but the HTML it received was before rendering:

66.249.75.21 - - [08/Dec/2014:12:11:51 +0000] "GET /owner/ HTTP/1.1" 200 6292 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.75.37 - - [08/Dec/2014:12:11:51 +0000] "GET /owner/?_escaped_fragment_= HTTP/1.1" 200 9401 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

In fact, I am ALWAYS seeing two requests from Googlebot - One with and without the escaped fragment.
I'm seeing other services without the escaped fragment as well: Google Page Speed Insights, Google-StructuredDataTestingTool, Google Web Preview.

A few that I added myself: (tumblr|slackbot|xml-sitemaps|google-structureddatatestingtool). Definitely need tubmlr :) It's a shame we have to go with a whitelist approach, because there are so many services where my site is just invisible.

@intellix I guess that in the first request google find the and then do a second request with escaped_fragment. Before the first request it did not know that it needs to use escaped_fragment

@varuzhnikov if we could refactor to remove if statements, that would be ideal. Any idea on the best way to do that? We haven't had any problems with this configuration yet.

@intellix @csbenjamin is correct. The first request see's your tag and re-requests the page with the escaped fragment parameter. "Fetch as Google" does not follow the escaped fragment, and that is a known bug. If you are seeing services that do not send two requests (one with escaped fragment), then we should add them to the whitelist of user agents.

I use self hosted prerender and nginx server/angularjs html application but when try share contentn on facebook or linked I always got root page (index.html) /

nginx conf
set $prerender "85.10.211.83:4000";
rewrite .* /$scheme://$host$request_uri? break;
proxy_pass http://$prerender;

just for review:
http://85.10.211.83:4000/http://novi.bktvnews.com:3030/#!/
and single pages is same
http://85.10.211.83:4000/http://novi.bktvnews.com:3030/#!/grafika-iz-dalarne-na-putu-vas

in app.js
I have

$locationProvider.hashPrefix('!');

and in index.html template

we had a AWS nginx proxy setup and were having timeout issues - service.prerender.io could not be resolved (60: Operation timed out). We were able to get around this by using nginx upstream:

upstream pre-render {
   server service.prerender.io;
}

and the if in location block changed to:

if ($prerender = 1) {
   rewrite ^(.*)$ /https://$server_name..... break;
   proxy_pass http://pre-render;
}

I use the official server :"service.prerender.io" and it works, but when i try to use my own server ,it didn't. I have test it with http://myserver.com/http://www.google.com and it work with my server which i think it mean the server work. But when i replace "service.prerender.io" with "myserver.com" , it didn't work and got an error with 502 Bad Gateway . Anyone know why?

I use nginx like a reverse proxy and i would like to use prerender. I don't know how to adapt the nginx.conf especially this section:

location / {
try_files $uri @prerender;
}

I can't use the "try_files" instruction as i use "proxy_pass" instruction.

Do you know how to do ? Thank you

@creatorkuang it'd be hard to know without seeing your configuration. myserver.com needs to be running the open-source prerender node server and on the right port (by default it's 3000), and your nginx prerender middleware has to be configured to proxy there if the prerender variable = 1. E.g. proxy_pass http://myserver.com; But I've only tried the open-source server from localhost:3000.

@geolart I think some people above commented that you can do that by moving your proxy_pass line into the location @prerender block if the prerender variable = 0 (instead of rewrite .* /index.html break;). Let me know if it doesn't work for you though.

@rkulla Thanks to you it's work for me !
Thank you Rhulla !

When also using nginx as a reverse proxy via proxy_pass http://myupstream, I had an issue where if I do proxy_set_header X-Prerender-Token XXX in the location @prerender block, it reset my other proxy_set_header lines, causing 'http://myupstream' to be treated as a literal URL. However, things work fine if I redefine my proxy_set_header 'Host', etc in the same block as X-Prerender-Token -- either all in the location / or all in the location @prerender, but not divided.

Send me an email at support@prerender.io if anyone has any issues. I don't get notified when someone comments on this gist.

As per Facebook official documentation, you should also add the user agent 'facebot' to $http_user_agent

https://developers.facebook.com/docs/sharing/best-practices#crawl

I have monitored many requests that come with Googlebot and AdsBot-Google-Mobile as User-Agent, so I added it to my list.

There is an exhaustive of those User-Agents somewhere?

You might want to add 'svg' to the list of extensions not prerendered.

@evandhoffman we have the same issue when hosting on Heroku, which in turn seems to be using Amazon ELB. $scheme is always HTTP. As an alternative to hardcoding we decided to use $http_x_forwarded_proto instead of $scheme.

@MrGamer you don't want to add Googlebot (or any other crawlers that support the escaped fragment protocol) to the user agent list. You could get penalized for cloaking. You want Google to continue using the escaped fragment protocol

@ermakovich great idea!

Send me an email at support@prerender.io if anyone has any issues. I don't get notified when someone comments on this gist.

For nginx proxy_pass users: do two things.

1. Comment out: following two lines in server block

   root /path/to/your/root;

   index index.html;

2. In the last if block ("if ($prerender = 0) {....."):

replace #rewrite .* /index.html break; with your proxy_pass as shown below (my application is running on port 3009).

proxy_pass http://127.0.0.1:3009;

Does anyone have an example of something like this working for Meteor ?

@matfin see https://gist.github.com/blaind/12e53d5d9aa77c9fd841 for a meteor config. Had to use global proxy configs for both (can't put them in if's), but prerender.io seems to accept upgraded HTTP 1.1 too.

Please add redditbot to the list of user agents.

There's also trailing whitespace in 6 lines, might want to remove that.

@thoop I'm configuring prerender using nginx with nodejs backened server.Below is my nginx.conf file:-

server {
listen 80;
server_name _;

    location / {        
    try_files $uri @prerender;         
  }

   location @prerender {
    proxy_set_header X-Prerender-Token your_token;

    set $prerender 0;        

    if ($http_user_agent ~* "baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator") {
        set $prerender 1;
    }
    if ($args ~ "_escaped_fragment_") {           
        set $prerender 1;
    }
    if ($http_user_agent ~ "Prerender") {
        set $prerender 0;
    }
    if ($uri ~ "\.(js|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent|ttf|woff)") {
        set $prerender 0;
    }       

    #resolve using Google's DNS server to force DNS resolution and prevent caching of IPs
    resolver 8.8.8.8;
    if ($prerender = 1) {            
        #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
        set $prerender "localhost:8000";
        rewrite .* /$scheme://$host$request_uri? break;
        proxy_pass http://localhost:8000;
    }    

    if ($prerender = 0) {
        proxy_pass http://localhost:3000;

    }
}

    #error_page  404              /404.html;

    # redirect server error pages to the static page /50x.html
    #
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   html;
    }


}

But I am seeing JavaScript and not the static HTML.I had the checked Common Problem section,It has mentioned Try moving your Prerender middleware configuration higher in your file,I want to know what is
configuration code for nginx that i need to move to higher section of file.

I've posted a small project that demonstrates an implementation of how to use this without violating the "If Is Evil" principle:
https://github.com/thabofletcher/prerender.io-nginx

This allows any configuration for your app, including rewrites, try_files, and proxy_path, all outside if blocks. I believe it can be consolidated into a single gist/config, but I prefer the use of includes for readability.

This is my url :http://example.com/customer/#/index,I have already done prerender configuration.How do i access the prerender page of http://example.com/customer/#/index, what should be my url format

What about: http://googlewebmastercentral.blogspot.com.br/2015/10/deprecating-our-ajax-crawling-scheme.html

+1 for the question from @thiagofesta ! (what do you think @thoop ?)

@thiagofesta @asquel don't believe in this. It's not as nice as google tells. We shot ourselves in feets when we changed our static title and meta description into dynamic using AngularJS <meta name="description" content="{{ pageDescription }}" />. Now we have { pageDescription }} as page description in google results and Alexa ranks raised up half milion points :) So we want to start to use prerender.io soon.

Correction:
I have to appologize google and have to admit to our mistake. We used <meta name="fragment" content="!"> and in this case google expects that you provide snapshot. After removing <meta name="fragment" content="!"> site and meta description were rendered properly. Other thing is that prerendering is good for other services and crawlers.

So when you depend on google rendering only, you should remember to delete <meta name="fragment" content="!"> meta tag!

@thabofletcher I used a similar configuration but when I got rid of

if ($prerender = 0) {
  rewrite .* /index.html break;
}

and replaced it by a single try_files directive, Prerender started getting 404 errors while I got a 202 cURLing the same URL. I wrongly assumed this would work as long as $prerender != 1, but it doesn't when proxied through Prerender. Have you ever faced this issue with your configuration?

edit: the explanation is here, IfIsEvil (nginx docs)

I've made a fork and updated it with the following user agents baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest|slackbot|vkShare|Slack-ImgProxy|Slackbot-LinkExpanding|Site Analyzer|SiteAnalyzerBot|Viber|Whatsapp|Telegram|W3C_Validator.

These now also include stuff as Viber, WhatsApp, and Telegram which all try to parse the og tags

@ermakovich thanks for the hack for heroku!

If you are using the Heroku Buildpack: NGINX (https://elements.heroku.com/buildpacks/ryandotsmith/nginx-buildpack), just replace

rewrite .* /$scheme://$host$request_uri? break;

by

rewrite .* /$http_x_forwarded_proto://$host$request_uri? break;

Hope that will be useful !

I'm using this setup with an Angular application and works ok except to for the root (/) path that never goes throw prerender.
I think the problem is with try_files that found the index.html and serve it without going to the @prerender section.

someone have this issue ? how to resolve it ?

Been having issues with prerender dying and having my entire site disappear from Google when everything returns 504 errors (my fault for not having server alerts of any kind). I think in such a case where prerender just dies, we should have a fallback so google at least gets served your HTML/JS (they just won't for API calls to resolve, as prerender is there for).

Example error that'll bring your prerender service down

2015-12-10T17:00:31.669Z started phantom
events.js:85
      throw er; // Unhandled 'error' event
            ^
Error: channel closed
    at ChildProcess.target.send (child_process.js:414:26)
    at sendHelper (cluster.js:676:8)
    at RoundRobinHandle.handoff (cluster.js:198:3)
    at RoundRobinHandle.distribute (cluster.js:184:20)

Been playing around with how to get error_page working with a proxy and this is what works.:

server {
    # Add this line, to redirect 50x errors back to index.html
    error_page 500 502 503 504 =200 /index.html;

    location / {
        index     index.html;
        try_files $uri @prerender;
    }

    location @prerender {
        # Add this line inside @prerender location
        proxy_intercept_errors on;
    }
}

Why not just simply do this vs. the rewrite?

proxy_pass http://service.prerender.io/$scheme://$host$request_uri?;

Google has deprecated the escaped fragment scheme:
https://googlewebmastercentral.blogspot.nl/2015/10/deprecating-our-ajax-crawling-scheme.html

I think googlebot should be added to the whitelist, the escaped fragment scheme will probably still used by other bots like bing.

@marcelpanse Google is still not doing a good enough job when crawling javascript, which they even admitted in their last Google Webmaster Hangout. Google still follows the escaped fragment crawling protocol like they did before so Prerendering pages for Google is still the recommended way of doing things. Adding googlebot to the whitelist could get your website penalized for cloaking so it's not recommended. You should support the escaped fragment crawling protocol for Googlebot.

@thedug The rewrite is the way that it is because nginx caches IP addresses. If an IP address changes on our end, then your prerendering could stop working. The way we have it with the variable will prevent nginx from caching IP addresses to prevent this problem.

I don't get any notifications when people comment on this thread so please email me at todd@prerender.io if anyone has any questions.

You should move block of If -- set -- out of location to server definition level. Or you'll get a lot of problems with "if is Evil" in Nginx.

Like this:

server {
    listen 80;
    server_name example.com;

    root   /path/to/your/root;
    index  index.html;

        set $prerender 0;
        if ($http_user_agent ~* "baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator") {
            set $prerender 1;
        }
        if ($args ~ "_escaped_fragment_") {
            set $prerender 1;
        }
        if ($http_user_agent ~ "Prerender") {
            set $prerender 0;
        }
        if ($uri ~ "\.(js|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent|ttf|woff)") {
            set $prerender 0;
        }

    location / {
        try_files $uri @prerender;
    }

    location @prerender {
        #proxy_set_header X-Prerender-Token YOUR_TOKEN;

        #resolve using Google's DNS server to force DNS resolution and prevent caching of IPs
        resolver 8.8.8.8;

        if ($prerender = 1) {

            #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
            set $prerender "service.prerender.io";
            rewrite .* /$scheme://$host$request_uri? break;
            proxy_pass http://$prerender;
        }
        if ($prerender = 0) {
            rewrite .* /index.html break;
        }
    }
}

Hi,

If i am browsing any webpage so 1st time with ?_escaped_fragment redirecting 301 & from next time 200 OK . this is happening for all pages....

My nginx config file for prerender is...

location @prerender {
#proxy_set_header X-Prerender-Token YOUR_TOKEN;

    set $prerender 0;
    if ($http_user_agent ~* "baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator") {
        set $prerender 1;
    }
    if ($args ~ "_escaped_fragment_") {
        set $prerender 1;
    }
    if ($http_user_agent ~ "Prerender") {
        set $prerender 0;
    }
    if ($uri ~ "\.(js|ico|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent|ttf|woff)") {
        set $prerender 0;
    }

    #resolve using Google's DNS server to force DNS resolution and prevent caching of IPs
    resolver 8.8.8.8;

    if ($prerender = 1) {

        #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
        set $prerender "192.168.152.191:3000";
             rewrite .* /$scheme://$server_name$request_uri? break;
        proxy_pass http://$prerender;

    }
    if ($prerender = 0) {
        rewrite .* /index.html break;
    }
}

    location ~* \.(gif|jpg|png|ico)$ {
            expires 30d;
    }

}

@thoop, i found article where google disclosed they have stopped to support official for escaped_fragment (http://searchengineland.com/google-has-deprecated-their-ajax-crawling-scheme-233402) . So now what is the alternative for google crowling now ?

Thanks in a lot!

Hi Guys,

I'm trying to use this config, but keep returning 404 error. I'll appreciate any help in having a quick look.

server {
listen 443;
server_name localhost;

location / {
    root   /usr/share/nginx/html;
    index  index.html index.htm;
    try_files $uri @prerender;
}
location @prerender {
        proxy_intercept_errors on;
        proxy_set_header X-Prerender-Token our_token;

        set $prerender 0;
        if ($http_user_agent ~* "baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link     preview|showyoubot|outbrain|pinterest|slackbot|vkShare|Slack-ImgProxy|Slackbot-LinkExpanding|Site Analyzer|SiteAnalyzerBot|Viber|Whatsapp|Telegram|W3C_Validator") {
            set $prerender 1;
        }
        if ($args ~ "_escaped_fragment_") {
            set $prerender 1;
        }
        if ($http_user_agent ~ "Prerender") {
            set $prerender 0;
        }
        if ($uri ~ "\.(js|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent|ttf|woff)") {
            set $prerender 0;
        }

        #resolve using Google's DNS server to force DNS resolution and prevent caching of IPs
        resolver 8.8.8.8;

        #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
        set $prerender "service.prerender.io";
        rewrite .* /$scheme://$host$request_uri? break;
        proxy_pass http://$prerender;

        if ($prerender = 1) {

            #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
            set $prerender "service.prerender.io";
            rewrite .* /$scheme://$host$request_uri? break;
            proxy_pass http://$prerender;
        }
        if ($prerender = 0) {
            rewrite .* /index.html break;
        }
    }

}

lethaldose's solution above worked for me

Hey there,

i am trying to use prerender with nginx and our angular.js app.
The setting is as follows: i have docker running on the server. Nginx is used as a reverse proxy to forward the requests to the docker-container. Inside the container is another Nginx running, which serves the web app.
Now i do not know exactly where i should put the prerender config. Following i post both nginx-configs for clarity and maybe some more convenience:

This is the nginx-config which proxies to the docker-container

server {
        listen                  80;
        return 301              https://$host$request_uri;
}

### DEVELOP ###
server {
        listen                  443 ssl;
        ssl_certificate         /path/to/file
        ssl_certificate_key     /path/to/file
        server_name             develop.vidatio.mediacube.at;

        location / {
                proxy_pass http://localhost:6000;
        }

        location /.well-known {
                root /var/www/do.not.delete.vidatio.com;
        }
}

and this is the currently working nginx-config inside the container:

worker_processes  1;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;

    server {
        listen       80;
        server_name  localhost;

        location / {
            root /var/www/build;
            try_files $uri /statics/master.html;
        }

        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
}

i tried to get it to work like this but it doesnt:

http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;

    server {
        listen       80;
        server_name  localhost;

        root /var/www/build;

        location / {
            try_files $uri @prerender /statics/master.html;
        }

        location @prerender {
            proxy_set_header X-Prerender-Token --TOKEN--

            set $prerender 0;
            if ($http_user_agent ~* "baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator") {
                set $prerender 1;
            }
            if ($args ~ "_escaped_fragment_") {
                set $prerender 1;
            }
            if ($http_user_agent ~ "Prerender") {
                set $prerender 0;
            }
            if ($uri ~ "\.(js|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent|ttf|woff)") {
                set $prerender 0;
            }

            #resolve using Google's DNS server to force DNS resolution and prevent caching of IPs
            resolver 8.8.8.8;

            if ($prerender = 1) {

                #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
                set $prerender "service.prerender.io";
                rewrite .* /$scheme://$host$request_uri? break;
                proxy_pass http://$prerender;
            }
            if ($prerender = 0) {
                rewrite .* /statics/master.html break;
            }
        }

        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
}

Thanks in advance!

if other people like me tried to proxy_pass if $prerender = 0 then just remove the if condition as it acts like else
You won't get any nginx config errors

Hope it helps

Hello, guys!

I've started prerender server listening port 3000, in my production machine, used this config file.
So, I've replaced this part to localhost:3000:

if ($prerender = 1) {
    #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
    set $prerender "service.prerender.io";
    rewrite .* /$scheme://$host$request_uri? break;
    proxy_pass http://$prerender;
}

if ($prerender = 1) {            
    #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
    set $prerender "localhost:3000";
    rewrite .* /$scheme://$host$request_uri? break;
    proxy_pass http://$prerender;
}

When I requesting home page from external client host, I'm always getting 504 and no rendering:
Fragment from Nginx (tail -f /var/log/nginx/access.log):

213.21.36.130 - - [26/Apr/2016:08:08:46 +0000] "GET / HTTP/1.1" 504 0 "-" "twitterbot"

When I requesting home page directly from production server, everything is working:
curl -A 'twitterbot' http://localhost:3000/http://my-website.com

When I using set $prerender "service.prerender.io";, everyting is also working.

It seems, like I have problem with local prerenderer…

Also, I don't understand, how this configuration passes requests to meteor in case of non-bot.
As far, as I understood this fragment:

if ($prerender = 0) {
    rewrite .* /index.html break;
}

It loads static index.html…but we are using Meteor…

My initial configuration of nginx without prerender is taken from guide How To Deploy a Meteor.js Application on Ubuntu 14.04 with Nginx

Anyone having the same issue?
Thanks in advance!

@intellix can you shed some light on prerender dying on your site? Know what the root cause was? Find a resolution you're happy with?

What about php-fpm?

Yes, how could we make this play nicely with PHP?

prerender get response of 304,how can i fix it?

I hit a problem with a redirect loop. This can happen if you have a redirect from http to https. To detect this you can curl your site curl -A "facebookexternalhit" -v https://example.com/. If it returns 302 with Location: https://example.com/ then you have a redirect loop.

To fix this you need to update this row
rewrite .* /$scheme://$host$request_uri? break;
to
rewrite .* /https://$host$request_uri? break;

This situation can occur when you have a setup where you have one server/proxy handling the SSL termination and http->https redirect and a separate web server that is just an http server running nginx with prerender configuration.

How can this be mixed with open_file_cache?

Confused about why we need a long blacklist of extensions. Wouldn't we just want a whitelist instead? Such as .html, .htm, and no extension at all?

how would this work if your nginx has an upstream proxy?
e.g.

upstream bfa_clj {
   server 0.0.0.0:3000;
}
server {
    listen 80 default_server;
    listen [::]:80 default_server ipv6only=on;

    root /usr/share/nginx/html;
    index index.html index.htm;

    # Make site accessible from http://localhost/
    server_name localhost;

    location / {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_redirect off;
        proxy_pass http://bfa_clj;
        break;
    }
}

I don't get any notifications when people comment on this thread so please email me at support@prerender.io if anyone has any questions. We're happy to help you get set up.

I wonder why bingbot is not included in the list not like Stanback's one.
I tested with Fetch as Bingbot tool.
I also sent an e-mail, Thanks.

If chain of http_user_agent match but my url contains #! can I replace this with ?_escaped_fragment_=?

Please add WhatsApp to user agent chain

@thoop What do you think about the improvment propsed by @rogierslag ?
baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest|slackbot|vkShare|Slack-ImgProxy|Slackbot-LinkExpanding|Site Analyzer|SiteAnalyzerBot|Viber|Whatsapp|Telegram|W3C_Validator

Also i think that these:
Validator.nu|W3C-checklink|W3C-mobileOK|W3C_I18n-Checker|NING|FeedValidator|Jigsaw|W3C_Unicorn
should be added as well.
https://validator.w3.org/services#list

for people that trying to use this config on own prerender servers, you must just replace:

set $prerender "service.prerender.io";
to
set $prerender "127.0.0.1:3000";// one wird thing localhost didn't worked for me so it must be 127.0.0.1:<YOUR_PRERENDER_PORT>

@thoop
server {
listen 80;
listen [::]:80 default_server ipv6only=on;
return 301 https://$host$request_uri;
}

server {
    listen 443;
    server_name example.me;

    ssl on;
    # Use certificate and key provided by Let's Encrypt:
    ssl_certificate /etc/letsencrypt/live/example.me/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/example.me/privkey.pem;
    ssl_session_timeout 5m;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_prefer_server_ciphers on;
    ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';

    # Pass requests for / to localhost:8080:
    location / {
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-NginX-Proxy true;
            proxy_pass http://localhost:8080/;
            proxy_ssl_session_reuse off;
            proxy_set_header Host $http_host;
            proxy_cache_bypass $http_upgrade;
            proxy_redirect off;
    }

}
The above is my Nginx config file am using proxy with https. So how can I configure the above prerender and it's MEAN stack application. What should I give in the path to your root section either my app folder or public folder. Thanks

I had to add googlebot for google's webmaster tool's "Fetch as Google" to work.

https://www.google.com/webmasters/tools/googlebot-fetch-details

Hi
Anyone knows how to do it on nginx on docker and AWS?

This does not include Google +. As stated in the Google+ docs, the user agent contains:

Google (+https://developers.google.com/+/web/snippet/)

which should be added to the list of possible user agents.

Hi Guys,

I was running four node servers using

Configuring multiple applications on same port using nginx reverse proxy based on request urls

its working but the bundles files cannot loaded. I am stuck. Please Help

on Browser Console

Failed to load resource: the server responded with a status of 404 (Not Found)
http://localhost/inline.bundle.js Failed to load resource: the server responded with a status of 404 (Not Found)
http://localhost/scripts.bundle.js Failed to load resource: the server responded with a status of 404 (Not Found)
http://localhost/vendor.bundle.js Failed to load resource: the server responded with a status of 404 (Not Found)
http://localhost/main.bundle.js Failed to load resource: the server responded with a status of 404 (Not Found)
http://localhost/styles.bundle.css Failed to load resource: the server responded with a status of 404 (Not Found)

I don't get any notifications when people comment on this thread so please email me at support@prerender.io if anyone has any questions. We're happy to help you get set up.

Some modifications, especially if removal with map, available on this fork : https://gist.github.com/regilero/dbddb029381dd3b781bc2906a5533620

is it possible to avoid prerendering some specific files? as I have a bower dependency that, when is prerendered, is adding an html tag that is causing the bot to take an incorrect Title name of the page...(ngnix config)

Hi, i have 504 issue.
already set nginx as written above & add metatag

As stated in the Yahoo! Help, the Yahoo bot identifies itself as:

Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Therefore adding Slurp to the list would handle this bot correctly.

How would one use uWSGI with this config? We have a django site with angular front-end, and we would like to cache static files.

I've trouble with whatsapp, seems that not full url is catch: I see log of nginx and when whatsapp's crawler explore my url this match only / instead of full path

Hi everyone,

I have an issue with my config, google crowl only my home page, even i suggest the other pages it's shows the home page content.

Also can i put a config for node.js and another in nginx in the same time ?

Many thanks in advance !

  server {
    listen 443;

server_name www.test.com test.com;
server_tokens off;


ssl on;
ssl_certificate /etc/nginx/ssl/test.com.chained.crt;
ssl_certificate_key /etc/nginx/ssl/test.com.key;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:30m;
ssl_session_timeout 1d;
ssl_session_tickets off;
ssl_dhparam /etc/nginx/cert/dhparam.pem;
ssl_ciphers 'ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS:EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA RC4 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS !RC4';

ssl_stapling on;
ssl_stapling_verify on;
ssl_trusted_certificate /etc/nginx/ssl/trustchain.crt;
resolver 8.8.8.8 8.8.4.4;

location / {
try_files $uri @prerender;
}

 location @prerender {
 proxy_set_header X-Prerender-Token xxxxxxxxxxxxxxxxxx;
 proxy_set_header X-Real-IP $remote_addr;
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 proxy_set_header X-Forwarded-Proto $scheme;
 add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
 proxy_http_version 1.1;
 proxy_set_header Upgrade $http_upgrade;
 proxy_set_header Connection "upgrade";
 #proxy_set_header Host $host;
 server_tokens off;

set $prerender 0;
 if ($http_user_agent ~* "googlebot|baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link 
 preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator") {
 set $prerender 1;
 }

 if ($args ~ "_escaped_fragment_") {
  set $prerender 1;
 }
 if ($http_user_agent ~ "Prerender") {
   set $prerender 0;
 }
 if ($uri ~ "\.
      (js|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|
       swf|dat|dmg|iso|flv|m4v|torrent|ttf|woff|svg|eot)") {
   set $prerender 0;
 }
   #resolve using Google's DNS server to force DNS resolution and prevent caching of IPs
    resolver 8.8.8.8;

   if ($prerender = 1) {
  #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
  set $prerender "service.prerender.io";
   rewrite .* /https://$host$request_uri? break;
   proxy_pass http://$prerender;
   }
   if ($prerender = 0) {
     proxy_pass http://127.0.0.1:3000;

   }
 }

 }

 server {
   listen  80;
   server_name    www.test.com test.com;
   return         301 https://www.test.cmo$request_uri;
   if ($host = 'test.com') {
   return 301 https://www.test.com$request_uri;
   }

}