Official prerender.io nginx.conf for nginx

nginx.conf

# Change YOUR_TOKEN to your prerender token
# Change example.com (server_name) to your website url
# Change /path/to/your/root to the correct value

server {
    listen 80;
    server_name example.com;
 
    root   /path/to/your/root;
    index  index.html;

    location / {
        try_files $uri @prerender;
    }
 
    location @prerender {
        proxy_set_header X-Prerender-Token YOUR_TOKEN;
        
        set $prerender 0;
        if ($http_user_agent ~* "googlebot|bingbot|yandex|baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest\/0\.|pinterestbot|slackbot|vkShare|W3C_Validator|whatsapp") {
            set $prerender 1;
        }
        if ($args ~ "_escaped_fragment_") {
            set $prerender 1;
        }
        if ($http_user_agent ~ "Prerender") {
            set $prerender 0;
        }
        if ($uri ~* "\.(js|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent|ttf|woff|svg|eot)") {
            set $prerender 0;
        }
        
        #resolve using Google's DNS server to force DNS resolution and prevent caching of IPs
        resolver 8.8.8.8;
 
        if ($prerender = 1) {
            
            #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
            set $prerender "service.prerender.io";
            rewrite .* /$scheme://$host$request_uri? break;
            proxy_pass http://$prerender;
        }
        if ($prerender = 0) {
            rewrite .* /index.html break;
        }
    }
}

condition

    if ($uri ~ ".js|.css|.xml|.less|.png|.jpg|.jpeg|.gif|.pdf|.doc|.txt|.ico|.rss|.zip|.mp3|.rar|.exe|.wmv|.doc|.avi|.ppt|.mpg|.mpeg|.tif|.wav|.mov|.psd|.ai|.xls|.mp4|.m4a|.swf|.dat|.dmg|.iso|.flv|.m4v|.torrent") {

incorrect because

    $uri ~ ".ico"

mean "string consist of one any character THEN 'ico'" that matched the URL /icon/ and it is wrong

I suggest this one:

    if ($uri ~ "\.(js|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent)$") {

that means "string is finished dot and one of strings 'js' OR 'css' etc"

You're right, thanks. I'll update that.

How about html5mode of angularjs? Is prerender supports html5mode, would it be enough to add googlebot to the if statement?

        if ($http_user_agent ~* "baiduspider|twitterbot|facebookexternalhit|googlebot") {
            set $prerender 1;
        }

Google's recommended way is to use escaped_fragment. In theory, yes we should add it to the user agent check, but in reality we wouldn't want Google to request the same URL using another user agent and then think the user is cloaking because the content is different. So we try to stay on the safe side.

Confused... why isn't googlebot listed?

@sansmischevia : because googlebot uses the escaped_fragment, which is listed.

If users are using HTML5 pushState, surely Google will request the URL's without escaped_fragment ?

Search bots look for --> <meta name="fragment" content="!" /> in the head tag. Read https://developers.google.com/webmasters/ajax-crawling/docs/specification
Pages without hash fragments It may be impossible or undesirable for some pages....

I'm not too sure on

  rewrite .* /$scheme://example.com$request_uri? break;

Am I only replacing the example.com ? Could we not use the $server_name variable for this?

  rewrite .* /$scheme://$server_name$request_uri? break;

If our "location /" block is where we typically specify a reverse proxy with proxy_pass, should I assume we would essentially add that to the "if (@prerender = 0)" section?

@sentient good idea :)

@akoumjian yes, in that case you would do your own proxy_pass "if @prerender = 0"

I added this since we were seeing issues where nginx was caching IPs and hitting servers that might have been taken out of our load balancer rotation:

#resolve using Google's DNS server
resolver 8.8.8.8;

#setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
set $prerender "service.prerender.io";
rewrite .* /$scheme://$server_name$request_uri? break;
proxy_pass http://$prerender;

With nginx 1.4.6 (ubuntu) I get an error when I try to restart nginx service or reload the configuration.

nginx: [emerg] "resolver" directive is not allowed here in /etc/nginx/sites-enabled/nginx.conf

I had to move resolver out of the conditional for it to work but this isn't ideal. Any ideas?

I'm wondering if $host is not a better choice than $server_name in

rewrite .* /$scheme://$server_name$request_uri? break;

There can be multiple $server_name values in a single server{...} but $host is the one in the HTTP requests (and falls back to $server_name anyway)

@thoop @patrickng I too have the same issue ('resolver' directive is not allowed here)

Sorry, I wish github would send notifications for comments on gists :(

@fergusg we used to have it set to $host but recently changed it to $server_name.

I'll look into a better solution for the resolver in the if-conditional.

Vkontakte social network uses this - "Mozilla/5.0 (compatible; vkShare; +http://vk.com/dev/Share)"
UserAgent for sharing functionality. Therefore you need to add 'vkshare' to detect list

pinterest|vkshare

See embeding docs. Sharing uses the same userAgent

If behind a load balancer, the $scheme var may not be set right - if the LB is doing SSL termination, the scheme on the machine behind the box may be http. Prerender service would then try to access http://, but get bounced to https://, which in my case did not work. I had to hard-code https:// in there.

Any update on @patrickng resolver issue? I took out the conditional as well.

@leorue can you try the new nginx config? I just updated it to move the resolver outside of the "if" statement.

I just changed $server_name back to $host. Hopefully that clears up any issues with the server name not being the actual url of your site.

Hi, do I have to install something on my webserver running nginx as frontend HTTP server, or do I simply need to add this snippet to my vhost .conf ?

You should just be able to add this snippet to your .conf file. Email me at todd@prerender.io if you're having any problems with it. Github doesn't send notifications on gists so I'll be able to help you more quickly over email.

http://wiki.nginx.org/IfIsEvil

Directive if has problems when used in location context, in some cases it doesn't do what you expect but something completely different instead. In some cases it even segfaults. It's generally a good idea to avoid it if possible.

The only 100% safe things which may be done inside if in location context are:

return ...;
rewrite ... last;
Anything else may possibly cause unpredictable behaviour, including potential SIGSEGV.

It is important to note that the behaviour of if is not inconsistent, given two identical requests it will not randomly fail on one and work on the other, with proper testing and understanding ifs can be used. The advice to use other directives where available still very much apply, though.

Google doesn't use ?escaped_fragment= for all of it's services. It might do for it's indexer but for instance when I use it from the Webmaster tools "Fetch as Google", it correctly renders it but the HTML it received was before rendering:

66.249.75.21 - - [08/Dec/2014:12:11:51 +0000] "GET /owner/ HTTP/1.1" 200 6292 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.75.37 - - [08/Dec/2014:12:11:51 +0000] "GET /owner/?_escaped_fragment_= HTTP/1.1" 200 9401 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

In fact, I am ALWAYS seeing two requests from Googlebot - One with and without the escaped fragment.
I'm seeing other services without the escaped fragment as well: Google Page Speed Insights, Google-StructuredDataTestingTool, Google Web Preview.

A few that I added myself: (tumblr|slackbot|xml-sitemaps|google-structureddatatestingtool). Definitely need tubmlr :) It's a shame we have to go with a whitelist approach, because there are so many services where my site is just invisible.

@intellix I guess that in the first request google find the and then do a second request with escaped_fragment. Before the first request it did not know that it needs to use escaped_fragment

@varuzhnikov if we could refactor to remove if statements, that would be ideal. Any idea on the best way to do that? We haven't had any problems with this configuration yet.

@intellix @csbenjamin is correct. The first request see's your tag and re-requests the page with the escaped fragment parameter. "Fetch as Google" does not follow the escaped fragment, and that is a known bug. If you are seeing services that do not send two requests (one with escaped fragment), then we should add them to the whitelist of user agents.

I use self hosted prerender and nginx server/angularjs html application but when try share contentn on facebook or linked I always got root page (index.html) /

nginx conf
set $prerender "85.10.211.83:4000";
rewrite .* /$scheme://$host$request_uri? break;
proxy_pass http://$prerender;

just for review:
http://85.10.211.83:4000/http://novi.bktvnews.com:3030/#!/
and single pages is same
http://85.10.211.83:4000/http://novi.bktvnews.com:3030/#!/grafika-iz-dalarne-na-putu-vas

in app.js
I have

$locationProvider.hashPrefix('!');

and in index.html template

we had a AWS nginx proxy setup and were having timeout issues - service.prerender.io could not be resolved (60: Operation timed out). We were able to get around this by using nginx upstream:

upstream pre-render {
   server service.prerender.io;
}

and the if in location block changed to:

if ($prerender = 1) {
   rewrite ^(.*)$ /https://$server_name..... break;
   proxy_pass http://pre-render;
}

I use the official server :"service.prerender.io" and it works, but when i try to use my own server ,it didn't. I have test it with http://myserver.com/http://www.google.com and it work with my server which i think it mean the server work. But when i replace "service.prerender.io" with "myserver.com" , it didn't work and got an error with 502 Bad Gateway . Anyone know why?

I use nginx like a reverse proxy and i would like to use prerender. I don't know how to adapt the nginx.conf especially this section:

location / {
try_files $uri @prerender;
}

I can't use the "try_files" instruction as i use "proxy_pass" instruction.

Do you know how to do ? Thank you

@creatorkuang it'd be hard to know without seeing your configuration. myserver.com needs to be running the open-source prerender node server and on the right port (by default it's 3000), and your nginx prerender middleware has to be configured to proxy there if the prerender variable = 1. E.g. proxy_pass http://myserver.com; But I've only tried the open-source server from localhost:3000.

@geolart I think some people above commented that you can do that by moving your proxy_pass line into the location @prerender block if the prerender variable = 0 (instead of rewrite .* /index.html break;). Let me know if it doesn't work for you though.

@rkulla Thanks to you it's work for me !
Thank you Rhulla !

When also using nginx as a reverse proxy via proxy_pass http://myupstream, I had an issue where if I do proxy_set_header X-Prerender-Token XXX in the location @prerender block, it reset my other proxy_set_header lines, causing 'http://myupstream' to be treated as a literal URL. However, things work fine if I redefine my proxy_set_header 'Host', etc in the same block as X-Prerender-Token -- either all in the location / or all in the location @prerender, but not divided.

Send me an email at support@prerender.io if anyone has any issues. I don't get notified when someone comments on this gist.

As per Facebook official documentation, you should also add the user agent 'facebot' to $http_user_agent

https://developers.facebook.com/docs/sharing/best-practices#crawl

I have monitored many requests that come with Googlebot and AdsBot-Google-Mobile as User-Agent, so I added it to my list.

There is an exhaustive of those User-Agents somewhere?

You might want to add 'svg' to the list of extensions not prerendered.

@evandhoffman we have the same issue when hosting on Heroku, which in turn seems to be using Amazon ELB. $scheme is always HTTP. As an alternative to hardcoding we decided to use $http_x_forwarded_proto instead of $scheme.

@MrGamer you don't want to add Googlebot (or any other crawlers that support the escaped fragment protocol) to the user agent list. You could get penalized for cloaking. You want Google to continue using the escaped fragment protocol

@ermakovich great idea!

Send me an email at support@prerender.io if anyone has any issues. I don't get notified when someone comments on this gist.

For nginx proxy_pass users: do two things.

1. Comment out: following two lines in server block

   root /path/to/your/root;

   index index.html;

2. In the last if block ("if ($prerender = 0) {....."):

replace #rewrite .* /index.html break; with your proxy_pass as shown below (my application is running on port 3009).

proxy_pass http://127.0.0.1:3009;

Does anyone have an example of something like this working for Meteor ?

@matfin see https://gist.github.com/blaind/12e53d5d9aa77c9fd841 for a meteor config. Had to use global proxy configs for both (can't put them in if's), but prerender.io seems to accept upgraded HTTP 1.1 too.

Please add redditbot to the list of user agents.

There's also trailing whitespace in 6 lines, might want to remove that.

@thoop I'm configuring prerender using nginx with nodejs backened server.Below is my nginx.conf file:-

server {
listen 80;
server_name _;

    location / {        
    try_files $uri @prerender;         
  }

   location @prerender {
    proxy_set_header X-Prerender-Token your_token;

    set $prerender 0;        

    if ($http_user_agent ~* "baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator") {
        set $prerender 1;
    }
    if ($args ~ "_escaped_fragment_") {           
        set $prerender 1;
    }
    if ($http_user_agent ~ "Prerender") {
        set $prerender 0;
    }
    if ($uri ~ "\.(js|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent|ttf|woff)") {
        set $prerender 0;
    }       

    #resolve using Google's DNS server to force DNS resolution and prevent caching of IPs
    resolver 8.8.8.8;
    if ($prerender = 1) {            
        #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
        set $prerender "localhost:8000";
        rewrite .* /$scheme://$host$request_uri? break;
        proxy_pass http://localhost:8000;
    }    

    if ($prerender = 0) {
        proxy_pass http://localhost:3000;

    }
}

    #error_page  404              /404.html;

    # redirect server error pages to the static page /50x.html
    #
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   html;
    }


}

But I am seeing JavaScript and not the static HTML.I had the checked Common Problem section,It has mentioned Try moving your Prerender middleware configuration higher in your file,I want to know what is
configuration code for nginx that i need to move to higher section of file.

I've posted a small project that demonstrates an implementation of how to use this without violating the "If Is Evil" principle:
https://github.com/thabofletcher/prerender.io-nginx

This allows any configuration for your app, including rewrites, try_files, and proxy_path, all outside if blocks. I believe it can be consolidated into a single gist/config, but I prefer the use of includes for readability.

This is my url :http://example.com/customer/#/index,I have already done prerender configuration.How do i access the prerender page of http://example.com/customer/#/index, what should be my url format

What about: http://googlewebmastercentral.blogspot.com.br/2015/10/deprecating-our-ajax-crawling-scheme.html

+1 for the question from @thiagofesta ! (what do you think @thoop ?)

@thiagofesta @asquel don't believe in this. It's not as nice as google tells. We shot ourselves in feets when we changed our static title and meta description into dynamic using AngularJS <meta name="description" content="{{ pageDescription }}" />. Now we have { pageDescription }} as page description in google results and Alexa ranks raised up half milion points :) So we want to start to use prerender.io soon.

Correction:
I have to appologize google and have to admit to our mistake. We used <meta name="fragment" content="!"> and in this case google expects that you provide snapshot. After removing <meta name="fragment" content="!"> site and meta description were rendered properly. Other thing is that prerendering is good for other services and crawlers.

So when you depend on google rendering only, you should remember to delete <meta name="fragment" content="!"> meta tag!

@thabofletcher I used a similar configuration but when I got rid of

if ($prerender = 0) {
  rewrite .* /index.html break;
}

and replaced it by a single try_files directive, Prerender started getting 404 errors while I got a 202 cURLing the same URL. I wrongly assumed this would work as long as $prerender != 1, but it doesn't when proxied through Prerender. Have you ever faced this issue with your configuration?

edit: the explanation is here, IfIsEvil (nginx docs)

I've made a fork and updated it with the following user agents baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest|slackbot|vkShare|Slack-ImgProxy|Slackbot-LinkExpanding|Site Analyzer|SiteAnalyzerBot|Viber|Whatsapp|Telegram|W3C_Validator.

These now also include stuff as Viber, WhatsApp, and Telegram which all try to parse the og tags

@ermakovich thanks for the hack for heroku!

If you are using the Heroku Buildpack: NGINX (https://elements.heroku.com/buildpacks/ryandotsmith/nginx-buildpack), just replace

rewrite .* /$scheme://$host$request_uri? break;

by

rewrite .* /$http_x_forwarded_proto://$host$request_uri? break;

Hope that will be useful !

I'm using this setup with an Angular application and works ok except to for the root (/) path that never goes throw prerender.
I think the problem is with try_files that found the index.html and serve it without going to the @prerender section.

someone have this issue ? how to resolve it ?

Been having issues with prerender dying and having my entire site disappear from Google when everything returns 504 errors (my fault for not having server alerts of any kind). I think in such a case where prerender just dies, we should have a fallback so google at least gets served your HTML/JS (they just won't for API calls to resolve, as prerender is there for).

Example error that'll bring your prerender service down

2015-12-10T17:00:31.669Z started phantom
events.js:85
      throw er; // Unhandled 'error' event
            ^
Error: channel closed
    at ChildProcess.target.send (child_process.js:414:26)
    at sendHelper (cluster.js:676:8)
    at RoundRobinHandle.handoff (cluster.js:198:3)
    at RoundRobinHandle.distribute (cluster.js:184:20)

Been playing around with how to get error_page working with a proxy and this is what works.:

server {
    # Add this line, to redirect 50x errors back to index.html
    error_page 500 502 503 504 =200 /index.html;

    location / {
        index     index.html;
        try_files $uri @prerender;
    }

    location @prerender {
        # Add this line inside @prerender location
        proxy_intercept_errors on;
    }
}

Why not just simply do this vs. the rewrite?

proxy_pass http://service.prerender.io/$scheme://$host$request_uri?;

Google has deprecated the escaped fragment scheme:
https://googlewebmastercentral.blogspot.nl/2015/10/deprecating-our-ajax-crawling-scheme.html

I think googlebot should be added to the whitelist, the escaped fragment scheme will probably still used by other bots like bing.

@marcelpanse Google is still not doing a good enough job when crawling javascript, which they even admitted in their last Google Webmaster Hangout. Google still follows the escaped fragment crawling protocol like they did before so Prerendering pages for Google is still the recommended way of doing things. Adding googlebot to the whitelist could get your website penalized for cloaking so it's not recommended. You should support the escaped fragment crawling protocol for Googlebot.

@thedug The rewrite is the way that it is because nginx caches IP addresses. If an IP address changes on our end, then your prerendering could stop working. The way we have it with the variable will prevent nginx from caching IP addresses to prevent this problem.

I don't get any notifications when people comment on this thread so please email me at todd@prerender.io if anyone has any questions.

You should move block of If -- set -- out of location to server definition level. Or you'll get a lot of problems with "if is Evil" in Nginx.

Like this:

server {
    listen 80;
    server_name example.com;

    root   /path/to/your/root;
    index  index.html;

        set $prerender 0;
        if ($http_user_agent ~* "baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator") {
            set $prerender 1;
        }
        if ($args ~ "_escaped_fragment_") {
            set $prerender 1;
        }
        if ($http_user_agent ~ "Prerender") {
            set $prerender 0;
        }
        if ($uri ~ "\.(js|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent|ttf|woff)") {
            set $prerender 0;
        }

    location / {
        try_files $uri @prerender;
    }

    location @prerender {
        #proxy_set_header X-Prerender-Token YOUR_TOKEN;

        #resolve using Google's DNS server to force DNS resolution and prevent caching of IPs
        resolver 8.8.8.8;

        if ($prerender = 1) {

            #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
            set $prerender "service.prerender.io";
            rewrite .* /$scheme://$host$request_uri? break;
            proxy_pass http://$prerender;
        }
        if ($prerender = 0) {
            rewrite .* /index.html break;
        }
    }
}

Hi,

If i am browsing any webpage so 1st time with ?_escaped_fragment redirecting 301 & from next time 200 OK . this is happening for all pages....

My nginx config file for prerender is...

location @prerender {
#proxy_set_header X-Prerender-Token YOUR_TOKEN;

    set $prerender 0;
    if ($http_user_agent ~* "baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator") {
        set $prerender 1;
    }
    if ($args ~ "_escaped_fragment_") {
        set $prerender 1;
    }
    if ($http_user_agent ~ "Prerender") {
        set $prerender 0;
    }
    if ($uri ~ "\.(js|ico|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent|ttf|woff)") {
        set $prerender 0;
    }

    #resolve using Google's DNS server to force DNS resolution and prevent caching of IPs
    resolver 8.8.8.8;

    if ($prerender = 1) {

        #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
        set $prerender "192.168.152.191:3000";
             rewrite .* /$scheme://$server_name$request_uri? break;
        proxy_pass http://$prerender;

    }
    if ($prerender = 0) {
        rewrite .* /index.html break;
    }
}

    location ~* \.(gif|jpg|png|ico)$ {
            expires 30d;
    }

}

@thoop, i found article where google disclosed they have stopped to support official for escaped_fragment (http://searchengineland.com/google-has-deprecated-their-ajax-crawling-scheme-233402) . So now what is the alternative for google crowling now ?

Thanks in a lot!

Hi Guys,

I'm trying to use this config, but keep returning 404 error. I'll appreciate any help in having a quick look.

server {
listen 443;
server_name localhost;

location / {
    root   /usr/share/nginx/html;
    index  index.html index.htm;
    try_files $uri @prerender;
}
location @prerender {
        proxy_intercept_errors on;
        proxy_set_header X-Prerender-Token our_token;

        set $prerender 0;
        if ($http_user_agent ~* "baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link     preview|showyoubot|outbrain|pinterest|slackbot|vkShare|Slack-ImgProxy|Slackbot-LinkExpanding|Site Analyzer|SiteAnalyzerBot|Viber|Whatsapp|Telegram|W3C_Validator") {
            set $prerender 1;
        }
        if ($args ~ "_escaped_fragment_") {
            set $prerender 1;
        }
        if ($http_user_agent ~ "Prerender") {
            set $prerender 0;
        }
        if ($uri ~ "\.(js|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent|ttf|woff)") {
            set $prerender 0;
        }

        #resolve using Google's DNS server to force DNS resolution and prevent caching of IPs
        resolver 8.8.8.8;

        #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
        set $prerender "service.prerender.io";
        rewrite .* /$scheme://$host$request_uri? break;
        proxy_pass http://$prerender;

        if ($prerender = 1) {

            #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
            set $prerender "service.prerender.io";
            rewrite .* /$scheme://$host$request_uri? break;
            proxy_pass http://$prerender;
        }
        if ($prerender = 0) {
            rewrite .* /index.html break;
        }
    }

}

lethaldose's solution above worked for me

Hey there,

i am trying to use prerender with nginx and our angular.js app.
The setting is as follows: i have docker running on the server. Nginx is used as a reverse proxy to forward the requests to the docker-container. Inside the container is another Nginx running, which serves the web app.
Now i do not know exactly where i should put the prerender config. Following i post both nginx-configs for clarity and maybe some more convenience:

This is the nginx-config which proxies to the docker-container

server {
        listen                  80;
        return 301              https://$host$request_uri;
}

### DEVELOP ###
server {
        listen                  443 ssl;
        ssl_certificate         /path/to/file
        ssl_certificate_key     /path/to/file
        server_name             develop.vidatio.mediacube.at;

        location / {
                proxy_pass http://localhost:6000;
        }

        location /.well-known {
                root /var/www/do.not.delete.vidatio.com;
        }
}

and this is the currently working nginx-config inside the container:

worker_processes  1;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;

    server {
        listen       80;
        server_name  localhost;

        location / {
            root /var/www/build;
            try_files $uri /statics/master.html;
        }

        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
}

i tried to get it to work like this but it doesnt:

http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;

    server {
        listen       80;
        server_name  localhost;

        root /var/www/build;

        location / {
            try_files $uri @prerender /statics/master.html;
        }

        location @prerender {
            proxy_set_header X-Prerender-Token --TOKEN--

            set $prerender 0;
            if ($http_user_agent ~* "baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator") {
                set $prerender 1;
            }
            if ($args ~ "_escaped_fragment_") {
                set $prerender 1;
            }
            if ($http_user_agent ~ "Prerender") {
                set $prerender 0;
            }
            if ($uri ~ "\.(js|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent|ttf|woff)") {
                set $prerender 0;
            }

            #resolve using Google's DNS server to force DNS resolution and prevent caching of IPs
            resolver 8.8.8.8;

            if ($prerender = 1) {

                #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
                set $prerender "service.prerender.io";
                rewrite .* /$scheme://$host$request_uri? break;
                proxy_pass http://$prerender;
            }
            if ($prerender = 0) {
                rewrite .* /statics/master.html break;
            }
        }

        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
}

Thanks in advance!

if other people like me tried to proxy_pass if $prerender = 0 then just remove the if condition as it acts like else
You won't get any nginx config errors

Hope it helps

Hello, guys!

I've started prerender server listening port 3000, in my production machine, used this config file.
So, I've replaced this part to localhost:3000:

if ($prerender = 1) {
    #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
    set $prerender "service.prerender.io";
    rewrite .* /$scheme://$host$request_uri? break;
    proxy_pass http://$prerender;
}

if ($prerender = 1) {            
    #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
    set $prerender "localhost:3000";
    rewrite .* /$scheme://$host$request_uri? break;
    proxy_pass http://$prerender;
}

When I requesting home page from external client host, I'm always getting 504 and no rendering:
Fragment from Nginx (tail -f /var/log/nginx/access.log):

213.21.36.130 - - [26/Apr/2016:08:08:46 +0000] "GET / HTTP/1.1" 504 0 "-" "twitterbot"

When I requesting home page directly from production server, everything is working:
curl -A 'twitterbot' http://localhost:3000/http://my-website.com

When I using set $prerender "service.prerender.io";, everyting is also working.

It seems, like I have problem with local prerenderer…

Also, I don't understand, how this configuration passes requests to meteor in case of non-bot.
As far, as I understood this fragment:

if ($prerender = 0) {
    rewrite .* /index.html break;
}

It loads static index.html…but we are using Meteor…

My initial configuration of nginx without prerender is taken from guide How To Deploy a Meteor.js Application on Ubuntu 14.04 with Nginx

Anyone having the same issue?
Thanks in advance!

@intellix can you shed some light on prerender dying on your site? Know what the root cause was? Find a resolution you're happy with?

What about php-fpm?

Yes, how could we make this play nicely with PHP?

prerender get response of 304,how can i fix it?

I hit a problem with a redirect loop. This can happen if you have a redirect from http to https. To detect this you can curl your site curl -A "facebookexternalhit" -v https://example.com/. If it returns 302 with Location: https://example.com/ then you have a redirect loop.

To fix this you need to update this row
rewrite .* /$scheme://$host$request_uri? break;
to
rewrite .* /https://$host$request_uri? break;

This situation can occur when you have a setup where you have one server/proxy handling the SSL termination and http->https redirect and a separate web server that is just an http server running nginx with prerender configuration.

How can this be mixed with open_file_cache?

Confused about why we need a long blacklist of extensions. Wouldn't we just want a whitelist instead? Such as .html, .htm, and no extension at all?

how would this work if your nginx has an upstream proxy?
e.g.

upstream bfa_clj {
   server 0.0.0.0:3000;
}
server {
    listen 80 default_server;
    listen [::]:80 default_server ipv6only=on;

    root /usr/share/nginx/html;
    index index.html index.htm;

    # Make site accessible from http://localhost/
    server_name localhost;

    location / {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_redirect off;
        proxy_pass http://bfa_clj;
        break;
    }
}

I don't get any notifications when people comment on this thread so please email me at support@prerender.io if anyone has any questions. We're happy to help you get set up.

I wonder why bingbot is not included in the list not like Stanback's one.
I tested with Fetch as Bingbot tool.
I also sent an e-mail, Thanks.

If chain of http_user_agent match but my url contains #! can I replace this with ?_escaped_fragment_=?

Please add WhatsApp to user agent chain

@thoop What do you think about the improvment propsed by @rogierslag ?
baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest|slackbot|vkShare|Slack-ImgProxy|Slackbot-LinkExpanding|Site Analyzer|SiteAnalyzerBot|Viber|Whatsapp|Telegram|W3C_Validator

Also i think that these:
Validator.nu|W3C-checklink|W3C-mobileOK|W3C_I18n-Checker|NING|FeedValidator|Jigsaw|W3C_Unicorn
should be added as well.
https://validator.w3.org/services#list

for people that trying to use this config on own prerender servers, you must just replace:

set $prerender "service.prerender.io";
to
set $prerender "127.0.0.1:3000";// one wird thing localhost didn't worked for me so it must be 127.0.0.1:<YOUR_PRERENDER_PORT>

@thoop
server {
listen 80;
listen [::]:80 default_server ipv6only=on;
return 301 https://$host$request_uri;
}

server {
    listen 443;
    server_name example.me;

    ssl on;
    # Use certificate and key provided by Let's Encrypt:
    ssl_certificate /etc/letsencrypt/live/example.me/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/example.me/privkey.pem;
    ssl_session_timeout 5m;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_prefer_server_ciphers on;
    ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';

    # Pass requests for / to localhost:8080:
    location / {
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-NginX-Proxy true;
            proxy_pass http://localhost:8080/;
            proxy_ssl_session_reuse off;
            proxy_set_header Host $http_host;
            proxy_cache_bypass $http_upgrade;
            proxy_redirect off;
    }

}
The above is my Nginx config file am using proxy with https. So how can I configure the above prerender and it's MEAN stack application. What should I give in the path to your root section either my app folder or public folder. Thanks

I had to add googlebot for google's webmaster tool's "Fetch as Google" to work.

https://www.google.com/webmasters/tools/googlebot-fetch-details

Hi
Anyone knows how to do it on nginx on docker and AWS?

This does not include Google +. As stated in the Google+ docs, the user agent contains:

Google (+https://developers.google.com/+/web/snippet/)

which should be added to the list of possible user agents.

Hi Guys,

I was running four node servers using

Configuring multiple applications on same port using nginx reverse proxy based on request urls

its working but the bundles files cannot loaded. I am stuck. Please Help

on Browser Console

Failed to load resource: the server responded with a status of 404 (Not Found)
http://localhost/inline.bundle.js Failed to load resource: the server responded with a status of 404 (Not Found)
http://localhost/scripts.bundle.js Failed to load resource: the server responded with a status of 404 (Not Found)
http://localhost/vendor.bundle.js Failed to load resource: the server responded with a status of 404 (Not Found)
http://localhost/main.bundle.js Failed to load resource: the server responded with a status of 404 (Not Found)
http://localhost/styles.bundle.css Failed to load resource: the server responded with a status of 404 (Not Found)

I don't get any notifications when people comment on this thread so please email me at support@prerender.io if anyone has any questions. We're happy to help you get set up.

Some modifications, especially if removal with map, available on this fork : https://gist.github.com/regilero/dbddb029381dd3b781bc2906a5533620

is it possible to avoid prerendering some specific files? as I have a bower dependency that, when is prerendered, is adding an html tag that is causing the bot to take an incorrect Title name of the page...(ngnix config)

Hi, i have 504 issue.
already set nginx as written above & add metatag

As stated in the Yahoo! Help, the Yahoo bot identifies itself as:

Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Therefore adding Slurp to the list would handle this bot correctly.

How would one use uWSGI with this config? We have a django site with angular front-end, and we would like to cache static files.

I've trouble with whatsapp, seems that not full url is catch: I see log of nginx and when whatsapp's crawler explore my url this match only / instead of full path

Hi everyone,

I have an issue with my config, google crowl only my home page, even i suggest the other pages it's shows the home page content.

Also can i put a config for node.js and another in nginx in the same time ?

Many thanks in advance !

  server {
    listen 443;

server_name www.test.com test.com;
server_tokens off;


ssl on;
ssl_certificate /etc/nginx/ssl/test.com.chained.crt;
ssl_certificate_key /etc/nginx/ssl/test.com.key;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:30m;
ssl_session_timeout 1d;
ssl_session_tickets off;
ssl_dhparam /etc/nginx/cert/dhparam.pem;
ssl_ciphers 'ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS:EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA RC4 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS !RC4';

ssl_stapling on;
ssl_stapling_verify on;
ssl_trusted_certificate /etc/nginx/ssl/trustchain.crt;
resolver 8.8.8.8 8.8.4.4;

location / {
try_files $uri @prerender;
}

 location @prerender {
 proxy_set_header X-Prerender-Token xxxxxxxxxxxxxxxxxx;
 proxy_set_header X-Real-IP $remote_addr;
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 proxy_set_header X-Forwarded-Proto $scheme;
 add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
 proxy_http_version 1.1;
 proxy_set_header Upgrade $http_upgrade;
 proxy_set_header Connection "upgrade";
 #proxy_set_header Host $host;
 server_tokens off;

set $prerender 0;
 if ($http_user_agent ~* "googlebot|baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link 
 preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator") {
 set $prerender 1;
 }

 if ($args ~ "_escaped_fragment_") {
  set $prerender 1;
 }
 if ($http_user_agent ~ "Prerender") {
   set $prerender 0;
 }
 if ($uri ~ "\.
      (js|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|
       swf|dat|dmg|iso|flv|m4v|torrent|ttf|woff|svg|eot)") {
   set $prerender 0;
 }
   #resolve using Google's DNS server to force DNS resolution and prevent caching of IPs
    resolver 8.8.8.8;

   if ($prerender = 1) {
  #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
  set $prerender "service.prerender.io";
   rewrite .* /https://$host$request_uri? break;
   proxy_pass http://$prerender;
   }
   if ($prerender = 0) {
     proxy_pass http://127.0.0.1:3000;

   }
 }

 }

 server {
   listen  80;
   server_name    www.test.com test.com;
   return         301 https://www.test.cmo$request_uri;
   if ($host = 'test.com') {
   return 301 https://www.test.com$request_uri;
   }

}

Seems like this guy has a working approach using map instead of IFs
https://gist.github.com/regilero/dbddb029381dd3b781bc2906a5533620

Code review failed, one time you put boolean in a variable, other time a string 😂

Sorry, n00b question.

If I have configured my nginx.conf file correctly, when I hit https://service.prerender.io/https://myurl (which displays correctly), will this request show up on https://prerender.io? Currently I see the message "We haven't seen a request with your Prerender token yet."

Thanks.

Hi All,
It doesn't work on my server: Centos 6.5
Below are my config on nginx: it returns 200 status

server {
  listen       80;
  server_name  {{ frontend_server_name }};

  charset UTF-8;
  access_log  /var/log/nginx/skill-test.frontend.access.log;
  error_log  /var/log/nginx/skill-test.frontend.error.log;
  root   {{ remote_stage_dir }}/frontend;
  index  index.html;

  location / {
    try_files $uri @prerender;
  }

  location @prerender {
      #proxy_set_header X-Prerender-Token YOUR_TOKEN;

      set $prerender 0;
      if ($http_user_agent ~* "baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator") {
          set $prerender 1;
      }
      if ($args ~ "_escaped_fragment_") {
          set $prerender 1;
      }
      if ($http_user_agent ~ "Prerender") {
          set $prerender 0;
      }
      if ($uri ~* "\.(js|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent|ttf|woff|svg|eot)") {
          set $prerender 0;
      }

      #resolve using Google's DNS server to force DNS resolution and prevent caching of IPs
      resolver 8.8.8.8;

      if ($prerender = 1) {
          #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
          set $prerender "127.0.0.1:3000";
          rewrite .* /$scheme://$host$request_uri? break;
          proxy_pass http://$prerender;
      }
      if ($prerender = 0) {
          rewrite .* /index.html break;
      }
  }

Is there a way to set YOUR_TOKEN as an environment variable?

@mdeolmos

You can do that using the lua-nginx-module

Hello, i see that SkypeUriPreview and other new crawlers are missing (eg: WhatsApp)
I found a decent list here: https://github.com/monperrus/crawler-user-agents/blob/master/crawler-user-agents.json

@sam-untapt Change your browser's user agent string (there are extensions for this) to look like a bot, then try the request. The pre-render shouldn't be hit if you're browsing normally.

Working w/ a hosting provider, we couldn't use the above nginx.conf as is. (Restricted to what we can add within a single location block.) Below is the alternative config for inside a location{} block -- not server{}.
Note Our case is not for an SPA, and therefore we removed the if around _escaped_fragment.

proxy_set_header X-Prerender-Token YOUR_TOKEN;  
# Using prerender or not as a variable forces DNS resolution since nginx caches IPs and doesn't play well with load balancing
set $prerender 0;

# Prerender if it's a search bot that doesn't support JavaScript AND the URL is SPA/JavsScript-rendered
if ($http_user_agent ~* "baiduspider|facebookexternalhit|googlebot|twitterbot|rogerbot|linkedinbot|embedly|bingbot|Baiduspider|Yahoo|YahooSeeker|quora link preview|showyoubot|outbrain|pinterest|applebot|slackbot|vkShare|W3C_Validator") {
  # Old Google-recommended URL parameter to flag SPA
  if ($args ~ "_escaped_fragment_") {
	set $prerender 1;
  }
  # A specific URL that you know is JS-rendered
  if ($uri ~* "^/api-reference") {
	set $prerender 1;
  }
}

# Don't prerender for the prerender bot's own user agent though
if ($http_user_agent ~ "Prerender") {
  set $prerender 0;
}
# Don't prerender media files (they don't contain JavaScript)
if ($uri ~ "\.(js|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent|ttf|woff)") {
  set $prerender 0;
}

set $prerenderprotocol "http";
set $prerenderservice "http://service.prerender.io";

if ($http_x_forwarded_proto = 'https') {
  set $prerenderprotocol "https";
  set $prerenderservice "https://service.prerender.io";
}

# Resolve using Google's DNS server to force DNS resolution and prevent caching of IPs
resolver 8.8.8.8;

# Do the proxy if the request met the tests
if ($prerender = 1) {
  rewrite .* /$prerenderprotocol://$host$request_uri? break;
  proxy_pass $prerenderservice;
}

I tried with the follow configuration and it didn't work.
Prerender.io page stays with the message "We haven't seen a request with your Prerender token yet."
Obviously I replaced YOUR_TOKEN with mine
Someone can help me ?

server {
    listen       80;
    server_name  localhost;

root /usr/share/nginx/html;
index index.html index.htm;

location / {
   try_files $uri $uri/ /index.html =404;
}

location @prerender {
    proxy_set_header X-Prerender-Token YOUR_TOKEN;
    
    set $prerender 0;
    if ($http_user_agent ~* "baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator") {
        set $prerender 1;
    }
    if ($args ~ "_escaped_fragment_") {
        set $prerender 1;
    }
    if ($http_user_agent ~ "Prerender") {
        set $prerender 0;
    }
    if ($uri ~* "\.(js|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent|ttf|woff|svg|eot)") {
        set $prerender 0;
    }
    
    #resolve using Google's DNS server to force DNS resolution and prevent caching of IPs
    resolver 8.8.8.8;

    if ($prerender = 1) {
        
        #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
        set $prerender "service.prerender.io";
        rewrite .* /$scheme://$host$request_uri? break;
        proxy_pass http://$prerender;
    }
    if ($prerender = 0) {
        rewrite .* /index.html break;
    }
}	
}

Hey, I just read ajax-crawling and the usage of escaped_fragment is deprecated: https://developers.google.com/webmasters/ajax-crawling/docs/specification

I guess
if ($args ~ "escaped_fragment") {
set $prerender 1;
}
could be removed from the gist then!?

Why try_files?

Using try_files the location @prerender will never be fired!
I was looking the following behaviour:

If any page is accessed from the browser, nginx serve it statically/locally. If that page is accessed from some crawler, it's served by the proxy.

The official snippet seems not to do that.

Am I crazy?

From "Deprecating our AJAX crawling scheme" (https://webmasters.googleblog.com/2017/12/rendering-ajax-crawling-pages.html)

Given these advances, in the second quarter of 2018, we'll be switching to rendering these pages on Google's side, rather than on requiring that sites do this themselves. In short, we'll no longer be using the AJAX crawling scheme.

So what do you think? Did you guys do some tests with "escaped_fragment" condition?

+1 to @jeerbl for pointing out that this config file is missing Google Plus. The Google+ Snippet Fetcher User Agent is:

Google (+https://developers.google.com/+/web/snippet/)

For my part I added it to my config file UserAgent regex as developers\.google\.com

Also, if you want to use the Structured Data Testing Tool, you will want to add Google-Structured-Data-Testing-Tool to the regex as well.

Final example:

if ($http_user_agent ~* "baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator|developers\.google\.com|Google-Structured-Data-Testing-Tool") {
  set $prerender 1;
 }

Please add redditbot and Discordbot to the list of user agents.

if we use proxy for caching requests can we use proxy_cache_bypass based on user agent?

The goal is that any refresh request made by prerender always takes latest version of the content. Any suggestions?

@ddereszewski

You may have already found a solution for this but I just ran into this too. Rather than cache based on user-agent (which didn't seem efficient as there are lots of possible UA strings), I use the same $prerender variable as the input to proxy_cache_bypass.

Edit:
Also added proxy_no_cache $prerender so that the response is not added to the cache in these cases either.

I tried adding this config and it didn't work right away because of my try_files:

location / { try_files $uri @prerender $uri/ =404; }

I ended up having to remove $uri/ =404 and it worked, but now non-existent pages returned index.html! I don't want this behavior, and narrowed it down to removing that pesky

if ($prerender = 0) { rewrite .* /index.html break; }

Why does this even exist here? Totally confusing.

I am getting Error: 2019/02/06 18:03:19 [error] 7#7: *17 service.prerender.io could not be resolved (110: Operation timed out), client: 10.0.101.66, server: , request: "GET / HTTP/1.1" with nginx as a sidecar in ECS. Please follow the nginx conf below.

Command: curl -vvv --header "User-Agent:googlebot" https://example.com

upstream backend {
  server example:3000;
}

map $http_upgrade $connection_upgrade {

  default upgrade;
  '' close;
}

server {

  listen 80;

  add_header X-Whom $hostname;
  proxy_buffering off;
  proxy_buffer_size 128k;
  proxy_buffers 4 256k;
  proxy_busy_buffers_size 256k;

location / {

    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade; #for websockets
    proxy_set_header Connection $connection_upgrade;
    proxy_set_header X-Forwarded-For $remote_addr;
    proxy_set_header Host $host;

    access_log /dev/stdout apm;
    error_log stderr notice;

    try_files $uri @prerender;
  }
  location @prerender {

    proxy_set_header X-Prerender-Token "TOKEN_VALUE";
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade; #for websockets
    proxy_set_header Connection $connection_upgrade;
    proxy_set_header X-Forwarded-For $remote_addr;
    proxy_set_header Host $host;
    
    access_log /dev/stdout apm;
    error_log stderr notice;

    set $prerender 0;
    if ($http_user_agent ~* "googlebot|bingbot|baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator|WhatsApp|Twitterbot|showyoubot|outbrain|vkShare|Slack-ImgProxy|Slackbot-LinkExpanding|Site Analyzer|SiteAnalyzerBot|Viber|Whatsapp|Telegram|W3C_Validator") {

      set $prerender 1;
    }


    resolver 8.8.8.8;

    if ($prerender = 1) {

      set $prerender "service.prerender.io";
      #rewrite .* /https://$host$request_uri? break;
      rewrite .* $host$request_uri? break;
      #proxy_pass http://$prerender;
      proxy_pass http://$prerender;
    }
    if ($prerender = 0) {

      proxy_pass http://backend;
    }

  }

}

location ~* /article/([a-z]|[A-Z]|[0-9]|[-]) {
rewrite ^/article/((([a-z]|[A-Z]|[0-9]|[-])+))?$ /article.html?$1 break;
}

how can render whit try_files inside on this route?, i want this rewrite for webpack url, but i can't access from prerender, any idea?

how come DuckDuckBot isn't in the list of crawlers?

doc appears twice in the extension regexp

For anybody having issues setting up their own pre-render server with NGINX and/or getting 502 errors.

Check your NGINX HTTPS and HTTPS logs for more information on the issue.

if "502 Bad Gateway" error throws on centos api url for api gateway proxy pass on NGINX, run following command to solve the issue:
sudo setsebool -P httpd_can_network_connect 1

If you have any issue with AWS loadbalancer https redirect you will need to force it to https

#rewrite .* /$scheme://$host$request_uri? break;
rewrite .* /https://$host$request_uri? break;

Will it work with cloudflare DNS?

I was using Prerender with Wordpress and HTTPS protocol and I had to consult Prerender.io to get it to work. I had to remove $http_user_agent ~ "Prerender" condition to be able to use try_files as can be seen in this gist https://gist.github.com/torava/324af0b11fafe6c5e1cc8e02121af608

"502 Bad Gateway" - check resolve dns 8.8.8.8 in your network. It was forbidden on my network and this did not work.

for my nginx config I was unable to include a try_files in an if statement:
It fails with "not allowed"

if ($prerender = 1) {
            
            #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
            set $prerender "service.prerender.io";
            rewrite .* /$scheme://$host$request_uri? break;
            proxy_pass http://$prerender;
        }
        if ($prerender = 0) {
            try_files $uri $uri/ /index.php?$query_string;
        }

I have excluded the if($prerender = 0) {...

Now I just do the try_files after the first if

if ($prerender = 1) {
            
            #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
            set $prerender "service.prerender.io";
            rewrite .* /$scheme://$host$request_uri? break;
            proxy_pass http://$prerender;
        }
        
        try_files $uri $uri/ /index.php?$query_string;

Is this fine?

Thank you for this @thoop! Similar to slackbot, I request you to please add discordbot to the list of bots - https://developers.whatismybrowser.com/useragents/parse/572332-discord-bot

I have this site: https://estimationpoker.de. It is written in Angular. Unfortunately all subsites can be crawled by google like /impressum or something except the root site. How is this possible? I am using the nginx config from here.

Hi all, I use prerender in my project for SEO. the nginx.conf is following:
`location / {
root /dist;
try_files $uri @prerender;
# try_files $uri $uri/ /index.html;
# index index.html index.htm;

    }

    location @prerender {
        # proxy_set_header X-Prerender-Token YOUR_TOKEN;
        
        set $prerender 0;
        if ($http_user_agent ~* "googlebot|bingbot|yandex|baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest\/0\.|pinterestbot|slackbot|vkShare|W3C_Validator|whatsapp") {
            set $prerender 1;
        }
        if ($args ~ "_escaped_fragment_") {
            set $prerender 1;
        }
        if ($http_user_agent ~ "Prerender") {
            set $prerender 0;
        }
        if ($uri ~* "\.(js|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent|ttf|woff|svg|eot)") {
            set $prerender 0;
        }
        
        #resolve using Google's DNS server to force DNS resolution and prevent caching of IPs
        resolver 8.8.8.8;

        if ($prerender = 1) {
            #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
            set $prerender "127.0.0.1:3000";
            rewrite .* /$scheme://$host$request_uri? break;
            proxy_pass http://$prerender;
            break;
        }
        try_files $uri $uri/ /index.html;
    }`

when I using the command: curl http://localhost:3000/render?url=http://localhost
The result is right. But when I using the command: curl http://localhost:3000/render?url=http://localhost/help
The result is 404. Does anyone know the reason? Please tell me. Thanks in advance!

Hi, can someone help me, I added GTmetrix to my http_user_agent, but it is not working. Google and others work though.

`
location @prerender {
proxy_set_header X-Prerender-Token XXXXXXXXX;

     set $prerender 0;
     if ($http_user_agent ~* "baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator|GTmetrix|screaming frog seo spider|screamingfrogseospider|chrome-lighthouse") {
         set $prerender 1;
     }
     if ($args ~ "_escaped_fragment_|prerender=1") {
         set $prerender 1;
     }
     if ($http_user_agent ~ "Prerender") {
         set $prerender 0;
     }

     if ($prerender = 1) {
         rewrite .* /?url=$scheme://$host$request_uri? break;
         proxy_pass http://prerender.freyagriculturalproducts.com;
     }

     if ($prerender = 0) {
         proxy_pass http://127.0.0.1:5080;
     }
 }

`