Prerender IIS Web.config

web.config

<?xml version="1.0"?>
<configuration>
    <system.webServer>
        <httpProtocol>
            <customHeaders>
                <add name="X-Prerender-Token" value="XXXXXXXXXXXXX" />
            </customHeaders>
        </httpProtocol>
        <rewrite>
            <rules>
                <!--# Only proxy the request to Prerender if it's a request for HTML-->
                <rule name="Prerender" stopProcessing="true">
                    <match url="^(?!.*?(\.js|\.css|\.xml|\.less|\.png|\.jpg|\.jpeg|\.gif|\.pdf|\.doc|\.txt|\.ico|\.rss|\.zip|\.mp3|\.rar|\.exe|\.wmv|\.doc|\.avi|\.ppt|\.mpg|\.mpeg|\.tif|\.wav|\.mov|\.psd|\.ai|\.xls|\.mp4|\.m4a|\.swf|\.dat|\.dmg|\.iso|\.flv|\.m4v|\.torrent))(.*)" ignoreCase="false" />
                    <conditions logicalGrouping="MatchAny">
                        <add input="{HTTP_USER_AGENT}" pattern="baiduspider|facebookexternalhit|twitterbot" />
                        <add input="{QUERY_STRING}" pattern="_escaped_fragment_" ignoreCase="false" />
                    </conditions>
                    <action type="Rewrite" url="http://service.prerender.io/http://yourwebsite.com/{R:2}" />
                </rule>
            </rules>
        </rewrite>
    </system.webServer>
</configuration>

@r3plica I tried out your example, and the notice is finally removed on the prerender site. I did have to put "http" in front of {HTTP_HOST}/{R:0} in order to make it work.

<action type="Rewrite" url="http://service.prerender.io/http://{HTTP_HOST}{REQUEST_URI}" appendQueryString="false" />

The changes were very minor and I don't really understand why the notice is gone now, but the previous code didn't.
Thanks!

Kudos to @r3plica and @Sniels for figuring this out. Took me a while to get it working but eventually got it to work. I think @blowsie needs to update his original web.config code snippet. Thanks guys!

I've got an angularjs site running on iis. Got the "You need to install your Prerender token..." to go away, but I get nothing but 504 errors under the crawl stats. This is not an azure site so i'm not sure the applcationHost.xdt configuration is applicable.

  <httpProtocol>
      <customHeaders>
        <add name="X-Prerender-Token" value="<your_token>" />
      </customHeaders>
    </httpProtocol>

    <rewrite>

      <rules>
        <!--# Only proxy the request to Prerender if it's a request for HTML-->


        <rule name="Prerender" stopProcessing="true">
          <match url="^(?!.*?(\.js|\.css|\.xml|\.less|\.png|\.jpg|\.jpeg|\.gif|\.pdf|\.doc|\.txt|\.ico|\.rss|\.zip|\.mp3|\.rar|\.exe|\.wmv|\.doc|\.avi|\.ppt|\.mpg|\.mpeg|\.tif|\.wav|\.mov|\.psd|\.ai|\.xls|\.mp4|\.m4a|\.swf|\.dat|\.dmg|\.iso|\.flv|\.m4v|\.torrent))(.*)" ignoreCase="false" />
          <conditions logicalGrouping="MatchAny">
            <add input="{HTTP_USER_AGENT}" pattern="baiduspider|facebookexternalhit|twitterbot|googlebot" />
            <add input="{QUERY_STRING}" pattern="(.*)_escaped_fragment_=(.*)" ignoreCase="false" />
          </conditions>
          <action type="Rewrite" url="http://service.prerender.io/http://{HTTP_HOST}{REQUEST_URI}"  appendQueryString="false" />
          <serverVariables>
            <set name="HTTP_X_PRERENDER_TOKEN" value="<your_token>" />
          </serverVariables>
        </rule>

        <rule name="Main Rule" stopProcessing="true">
          <match url=".*" />
          <conditions logicalGrouping="MatchAll">
            <add input="{REQUEST_FILENAME}" matchType="IsFile" negate="true" />
            <add input="{REQUEST_FILENAME}" matchType="IsDirectory" negate="true" />
          </conditions>
          <action type="Rewrite" url="/" />
        </rule>
      </rules>
    </rewrite>

This worked for us. Our rewrite is as follows:

    <rule name="Seo rewrite rule" stopProcessing="true">
      <serverVariables>
        <set name="HTTP_X_PRERENDER_TOKEN" value="[...your token...]" />
      </serverVariables>
      <conditions>
        <add input="{QUERY_STRING}" pattern="(.*)_escaped_fragment_=(.*)" />
      </conditions>
      <action type="Rewrite" url="http://service.prerender.io/http://{HTTP_HOST}{REQUEST_URI}" appendQueryString="false" />
    </rule>

As far as the "allowableServerVariables", we handled this through the IIS and clicking either the Default Web Site of the Site itself, then clicking the URL Rewrite module icon, then select View Server Variables, and add the HTTP_X_PRERENDER_TOKEN variable name to the list.

With this in place, we were now able to append the escaped_fragment querystring to our urls and see the rewritten results after coming back through the server. You should also be able to verify in your Account page on PreRender.io that the pages are being cached.

HTH,
Randy

it worked for me just fine, but when i add this:

from Snail answer, it stop working...

@JT00 did you end up getting it to work? I too am not on Azure.

Incase anyone was also not using Azure and hitting constant 404 errors after setting everything up as per the suggestions above, I was able to get it working by installing the IIS ARR Module, then going into the IIS Rewrite rules for my site in IIS itself and when attempting to add a rule of type "Reverse Proxy Rule", it asked me if I wanted to enable reverse proxy, clicked yes. You dont need to actually add a rule, as now the ARR reverse proxy feature is installed.

More info on this solution here:
http://stackoverflow.com/questions/8281826/url-rewrite-causing-404
http://weblogs.asp.net/owscott/creating-a-reverse-proxy-with-url-rewrite-for-iis

For those of you who struggled for a long time to get the reverse proxy element of this working and still couldn't figure it out, I opted for the "poor man's solution" using a web request to retrieve the compiled HTML from the hosted prerender.io service

n.b. You CAN test this method locally and it will work :)

Web config rewrite rule:

<rewrite>
      <rules>
        <clear />
        <rule name="Prerender" enabled="true" stopProcessing="true">
          <match url="^(?!.*?(\.js|\.css|\.xml|\.html|\.less|\.png|\.jpg|\.jpeg|\.gif|\.pdf|\.doc|\.txt|\.ico|\.rss|\.zip|\.mp3|\.rar|\.exe|\.wmv|\.doc|\.avi|\.ppt|\.mpg|\.mpeg|\.tif|\.wav|\.mov|\.psd|\.ai|\.xls|\.mp4|\.m4a|\.swf|\.dat|\.dmg|\.iso|\.flv|\.woff|\.ttf|\.m4v|\.svg|\.torrent))(.*)" ignoreCase="false" />
          <conditions logicalGrouping="MatchAll">
            <add input="{HTTP_USER_AGENT}" pattern="baiduspider|facebookexternalhit|twitterbot|googlebot" />
            <add input="{QUERY_STRING}" pattern="(.*)_escaped_fragment_=(.*)" ignoreCase="false" />
          </conditions>
          <!-- Rewrite the URL to a controller in our application that will do all of the prerendering legwork -->
          <action type="Rewrite" url="/Home/Snapshot?path={REQUEST_URI}" appendQueryString="false" />
        </rule>
        <rule name="angularjs" stopProcessing="true">
          <match url=".*" />
          <conditions logicalGrouping="MatchAll">
            <add input="{REQUEST_FILENAME}" matchType="IsFile" negate="true" />
            <add input="{REQUEST_FILENAME}" matchType="IsDirectory" negate="true" />
            <add input="{REQUEST_URI}" pattern="^/(api)" negate="true" />
          </conditions>
          <action type="Rewrite" url="/" />
        </rule>
    </rules>
</rewrite>

Then in a controller of your choice (I opted for Home), create a new Snapshot action as below:

public ActionResult Snapshot(string path)
{
    string fullUrl = default(string);
    string host = "https://example.com"; // CHANGE THIS or use Request.Url.Host
    fullUrl = string.Format("{0}{1}", host, path);

    string prerenderIoUrl = string.Format("{0}{1}", "http://service.prerender.io/", fullUrl);

    string prerenderIoApiKey = "[YOUR TOKEN HERE]";

    var request = WebRequest.Create(prerenderIoUrl);
    request.Headers.Add("X-Prerender-Token", prerenderIoApiKey);

    var response = request.GetResponse();
    var responseStream = response.GetResponseStream();
    var reader = new StreamReader(responseStream);
    string html = reader.ReadToEnd();

    reader.Close();
    response.Close();
    return Content(html);            
}

Ensure you have put your Prerender token and host values into the code above.

Test by changing your user agent in Chrome dev tools to googlebot and appending ?_escaped_fragment_= onto the URL of your site. When you right click and view "Page Source", it should have fully rendered your HTML between your ng-view tags. Also you can use Fiddler to see the outgoing request to the service.prerender.io url.

Have verified that this method is hitting my Prerender.io account and new pages are cached and previously cached pages are hit.

If anyone ever encounters this issue, I've tried the stuff above and got it working after many hours by mixing some things up.

1. In your Web App root directory (/site/wwwroot), add or update your web.config file with this:

<configuration>
  <system.webServer>     
    <rewrite>     
        <rules>
            <rule name="Prerender" stopProcessing="true">
                <serverVariables>
                    <set name="HTTP_X_PRERENDER_TOKEN" value="**YOUR TOKEN**" />
                </serverVariables>
                <match url="^(?!.*?(\.js|\.css|\.xml|\.less|\.png|\.jpg|\.jpeg|\.gif|\.pdf|\.doc|\.txt|\.ico|\.rss|\.zip|\.mp3|\.rar|\.exe|\.wmv|\.doc|\.avi|\.ppt|\.mpg|\.mpeg|\.tif|\.wav|\.mov|\.psd|\.ai|\.xls|\.mp4|\.m4a|\.swf|\.dat|\.dmg|\.iso|\.flv|\.m4v|\.torrent))(.*)" ignoreCase="false" />
                <conditions logicalGrouping="MatchAny">
                    <add input="{HTTP_USER_AGENT}" pattern="bingbot|googlebot|baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator" />
                </conditions>
                <action type="Rewrite" url="https://service.prerender.io/https://{HTTP_HOST}{REQUEST_URI}" appendQueryString="false" />
            </rule>
        </rules>
    </rewrite>

  </system.webServer>
</configuration>

2. In /site/ create a file named applicationHost.xdt and add this content:

<?xml version="1.0"?>
<configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
    <system.webServer>
        <proxy xdt:Transform="InsertIfMissing" enabled="true" preserveHostHeader="false" reverseRewriteHostInResponseHeaders="false" />
        <rewrite>
            <allowedServerVariables>
                <add name="HTTP_X_PRERENDER_TOKEN" xdt:Transform="InsertIfMissing" />
            </allowedServerVariables>
        </rewrite>
    </system.webServer>
</configuration>

**3. Restart your Web App service (click 'Stop' and 'Start', not only 'Restart').

4. Test with a tool like: https://technicalseo.com/tools/fetch-render/

Hope this helps someone.

This worked for us. Our rewrite is as follows:

        <rule name="Seo rewrite rule" stopProcessing="true">
          <serverVariables>
            <set name="HTTP_X_PRERENDER_TOKEN" value="[...your token...]" />
          </serverVariables>
          <conditions>
            <add input="{QUERY_STRING}" pattern="(.*)_escaped_fragment_=(.*)" />
          </conditions>
          <action type="Rewrite" url="http://service.prerender.io/http://{HTTP_HOST}{REQUEST_URI}" appendQueryString="false" />
        </rule>

As far as the "allowableServerVariables", we handled this through the IIS and clicking either the Default Web Site of the Site itself, then clicking the URL Rewrite module icon, then select View Server Variables, and add the HTTP_X_PRERENDER_TOKEN variable name to the list.

With this in place, we were now able to append the escaped_fragment querystring to our urls and see the rewritten results after coming back through the server. You should also be able to verify in your Account page on PreRender.io that the pages are being cached.

HTH,
Randy

Hi!!

I spend a lot days for the solution, but i add the variable and works fine!!!

Solution for me :

Only exist the web.config with the configuration (Htm5 redirect and this) , i dont need it applicationHos.xdt.:

---

<rewrite>
          <rules>
              <!--# Only proxy the request to Prerender if it's a request for HTML-->
              <rule name="SEO" stopProcessing="true">
              <match url="^(?!.*?(\.js|\.css|\.xml|\.html|\.less|\.png|\.jpg|\.jpeg|\.gif|\.pdf|\.doc|\.txt|\.ico|\.rss|\.zip|\.mp3|\.rar|\.exe|\.wmv|\.doc|\.avi|\.ppt|\.mpg|\.mpeg|\.tif|\.wav|\.mov|\.psd|\.ai|\.xls|\.mp4|\.m4a|\.swf|\.dat|\.dmg|\.iso|\.flv|\.woff|\.ttf|\.m4v|\.svg|\.torrent))(.*)" ignoreCase="false" />
              <conditions logicalGrouping="MatchAny">
                  <add input="{HTTP_USER_AGENT}" pattern="facebookexternalhit|twitterbot|googlebot|whatsapp" />
                  <add input="{QUERY_STRING}" pattern="(.*)_escaped_fragment_=(.*)" ignoreCase="false" />
              </conditions>
                  <action type="Rewrite" url="http://service.prerender.io/http://{HTTP_HOST}{REQUEST_URI}" appendQueryString="false" /> 
              <serverVariables>
                <set name="HTTP_X_PRERENDER_TOKEN" value="yourtoken" /> 
              </serverVariables>
              </rule>
              <rule name="Redirect To Index" stopProcessing="true">
                  <match url=".*" />
                  <conditions>
                      <add input="{REQUEST_FILENAME}" matchType="IsFile" negate="true" />
                      <add input="{REQUEST_FILENAME}" matchType="IsDirectory" negate="true" />
                  </conditions>
                  <action type="Rewrite" url="/" />
              </rule> 
          </rules>
      </rewrite>

---

The next step was as you indicates ; add into Rewrite module icon, then select View Server Variables, the new variable name "HTTP_X_PRERENDER_TOKEN" and works !! amazing =)!!

Thnks for your time.

I want to share my test options ;

1.-https://technicalseo.com/tools/fetch-render/
2.-https://developers.facebook.com/tools/debug

Any ask or something im here