Skip to content

Instantly share code, notes, and snippets.

@thoop

thoop/.htaccess

Last active January 27, 2024 14:07
Show Gist options
  • Save thoop/8072354 to your computer and use it in GitHub Desktop.
Save thoop/8072354 to your computer and use it in GitHub Desktop.
Download ZIP
Official prerender.io .htaccess for Apache.
Raw
.htaccess
# Change YOUR_TOKEN to your prerender token
# Change http://example.com (at the end of the last RewriteRule) to your website url
<IfModule mod_headers.c>
RequestHeader set X-Prerender-Token "YOUR_TOKEN"
</IfModule>
<IfModule mod_rewrite.c>
RewriteEngine On
<IfModule mod_proxy_http.c>
RewriteCond %{HTTP_USER_AGENT} googlebot|bingbot|yandex|baiduspider|facebookexternalhit|twitterbot|rogerbot|linkedinbot|embedly|quora\ link\ preview|showyoubot|outbrain|pinterest\/0\.|pinterestbot|slackbot|vkShare|W3C_Validator [NC,OR]
RewriteCond %{QUERY_STRING} _escaped_fragment_
# Only proxy the request to Prerender if it's a request for HTML
RewriteRule ^(?!.*?(\.js|\.css|\.xml|\.less|\.png|\.jpg|\.jpeg|\.gif|\.pdf|\.doc|\.txt|\.ico|\.rss|\.zip|\.mp3|\.rar|\.exe|\.wmv|\.doc|\.avi|\.ppt|\.mpg|\.mpeg|\.tif|\.wav|\.mov|\.psd|\.ai|\.xls|\.mp4|\.m4a|\.swf|\.dat|\.dmg|\.iso|\.flv|\.m4v|\.torrent|\.ttf|\.woff|\.svg))(.*) http://service.prerender.io/http://example.com/$2 [P,L]
</IfModule>
</IfModule>
Copy link

ghost commented Aug 17, 2016

@simonpeters I'm having the same problem with the canonical URL appearing below a facebook post.
Have you had any luck with getting it to show the fetched URL?

@viking2917
Copy link

With google having deprecated the escaped_fragment syntax (https://webmasters.googleblog.com/2015/10/deprecating-our-ajax-crawling-scheme.html), anecdotally in my apache logs I see a lot of requests from Google coming without the escaped_fragment parameter. I added |googlebot| to my .htaccess for redirection to pre-render and my cache hits went way up. Anyone else have similar experience? Is there any particular reason NOT to have googlebot explicitly listed?

@mustivar
Copy link

Hi I have an application developed with spring boot + angularjs
I serve it through Apache with ProxyPass
With suggested rewrite rule I can redirect it to prerender.io but unfortunately angular views cannot be rendered.
In rendered body all I can see is top level divs. It seems views for that divs cannot be loaded. (I use ui-router for views)
Also I cannot see any script and .js includes in rendered page(I think that is how prerender does that)

I tried every solution that is suggested here and on web but could not succeed to make it work.

Do you have any suggestions for me? Thanks.

@russellwark
Copy link

russellwark commented Aug 31, 2016 •
edited

I'm having some issues with my .htaccess file for a site I've been working on - basically, it's a WordPress site with an Angular theme, hence the need for Prerender.io. What's happening is that the site is throwing a 404 every time the page is accessed, yet the page still displays and can be refreshed. Since it's throwing a 404, Prerender.io isn't picking it up for caching. This is only in HTML5 'pretty URLs' mode - it doesn't throw a 404 if you access the page with the hashbang.

Here's my .htaccess file - any suggestions would be very much appreciated.

<IfModule mod_deflate.c>
    <IfModule mod_headers.c>
        Header append Vary User-Agent env=!dont-vary
    </IfModule>
        AddOutputFilterByType DEFLATE text/css text/x-component application/x-javascript application/javascript text/javascript text/x-js text/html text/richtext image/svg+xml text/plain text/xsd text/xsl text/xml image/x-icon application/json
    <IfModule mod_mime.c>
        AddOutputFilter DEFLATE js css htm html xml
    </IfModule>
</IfModule>

Options -MultiViews

RequestHeader set X-Prerender-Token "{prerender_token}"

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} baiduspider|facebookexternalhit|twitterbot|rogerbot|linkedinbot|embedly|bingbot|Baiduspider|Yahoo|YahooSeeker|quora\ link\ preview|showyoubot|outbrain|pinterest|applebot [NC,OR]
RewriteCond %{QUERY_STRING} _escaped_fragment_

RewriteRule ^(?!.*?(\.js|\.css|\.xml|\.less|\.png|\.jpg|\.jpeg|\.gif|\.pdf|\.doc|\.txt|\.ico|\.rss|\.zip|\.mp3|\.rar|\.exe|\.wmv|\.doc|\.avi|\.ppt|\.mpg|\.mpeg|\.tif|\.wav|\.mov|\.psd|\.ai|\.xls|\.mp4|\.m4a|\.swf|\.dat|\.dmg|\.iso|\.flv|\.m4v|\.torrent|\.ttf|\.woff))(.*) http://service.prerender.io/http://%{HTTP_HOST}/$2 [P,L]
</IfModule>

RewriteEngine On  
RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} -f [OR]
RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} -d
RewriteRule ^ - [L]

RewriteRule ^ /index.php [L]

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d

RewriteCond %{HTTP_USER_AGENT} libwww-perl.* 
RewriteRule .* ? [F,L]

</IfModule>

<IfModule mod_expires.c>
ExpiresActive On
ExpiresByType image/jpg "access 1 year"
ExpiresByType image/jpeg "access 1 year"
ExpiresByType image/gif "access 1 year"
ExpiresByType image/png "access 1 year"
ExpiresByType text/css "access 1 month"
ExpiresByType text/html "access 1 month"
ExpiresByType application/pdf "access 1 month"
ExpiresByType text/x-javascript "access 1 month"
ExpiresByType application/x-shockwave-flash "access 1 month"
ExpiresByType image/x-icon "access 1 year"
ExpiresDefault "access 1 month"
</IfModule>

@maa42
Copy link

maa42 commented Sep 5, 2016

Thanks everyone for posting on this forum. I'm on AngularJS and apache, #! URLs. I noticed today that google was not crawling my site. I made the following changes: Per other comments, converted the #! RewriteCond %{QUERY_STRING} escaped_fragment=(%2F|/)(.). I also added the google bots to the list being the test searches did not work till I added the bots. Finally, I'm https so the url definition at the end of the file has to be HTTPS! Good luck everyone

DirectoryIndex index.html

RequestHeader set X-Prerender-Token "Your Token"

RewriteEngine On RewriteCond %{HTTP_USER_AGENT} baiduspider|googlebot|googlebot-mobile|bingbot|facebookexternalhit|twitterbot|rogerbot|linkedinbot|embedly|quora\ link\ preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator [NC,OR] RewriteCond %{QUERY_STRING} _escaped_fragment_=(\%2F|/)_(._) 
# Only proxy the request to Prerender if it's a request for HTML
RewriteRule ^(?!.*?(\.js|\.css|\.xml|\.less|\.png|\.jpg|\.jpeg|\.gif|\.pdf|\.doc|\.txt|\.ico|\.rss|\.zip|\.mp3|\.rar|\.exe|\.wmv|\.doc|\.avi|\.ppt|\.mpg|\.mpeg|\.tif|\.wav|\.mov|\.psd|\.ai|\.xls|\.mp4|\.m4a|\.swf|\.dat|\.dmg|\.iso|\.flv|\.m4v|\.torrent|\.ttf|\.woff))(.*) http://service.prerender.io/https://www.leaguescience.com/$2 [P,L]

@thoop
Copy link
Author

thoop commented Oct 11, 2016

I don't get notified when someone comments on this gist. Send us an email at support@prerender.io if anyone has any questions or issues.

@Martin-Eckleben
Copy link

THX @benceg this was what i was looking for for two days!

Anyone suffering with Angular/Worpress/prerender.io/.htaccess combos look at his comment :)

@Dizzyspb
Copy link

Dizzyspb commented Nov 3, 2016 •
edited

.htaccess

RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} -f [OR]
RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} -d
# Go to it as is
RewriteRule ^ - [L]

  RewriteCond %{REQUEST_URI} ^.*/$
  RewriteCond %{REQUEST_URI} !^/$
  RewriteRule ^(.*)/$ $1 [R,QSA,L]      

RequestHeader set X-Prerender-Token "%MYTOKEN%"

RewriteCond %{HTTP_USER_AGENT} baiduspider|facebookexternalhit|twitterbot|rogerbot|linkedinbot|embedly|quora\ link\ preview|showyoubot|outbrain|pinterest [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Yandex(Bot|Images|Video|Media) [NC,OR]
RewriteCond %{QUERY_STRING} _escaped_fragment_

RewriteRule ^(.*)$ http://service.prerender.io/http://%{HTTP_HOST}$1 [P,L]

# Accept everything on index.html
RewriteRule ^ /index.html`

in index.html

But robots can't see my site.
On prerender.io i have 67 cached pages.

Could you please help me.

@varunkumarg1988
Copy link

guys need help here,

Where do I need to place .htaccess file ?

I am using hashbang(#!) URL methodology and the steps I followed are as follows. Please suggest if there needs to be any changes

  1. added in app.js file, $locationProvider.hashPrefix('!');
  2. enabled 3 modules on apache server (mod_headers.c, mod_rewrite.c, mod_proxy_http.c)
  3. created an .htaccess file

.htaccess file

RequestHeader set X-Prerender-Token "XXXXXXXXXXXXXXXXX" RewriteEngine On 
<IfModule mod_proxy_http.so>
    RewriteCond %{HTTP_USER_AGENT} baiduspider|facebookexternalhit|twitterbot|rogerbot|linkedinbot|embedly|quora\ link\ preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator [NC,OR]
    RewriteCond %{QUERY_STRING} _escaped_fragment_
    
    # Only proxy the request to Prerender if it's a request for HTML
    RewriteRule ^(?!.*?(\.js|\.css|\.xml|\.less|\.png|\.jpg|\.jpeg|\.gif|\.pdf|\.doc|\.txt|\.ico|\.rss|\.zip|\.mp3|\.rar|\.exe|\.wmv|\.doc|\.avi|\.ppt|\.mpg|\.mpeg|\.tif|\.wav|\.mov|\.psd|\.ai|\.xls|\.mp4|\.m4a|\.swf|\.dat|\.dmg|\.iso|\.flv|\.m4v|\.torrent|\.ttf|\.woff))(.*) http://service.prerender.io/http://localhost/$2 [P,L]
</IfModule>

Now, the question is where do I need to place the .htaccess file and what other changes required in .htaccess configuartion. open for any suggestions.

Thank you

@avasbrewer
Copy link

This forum helps me in my web design china base site. It still need some work.

@iit2011081
Copy link

I am using zf2 and have apache installed on server. I have added the following lines in my .htaccess file.

RewriteEngine On 
<IfModule mod_proxy_http.c>
    RewriteCond %{HTTP_USER_AGENT} baiduspider|facebookexternalhit|twitterbot|rogerbot|linkedinbot|embedly|quora\ link\ preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator [NC,OR]
    RewriteCond %{QUERY_STRING} _escaped_fragment_
    
    # Only proxy the request to Prerender if it's a request for HTML
    RewriteRule ^(?!.*?(\.js|\.css|\.xml|\.less|\.png|\.jpg|\.jpeg|\.gif|\.pdf|\.doc|\.txt|\.ico|\.rss|\.zip|\.mp3|\.rar|\.exe|\.wmv|\.doc|\.avi|\.ppt|\.mpg|\.mpeg|\.tif|\.wav|\.mov|\.psd|\.ai|\.xls|\.mp4|\.m4a|\.swf|\.dat|\.dmg|\.iso|\.flv|\.m4v|\.torrent|\.ttf|\.woff))(.*) http://service.prerender.io/%{REQUEST_SCHEME}://%{HTTP_HOST}/$2 [P,L]
</IfModule>

But still it is not redirecting my url to corresponding prerender url. Even I have changed the User Agent to google bot and added the escaped_fragment as query paramter. Am I testing correctly or there is something wrong with this htaccess code ?

@iit2011081
Copy link

How to pass username and password in htaccess if I am using basic auth

@greenfm
Copy link

greenfm commented Dec 11, 2016

Is there a way to have the page cached when a user visits a page? or should this be automatic and that I'm doing it wrong?

@SherifElfadaly
Copy link

It doesn't work when sharing in whatsapp even when adding the Whatsapp user agent.

<IfModule mod_headers.c>
    RequestHeader set X-Prerender-Token "%MYTOKEN%"
</IfModule>

<IfModule mod_rewrite.c>

  RewriteEngine On
    # If requested resource exists as a file or directory
      # (REQUEST_FILENAME is only relative in virtualhost context, so not usable)
        RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} -f [OR]
        RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} -d
      # Only exception is /index.htm, /index.html
        RewriteCond %{REQUEST_URI} !/index\.html?
      # Go to it as is
        RewriteRule ^ - [L]
    # If non existent
      # If path ends with / and is not just a single /, redirect to without the trailing /
        RewriteCond %{REQUEST_URI} ^.*/$
        RewriteCond %{REQUEST_URI} !^/$
        RewriteRule ^(.*)/$ $1 [R,QSA,L]
      # If path that is not empty or / or /index.htm or /index.html, redirect to /#!/path
        RewriteCond %{REQUEST_URI} !(/index\.html?|/|)$
        RewriteRule ^(.*)$ /#!$1 [R,QSA,NE,L]
      # If not /, redirect to it.
        RewriteCond %{REQUEST_URI} !^/$
        RewriteRule ^ / [R,QSA,L]

  # Handle Prerender.io
    RequestHeader set X-Prerender-Token "%MYTOKEN%"

    RewriteCond %{HTTP_USER_AGENT} baiduspider|facebookexternalhit|twitterbot|Whatsapp|rogerbot|linkedinbot|embedly|quora\ link\ preview|showyoubot|outbrain|pinterest [NC,OR]
    RewriteCond %{QUERY_STRING} _escaped_fragment_
    RewriteCond %{QUERY_STRING} _escaped_fragment_=([^&]*)

  # Proxy the request
    RewriteRule ^ http://service.prerender.io/http://%{HTTP_HOST}%?_escaped_fragment_=%1 [P,L]
    
</IfModule>

@denitto
Copy link

denitto commented Mar 27, 2017

interesting note I found re: iMessage User Agent. Hope this helps someone!

iMessage User Agent

@thoop
Copy link
Author

thoop commented Jun 16, 2017

I don't get notified when someone comments on this gist. Send us an email at support@prerender.io if anyone has any questions or issues.

@thucnd
Copy link

thucnd commented Jul 5, 2017 •
edited

it doesn't work on my site

RewriteEngine On
RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} -f [OR]
RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} -d
RewriteRule ^ - [L]

RequestHeader set X-Prerender-Token "mytoken"

RewriteCond %{HTTP_USER_AGENT} baiduspider|facebookexternalhit|twitterbot|rogerbot|linkedinbot|embedly|quora\ link\ preview|showyoubot|outbrain|pinterest [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Yandex(Bot|Images|Video|Media) [NC,OR]
RewriteCond %{QUERY_STRING} _escaped_fragment_

RewriteRule ^(.*)$ http://service.prerender.io/http://%{HTTP_HOST}$1 [P,L]

# Allow Robots.txt to pass through
RewriteRule ^robots.txt - [L]
RewriteRule ^ /index.html [L]

please help me to check config

@sashatexb
Copy link

sashatexb commented Jan 6, 2018 •
edited

If you have prerender working and running for all pages except the root(home) page.
Your .htaccess might have the URL set to /index.php or /index.shtml for the homepage. If so, you could change your rewrite rule to this:

RewriteRule ^(?!.*?(\.js|\.css|\.xml|\.less|\.png|\.jpg|\.jpeg|\.gif|\.pdf|\.doc|\.txt|\.ico|\.rss|\.zip|\.mp3|\.rar|\.exe|\.wmv|\.doc|\.avi|\.ppt|\.mpg|\.mpeg|\.tif|\.wav|\.mov|\.psd|\.ai|\.xls|\.mp4|\.m4a|\.swf|\.dat|\.dmg|\.iso|\.flv|\.m4v|\.torrent|\.ttf|\.woff))(index\.php)?(.*) http://service.prerender.io/http://www.example.com$3 [P,L]

I added (index.php or index.shtml)? to capture that out if found and changed the $2 to $3. Don't forget to change the http://www.example.com to your domain.

@atomalak
Copy link

atomalak commented Mar 7, 2018 •
edited

Hi i have used prerender for my angular project i did everything for activate prerender however not success

My web site url is:http://ugurdogrusoy.com/preparing/

firstly addred

Then added prerender token inside .htaccess you can see rule and token below

RewriteEngine On # If requested resource exists as a file or directory # (REQUEST_FILENAME is only relative in virtualhost context, so not usable) RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} -f [OR] RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} -d # Go to it as is RewriteRule ^ - [L] 
# If non existent
# If path ends with / and is not just a single /, redirect to without the trailing /
RewriteCond %{REQUEST_URI} ^.*/$
RewriteCond %{REQUEST_URI} !^/$
RewriteRule ^(.*)/$ $1 [R,QSA,L]      

# Handle Prerender.io
RequestHeader set X-Prerender-Token "MY_TOKEN"
   

RewriteCond %{HTTP_USER_AGENT} baiduspider|facebookexternalhit|twitterbot|rogerbot|linkedinbot|embedly|quora\ link\ preview|showyoubot|outbrain|pinterest [NC,OR]
RewriteCond %{QUERY_STRING} _escaped_fragment_

# Proxy the request
RewriteRule ^(.*)$ http://service.prerender.io/http://%{HTTP_HOST}/preparing/$1 [P,L]

# If non existent
# Accept everything on index.html
RewriteRule ^ /index.html

i can't escape We haven't seen a request with your prerender token yet

where have i done wrong you can help me ?

Thanks

@Bersam
Copy link

Bersam commented May 27, 2018

Any Idea how to integrate that with tomcat? seems P flag in Tomcat doesn't work.

@mkosir
Copy link

mkosir commented Dec 29, 2018 •
edited

If anybody else is having a problem with always returning index file if it's not a crawler, so your SPA router can catch 404, I created a small gist -
https://gist.github.com/markokosir/77e8df3b7cf0b00023178c907f5c998e

@d8bauxit8
Copy link

A warning to anyone using vector files. You will need to add a |\.svg capture group to the proxy rule.

@BorisPocatko
Copy link

If anyone needs to redirect .html files only (skipping extensions .xhtml,.shtml,.phtml,.jp,.cgi,.ppl,.pl,.perl,.php)
Here is my fork: https://gist.github.com/BorisPocatko/77f8f3812d05b934556854267a48150f

@Volodymyrkohut
Copy link

Volodymyrkohut commented Feb 21, 2019 •
edited

.htaccess

RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} -f [OR]
RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} -d
# Go to it as is
RewriteRule ^ - [L]

  RewriteCond %{REQUEST_URI} ^.*/$
  RewriteCond %{REQUEST_URI} !^/$
  RewriteRule ^(.*)/$ $1 [R,QSA,L]      

RequestHeader set X-Prerender-Token "%MYTOKEN%"

RewriteCond %{HTTP_USER_AGENT} baiduspider|facebookexternalhit|twitterbot|rogerbot|linkedinbot|embedly|quora\ link\ preview|showyoubot|outbrain|pinterest [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Yandex(Bot|Images|Video|Media) [NC,OR]
RewriteCond %{QUERY_STRING} _escaped_fragment_

RewriteRule ^(.*)$ http://service.prerender.io/http://%{HTTP_HOST}$1 [P,L]

# Accept everything on index.html
RewriteRule ^ /index.html`

in index.html

But robots can't see my site.
On prerender.io i have 67 cached pages.

Could you please help me.

@Dizzyspb Did you fix it?

@bobbymart1n
Copy link

bobbymart1n commented May 7, 2019 •
edited

Hey all, I'm running into a problem when trying to Rewrite part of the URL being cached in prerender.io. I've tried removing that part of the URL using:
Rewrite ^root-directory/parent-directory/dist/(.*)$ $1 [R,L]
Doing this inside of:

<IfModule mod_proxy_http.c>
        Rewrite ^root-directory/parent-directory/dist/(.*)$ $1 [R,L]
        RewriteCond %{HTTP_USER_AGENT} baiduspider|facebookexternalhit|twitterbot|rogerbot|linkedinbot|embedly|quora\ link\ preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator [NC,OR]
        RewriteCond %{QUERY_STRING} _escaped_fragment_

        # Only proxy the request to Prerender if it's a request for HTML
        RewriteRule ^(?!.*?(\.js|\.css|\.xml|\.less|\.png|\.jpg|\.jpeg|\.gif|\.pdf|\.doc|\.txt|\.ico|\.rss|\.zip|\.mp3|\.rar|\.exe|\.wmv|\.doc|\.avi|\.ppt|\.mpg|\.mpeg|\.tif|\.wav|\.mov|\.psd|\.ai|\.xls|\.mp4|\.m4a|\.swf|\.dat|\.dmg|\.iso|\.flv|\.m4v|\.torrent|\.ttf|\.woff))(.*) http://service.prerender.io/https://www.example.com/$2 [P,L]
    </IfModule>

Doing this on the live server results in TOO_MANY_REDIRECTS. Is there anyway to solve this? I've changed the https://www.example.com/$2 to represent my sites URL

@eash-veera
Copy link

Hi i have used prerender for my angular project i did everything for activate prerender however not success

My web site url is:http://ugurdogrusoy.com/preparing/

firstly addred

Then added prerender token inside .htaccess you can see rule and token below

RewriteEngine On # If requested resource exists as a file or directory # (REQUEST_FILENAME is only relative in virtualhost context, so not usable) RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} -f [OR] RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} -d # Go to it as is RewriteRule ^ - [L]

# If non existent
# If path ends with / and is not just a single /, redirect to without the trailing /
RewriteCond %{REQUEST_URI} ^.*/$
RewriteCond %{REQUEST_URI} !^/$
RewriteRule ^(.*)/$ $1 [R,QSA,L]      

# Handle Prerender.io
RequestHeader set X-Prerender-Token "MY_TOKEN"
   

RewriteCond %{HTTP_USER_AGENT} baiduspider|facebookexternalhit|twitterbot|rogerbot|linkedinbot|embedly|quora\ link\ preview|showyoubot|outbrain|pinterest [NC,OR]
RewriteCond %{QUERY_STRING} _escaped_fragment_

# Proxy the request
RewriteRule ^(.*)$ http://service.prerender.io/http://%{HTTP_HOST}/preparing/$1 [P,L]

# If non existent
# Accept everything on index.html
RewriteRule ^ /index.html

i can't escape We haven't seen a request with your prerender token yet

where have i done wrong you can help me ?

Thanks

I have had the same issue, our site is using react, and we even tried to shift the config file further up to the root. Still the same issue. Any luck with this ?

@yokoishioka
Copy link

you are awesome!! i almost wept when i finally saw the token not used message go away. thanks!!

@arnotixe
Copy link

Just adding 2 cents:
On my hosting provider (Dreamhost), the [NC,OR] yielded strange results when just copy/pasting. On RewriteCond with only one line, I had to write it as
RewriteCond … googlebot|yandex|…|…|… [NC]
(re-writing the RewriteCond with one per line also worked, but not with [OR] on the first line. This would work:)

RewriteCond … googlebot [NC]
RewriteCond … yandex [NC,OR]
RewriteCond … WhatsApp [NC,OR]

note the seemingly missing OR on the first line

On the other hand, my second cent is that last WhatsApp entry - turns out WhatsApp does the scraping directly from inside the app (at least it did on my android phone today ;) So my full line is now
RewriteCond %{HTTP_USER_AGENT} googlebot|bingbot|yandex|baiduspider|facebookexternalhit|twitterbot|rogerbot|linkedinbot|embedly|quora\ link\ preview|showyoubot|outbrain|pinterest\/0\.|pinterestbot|slackbot|vkShare|W3C_Validator|WhatsApp [NC]

(And my full htaccess

    RewriteEngine On
        
        # https://stackoverflow.com/questions/18406156/redirect-all-to-index-php-using-htaccess
        RewriteCond %{REQUEST_FILENAME} !-f
        RewriteCond %{REQUEST_FILENAME} !-d
        RewriteCond %{HTTP_USER_AGENT} googlebot|bingbot|yandex|baiduspider|facebookexternalhit|twitterbot|rogerbot|linkedinbot|embedly|quora\ link\ preview|showyoubot|outbrain|pinterest\/0\.|pinterestbot|slackbot|vkShare|W3C_Validator|WhatsApp [NC]
        #        RewriteCond %{HTTP_USER_AGENT} facebookexternalhit|googlebot [NC]   MUST BE WRITTEN WITHOUT OR
        #        RewriteCond %{HTTP_USER_AGENT} googlebot [NC]
        #        RewriteCond %{HTTP_USER_AGENT} facebookexternalhit [NC,OR]          'OR' IS FOR SECOND LINE (AND THIRD AND FOURTH ETC. WON'T WORK ON FIRST LINE)
        RewriteRule ^(.*)$ opengraph.php?q=$1 [NC,L,QSA]

        RewriteCond %{REQUEST_FILENAME} !-f
        RewriteCond %{REQUEST_FILENAME} !-d
        RewriteRule ^(.*)$ redir.php?orig_path=$1 [NC,L,QSA]

</IfModule>

@tyluudinh
Copy link

Angular 7 + Apache2: work for me. Ping me if you have any issue

.htaccess: https://gist.github.com/tyluudinh/2149d6cc62219c141f619682b5378c47

.apache2.conf: https://gist.github.com/tyluudinh/bc257a1aa2a4b8826ff28916238cc27d

@Kotstar1
Copy link

@tyluudinh This worked for me, thank you 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment