Apex Record Redirects

README.md

Apex Record Redirects

An apex (or root) record of a zone cannot be a CNAME by specification. This has implications for domains that try to integrate with virtually any cloud platform (AWS, Azure, Heroku, etc.). A common pattern to mitigate this issue is to perform a HTTP 301 redirect from the apex record to a subdomain (usually www.domain.com).

On the other hand, using a single cloud provider for DNS and underlying infrastructure may yield proprietary solutions. For example, AWS provides a Route 53 alias record which can be used at the zone apex.

Alexa Script

To understand the prevalence of the apex redirect pattern this script will test the top 1000 Alexa rank domains (as of 3/11/16).

Prerequisities

You need a python 2.7 installation (or virtualenv). Install the python dependencies via pip:

$ pip install -r requirements.txt

Usage

$ python alexa.py <limit>

Results

Top 10

$ python alexa.py 10
Checking Alexa top 10 sites
https://www.google.com/?gws_rd=ssl => 200
https://www.wikipedia.org/ => 200
https://www.google.co.in/?gws_rd=ssl => 200
https://www.facebook.com/ => 200
https://twitter.com/ => 200
http://www.amazon.com/ => 200
https://www.yahoo.com/ => 200
https://www.youtube.com/ => 200
http://www.qq.com/ => 200
http://baidu.com/ => 200
0 errors or timeouts
8/10 sites perform apex redirect to subdomain

Note: baidu.com performs a client side redirect

Top 100

5 errors or timeouts
76/95 sites perform apex redirect to subdomain

Top 1000

72 errors or timeouts
719/928 sites perform apex redirect to subdomain

Known Issues

• Client side redirection is not detected
• 5-10% error rate are common (the GFW may be responsible)

alexa.py

#!/usr/bin/env python

from __future__ import print_function

import sys

import grequests
import tldextract

try:
    LIMIT = int(sys.argv[1])
except IndexError:
    LIMIT = 1000

USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'

def urls():
    count = 0

    with open('./top-1000.csv') as f:
        for line in f:
            if count >= LIMIT:
                break

            _, site = line.strip().split(',')

            # Need to ensure we hit an apex record
            _, domain, suffix = tldextract.extract(site)
            yield grequests.get('http://%s.%s' % (domain, suffix),
                                headers={'User-Agent': USER_AGENT}, timeout=5)

            count += 1

redirects, total = 0, 0

print('Checking Alexa top %s sites' % LIMIT)

for response in grequests.imap(urls(), size=32):
    print('%s => %s' % (response.url, response.status_code))

    # Count redirection to a subdomain of any sort
    sub, _, _ = tldextract.extract(response.url)
    if len(sub):
        redirects += 1

    # Drop client or server errors from our counts
    if response.status_code <= 400:
        total +=1

print('%d errors or timeouts' % (LIMIT - total))
print('%d/%d sites perform apex redirect to subdomain' % (redirects, total))

# vim: tabstop=4 expandtab shiftwidth=4 softtabstop=4

requirements.txt

grequests
tldextract

top-1000.csv