Experiments to determine why CSS doesn't update on v7.0

gistfile1.txt

* Hypothesis: takes two builds to trigger css rebuild
  - experiment: push css change commit, wait for build, push another, wait for build, see if fixed
  - result: did not fix the problem
  - observation: the hypothesis was contradicted by stable working

* Hypothesis: css is not rebuilt by the server at all
  - experiment: curl all css files generated before and after change and compare them
    - curl -v https://docs.citusdata.com/en/v7.0/_static/css/theme.css > theme-7-3c64d3cf.css
      * TLS 1.2 connection using TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
      * Server certificate: citusdata.com
      * Server certificate: CloudFlare Inc ECC CA-2
      * Server certificate: Baltimore CyberTrust Root
      > GET /en/v7.0/_static/css/theme.css HTTP/1.1
      > Host: docs.citusdata.com
      > User-Agent: curl/7.43.0
      > Accept: */*
      >
      < HTTP/1.1 200 OK
      < Date: Thu, 07 Sep 2017 16:17:20 GMT
      < Content-Type: text/css
      < Transfer-Encoding: chunked
      < Connection: keep-alive
      < Set-Cookie: __cfduid=d61f10194e94f1e1cde0f4639013d6a991504801040; expires=Fri, 07-Sep-18 16:17:20 GMT; path=/; domain=.citusdata.com; HttpOnly
      < Last-Modified: Thu, 07 Sep 2017 01:03:22 GMT
      < Vary: Accept-Encoding
      < ETag: W/"59b09ada-1d256"
      < X-Cname-TryFiles: True
      < X-Served: Nginx
      < X-Deity: web02
      < CF-Cache-Status: HIT
      < Expires: Thu, 07 Sep 2017 20:17:20 GMT
      < Cache-Control: public, max-age=14400
      < Server: cloudflare-nginx
      < CF-RAY: 39aafa471be7558e-ORD
    - curl -v https://docs.citusdata.com/en/v7.0/_static/css/theme.css > theme-7-c1b70c5c.css
      * TLS 1.2 connection using TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
      * Server certificate: citusdata.com
      * Server certificate: CloudFlare Inc ECC CA-2
      * Server certificate: Baltimore CyberTrust Root
      > GET /en/v7.0/_static/css/theme.css HTTP/1.1
      > Host: docs.citusdata.com
      > User-Agent: curl/7.43.0
      > Accept: */*
      >
      < HTTP/1.1 200 OK
      < Date: Thu, 07 Sep 2017 16:22:01 GMT
      < Content-Type: text/css
      < Transfer-Encoding: chunked
      < Connection: keep-alive
      < Set-Cookie: __cfduid=d923f6b42195b15193981415bbd16b47c1504801321; expires=Fri, 07-Sep-18 16:22:01 GMT; path=/; domain=.citusdata.com; HttpOnly
      < Last-Modified: Thu, 07 Sep 2017 01:03:22 GMT
      < Vary: Accept-Encoding
      < ETag: W/"59b09ada-1d256"
      < X-Cname-TryFiles: True
      < X-Served: Nginx
      < X-Deity: web02
      < CF-Cache-Status: HIT
      < Expires: Thu, 07 Sep 2017 20:22:01 GMT
      < Cache-Control: public, max-age=14400
      < Server: cloudflare-nginx
      < CF-RAY: 39ab01236e3a10e7-ORD
  - result: `diff theme-7-*` shows them to be identical
    Also with the new css cherry picked to 7.0:
    curl https://docs.citusdata.com/en/v7.0/_static/css/theme.css | md5
      5fc5806c4348a742816c6c2ebe268387
    curl https://docs.citusdata.com/en/stable/_static/css/theme.css | md5
      2846b1959983a85a4ce6a31988c4d0fd
    curl https://docs.citusdata.com/en/latest/_static/css/theme.css | md5
      2846b1959983a85a4ce6a31988c4d0fd
  - observation: why does the grid still work? Perhaps other css files are being requested (flexboxgrid)

* Hypothesis: flexboxgrid.min.css is not being combined into theme.css, but served separately
  - experiment: on the latest version, check network tab
  - result: yes, https://docs.citusdata.com/en/latest/_static/css/flexboxgrid.min.css is being requested

* Hypothesis: I wrote incorrect sass syntax in _themes/sass/_theme_layout.sass for my change
  - experiment: run the process locally to build theme.css, use full verbosity and look for warnings or errors
      $ grunt sass:dev
      Running "sass:dev" (sass) task
      Done, without errors.
  - result: no errors or warnings
  - observation: the experiment was suspect because the css has always rendered properly locally

* Hypothesis: read the docs never regenerates css on the 7.0 version
  - experiment: curl the css again and inspect the Last-Modified header
      - curl -v https://docs.citusdata.com/en/v7.0/_static/css/theme.css
        *   Trying 104.25.46.11...
        * Connected to docs.citusdata.com (104.25.46.11) port 443 (#0)
        * TLS 1.2 connection using TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
        * Server certificate: citusdata.com
        * Server certificate: CloudFlare Inc ECC CA-2
        * Server certificate: Baltimore CyberTrust Root
        > GET /en/v7.0/_static/css/theme.css HTTP/1.1
        > Host: docs.citusdata.com
        > User-Agent: curl/7.43.0
        > Accept: */*
        >
        < HTTP/1.1 200 OK
        < Date: Thu, 07 Sep 2017 17:39:06 GMT
        < Content-Type: text/css
        < Transfer-Encoding: chunked
        < Connection: keep-alive
        < Set-Cookie: __cfduid=d708c38039a4ce3025fa35a50638687251504805946; expires=Fri, 07-Sep-18 17:39:06 GMT; path=/; domain=.citusdata.com; HttpOnly
        < Last-Modified: Thu, 07 Sep 2017 01:03:22 GMT
        < Vary: Accept-Encoding
        < ETag: W/"59b09ada-1d256"
        < X-Cname-TryFiles: True
        < X-Served: Nginx
        < X-Deity: web02
        < CF-Cache-Status: HIT
        < Expires: Thu, 07 Sep 2017 21:39:06 GMT
        < Cache-Control: public, max-age=14400
        < Server: cloudflare-nginx
        < CF-RAY: 39ab720cc8f455ca-ORD
        <
  - result: the header has the same value for both times the css was generated.
            However this may be because RTD is using file timestamps from a git commit
      is: Last-Modified: Thu, 07 Sep 2017 01:03:22 GMT
          Date: Thu, 07 Sep 2017 17:39:06 GMT
      vs: Last-Modified: Thu, 07 Sep 2017 01:03:22 GMT
          Date: Thu, 07 Sep 2017 16:22:01 GMT
  - observation:
      latest worked right the first time:
        Last-Modified: Wed, 06 Sep 2017 19:14:39 GMT

  - experiment 2: push another tiny css change on top of a
                  reverted 7.0 and see if it appears -
      change ".wy-side-nav-search input[type=text]" border color to #222
  - result: v7.0 reflects the change but stable does not!
      stable: Last-Modified: Wed, 06 Sep 2017 18:50:15 GMT
      v7.0:   Last-Modified: Thu, 07 Sep 2017 17:54:59 GMT

* Hypothesis: CloudFlare is caching the CSS between sphinx and the browser
  - observation: I noticed that "CF-Cache-Status: HIT" is present in headers
  - experiment: disable caching on cloudflare for docs.citusdata.com