Skip to content

Instantly share code, notes, and snippets.

@searls

searls/wat.md

Last active October 5, 2020 23:54
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save searls/9a398d13a35baebfe12937936944f1a0 to your computer and use it in GitHub Desktop.
Save searls/9a398d13a35baebfe12937936944f1a0 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
wat.md

Debugging a CORS header issue

Just wasted a bunch of time thinking that our static-rails gem was messing up CORS headers. Turned out that AWS Cloudfront just randomly choked on a handful of assets across our properties and the only way to fix it was to manually invalidate (read: cache-bust) them and force it to go and refetch them.

This particular font served by AWS cloudfront, when requested by Chrome, is not returning the CORS Access-Control-Allow-Origin header.

The bad request:

This is the cURL command provided by right-click copy in Chrome devtools with --verbose tacked on to see the header:

curl --verbose 'https://cdn-blog.testdouble.com/webfonts/SourceSansPro-woff2-Semibold.woff2' \
  -H 'authority: cdn-blog.testdouble.com' \
  -H 'origin: https://blog.testdouble.com' \
  -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_16_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36' \
  -H 'dnt: 1' \
  -H 'accept: */*' \
  -H 'sec-fetch-site: same-site' \
  -H 'sec-fetch-mode: cors' \
  -H 'sec-fetch-dest: font' \
  -H 'referer: https://cdn-blog.testdouble.com/main.63250acfa54004d69440811a8bff5b7e7e4507dedc7e6626fd4a7e3c7e97a2bdc5edd004ffffb4e6985bb66032d0ac49a3008c619056980cf84807657b61fa54.css' \
  -H 'accept-language: en-US,en;q=0.9' \
  -H 'range: bytes=15971-15971' \
  -H 'if-range: Fri, 11 Sep 2020 20:29:51 GMT' \
  --compressed

These headers come back:

< HTTP/2 200 
< content-type: application/font-woff2
< content-length: 89668
< server: Cowboy
< date: Sun, 04 Oct 2020 16:29:43 GMT
< last-modified: Thu, 01 Oct 2020 16:43:28 GMT
< cache-control: public; max-age=31536000
< x-request-id: 8a070b5b-2023-4425-9888-1c4d69878002
< x-runtime: 0.002611
< strict-transport-security: max-age=31536000
< via: 1.1 vegur, 1.1 d365d3bc6fd19afdef198b27dff058b7.cloudfront.net (CloudFront)
< x-cache: Hit from cloudfront
< x-amz-cf-pop: ORD53-C1
< x-amz-cf-id: _94ded0e3M0gea2fpTX_WNbqbElr9x5g5p4UFp0q-Dab6EqX1-Jj0A==
< age: 90903

Test 1: Not sending --compressed

The same cURL, but with --compressed removed does return access-control-allow-origin: * as configured:

< HTTP/2 200 
< content-type: application/font-woff2
< content-length: 89668
< server: Cowboy
< date: Sun, 04 Oct 2020 15:07:19 GMT
< access-control-allow-origin: *
< access-control-allow-methods: GET, POST, PATCH, PUT
< access-control-expose-headers: 
< access-control-max-age: 7200
< last-modified: Thu, 01 Oct 2020 16:43:28 GMT
< cache-control: public; max-age=31536000
< x-request-id: cb4d28a4-26c9-44cb-964c-b636f7fe78c5
< x-runtime: 0.007012
< strict-transport-security: max-age=31536000
< via: 1.1 vegur, 1.1 bcca980c8c3bc3b385e284d2276b6faa.cloudfront.net (CloudFront)
< x-cache: Hit from cloudfront
< x-amz-cf-pop: ORD53-C1
< x-amz-cf-id: 4brHGF2_R57KzgU5urVHPBaK9speibw4YqQdtDTgNRJEAPYnQzQWYw==
< age: 95913

Test 2: The origin itself

If you cURL the origin itself (hitting our heroku dyno serving the asset via the static-rails gem:


```sh
curl --verbose 'https://blog.testdouble.com/webfonts/SourceSansPro-woff2-Semibold.woff2' \
  -H 'authority: cdn-blog.testdouble.com' \
  -H 'origin: https://blog.testdouble.com' \
  -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_16_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36' \
  -H 'dnt: 1' \
  -H 'accept: */*' \
  -H 'sec-fetch-site: same-site' \
  -H 'sec-fetch-mode: cors' \
  -H 'sec-fetch-dest: font' \
  -H 'referer: https://cdn-blog.testdouble.com/main.63250acfa54004d69440811a8bff5b7e7e4507dedc7e6626fd4a7e3c7e97a2bdc5edd004ffffb4e6985bb66032d0ac49a3008c619056980cf84807657b61fa54.css' \
  -H 'accept-language: en-US,en;q=0.9' \
  -H 'range: bytes=15971-15971' \
  -H 'if-range: Fri, 11 Sep 2020 20:29:51 GMT' \
  --compressed

Then you also get the right header:

< HTTP/1.1 206 Partial Content
< Server: Cowboy
< Date: Mon, 05 Oct 2020 17:46:29 GMT
< Connection: keep-alive
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Methods: GET, POST, PATCH, PUT
< Access-Control-Expose-Headers: 
< Access-Control-Max-Age: 7200
< Last-Modified: Thu, 01 Oct 2020 16:43:28 GMT
< Content-Type: application/font-woff2
< Cache-Control: public; max-age=31536000
< Content-Range: bytes 15971-15971/89668
< Set-Cookie: _csrf_token=cRd0OacDmMssgBV57bg%2BRbJduWXSjO5Td1LRcELfCAV6VwVTieLzSaV2Uwnf2fEZCpWZA%2BAaCzRS61ja3vvVXg%3D%3D; path=/; secure
< Set-Cookie: _test_double_web_session=J56WIuBqzVJK9BaA96KUCcOYAZxkbWCfy3ST%2FvT3A8PAkrMcvxBQtw38lWHBP5nu9Nk66fYejk9dbeHERfMJrIekzQU6kR0PACwjJgYzHqftxze4mpoy1iCrPDYKW0Qy7DC58CM%2B2IomVDJKtUgm%2BNTDJemMTekOY4UnZ07mOis3%2FRHGq8DVqFaUrOGFUUqAiQpFJwYvaBX8IYBGqfvsDyOGGHjzDMA2wpHxH6E70j6OeWlsX%2FWZnFywaACh88XeW2a2WZhzg8NaqyqoAncO3PrXnlVEEsUGQc6asXiKi1A%3D--eG6gyVsyTbz90uUa--aSdAG7qBNzvDqVJ%2F4nSytQ%3D%3D; path=/; secure; HttpOnly
< X-Request-Id: bad30a08-a74b-49f6-93d7-c98d2898c1fb
< X-Runtime: 0.001596
< Strict-Transport-Security: max-age=31536000
< Vary: Origin
< Content-Length: 1
< Via: 1.1 vegur

Test 3: trying another file

Fortunately there are a bunch of very similar fonts all in this directory so I can just try Bold instead of SemiBold and see if that works:

curl --verbose 'https://cdn-blog.testdouble.com/webfonts/SourceSansPro-woff2-Bold.woff2' \
  -H 'authority: cdn-blog.testdouble.com' \
  -H 'origin: https://blog.testdouble.com' \
  -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_16_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36' \
  -H 'dnt: 1' \
  -H 'accept: */*' \
  -H 'sec-fetch-site: same-site' \
  -H 'sec-fetch-mode: cors' \
  -H 'sec-fetch-dest: font' \
  -H 'referer: https://cdn-blog.testdouble.com/main.63250acfa54004d69440811a8bff5b7e7e4507dedc7e6626fd4a7e3c7e97a2bdc5edd004ffffb4e6985bb66032d0ac49a3008c619056980cf84807657b61fa54.css' \
  -H 'accept-language: en-US,en;q=0.9' \
  -H 'range: bytes=15971-15971' \
  -H 'if-range: Fri, 11 Sep 2020 20:29:51 GMT' \
  --compressed

And sure enough, it also returns the right headers

< HTTP/2 200 
< content-type: application/font-woff2
< content-length: 89076
< server: Cowboy
< date: Mon, 05 Oct 2020 17:34:58 GMT
< access-control-allow-origin: *
< access-control-allow-methods: GET, POST, PATCH, PUT
< access-control-expose-headers: 
< access-control-max-age: 7200
< last-modified: Thu, 01 Oct 2020 16:43:28 GMT
< cache-control: public; max-age=31536000
< x-request-id: 60b2c877-a272-4c83-a5fa-ddf3f49f7f12
< x-runtime: 0.002382
< strict-transport-security: max-age=31536000
< via: 1.1 vegur, 1.1 f0312eca85d338806221bc299acb4e0b.cloudfront.net (CloudFront)
< x-cache: Hit from cloudfront
< x-amz-cf-pop: IAD79-C3
< x-amz-cf-id: w40YyvStgPGVqYQ3f1ubsC9Uy9SQWQXvOP1EDWkAbSiF3TCCtXRvkg==
< age: 867
<

Test 4: invalidating the file in AWS and trying again

So next I tried invalidating the path:

Invalidation ID: I2IZII663B3SL5
Status: Completed
Date and Time Created: 2020-10-05 13:50 UTC-4
Object Paths: /webfonts/SourceSansPro-woff2-Semibold.woff2

And sure enough the same cURL at the top magically works now:

< HTTP/2 206 
< content-type: application/font-woff2
< content-length: 1
< server: Cowboy
< date: Mon, 05 Oct 2020 17:53:46 GMT
< access-control-allow-origin: *
< access-control-allow-methods: GET, POST, PATCH, PUT
< access-control-expose-headers: 
< access-control-max-age: 7200
< last-modified: Thu, 01 Oct 2020 16:43:28 GMT
< cache-control: public; max-age=31536000
< x-request-id: e89ff3fa-7bc8-4651-8f7c-ac1e83c8e56d
< x-runtime: 0.001693
< strict-transport-security: max-age=31536000
< via: 1.1 vegur, 1.1 c93cdf0926e57254c4cc150bcbedb97c.cloudfront.net (CloudFront)
< content-range: bytes 15971-15971/89668
< x-cache: Miss from cloudfront
< x-amz-cf-pop: IAD79-C3
< x-amz-cf-id: wubkzUCUWgt_yCA98YRtpteko3Q6YSu-xrYX4mXqN7IwTPSV6vpzlA==
<

Neat score: 👌👌👌👌

@kkuchta
Copy link

kkuchta commented Oct 5, 2020 •
edited

I'm not gonna pretend to know if this is your issue, but an anecdote when I ran into the exact same issue (cors + font + rails + cloudfront):

When you make an http request to a server, it'll only return the access-control response headers if the Origin: header was present in the request. No origin request header, no access-control response header.

Cloudfront has a list of request headers where, if they vary between requests, those requests are cached separately. Eg if two requests come in to cloudfront with everything the same except the User-Agent header is different, cloudfront considers them the same request and the second one receives the cached response of the first.

So what happened to me is:

  1. I deploy the update to my rails app that includes a new font asset for the first time.
  2. The first request to cloudfront for those fonts happens to come with no origin header.
    3. Cloudfront dutifully passed the request with no origin header to the rails app
    4. Rails correctly returns the response with no access-control headers
    5. Cloudfront caches the response with no access-control headers and returns it to the requester
  3. The second request to cloudfront for those fonts happens, and does include an origin header
    7. Cloudfront sees that this request is identical to the first one as far as all the headers it cares about are concerned. The origin header is different, but it doesn't care.
    8. Cloudfront returns the cached response, which incorrectly does not include the access-control headers.
  4. Whoever made that request in step 6 now sees a cors error.

This was maddening to debug. It's non-deterministic, since it depends on what the first http request to cloudfront is. And since cloudfront keeps a bunch of independent, geographically-distributed caches, some of them will have this issue and some won't (depending on the first request each cache received). So I saw issues where I could repro the cors bug on my laptop, but not on my phone (which was hitting a different cloudfront edge cache).

Anyway, invalidating the cloudfront cache fixed it in the short term from one location, but didn't fix the underlying issue.

The ultimate fix was configuring cloudfront to understand that two requests with different "Origin" headers should be cached separately. You do this via the "Cache Policy" settings for the CF distribution.

I usually try to avoid responding to people complaining about bugs on twitter with "well have you tried...", but this one took me weeks of hair-pulling to figure out, so on the off-chance it's related to your issue, I hope it's useful.

@searls
Copy link
Author

searls commented Oct 5, 2020

Damn, great write up! Going to do this right now. Thanks @kkuchta!!

@kkuchta
Copy link

kkuchta commented Oct 5, 2020

Glad I could help! Hope it solves your issue and you don't have to spend the weeks of "I fixed it - wait, no, it's back again" that I had to go through. :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment