Skip to content

Instantly share code, notes, and snippets.

@minaguib

minaguib/sectigo.md

Last active Nov 5, 2020
Embed
What would you like to do?
Sectigo/Comodo/USERTrust/AddTrust Certificate Expiry May 30 2020 notes

Comodo/Sectigo/USERTrust/AddTrust root certificate expiry

On May 30th 2020 (10:48am GMT / 6:48am EDT / 3:48am PDT), several root & intermediate certificates that were part of the Comodo family expired.

The expired certificates

The following certificates expired on May 30 10:48:38 2020 GMT:

Impact

Thousands of web sites / services / APIs which present certificates that were issued by the vendor may have experienced trouble negotiating incoming secure connections from their clients. Exact issues, if any, depended on a mix of server configuration, client version, and client configuration.

A client validating any of those certificates had access to 3 "paths" to do so successfully. One of those paths (A) became invalid after the above certificates expired:

Trust Chain Path A: (INVALID)

Trust Chain Path B: (VALID)

  • USERTrust RSA Certification Authority (Type:Root) (Serial:01:fd:6d:30:fc:a3:ca:51:a8:1b:bc:64:0e:35:03:2d)
  • Sectigo RSA DV/OV/EV Secure Server CA (Type:Intermediate)
  • End Entity (Type:Leaf Certificate)

Trust Chain Path C: (VALID)

Problems and solutions

Client reports certificate not issued by trusted Certificate Authority

To address:

Client-side:

Ensure client updates their local CA roots trust store to include one/both of the (Type:Root) certificates in either of the Path B or Path C cases above

Server-side:

In the chain bundle you serve, include the (Type:Intermediate) certificates in path B and/or C above to encourage your clients to reach one of the two valid (Type:Root) certificates in their local CA roots trust store.

OpenSSL-based clients reporting certificate is expired

If a server is sending a bundled chain of certificates and it includes the above NOW EXPIRED certificates from Chain Path A, AND the client is using an older OpenSSL version (<1.1.0), AND the client's local CA roots trust store has the "AddTrust" root certificate, it will trigger a bug where the client will report an Expired error.

To address, do any/some/most of the following:

Server-side:

Update the server to no longer include the above 2 expired certificates in the chain bundle.

Client-side:

Update the OpenSSL library software version.

Edit the local CA roots trust store and remove/disable/blacklist the "AddTrust External CA Root" root certificate.

More details

@asurak

This comment has been minimized.

Copy link

@asurak asurak commented May 30, 2020

There is a path towards AAA Certificates that does not require using the USERTrust cert as a root - https://ssl-tools.net/subjects/cd30d24c343a82ab1f0570158ad7a107762992e9. Since some old clients will not recognise it as a trusted root CA, going with the path to AAA is safer. (we had to do it for Algolia production systems)

@minaguib

This comment has been minimized.

Copy link
Owner Author

@minaguib minaguib commented May 30, 2020

Fascinating, will dig in and update as necessary. Thanks @asurak

@minaguib

This comment has been minimized.

Copy link
Owner Author

@minaguib minaguib commented May 30, 2020

@asurak Thanks, done.

@minaguib

This comment has been minimized.

Copy link
Owner Author

@minaguib minaguib commented May 30, 2020

Also, TIL, client validation walking up the chain just looks for the issuer by CN. There can be multiple matches (from server-supplied bundle and/or local store), some valid some not... !!

@asurak

This comment has been minimized.

Copy link

@asurak asurak commented May 30, 2020

Great! The chain with AAA works also on old systems with old openssl, so it has the higher level of compatibility, if needed.

@kawesam

This comment has been minimized.

Copy link

@kawesam kawesam commented May 31, 2020

How do I delete this two files on ubuntu 16 to resolve the issue? Update the server to no longer include the above 2 expired certificates in the chain bundle.

@minaguib

This comment has been minimized.

Copy link
Owner Author

@minaguib minaguib commented May 31, 2020

@kawesam I'm not sure about Ubuntu/16 specifics.

If you're running an HTTPS server and you think you're sending the expired intermediate/root in the chain, check with https://whatsmychaincert.com/ and if that's the case, it depends on the web server you're using, but it'll typically involve editing a PEM file and removing some certs from it.

If you're running a client on an Ubuntu 16 server, see if the instructions at the bottom of Andrew's blog post help

@kamilbednarz

This comment has been minimized.

Copy link

@kamilbednarz kamilbednarz commented Jun 1, 2020

My band-aid for AWS Beanstalk, Amazon Linux (CentOS) & .ebextensions:

# On May 30, 2020, the AddTrust root certificate that is still used in some SSL cert authorization
# chains (e.g. our case: xxx.yyy.com). OpenSSL 1.0.x in Amazon Linux is not able to fall back
# from the expired resolution path to a valid one.
# Solution to this is to blacklist expired certificate, it won't be used for the cert validations.
 
# More details here:
# https://calnetweb.berkeley.edu/calnet-technologists/incommon-sectigo-certificate-service/addtrust-external-root-expiration-may-2020
 
files:
  /etc/pki/ca-trust/source/blacklist/add_trust.cer:
    mode: "000755"
    owner: root
    group: root
    content: |
      -----BEGIN TRUSTED CERTIFICATE-----
      MIIENjCCAx6gAwIBAgIBATANBgkqhkiG9w0BAQUFADBvMQswCQYDVQQGEwJTRTEU
      MBIGA1UEChMLQWRkVHJ1c3QgQUIxJjAkBgNVBAsTHUFkZFRydXN0IEV4dGVybmFs
      IFRUUCBOZXR3b3JrMSIwIAYDVQQDExlBZGRUcnVzdCBFeHRlcm5hbCBDQSBSb290
      MB4XDTAwMDUzMDEwNDgzOFoXDTIwMDUzMDEwNDgzOFowbzELMAkGA1UEBhMCU0Ux
      FDASBgNVBAoTC0FkZFRydXN0IEFCMSYwJAYDVQQLEx1BZGRUcnVzdCBFeHRlcm5h
      bCBUVFAgTmV0d29yazEiMCAGA1UEAxMZQWRkVHJ1c3QgRXh0ZXJuYWwgQ0EgUm9v
      dDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALf3GjPm8gAELTngTlvt
      H7xsD821+iO2zt6bETOXpClMfZOfvUq8k+0DGuOPz+VtUFrWlymUWoCwSXrbLpX9
      uMq/NzgtHj6RQa1wVsfwTz/oMp50ysiQVOnGXw94nZpAPA6sYapeFI+eh6FqUNzX
      mk6vBbOmcZSccbNQYArHE504B4YCqOmoaSYYkKtMsE8jqzpPhNjfzp/haW+710LX
      a0Tkx63ubUFfclpxCDezeWWkWaCUN/cALw3CknLa0Dhy2xSoRcRdKn23tNbE7qzN
      E0S3ySvdQwAl+mG5aWpYIxG3pzOPVnVZ9c0p10a3CitlttNCbxWyuHv77+ldU9U0
      WicCAwEAAaOB3DCB2TAdBgNVHQ4EFgQUrb2YejS0Jvf6xCZU7wO94CTLVBowCwYD
      VR0PBAQDAgEGMA8GA1UdEwEB/wQFMAMBAf8wgZkGA1UdIwSBkTCBjoAUrb2YejS0
      Jvf6xCZU7wO94CTLVBqhc6RxMG8xCzAJBgNVBAYTAlNFMRQwEgYDVQQKEwtBZGRU
      cnVzdCBBQjEmMCQGA1UECxMdQWRkVHJ1c3QgRXh0ZXJuYWwgVFRQIE5ldHdvcmsx
      IjAgBgNVBAMTGUFkZFRydXN0IEV4dGVybmFsIENBIFJvb3SCAQEwDQYJKoZIhvcN
      AQEFBQADggEBALCb4IUlwtYj4g+WBpKdQZic2YR5gdkeWxQHIzZlj7DYd7usQWxH
      YINRsPkyPef89iYTx4AWpb9a/IfPeHmJIZriTAcKhjW88t5RxNKWt9x+Tu5w/Rw5
      6wwCURQtjr0W4MHfRnXnJK3s9EK0hZNwEGe6nQY1ShjTK3rMUUKhemPR5ruhxSvC
      Nr4TDea9Y355e6cJDUCrat2PisP29owaQgVR1EX1n6diIWgVIEM8med8vSTYqZEX
      c4g/VhsxOBi0cQ+azcgOno4uG+GMmIPLHzHxREzGBHNJdmAPx/i9F4BrLunMTA5a
      mnkPIAou1Z5jJh5VkpTYghdae9C8x49OhgQwODAeBggrBgEFBQcDAwYIKwYBBQUH
      AwEGCCsGAQUFBwMEDBZBZGRUcnVzdCBFeHRlcm5hbCBSb290
      -----END TRUSTED CERTIFICATE-----
 
commands:
  01_enable_dynamic_cert_files_evaluation:
    command: "update-ca-trust enable"
  02_extract_certs:
    command: "update-ca-trust extract"

But for non-beanstalk servers: it's a matter of placing this cert in the blacklist directory & re-generating ca-trust new files.

@emmm-dee

This comment has been minimized.

Copy link

@emmm-dee emmm-dee commented Jun 1, 2020

Well this made for an interesting start of the week :(

@ValZapod

This comment has been minimized.

Copy link

@ValZapod ValZapod commented Jun 2, 2020

@minaguib

This comment has been minimized.

Copy link
Owner Author

@minaguib minaguib commented Jun 2, 2020

@ValZapod Thanks. I noted COMODO RSA Certification Authority in the "expired certificates" list.

I'm thinking now of ways to better represent the variations in INVALID and VALID chain paths...

@ValZapod

This comment has been minimized.

Copy link

@ValZapod ValZapod commented Jun 2, 2020

https://gitlab.com/gnutls/gnutls/-/issues/1008 Fixed.
curl/curl#5496 Wontfix as it is upstreams problem (openssl and gnutls)
openssl/openssl#12006 Was already fixed some time ago... Original commit in my comment there.

@ValZapod

This comment has been minimized.

Copy link

@ValZapod ValZapod commented Jun 2, 2020

These also expired https://crt.sh/?id=1721082 https://crt.sh/?id=1733294 https://crt.sh/?id=1726036 https://crt.sh/?id=12716435 https://crt.sh/?id=12716435 some of them have the same public key, LOL (like all USERTrust RSA Certification Authority). Very strange. Is this normal? I thought they change it. Looks like this is all.

@ValZapod

This comment has been minimized.

Copy link

@ValZapod ValZapod commented Jun 8, 2020

I will also add that this https://crt.sh/?id=1720081 has the same private and public key as COMODO RSA Certification Authority https://crt.sh/?id=1044348 but is valid until 2038. So it is not technically expired. Also it is root, not intermediate. See https://crt.sh/?spkisha256=82b5f84daf47a59c7ab521e4982aefa40a53406a3aec26039efa6b2e0e7244c1

And with the main cert even (!) https://crt.sh/?id=162879063 it has the same public key and still not expired!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment