Skip to content

Instantly share code, notes, and snippets.

@minaguib

minaguib/sectigo.md

Last active Nov 5, 2020
Embed
What would you like to do?
Sectigo/Comodo/USERTrust/AddTrust Certificate Expiry May 30 2020 notes

Comodo/Sectigo/USERTrust/AddTrust root certificate expiry

On May 30th 2020 (10:48am GMT / 6:48am EDT / 3:48am PDT), several root & intermediate certificates that were part of the Comodo family expired.

The expired certificates

The following certificates expired on May 30 10:48:38 2020 GMT:

Impact

Thousands of web sites / services / APIs which present certificates that were issued by the vendor may have experienced trouble negotiating incoming secure connections from their clients. Exact issues, if any, depended on a mix of server configuration, client version, and client configuration.

A client validating any of those certificates had access to 3 "paths" to do so successfully. One of those paths (A) became invalid after the above certificates expired:

Trust Chain Path A: (INVALID)

Trust Chain Path B: (VALID)

  • USERTrust RSA Certification Authority (Type:Root) (Serial:01:fd:6d:30:fc:a3:ca:51:a8:1b:bc:64:0e:35:03:2d)
  • Sectigo RSA DV/OV/EV Secure Server CA (Type:Intermediate)
  • End Entity (Type:Leaf Certificate)

Trust Chain Path C: (VALID)

Problems and solutions

Client reports certificate not issued by trusted Certificate Authority

To address:

Client-side:

Ensure client updates their local CA roots trust store to include one/both of the (Type:Root) certificates in either of the Path B or Path C cases above

Server-side:

In the chain bundle you serve, include the (Type:Intermediate) certificates in path B and/or C above to encourage your clients to reach one of the two valid (Type:Root) certificates in their local CA roots trust store.

OpenSSL-based clients reporting certificate is expired

If a server is sending a bundled chain of certificates and it includes the above NOW EXPIRED certificates from Chain Path A, AND the client is using an older OpenSSL version (<1.1.0), AND the client's local CA roots trust store has the "AddTrust" root certificate, it will trigger a bug where the client will report an Expired error.

To address, do any/some/most of the following:

Server-side:

Update the server to no longer include the above 2 expired certificates in the chain bundle.

Client-side:

Update the OpenSSL library software version.

Edit the local CA roots trust store and remove/disable/blacklist the "AddTrust External CA Root" root certificate.

More details

@asurak

This comment has been minimized.

Copy link

@asurak asurak commented May 30, 2020

There is a path towards AAA Certificates that does not require using the USERTrust cert as a root - https://ssl-tools.net/subjects/cd30d24c343a82ab1f0570158ad7a107762992e9. Since some old clients will not recognise it as a trusted root CA, going with the path to AAA is safer. (we had to do it for Algolia production systems)

@minaguib

This comment has been minimized.

Copy link
Owner Author

@minaguib minaguib commented May 30, 2020

Fascinating, will dig in and update as necessary. Thanks @asurak

@minaguib

This comment has been minimized.

Copy link
Owner Author

@minaguib minaguib commented May 30, 2020

@asurak Thanks, done.

@minaguib

This comment has been minimized.

Copy link
Owner Author

@minaguib minaguib commented May 30, 2020

Also, TIL, client validation walking up the chain just looks for the issuer by CN. There can be multiple matches (from server-supplied bundle and/or local store), some valid some not... !!

@asurak

This comment has been minimized.

Copy link

@asurak asurak commented May 30, 2020

Great! The chain with AAA works also on old systems with old openssl, so it has the higher level of compatibility, if needed.

@kawesam

This comment has been minimized.

Copy link

@kawesam kawesam commented May 31, 2020

How do I delete this two files on ubuntu 16 to resolve the issue? Update the server to no longer include the above 2 expired certificates in the chain bundle.

@minaguib

This comment has been minimized.

Copy link
Owner Author

@minaguib minaguib commented May 31, 2020

@kawesam I'm not sure about Ubuntu/16 specifics.

If you're running an HTTPS server and you think you're sending the expired intermediate/root in the chain, check with https://whatsmychaincert.com/ and if that's the case, it depends on the web server you're using, but it'll typically involve editing a PEM file and removing some certs from it.

If you're running a client on an Ubuntu 16 server, see if the instructions at the bottom of Andrew's blog post help

@kamilbednarz

This comment has been minimized.

Copy link

@kamilbednarz kamilbednarz commented Jun 1, 2020

My band-aid for AWS Beanstalk, Amazon Linux (CentOS) & .ebextensions:

# On May 30, 2020, the AddTrust root certificate that is still used in some SSL cert authorization
# chains (e.g. our case: xxx.yyy.com). OpenSSL 1.0.x in Amazon Linux is not able to fall back
# from the expired resolution path to a valid one.
# Solution to this is to blacklist expired certificate, it won't be used for the cert validations.
 
# More details here:
# https://calnetweb.berkeley.edu/calnet-technologists/incommon-sectigo-certificate-service/addtrust-external-root-expiration-may-2020
 
files:
  /etc/pki/ca-trust/source/blacklist/add_trust.cer:
    mode: "000755"
    owner: root
    group: root
    content: |
      -----BEGIN TRUSTED CERTIFICATE-----
      MIIENjCCAx6gAwIBAgIBATANBgkqhkiG9w0BAQUFADBvMQswCQYDVQQGEwJTRTEU
      MBIGA1UEChMLQWRkVHJ1c3QgQUIxJjAkBgNVBAsTHUFkZFRydXN0IEV4dGVybmFs
      IFRUUCBOZXR3b3JrMSIwIAYDVQQDExlBZGRUcnVzdCBFeHRlcm5hbCBDQSBSb290
      MB4XDTAwMDUzMDEwNDgzOFoXDTIwMDUzMDEwNDgzOFowbzELMAkGA1UEBhMCU0Ux
      FDASBgNVBAoTC0FkZFRydXN0IEFCMSYwJAYDVQQLEx1BZGRUcnVzdCBFeHRlcm5h
      bCBUVFAgTmV0d29yazEiMCAGA1UEAxMZQWRkVHJ1c3QgRXh0ZXJuYWwgQ0EgUm9v
      dDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALf3GjPm8gAELTngTlvt
      H7xsD821+iO2zt6bETOXpClMfZOfvUq8k+0DGuOPz+VtUFrWlymUWoCwSXrbLpX9
      uMq/NzgtHj6RQa1wVsfwTz/oMp50ysiQVOnGXw94nZpAPA6sYapeFI+eh6FqUNzX
      mk6vBbOmcZSccbNQYArHE504B4YCqOmoaSYYkKtMsE8jqzpPhNjfzp/haW+710LX
      a0Tkx63ubUFfclpxCDezeWWkWaCUN/cALw3CknLa0Dhy2xSoRcRdKn23tNbE7qzN
      E0S3ySvdQwAl+mG5aWpYIxG3pzOPVnVZ9c0p10a3CitlttNCbxWyuHv77+ldU9U0
      WicCAwEAAaOB3DCB2TAdBgNVHQ4EFgQUrb2YejS0Jvf6xCZU7wO94CTLVBowCwYD
      VR0PBAQDAgEGMA8GA1UdEwEB/wQFMAMBAf8wgZkGA1UdIwSBkTCBjoAUrb2YejS0
      Jvf6xCZU7wO94CTLVBqhc6RxMG8xCzAJBgNVBAYTAlNFMRQwEgYDVQQKEwtBZGRU
      cnVzdCBBQjEmMCQGA1UECxMdQWRkVHJ1c3QgRXh0ZXJuYWwgVFRQIE5ldHdvcmsx
      IjAgBgNVBAMTGUFkZFRydXN0IEV4dGVybmFsIENBIFJvb3SCAQEwDQYJKoZIhvcN
      AQEFBQADggEBALCb4IUlwtYj4g+WBpKdQZic2YR5gdkeWxQHIzZlj7DYd7usQWxH
      YINRsPkyPef89iYTx4AWpb9a/IfPeHmJIZriTAcKhjW88t5RxNKWt9x+Tu5w/Rw5
      6wwCURQtjr0W4MHfRnXnJK3s9EK0hZNwEGe6nQY1ShjTK3rMUUKhemPR5ruhxSvC
      Nr4TDea9Y355e6cJDUCrat2PisP29owaQgVR1EX1n6diIWgVIEM8med8vSTYqZEX
      c4g/VhsxOBi0cQ+azcgOno4uG+GMmIPLHzHxREzGBHNJdmAPx/i9F4BrLunMTA5a
      mnkPIAou1Z5jJh5VkpTYghdae9C8x49OhgQwODAeBggrBgEFBQcDAwYIKwYBBQUH
      AwEGCCsGAQUFBwMEDBZBZGRUcnVzdCBFeHRlcm5hbCBSb290
      -----END TRUSTED CERTIFICATE-----
 
commands:
  01_enable_dynamic_cert_files_evaluation:
    command: "update-ca-trust enable"
  02_extract_certs:
    command: "update-ca-trust extract"

But for non-beanstalk servers: it's a matter of placing this cert in the blacklist directory & re-generating ca-trust new files.

@emmm-dee

This comment has been minimized.

Copy link

@emmm-dee emmm-dee commented Jun 1, 2020

Well this made for an interesting start of the week :(

@ValZapod

This comment has been minimized.

Copy link

@ValZapod ValZapod commented Jun 2, 2020

@minaguib

This comment has been minimized.

Copy link
Owner Author

@minaguib minaguib commented Jun 2, 2020

@ValZapod Thanks. I noted COMODO RSA Certification Authority in the "expired certificates" list.

I'm thinking now of ways to better represent the variations in INVALID and VALID chain paths...

@ValZapod

This comment has been minimized.

Copy link

@ValZapod ValZapod commented Jun 2, 2020

https://gitlab.com/gnutls/gnutls/-/issues/1008 Fixed.
curl/curl#5496 Wontfix as it is upstreams problem (openssl and gnutls)
openssl/openssl#12006 Was already fixed some time ago... Original commit in my comment there.

@ValZapod

This comment has been minimized.

Copy link

@ValZapod ValZapod commented Jun 2, 2020

These also expired https://crt.sh/?id=1721082 https://crt.sh/?id=1733294 https://crt.sh/?id=1726036 https://crt.sh/?id=12716435 https://crt.sh/?id=12716435 some of them have the same public key, LOL (like all USERTrust RSA Certification Authority). Very strange. Is this normal? I thought they change it. Looks like this is all.

@ValZapod

This comment has been minimized.

Copy link

@ValZapod ValZapod commented Jun 8, 2020

I will also add that this https://crt.sh/?id=1720081 has the same private and public key as COMODO RSA Certification Authority https://crt.sh/?id=1044348 but is valid until 2038. So it is not technically expired. Also it is root, not intermediate. See https://crt.sh/?spkisha256=82b5f84daf47a59c7ab521e4982aefa40a53406a3aec26039efa6b2e0e7244c1

And with the main cert even (!) https://crt.sh/?id=162879063 it has the same public key and still not expired!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.