If you’re like me, you've probably woken up on a Monday to find some of your services suddenly failing; perhaps it’s your mobile app failing to connect to your API or your images suddenly failing to load. In my case, some of our mobile apps (Android, specifically) were reporting errors trying to retrieve data from our APIs. After debugging for a bit I found that it was an SSL issue.
Opening the URL in question in Chrome or making requests on iOS, however, showed no issues, and I was greeted to the normal green lock icon representing a normal certificate validation.
My Issue
Modern browsers and HTTP clients, like the iOS one, have better TLS trust verification strategies than older HTTP clients, like cURL and the Android built-in one. They will build a chain of trust to the root instead of stopping and failing at the first expired certificate in the chain.
Chrome happily validates the certificate chain, recognizing the newer USERTrust Root Certificate.
cURL, however, fails validation saying the certificate has expired.
So what’s the problem and how do we debug further?
Well to investigate SSL issues we can rely on good old OpenSSL. You can use OpenSSL to fetch all certificate data for a given server by running:
# Hostname without protocol (no http/s)
openssl s_client -connect <hostname>:<port>
The output you will receive should contain a section containing the full list of certificates in the chain:
Certificate chain
0 s: ### No.0 is your own certificate, followed by all intermediate and root certificates
1 s:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
2 s:/C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority
i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
3 s:/C=US/ST=DE/L=Wilmington/O=Corporation Service Company/CN=Trusted Secure Certificate Authority 5
i:/C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority
Notice the AddTrust in the 1st and 2nd positions in the certificate chain. These certificates are now expired and are causing certificate expiration errors in cURL and the Android HTTP client.
How do we mitigate this?
The proper way to mitigate this is on the server-side is to update the SSL configuration of the server or the reverse-proxy if using TLS-termination.
You can check your certificate chain; put the URL of your site into https://whatsmychaincert.com and it will spit out useful information about your certificate chain issues. You can even put in your hostname (domain) and it will auto-remove the expired AddTrust certificates and produce a fixed .crt file you can put into your existing configuration as a drop-in replacement.
If you’re not a server administrator or you’re having issues with another platform and just want to fix your client until the server admin picks up the slack, you have a few options:
On Linux-based client environments (Ubuntu and others), you can edit the ca-certificates.conf file:
/etc/ca-certificates.conf
There you will find 2 references to AddTrust; a quick fix is to just prepend! Before each line, run update-ca-certificates to apply the changes afterward.
On macOS, a simple way to fix this for me was to edit /etc/ssl/cert.pem, find the AddTrust certificate, and completely remove it.
Make sure you only delete lines between and including:
### AddTrust AB
-----END CERTIFICATE-----
This should fix your issues; the cURL and other HTTP clients should now be happy to resolve the new USERTrust Root Certificate properly.
Hope you found this helpful and I saved you some headache, cheers!
Previously published at https://blog.codechem.com/fixing-the-addtrust-external-ca-root-expiration