If you’re like me, you've probably woken up on a Monday to find some of your services suddenly failing; perhaps it’s your mobile app failing to connect to your API or your images suddenly failing to load. In my case, some of our mobile apps (Android, specifically) were . After debugging for a bit I found that it was an SSL issue. reporting errors trying to retrieve data from our APIs Opening the URL in question in Chrome or making requests on iOS, however, showed no issues, and I was greeted to the normal green lock icon representing a normal certificate validation. My Issue Modern browsers and HTTP clients, like the iOS one, have better TLS trust verification strategies than older HTTP clients, like cURL and the Android built-in one. They will build a chain of trust to the root instead of stopping and failing at the first expired certificate in the chain. Chrome happily validates the certificate chain, recognizing the newer USERTrust Root Certificate. cURL, however, fails validation saying the certificate has expired. So what’s the problem and how do we debug further? Well to investigate SSL issues we can rely on good old OpenSSL. You can use OpenSSL to fetch all certificate data for a given server by running: # Hostname without protocol (no http/s)
openssl s_client -connect <hostname>: < > port The output you will receive should contain a section containing the full list of certificates in the chain: Certificate chain s: ### No is your own certificate, followed by all intermediate and root certificates s: O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
   i: O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root s: ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority
   i: O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root s: ST=DE/L=Wilmington/O=Corporation Service Company/CN=Trusted Secure Certificate Authority i: ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority 0 .0 1 /C=SE/ /C=SE/ 2 /C=US/ /C=SE/ 3 /C=US/ 5 /C=US/ Notice the in the 1st and 2nd positions in the certificate chain. These certificates are now expired and are causing errors in and the Android HTTP client. AddTrust certificate expiration cURL How do we mitigate this? The proper way to mitigate this is on the is to update the SSL configuration of the server or the reverse-proxy if using TLS-termination. server-side You can check your certificate chain; put the URL of your site into and it will spit out useful information about your certificate chain issues. You can even put in your (domain) and it will auto-remove the expired AddTrust certificates and produce a fixed .crt file you can put into your existing configuration as a drop-in replacement. https://whatsmychaincert.com hostname or you’re having issues with another platform and just want to fix your client until the server admin picks up the slack, you have a few options: If you’re not a server administrator On Linux-based client environments (Ubuntu and others), you can edit the file: ca-certificates.conf /etc/ca-certificates.conf There you will find 2 references to ; a quick fix is to just prepend! Before each line, run update-ca-certificates to apply the changes afterward. AddTrust On , a simple way to fix this for me was to edit /etc/ssl/cert.pem, find the certificate, and completely remove it. macOS AddTrust Make sure you only delete lines between and including: ### AddTrust AB
-----END CERTIFICATE----- This should fix your issues; the cURL and other HTTP clients should now be happy to resolve the new properly. USERTrust Root Certificate Hope you found this helpful and I saved you some headache, cheers! Previously published at https://blog.codechem.com/fixing-the-addtrust-external-ca-root-expiration

Chain

CodeChem

Fetch

Slack

2022 - HackerNoon Contributor of the Year - Algorithms

2022 - HackerNoon Contributor of the Year - Mvp

Work with us!

Nominated for 2022 - HackerNoon Contributor of the Year - Algorithms

Nominated for 2022 - HackerNoon Contributor of the Year - Mvp

Too Long; Didn't Read

Questions about cloud security? Get answers here.