Troubleshooting SSL Errors
How to check the details of an SSL certificate
If you're not sure if the certificate you're using is new, old, or what info is in it, you can use the "openssl" command with the 509 option to get you more info on a certificate, e.g.,
openssl x509 -in -text -noout
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 0 (0x0)
Signature Algorithm: md5WithRSAEncryption
Issuer: C=US, ST=Someprovince, L=Sometown, O=none, OU=none, CN=/emailAddress=webaster@localhost
Validity
Not Before: Jul 7 05:58:09 2009 GMT
Not After : Nov 21 05:58:09 2036 GMT
Subject: C=US, ST=Someprovince, L=Sometown, O=none, OU=none, CN=/emailAddress=webaster@localhost
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public Key: (1024 bit)
Modulus (1024 bit):
00:98:32:24:b9:13:30:7b:97:6f:dc:35:c2:a0:e8:
58:de:9f:88:d2:b0:86:1c:b9:67:1f:7d:b1:28:6c:
b8:52:47:f0:14:32:4c:36:62:14:a7:bf:d1:d1:47:
ed:bc:63:60:c7:ee:3c:22:58:69:4f:b4:93:8c:3e:
f5:02:54:85:70:24:c6:3e:b1:67:90:50:ab:18:64:
60:b2:46:e1:19:d3:b7:34:7b:48:c6:6b:bb:da:56:
54:92:c0:2e:39:81:f7:94:af:3d:51:2c:cf:4a:86:
9e:0c:30:60:82:6b:34:f6:9b:e4:01:9c:39:1b:4f:
68:61:fa:5f:7e:cc:f6:e0:ed
Exponent: 65537 (0x10001)
Signature Algorithm: md5WithRSAEncryption
20:86:28:65:d5:fe:d2:a9:0c:f9:c3:24:51:ff:de:ff:11:6e:
ec:b4:dc:b1:11:70:cb:37:52:45:e7:cb:f3:e9:e1:b3:3a:5d:
20:8b:49:07:43:a4:da:b8:17:bf:4b:84:89:d7:0e:18:34:b3:
0c:c2:0c:cd:56:2b:9e:e1:e0:6a:c4:7d:fc:3f:3e:89:8a:a5:
c9:a5:1a:1c:2c:7d:74:1c:34:54:b1:8b:36:2d:91:b6:38:79:
ad:e5:14:53:d2:32:ce:ee:89:ef:60:f3:98:0f:38:79:e3:07:
f5:e5:a5:ff:8f:6b:74:46:4c:89:7e:33:0e:66:cf:34:3b:c1:
e3:79
Similarly, you can manually connect to a remote host (e.g., )
to check if a certificate is self-signed or not:
openssl s_client -servername -host -port 443 | grep 'Verify return code'
where the output might look like:
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
verify return:1
depth=0 CN =
verify return:1
Verify return code: 0 (ok)
QUIT
DONE
We'd take note of the Verify return code: 0 (ok) status. The "code: 0" is good, as it means the certificate is correctly signed by a certificate authority.
I usually type "QUIT" to exit the current connection (this is a client connection to Apache, so you could create a request here). Else some systems might need Ctrl-D to send an "end-of-file" character to close it.
If you see:
Verify return code: 18 (self signed certificate)
then it means the certificate could be self-signed (you created the cert/key yourself), or the ca root bundle or chain have not been correctly installed.
How to check the details of a certificate request
If you have a certificate request, and want to see what information is set inside, you can use the "openssl" command with the "req" option to get you more info on the CSR.
openssl req -in /path/to/your/certificate.request -text -noout
Certificate Request:
Data:
Version: 0 (0x0)
Subject: C=CA, ST=Alberta, L=St. Albert, O=JBMC-Software, CN=www.testdomain.com/emailAddress=my@email.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public Key: (2048 bit)
Modulus (2048 bit):
00:c5:bb:20:60:a2:bd:c1:aa:10:76:f4:d5:46:73:
e5:c5:b1:f8:a5:ae:f8:f0:70:02:1e:52:fb:ef:e3:
d5:8b:75:d0:b8:d5:72:ad:63:11:10:4a:99:b4:41:
ab:06:91:6a:5e:68:3b:5e:e2:c3:cb:d6:51:19:cb:
25:5a:40:05:fe:13:1b:f0:87:49:0e:95:04:d5:81:
bb:7c:07:1b:3c:69:08:9e:ad:ef:9e:1e:97:a0:5e:
f5:66:c2:e0:f5:65:2d:6f:cb:eb:45:4d:f6:3b:e0:
33:43:78:48:30:14:5b:40:99:e0:e3:14:7e:46:5b:
11:d0:d4:d9:6d:bf:61:ec:02:a0:96:d5:f8:0c:30:
ae:ce:88:64:88:01:a1:83:03:90:68:67:16:2d:77:
6c:41:ca:c4:8a:42:1d:b6:1a:a2:c2:06:20:31:89:
36:ae:6a:03:9f:e6:4f:cc:97:a4:24:f4:a5:c6:c2:
c6:be:d7:73:2f:a6:56:89:28:83:e1:be:cf:b7:57:
70:66:e5:7f:28:ac:2b:e2:3e:1e:a8:79:4c:64:db:
d7:a2:9c:20:5c:b2:cb:92:e5:4d:07:45:b2:4c:9a:
a6:01:ea:49:88:f4:c6:fc:d0:45:52:92:39:32:3e:
93:54:6a:d8:3d:c4:42:d1:30:a3:45:89:86:9c:c8:
12:63
Exponent: 65537 (0x10001)
Attributes:
challengePassword :A challenge password
Signature Algorithm: md5WithRSAEncryption
a8:b2:06:3d:a0:56:86:72:53:9a:8b:9a:b3:1a:9f:4f:6f:27:
a2:d7:25:28:63:3b:39:3c:6d:f6:e0:66:43:1d:a6:1f:fc:2c:
af:af:11:04:80:5d:4a:29:03:1f:24:8e:c2:3e:65:36:75:82:
1e:6e:12:aa:68:fd:d7:8d:d5:8d:9c:12:74:6c:5e:8a:d1:af:
a4:da:12:6e:fa:df:fe:cf:9c:1d:22:cd:8c:91:2a:1e:67:1b:
bf:2f:95:7c:b2:4e:7d:e0:9b:41:d3:c5:30:e7:5e:04:91:e6:
95:0e:8d:da:4c:cd:86:ac:37:c7:95:d5:3a:0a:9f:5e:d8:1e:
9b:ea:27:11:7e:03:ff:b3:1a:66:da:ab:82:d2:ad:ad:8d:cd:
b4:cb:67:b0:99:63:ae:e1:b9:8d:13:26:b1:df:84:85:72:61:
a4:4f:82:9b:86:10:37:57:bc:d4:d8:6b:8b:f7:12:27:a3:ef:
b3:c0:4b:20:8d:bb:0d:18:64:f8:38:06:69:31:0c:2d:2d:86:
39:a1:98:26:7c:a7:02:98:e1:35:d8:ee:ca:a8:fe:80:29:d5:
5c:6f:55:d0:b5:12:9e:31:86:ed:8f:c2:1c:ff:ec:3a:9b:3b:
2c:c6:80:c2:1e:c7:a5:de:a0:ea:11:ef:f3:c7:df:74:8e:5e:
a4:b6:50:80
[root@server]#
DirectAdmin is not accepting my encrypted SSL key
If you have an RSA PRIVATE KEY with headers that look something like:
-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: DES-EDE3-CBC,D061ACF7F7A7D27F
then you would have a password encrypted key.
This will not work with DirectAdmin since it would require someone to type in a password each time Apache restarts.
You do not have to start over unless you've forgotten the password. You can actually convert this password protected key into a normal key by typing:
openssl rsa -in /path/to/your/key -out /the/new/key
where you'd replace the path values with appropriate values.
This will ask you for the password you specified when you created the original key.
It will then create a non-password-protected key that you can use for DA.
The LetsEncrypt DST Root CA X3 Expiration, Fixes, and ZeroSSL Support
Albeit a rare occurrence, CA root certificates do occasionally expire, and this can cause a multitude of SSL errors for clients using older OpenSSL versions and/or outdated ca-certificate installations.
The Public Key Infrastructure (PKI) facilitates SSL and public key encryption management, providing CIA (confidentiality, integrity, and authenticity) trust services. It includes a Truststore, which is a locally stored list of all 'trusted' root CA self-signed certificates stored directly on your OS. This establishes explicit trust for the root CAs listed.
Root CA certificates are necessary in validating a Chain of Trust in order to verify the authenticity of an organization claiming to own the domain for which the SSL is implemented. A trusted Registration Authority (RA) will create their own self-signed root CA certificate (which is trusted explicitly via your truststore), then, in order to guard it as much as possible, will store it offline and create/sign intermediate SSLs (sub CA certificates) with it (these must be verified). These intermediate SSLs are then permitted to sign your domain's SSLs on behalf of the root CA. If any of the SSLs in this Chain of Trust are broken, SSL errors may result.
So, what happens when a root CA certificate expires and how can we rectify the errors that result?
LetsEncrypt and ISRG
LetsEncrypt started out using a known and trusted CA, IdenTrust, to cross-sign their root certificate until their own ISRG root certificate became widely trusted. Now that it is, the existing root CA is expired. However, existing SSLs that were signed with this now expired root CA may show SSLs with different utilities if the computer you are connecting from has the expired root CA still listed in its truststore.
An example of this is recent connections via CustomBuild to DirectAdmin mirrors that have valid SSLs, but the chain includes the expired root certificate, so SSL errors like this excerpt from a CustomBuild run may result:
ERROR: cannot verify files-ca.directadmin.com's certificate, issued by '/C=US/O=Let\'s Encrypt/CN=R3':
Issued certificate has expired.
To connect to files-ca.directadmin.com insecurely, use `--no-check-certificate'.
SSLLABS.com/ssltest against the mirror shows this:
In trust store DST Root CA X3 Self-signed
Fingerprint SHA256: 0687260331a72403d909f105e69bcf0d32e1bd2493ffc6d9206d11bcd6770739
Pin SHA256: Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys=
RSA 2048 bits (e 65537) / SHA1withRSA
Valid until: Thu, 30 Sep 2021 14:01:15 UTC
EXPIRED
Weak or insecure signature, but no impact on root certificate
The cross-signed SSL shouldn't be a problem if the expired root SSL is removed from the TrustStore, or if OpenSSL v1.1.0 or later is in use rather than OpenSSL v1.0.2.
OpenSSL v1.1.0 has improved certificate chaining so that trusted store certificates are preferred over untrusted certificates, thus allowing a working certificate chain to be built.
So, if you're using an older version of OpenSSL, you have the following options to rectify this issue for connecting to domains that use the expired root certificate in their chain:
- Upgrade OpenSSL to v1.1.0 or later
- Upgrade the package ca-certificates
- Blacklist the expired root certificate in your TrustStore
If these errors are occurring on your own site, you can:
- Use --preferred-chain in ACME LetsEncrypt libraries, such as is used with DA's own
letsencrypt.shscript now, to specify the preferred chain to be built using the valid ISRG root certificate. - Use an alternate provider to issue your SSLs, but know that they, too, will have an expired root certificate one day, as this issue is not specific to LetsEncrypt only (recall the Comodo/Sectigo AddTrust External CA Expiration of May 2020?).
How to Upgrade OpenSSL
These upgrades would be handled by your OS. Reference your OS vendor's documentation for how to proceed here.
For example, for CentOS, Cloudlinux and other RHEL servers, you can use YUM or DNF to run updates: yum update or dnf update
Alternatively, you can update specific packages, via a package manager like dnf, or by compiling from source. Again, reference your OS vendor's documentation for instructions.
How to Upgrade the ca-certificates package
This, too, is vendor specific.
Check if your truststore contains the expired LetsEncrypt DST CA Root X3 certificate:
trust list | grep -A2 -B2 'X3'
If yes and if you're using CentOS or Cloudlinux, run this:
yum update ca-certificates
Then check that the expired certificate has been removed:
rpm -q -changelog ca-certificates | head -5
Output should be as follows:
[root@host4 ~]# rpm -q -changelog ca-certificates | head -5
* Tue Sep 14 2021 Bob Relyea <rrelyea@redhat.com> - 2021.2.50-72
- Fix expired certificate.
- Removing:
- # Certificate "DST Root CA X3"
[root@host4 ~]#
And wget/curl commands used by CustomBuild should now succeed:
[root@host4 ~]# wget https://files-ca.directadmin.com/services/custombuild/2.0/custombuild.tar.gz
--2021-11-11 09:17:11-- https://files-ca.directadmin.com/services/custombuild/2.0/custombuild.tar.gz
Resolving files-ca.directadmin.com (files-ca.directadmin.com)... 51.161.40.239, 2607:5300:60:1eb6::16
Connecting to files-ca.directadmin.com (files-ca.directadmin.com)|51.161.40.239|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 265985 (260K) [application/octet-stream]
Saving to: ‘custombuild.tar.gz.1’
100%[================================================================>] 265,985 --.-K/s in 0.08s
2021-11-11 09:17:11 (3.05 MB/s) - ‘custombuild.tar.gz.1’ saved [265985/265985]
[root@host4 ~]#
If using Debian:sudo apt-get --only-upgrade install ca-certificates orsudo apt-get install --reinstall ca-certificates
How to blacklist the expired certificate in your TrustStore
First, get the pkics11 id:
trust list | grep -A2 -B2 'X3'
Example:
[root@host4 ~]# trust list | grep -A2 -B2 'X3'
pkcs11:id=%c4%a7%b1%a4%7b%2c%71%fa%db%e1%4b%90%75%ff%c4%15%60%85%89%10;type=cert
type: certificate
label: DST Root CA X3
trust: anchor
category: authority
[root@host4 ~]#
Since this is a read-only certificate, it cannot be removed with the trust anchor --remove command:
[root@host4 ~]# trust anchor --remove "pkcs11:id=%c4%a7%b1%a4%7b%2c%71%fa%db%e1%4b%90%75%ff%c4%15%60%85%89%10;type=cert"
p11-kit: couldn't remove read-only certificate
[root@host4 ~]#
Instead, dump the certificate into a file so that it will be blacklisted in your TrustStore (be sure to confirm the appropriate path for your OS):
trust dump --filter "pkcs11:id=%c4%a7%b1%a4%7b%2c%71%fa%db%e1%4b%90%75%ff%c4%15%60%85%89%10;type=cert" > /etc/pki/ca-trust/source/blacklist/DST-root-CA-X3.p11-kit
Then, update the ca trust store:
update-ca-trust
Output expected will be similar to the following:
[root@host3 ~]# update-ca-trust
p11-kit: overriding trust for anchor in blacklist: DST-root-CA-X3.p11-kit
p11-kit: overriding trust for anchor in blacklist: DST-root-CA-X3.p11-kit
p11-kit: overriding trust for anchor in blacklist: DST-root-CA-X3.p11-kit
p11-kit: overriding trust for anchor in blacklist: DST-root-CA-X3.p11-kit
p11-kit: overriding trust for anchor in blacklist: DST-root-CA-X3.p11-kit
[root@host3 ~]#
Now, upon checking your TrustStore, you'll see that the certificate is blacklisted:
[root@host3 ~]# trust list | grep -A2 -B2 'X3'
p11-kit: overriding trust for anchor in blacklist: DST-root-CA-X3.p11-kit
pkcs11:id=%c4%a7%b1%a4%7b%2c%71%fa%db%e1%4b%90%75%ff%c4%15%60%85%89%10;type=cert
type: certificate
label: DST Root CA X3
trust: blacklisted
category: authority
[root@host3 ~]#
And the errors are gone:
[root@host3 ~]# wget https://files-ca.directadmin.com/services/custombuild/2.0/custombuild.tar.gz
--2021-11-11 03:24:49-- https://files-ca.directadmin.com/services/custombuild/2.0/custombuild.tar.gz
Resolving files-ca.directadmin.com (files-ca.directadmin.com)... 51.161.40.239, 2607:5300:60:1eb6::16
Connecting to files-ca.directadmin.com (files-ca.directadmin.com)|51.161.40.239|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 265985 (260K) [application/octet-stream]
Saving to: ‘custombuild.tar.gz’
100%[==================================>] 265,985 --.-K/s in 0.08s
2021-11-11 03:24:50 (2.99 MB/s) - ‘custombuild.tar.gz’ saved [265985/265985]
[root@host3 ~]#
How to use --preferred-chain with letsencrypt.sh
This allows you to specify which chain will take precedence, so that OpenSSL v1.0.2 won't end up using the untrusted, expired certificate for the chain. It will instead use the certificate you specify. This is done for you automatically via the letsencrypt.sh script, which sets the preferred-chain to "ISRG Root X1".
How to use an alternate provider- ZeroSSL
From the forum:
We have prepared an early tech preview of a ZeroSSL support. For this to work we released a new lego version (capable of issuing ZeroSSL certificates) and new letsencrypt version v2.0.24. Participation in this early preview is controlled by the existence of /root/.zerossl file. If this file is present DA will try issuing cert from ZeroSSL, if this file is absent everything will work as before (certs issued from LetsEncrypt).
If you would like to try it out please do the following:
da config-set default_acme_provider zerossl
/usr/local/directadmin/scripts/letsencrypt.sh request server.name.example.net
The following command can be used to confirm that a ZeroSSL certificate is now in use:
openssl s_client -connect example.net:2222 -servername example.net