urllib's CERTIFICATE_VERIFY_FAILED after installing newer OpenSSL in Ubuntu 20.04
Requests to our authorization providers started failing with
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)>. At the same time, when testing https calls with the
requests library, everything worked. It turns out the library Flask-Oauthlib uses
urllib.request.urlopen to talk a provider, and it looked like
urllib couldn’t get a hold of certificates, which made the request fail.
We recently updated from OpenSSL 1.1.1f to 1.1.1l, so it was natural to start there. After some digging through
openssl, I found the problem.
openssl is being installed, a value called
OPENSSLDIR can be configured. It specifies the default location for configuration and CA certificates. Ubuntu 20.04 comes preinstalled with OpenSSL 1.1.1f with
OPENSSLDIR set to
/usr/lib/ssl. This is the version we used until recently.
/usr/lib/ssl/certs is filled with symlinks for certificates residing in
/usr/share/ca-certificates/mozilla. However, after moving to OpenSSL 1.1.1l, we install it from source in our
Dockerfile, where we use the default
OPENSSLDIR which is
/usr/local/ssl. The folder
/usr/local/ssl/certs is empty. Which version of OpenSSL we are installing is not important, the difference is in the
OPENSSLDIR set in the pre-installed OpenSSL vs the default one when installing it yourself.
urllib expects certs to be under
OPENSSLDIR/certs so when it’s empty, using HTTPS fails.
Ways to solve this:
- Set the environment variable
/usr/lib/ssl/certsor other directory with the CA certificates. This overrides the default
- Add symlinks to the empty
/usr/local/ssl/certsfolder that point to
I initially thought a solution could be to use
usr/lib/ssl as the
--openssldir option when installing, however the docs do not recommend overwriting the preinstalled version location. The best solution might therefore be to use the
requests isn’t affected is because it uses the
certifi package which comes with it’s own certificate, which resides in its directory under Python’s
site-packages. You could therefore also point the environment variable
SSL_CERT_FILE (which also overrides OpenSSL’s default) to this certificate to make