Requests to our authorization providers started failing with urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)>. At the same time, when testing https calls with the requests library, everything worked. It turns out the library Flask-Oauthlib uses urllib.request.urlopen to talk a provider, and it looked like urllib couldn’t get a hold of certificates, which made the request fail.

We recently updated from OpenSSL 1.1.1f to 1.1.1l, so it was natural to start there. After some digging through cpython and openssl, I found the problem.

When openssl is being installed, a value called OPENSSLDIR can be configured. It specifies the default location for configuration and CA certificates. Ubuntu 20.04 comes preinstalled with OpenSSL 1.1.1f with OPENSSLDIR set to /usr/lib/ssl. This is the version we used until recently. /usr/lib/ssl/certs is filled with symlinks for certificates residing in /usr/share/ca-certificates/mozilla. However, after moving to OpenSSL 1.1.1l, we install it from source in our Dockerfile, where we use the default OPENSSLDIR which is /usr/local/ssl. The folder /usr/local/ssl/certs is empty. Which version of OpenSSL we are installing is not important, the difference is in the OPENSSLDIR set in the pre-installed OpenSSL vs the default one when installing it yourself.

urllib expects certs to be under OPENSSLDIR/certs so when it’s empty, using HTTPS fails.

Ways to solve this:

  1. Set the environment variable SSL_CERT_DIR to /usr/lib/ssl/certs or other directory with the CA certificates. This overrides the default OPENSSLDIR/certs value.
  2. Add symlinks to the empty /usr/local/ssl/certs folder that point to /usr/share/ca-certificates/mozilla.

I initially thought a solution could be to use usr/lib/ssl as the --openssldir option when installing, however the docs do not recommend overwriting the preinstalled version location. The best solution might therefore be to use the SSL_CERT_DIR variable.

The reason requests isn’t affected is because it uses the certifi package which comes with it’s own certificate, which resides in its directory under Python’s site-packages. You could therefore also point the environment variable SSL_CERT_FILE (which also overrides OpenSSL’s default) to this certificate to make urllib work.