SSL certificate issue - wildcard Let's Encrypt

RouterOS General
1

I’m using a wildcard Let’s Encrypt cert, and I have a ROS script that imports the chain.pem, cert.pem & privkey.pem files and restarts the www-ssl service setting cert.pem to it.

I just noticed, that one of my python scripts, that uses the requests library is unable to reach the REST API:

Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)')))

The script worked until recently without any issues.

I started digging and noticed this, the 10.0.0.1 is the mikrotik:

# openssl s_client -connect 10.0.0.1:443 -servername 10.0.0.1 </dev/null | openssl x509 -noout -issuer -subject
depth=0 CN = *.MYDOMAIN.net
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = *.MYDOMAIN.net
verify error:num=21:unable to verify the first certificate
verify return:1
depth=0 CN = *.MYDOMAIN.net
verify return:1
DONE
issuer=C = US, O = Let's Encrypt, CN = R11
subject=CN = *.MYDOMAIN.net

And here’s how it looks for other service that uses the same exact wildcard certificate, I believe it’s runnng nginx under the hood:

# openssl s_client -connect 10.0.0.201:443 -servername 10.0.0.201 </dev/null | openssl x509 -noout -issuer -subject
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R11
verify return:1
depth=0 CN = *.MYDOMAIN.net
verify return:1
DONE
issuer=C = US, O = Let's Encrypt, CN = R11
subject=CN = *.MYDOMAIN.net

Notice, that the mikrotik doesn’t appear to serve the whole chain.

This issue affects the following factory-ROS7 devices:

  • S53UG+M-5HaxD2HaxD (Chateau 5G ax), running 7.8
  • CRS310-8G+2S+, running 7.16.1
  • C52iG-5HaxD2HaxD (hap ax2), running 7.13.4

However, I also have the good old hap ac2 (RBD52G-5HacD2HnD) which runs factory-ROS6, (but is upgraded to 7.16.1). It does not present this problem:

# openssl s_client -connect 10.0.0.10:443 -servername 10.0.0.10 </dev/null | openssl x509 -noout -issuer -subject
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R11
verify return:1
depth=0 CN = *.MYDOMAIN.net
verify return:1
DONE
issuer=C = US, O = Let's Encrypt, CN = R11
subject=CN = *.MYDOMAIN.net

Following the “REST API Certificate - FullChain.pem?” topic I imported the isrgrootx1.pem & lets-encrypt-r3.pem. I also tried uploading the fullchain.pem from LE and using it on www-ssl but it did not help.

How can I solve this? Why only now did this problem started to appear?

2

It’s using the R11 certificate, not same as other thread.

Consult the LE web page: https://letsencrypt.org/certificates/

3

Huh, I don’t get it.

I downloaded https://letsencrypt.org/certs/2024/r11.pem - tried to import it, but nothing happened, as the R11 was already imported from the chain.pem file produced by LE renewal process.

I removed it and added if again from the aforementioned link.

Now it works.

I don’t get it… And I still don’t get why on ROS6 it was never an issue :thinking:

Edit: i just compared, the chain.pem from LE is the same exact file as the r11.pem. Looks like there’s a bug in the certificate handling/import process on Mikrotik.

Here’s the script I’m using for the SSL update process. The /mnt/sslprovisioner/ssl/MYDOMAIN.net/ directory is the Let’s Encrypt output dir.

/system script add dont-require-permissions=no name=update_ssl_certs owner=admin policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon \
  source={
        :local current [/certificate/get MYDOMAIN.net_cert invalid-after]
        :put "Downloading certs..."
        /system ssh-exec address=sslprovisioner.MYDOMAIN.net user=sslprovisioner command="cat /mnt/sslprovisioner/ssl/MYDOMAIN.net/cert.pem" output-to-file=cert.pem
        :delay 2
        /system ssh-exec address=sslprovisioner.MYDOMAIN.net user=sslprovisioner command="cat /mnt/sslprovisioner/ssl/MYDOMAIN.net/chain.pem" output-to-file=chain.pem
        :delay 2
        /system ssh-exec address=sslprovisioner.MYDOMAIN.net user=sslprovisioner command="cat /mnt/sslprovisioner/ssl/MYDOMAIN.net/fullchain.pem" output-to-file=fullchain.pem
        :delay 2
        /system ssh-exec address=sslprovisioner.MYDOMAIN.net user=sslprovisioner command="cat /mnt/sslprovisioner/ssl/MYDOMAIN.net/privkey.pem" output-to-file=privkey.pem
        :delay 2
        :put "Importing certs..."
        /certificate import file-name=chain.pem name="MYDOMAIN.net_chain" passphrase=""
        :delay 2
        /certificate import file-name=fullchain.pem name="MYDOMAIN.net_fullchain" passphrase=""
        :delay 2
        /certificate import file-name=cert.pem name="MYDOMAIN.net_cert" passphrase=""
        :delay 2
        /certificate import file-name=privkey.pem name="MYDOMAIN.net_privkey" passphrase=""
        :delay 2
        :local new [/certificate/get MYDOMAIN.net_fullchain invalid-after]
        :put "Checking if they are new..."
        if ($new != $current) do={
            :delay 2
            /ip service set www-ssl certificate=none disabled=yes
            :delay 2
            /ip service set api-ssl certificate=none disabled=yes
            :delay 2
            /ip service set www-ssl address=10.0.0.0/16 certificate=MYDOMAIN.net_cert disabled=no
            :delay 2
            /ip service set api-ssl address=10.0.0.0/16 certificate=MYDOMAIN.net_cert disabled=no port=8729
            :delay 2
            :log info "SSL Certificates updated!"
            :put "SSL Certificates updated!"
            /tool e-mail send to=shalak@MYDOMAIN.net subject="SSL updated. Valid till $[/certificate/get MYDOMAIN.net_fullchain invalid-after]" body="SSL updated at $[/system clock get date]. Old cert was valid till $current. New cert is valid till $[/certificate/get MYDOMAIN.net_fullchain invalid-after ]" tls=starttls
        } else={
          :log info "No new SSL certificates, doing nothing..."
          :put "No new SSL certificates, doing nothing..."
        }
      }

I recall that without the delays I had problems with this process… Maybe there’s some race condition there?

4

A race condition seems possible. IMO, it’s kinda already a “bug” that you need any delay. Commands should be synchronous. But… I do know that :delay is in sometime needed in /certificate. So maybe you’re right and a long delay might be needed, but 2 seconds seems like a long time already…so dunno.