NAT port 443 breaks SSL on webserver

zabu · Sat Apr 20, 2024 10:29 pm

Hello,
I’m just setting up my first mikrotik router CCR2004-1G-12S+2XS and…

TL;DR
If I forward port 443 to my webserver on the lan I can reach the server but I get a bad ssl error NET:ERR_CERT_COMMON_NAME_INVALID. If I click proceed anyway, I can load the address, so I’m guessing the NAT forwarding works.
This is my current conf (i've been trying random stuff for hours so may be a bit messy)

# 2024-04-20 20:38:36 by RouterOS 7.14.3
# software id = **ELIDED**
#
# model = CCR2004-1G-12S+2XS
# serial number = **ELIDED**
/interface bridge
add name=lan
add name=wan protocol-mode=none
/interface ethernet
set [ find default-name=sfp28-2 ] fec-mode=fec91
/ip pool
add name=dhcp_pool0 ranges=192.168.100.2-192.168.100.254
/ip dhcp-server
add address-pool=dhcp_pool0 interface=lan name=dhcp1
/ipv6 dhcp-server
add address-pool=v6pool interface=lan name=v6server
/port
set 0 name=serial0
set 1 name=serial1
/interface bridge port
add bridge=lan interface=sfp28-1
add bridge=wan interface=sfp28-2
add bridge=lan interface=sfp-sfpplus1
add bridge=lan interface=sfp-sfpplus2
add bridge=lan interface=sfp-sfpplus3
add bridge=lan interface=sfp-sfpplus4
add bridge=lan interface=sfp-sfpplus5
add bridge=lan interface=sfp-sfpplus6
add bridge=lan interface=sfp-sfpplus7
add bridge=lan interface=sfp-sfpplus8
add bridge=lan interface=sfp-sfpplus9
add bridge=lan interface=sfp-sfpplus10
add bridge=lan interface=sfp-sfpplus11
add bridge=lan interface=sfp-sfpplus12
add bridge=lan interface=ether1
/ipv6 settings
set accept-router-advertisements=yes
/ip address
add address=192.168.88.1/24 comment=defconf interface=ether1 network=\
    192.168.88.0
add address=192.168.100.1/24 interface=lan network=192.168.100.0
/ip dhcp-client
add interface=wan
/ip dhcp-server network
add address=192.168.100.0/24 gateway=192.168.100.1
/ip firewall filter
add action=accept chain=input connection-state=established,related
add action=accept chain=input protocol=icmp
add action=accept chain=input in-interface=lan
add action=drop chain=input
add action=fasttrack-connection chain=forward connection-state=\
    established,related hw-offload=yes
add action=accept chain=forward connection-state=established,related
add action=accept chain=forward out-interface=wan
add action=accept chain=forward connection-nat-state=dstnat
add action=drop chain=forward
/ip firewall nat
add action=masquerade chain=srcnat out-interface=wan
add action=masquerade chain=srcnat out-interface=lan src-address=\
    192.168.100.0/24
add action=dst-nat chain=dstnat dst-address-type=local dst-port=80,443 \
    in-interface=wan protocol=tcp to-addresses=192.168.100.2
/ip service
set telnet address=192.168.100.0/24 disabled=yes
set ftp address=192.168.100.0/24 disabled=yes
set www address=192.168.100.0/24 disabled=yes
set ssh address=192.168.100.0/24 port=65535
set www-ssl address=192.168.100.0/24
set api address=192.168.100.0/24 disabled=yes
set winbox address=192.168.100.0/24
set api-ssl address=192.168.100.0/24 disabled=yes
/ipv6 address
add from-pool=v6pool interface=lan
/ipv6 dhcp-client
add add-default-route=yes interface=wan pool-name=v6pool pool-prefix-length=\
    56 request=address,prefix use-interface-duid=yes
/ipv6 firewall filter
add action=accept chain=forward connection-state=established,related \
    in-interface=wan out-interface=lan
add action=drop chain=forward in-interface=wan out-interface=lan
/ipv6 nd
set [ find default=yes ] interface=lan managed-address-configuration=yes mtu=\
    1500 other-configuration=yes
/system clock
set time-zone-name=**ELIDED**
/system note
set show-at-login=no
/system package update
set channel=testing
/system routerboard settings
set enter-setup-on=delete-key

My setup
ISP <-> Mikrotik Router <-> pfSense <-> Web server.

Mikrotik router has ip 192.168.100.1
pfSense WAN has ip 192.168.100.2
The LAB subnet in pfSense is on 192.168.2.0/24
The web server is on 192.168.2.50

I have a letsencrypt wildcard cert setup for *.mydomain.com

The webserver is running in a proxmox vm with Nginx as reverse proxy, pointing several subdomain to internal services and vm’s.

I don’t have a static IP address, but I have my domain setup with cloudflare with an A record pointing to my public ipv4. In the webserver I have a cron job checking when my public ip changes and with the cloudflare API i update the A record to point it to the new IP.

This was working fine until I changed my ISP (init7, Switzerland) and I swapped the proprietary router from the previous ISP (Swisscom) with this Mikrotik.

In the previous router i would just forward 80 and 443 to pfSense and in pfSense i have NAT rules to forward to the Web server VM.

I tried the same setup but somehow the wildcard certificate is not working. I see there’s a certificate section in routerOs, so I’m guessing the router is trying to present a cert before forwarding? Dunno, that sounds dumb, but I’m out of ideas.

Ofc before bothering you nice folks I tried to search Google and this forum, but I guess port forwarding is normally confusing for newbies like me so it’s kinda hard to filter the results for my problem.

I found this viewtopic.php?t=114433 where the problem was mostly the ability to access the url from within the LAN (will need to sort that out too sooner or later, but the SSL is a bigger concern atm) & from where I tried the configuration posted by viewtopic.php?t=114433#p567262

And viewtopic.php?t=167632 where they said you can only make it work with by setting your static ip as dest in the NAT rule. I tried, even if that would not be a solution for me with a dynamic IP, but still got same SSL error.

Anyone can point me in the right direction?
Thanks and please don't flame me too hard if this was answered already somewhere else.

Sun Apr 21, 2024 6:06 am

NET:ERR_CERT_COMMON_NAME_INVALID

That means the name you put into your browser's address bar doesn't match the CN field of the certificate. Use your browser's certificate inspection tools to cross-check this.

This is my current conf (i've been trying random stuff for hours so may be a bit messy)

Yes, it is. Prepare for a thorough nit-picking.

/interface bridge add name=wan protocol-mode=none

This is a bad idea. The CCR2004 does not support two hardware-offloaded bridges, just one. If RouterOS happens to choose wrong and offload the WAN bridge, all LAN-side traffic will be forced through the CPU even when it could otherwise be bridged, as in a dumb Ethernet switch.

The default configuration was correct already for a configuration like this: the lone WAN-side interface shouldn't be part of any bridge. Note also that it solves the wish for a logical interface name by use of the interface list feature of RouterOS. If you don't like interface lists, you can rename the sfp28-2 port to "wan" instead.

/ip address add address=192.168.88.1/24 comment=defconf interface=ether1

This looks like an attempt to provide a "management port" for getting back into the router when something wrecks the config badly enough that your "lan" bridge won't respond. If so, you should remove ether1 from the bridge to make it truly independent. If not, this bit of "defconf" should be removed.

/ip firewall nat add action=masquerade chain=srcnat out-interface=lan src-address=192.168.100.0/24

Why in the world are you masquerading traffic out to the LAN?

The prior rule is all you need in normal use: masquerade traffic out to the WAN.

add action=dst-nat chain=dstnat dst-address-type=local dst-port=80,443…

I don't see what the dst-address-type bit is doing for you. Indeed, as I read your problem description, it appears outright wrong per the docs on this: the dst-address isn't explicitly assigned to the router itself. I don't see how you're getting the port-forward to work at all, TLS errors aside.

/ipv6 firewall filter …

You appear to have set up IPv6 on the WAN side, but you have removed nearly all of the default IPv6 firewall rules. In particular, there is no input filtering at all! This exposes your router's management services to anyone who can work out your /56 IPv6 address block, or who happens to stumble across it, as when port-scanning.

The web server is on 192.168.2.50…in a proxmox vm

Yes, on the other side of the pfSense firewall, but you're here pointing the finger at the MT firewall as the cause of the problem.

I'm not telling you you're wrong, but you're violating one of the laws of basic troubleshooting here. Put the VM into the DMZ between the two firewalls, then re-test. If it suddenly starts working properly, it never was the MT firewall; take it up on a pfSense forum. If it remains broken, then you have at least removed a point of confusion.

Regardless, I don't see why you'd want double firewalls on a public-facing web service. (Triple if you count whatever's running inside the web server VM.) Is it to be exposed, or not?

…and in pfSense i have NAT rules to forward to the Web server VM.

You have double NAT on top of all this? No wonder you're having difficulties; you bought and paid for a palette full of trouble, freight-delivered!

My read of your configuration before I came on this disclosure is that you were using straight routing past the pfSense, not NAT. You need NAT at the MT layer to map 192.168.100.0/24 addresses to the single public IPv4 address you get via DHCP, but that does not apply to your sanctum sanctorum 192.168.2.0/24 range.

I see there’s a certificate section in routerOs

Yes, but the typical use of that is minting local certs, not managing ones you get via Let's Encrypt.

You could set your shiny new CCR2004 up as a CA for your LAN's internal domain name (myhouse.lan, to pick an example) then install that CA root into all your local machines, allowing you to mint TLS and other certs under it which will be trusted, e.g. for www.myhouse.lan, pointing at the internal web server you're presently trying to get working for LAN-side TLS access. Or, since you have TLS WebFig enabled on this new router, set your local DNS up to point router.myhouse.lan back at this same router, then mint a TLS cert under that CA to allow TLS WebFig access from the LAN without certificate warnings.

I’m guessing the router is trying to present a cert before forwarding?

Without the dstnat port=443 rule, it would due to WebFig, but the dstnat chain takes precedence, routing traffic in from the WAN on that port to your Proxmox web server VM instead.

zabu · Sun Apr 21, 2024 12:11 pm

Thanks!
The nitpicking is very much appreciated. As you can imagine I'm very much clueless about networking so everything helps.

I actually thought I deleted this post since a few mins after creating it I realized that the *.mydomain.com cert was not including the simple 2nd level mydomain.com so I was getting a name mismatch. I felt dumber than usual, so I tried to delete the post.
Now I'm happy it didn't go through (maybe i hit the delete button while you were replying?) and I got some feedback on the messy conf.

This is my current conf btw

# 2024-04-21 10:32:31 by RouterOS 7.14.3
# software id = ******
#
# model = CCR2004-1G-12S+2XS
# serial number = *****
/interface bridge
add name=lan
add name=wan protocol-mode=none
/interface ethernet
set [ find default-name=sfp28-2 ] fec-mode=fec91
/ip pool
add name=dhcp_pool0 ranges=192.168.100.100-192.168.100.254
/ip dhcp-server
add address-pool=dhcp_pool0 interface=lan name=dhcp1
/ipv6 dhcp-server
add address-pool=v6pool interface=lan name=v6server
/port
set 0 name=serial0
set 1 name=serial1
/interface bridge port
add bridge=lan interface=sfp28-1
add bridge=wan interface=sfp28-2
add bridge=lan interface=sfp-sfpplus1
add bridge=lan interface=sfp-sfpplus2
add bridge=lan interface=sfp-sfpplus3
add bridge=lan interface=sfp-sfpplus4
add bridge=lan interface=sfp-sfpplus5
add bridge=lan interface=sfp-sfpplus6
add bridge=lan interface=sfp-sfpplus7
add bridge=lan interface=sfp-sfpplus8
add bridge=lan interface=sfp-sfpplus9
add bridge=lan interface=sfp-sfpplus10
add bridge=lan interface=sfp-sfpplus11
add bridge=lan interface=sfp-sfpplus12
add bridge=lan interface=ether1
/ipv6 settings
set accept-router-advertisements=yes
/ip address
add address=192.168.88.1/24 comment=defconf interface=ether1 network=\
    192.168.88.0
add address=192.168.100.1/24 interface=lan network=192.168.100.0
/ip dhcp-client
add interface=wan
/ip dhcp-server network
add address=192.168.100.0/24 gateway=192.168.100.1
/ip firewall filter
add action=accept chain=input connection-state=established,related
add action=accept chain=input protocol=icmp
add action=accept chain=input in-interface=lan
add action=drop chain=input
add action=accept chain=forward connection-state=established,related
add action=accept chain=forward out-interface=wan
add action=accept chain=forward connection-nat-state=dstnat
add action=fasttrack-connection chain=forward connection-state=\
    established,related hw-offload=yes
add action=drop chain=forward
/ip firewall nat
add action=masquerade chain=srcnat out-interface=wan
add action=masquerade chain=srcnat comment="hairpin nat" dst-address=\
    !192.168.100.1 src-address=192.168.100.0/24
add action=dst-nat chain=dstnat comment=WebServer dst-address=!192.168.100.1 \
    dst-address-type=local dst-port=80,443 protocol=tcp to-addresses=\
    192.168.100.2
add action=dst-nat chain=dstnat comment="Minecraft TCP" dst-address=\
    !192.168.100.1 dst-address-type=local dst-port=25565 protocol=tcp \
    to-addresses=192.168.100.2 to-ports=25565
add action=dst-nat chain=dstnat comment="Minecraft UDP" dst-address=\
    !192.168.100.1 dst-address-type=local dst-port=25565 protocol=udp \
    to-addresses=192.168.100.2 to-ports=25565
/ip route
add disabled=no distance=1 dst-address=192.168.2.0/24 gateway=192.168.100.2 \
    routing-table=main scope=30 suppress-hw-offload=no target-scope=10
/ip service
set telnet address=192.168.100.0/24 disabled=yes
set ftp address=192.168.100.0/24 disabled=yes
set www address=192.168.100.0/24 disabled=yes
set ssh address=192.168.100.0/24 port=65535
set www-ssl address=192.168.100.0/24
set api address=192.168.100.0/24 disabled=yes
set winbox address=192.168.100.0/24
set api-ssl address=192.168.100.0/24 disabled=yes
/ipv6 address
add from-pool=v6pool interface=lan
/ipv6 dhcp-client
add add-default-route=yes interface=wan pool-name=v6pool pool-prefix-length=\
    56 request=address,prefix use-interface-duid=yes
/ipv6 firewall filter
add action=accept chain=forward connection-state=established,related \
    in-interface=wan out-interface=lan
add action=drop chain=forward in-interface=wan out-interface=lan
/ipv6 nd
set [ find default=yes ] interface=lan managed-address-configuration=yes mtu=\
    1500 other-configuration=yes
/system clock
set time-zone-name=Europe/Zurich
/system health settings
set fan-min-speed-percent=5%
/system note
set show-at-login=no
/system package update
set channel=testing
/system routerboard settings
set enter-setup-on=delete-key

The base of this conf actually came from my ISP, they shared a configuration file here https://www.init7.net/de/support/router ... _fr_02.pdf (english starts on page3)

This is a bad idea. The CCR2004 does not support two hardware-offloaded bridges, just one. If RouterOS happens to choose wrong and offload the WAN bridge, all LAN-side traffic will be forced through the CPU even when it could otherwise be bridged, as in a dumb Ethernet switch.

Got it! I'll drop the WAN bridge. Actually I was about to make a post that all I get is 3 to 5 Gb on my 10G hardware, as I understand this may be a step into troubleshooting that?

You appear to have set up IPv6 on the WAN side, but you have removed nearly all of the default IPv6 firewall rules. In particular, there is no input filtering at all! This exposes your router's management services to anyone who can work out your /56 IPv6 address block, or who happens to stumble across it, as when port-scanning.

Oh great. That's what came from the ISP config, they provide a /48 fixed ipv6 subnet, which, as I'm sure you will be shocked to learn, I have no idea how to handle it. Ip6 is still rocket science for me, so I thought i should trust what they gave me. I need to start looking into it. Still basic firewalls rules should be on me, thanks for pointing this out.

You have double NAT on top of all this? No wonder you're having difficulties; you bought and paid for a palette full of trouble, freight-delivered!

Ye... I know. Tbh my previous ISP proprietary router was a super dumb machine, so it made sense to have a proper firewall solution for my lab subnet. Also, we're using pfsense at work (i'm a full stack developer) and my CTO & sysadmin said he'd like to offload some of his duties to me, so I wanted to get some practice with it on my home lab. Yes, as you can see one poor life choice after another, looks like i'll be a full stack-devops-sysAdmin unicorn.... what can go wrong. But I'd like to learn about networking, so I took the chance of a pay raise and someone that would teach me. We'll see how it goes.

Thanks for the help, I think the latest NAT conf is working, since i can access the webserver and the kids minecraft vm from the internet and from the lan. But i'm sure they're still a mess.

Sun Apr 21, 2024 12:40 pm

all I get is 3 to 5 Gb on my 10G hardware

You need to have everything dialed in to peg the meters on a 10G link with consumer-grade hardware. Disk, memory, packet sizes, drivers…everything. It's why it's taken so long for 10G to reach the consumer market; most PCs couldn't make decent use of it until recently, and then only at the high end.

My fastest results here are in the 6-7 Gbit/sec range to an RB4011 router running my iperf3 container, a CPU-based test, but I've red-lined a CRS328 in a pure switch chip test.

(9.4 Gbit/sec is 10 Gbit/sec minus packet overhead and such. It doesn't get a lot faster without tweaks like jumbo packets, which I didn't need to achieve the linked result.)

Ip6 is still rocket science for me

This guide of mine may be of some use to you, then.

I didn't link it earlier because it appeared you'd gotten the basics down, and it was written for a different ISP, but perhaps you can still squeeze some juice from it.

zabu · Sun Apr 21, 2024 1:02 pm

Awesome, I'll check it out.

I guess I'll mark this thread as solved since I can reach the webserver on 433 with my current configuration.

I'll check that default configuration link you posted earlier & try to sort out the mess in my conf and maybe post it in another thread with all my hardware to see if we could optimize more speed out of it.

Thanks for the help!

NAT port 443 breaks SSL on webserver [SOLVED]

NAT port 443 breaks SSL on webserver

Re: NAT port 443 breaks SSL on webserver

Re: NAT port 443 breaks SSL on webserver [SOLVED]

Re: NAT port 443 breaks SSL on webserver

Re: NAT port 443 breaks SSL on webserver

Who is online