Community discussions

MikroTik App
 
robkampen
newbie
Topic Author
Posts: 32
Joined: Mon Aug 05, 2019 10:44 pm

HTTPS server access from LAN fails, from WAN it works.

Wed Jan 04, 2023 11:06 am

Hi,
We have a number of webservers working in our network - they are on a separate subnet from the local LAN and Wifi users.
Accessing these websites from the WAN / Interent works fine with Qualsys SSL Labs giving our implementation on both IP4 and IPv6 an A+ rating.
When I try to load a page from a device connected internally to our LAN / Wifi / IOT subnets we get errors stating that the browser / server could not find a suitable protocol to establish service.
After many hours / much reading etc I have found that trying a curl request to the site shows what appears to be happening (or failing to happen)
I attach the file curl_tls_handshake.txt to show what happens from the WAN when all works as expected.
Then I attach curl_tls_handshake_broken.txt and one can see that initial contact is made but the reply fails. All my reading on these errors deal with server and client errors, which is not the problem here - it is a routing problem - NAT probably, although dstnat must be fine for the WAN connections to function.
So I guess a source nat issue but now I'm out of my depth.
I have revised my entire network set up over the last month to use what others on this forum recommend as best practice - to no avail.
A suitably scrubbed export file is also attached.
Any insights appreciated
TIA Rob
You do not have the required permissions to view the files attached to this post.
 
erlinden
Forum Guru
Forum Guru
Posts: 1920
Joined: Wed Jun 12, 2013 1:59 pm
Location: Netherlands

Re: HTTPS server access from LAN fails, from WAN it works.

Wed Jan 04, 2023 11:13 am

How is the url resolved? If with public IP address you have to have Hairpin NAT in place:
https://help.mikrotik.com/docs/display/ ... HairpinNAT
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 19106
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: HTTPS server access from LAN fails, from WAN it works.

Wed Jan 04, 2023 2:30 pm

erlinden, WRONG~ if the server is in a different subnet then the users, why is hairpin NAT required??

I must say I despise bloated youtube driven firewalls, and this one ranks way way up there.....
You know the person has no clue about firewalls and just copies crap to ones config with this line........

add action=accept chain=input comment="Reach clients behind NAT from WAN" connection-nat-state=srcnat,dstnat connection-state=new in-interface-list=WAN

Two issues,
a. WRONG chain
b. FORMATTED to defeat success

I would hate to imagine what else is fubared.

I do see the rule also stated in forward chain
add action=drop chain=forward comment="Reach clients behind NAT from WAN" connection-nat-state=!dstnat connection-state=new in-interface-list=WAN log=yes log-prefix="from WAN"

Needs to be modified to
add action=accept chain=forward comment="allow port forwarding" connection-nat-state=dstnat

As for the rest to complicated to fathom.
 
robkampen
newbie
Topic Author
Posts: 32
Joined: Mon Aug 05, 2019 10:44 pm

Re: HTTPS server access from LAN fails, from WAN it works.

Thu Jan 05, 2023 2:56 am

@erlinden
sorry forgot to mention that we use outside public DNS service as Mikrotik DNS is not up to the task and I don't want to run a linux/bind server internally. Used to do that a decade ago - lots of work not much gain. Thus all DNS is resolved by public NS and we need to deal with public IP addresses.
Also, I believe from reading forum and mikrotik documents that if the servers are on a separate subnet that no hairpin is needed. Added to that, trying to hairpin with four public IP addresses terminating on the router is just too mind-bendingly difficult.

@anav
Once again you shoot from the hip. I do admit my comments in the firewall rules are in error - sorry, they are only updated when I am nearing completion so that my backup support person can follow the rationale for what is in place. So I have gone through and updated the comments. Updated firewall export attached.

I have spent hours of trial and errors to get this network working - it is all done in a single device - rather than most of the mikrotik wiki examples which orient towards ISP or enterprise layouts and design and use multiple routers and switches. This network is for a charity / not for profit organisation, with limited financial resources.

So briefly, the following works:
A - Terminate two different ISP provided WAN via pppoe connections, one fibre with three IP4 addresses, one VDSL with a single IP4 address. Each also provides an IPv6 /56 delegated prefix which I use in the dual stack configuration I have engineered.
B - run two hardware servers, each with multiple Virtual Machines for 6 websites, 2 email servers, various portals to backend systems for our nationwide staff to access. These all have dual stack addresses - the IP4 addresses are static from a server subnet 192.168.128.0/24 with dstnat and ip/firewall/filter rules to control access, the IPv6 are global addresses for each virtual server from one of the /64 pools I create from the /56 delegated prefixes. One of the servers actually has two IPv6 global addresses, one from each pool.
C - Run four mikrotik hAPac as wireless Access Points, as well as providing ethernet connectivity to various IoT devices distributed around the building. These connect via trunk connections with four vLANs to provide isolation and security. Wifi, IoT, LAN and MGMT each have their own IP4 subnet and vlan id.
D - We have a Wifi vLAN on a IP4 subnet 192.168.131.0/24 used for staff to access the internet. They need to access the servers also and this is the bit that doesn't yet work, and the reason for this post.
E - we have a IoT vLAN also on a IP4 subnet 192.168.130.0/24 where SIP ATA provide telephone access, also have EFTPoS machines, and a windoze machine streaming spotify for internal music in our public client area.
F - a LAN subnet 192.168.129.0/24 for secure workstation and trusted machines have access.
All of this works reliably and securely.

Our only issue is that for our staff to access our internal servers, they currently need to turn off their wifi and utilise their cellphone data (hence access via WAN) to reach the https websites we host.

I have previously used internal DNS via mikrotik DNS to directly use the 192.168.x.x addresses to access our servers, however this became unworkable as we use DKIM and DMARC and need multiple DNS TXT records for each of the domains we host. Hence we went to using just external public IP addresses and external DNS service - and used hairpin. As hinted above, this worked for a single IP address and connection but when we added the second WAN connection and some additional IP addresses it got way too complex. So we moved the servers to their own subnet as recommended by forum articles and the prolific posting members.

I am sure there is something glaringly obvious that I'm missing, but after all the dozens of hours and days I have spent on this, I'm just not seeing it.

Thanks for any insights anyone can give.
Rob
You do not have the required permissions to view the files attached to this post.
 
robkampen
newbie
Topic Author
Posts: 32
Joined: Mon Aug 05, 2019 10:44 pm

Re: HTTPS server access from LAN fails, from WAN it works.

Wed Jan 11, 2023 10:34 am

Bumping this.
I still am unable to access our HTTPS servers or our IMAP/SMTP email servers from our LAN. All fine from the WAN side of things, but internally it either gives:
This site can’t provide a secure connection
webmail.domainname.org uses an unsupported protocol.
ERR_SSL_VERSION_OR_CIPHER_MISMATCH
which is the browser trying to be clever - as the same browser works just fine if we access the webserver domain from anywhere else.
Hence my using the curl access and results as shown in previous post above.
Trying to SSH from the LAN_VLAN (192.168.129.0/24) via the use of the external WAN accessible address also fails. Rather than accept the SSH key exchange it drops to asking for password, which it turns out is actually the router although it doesn't accept the correct password, but the router log shows a failed login failure from the source subnet address. Colour me confused.

From a LAN connected mail client to the external domainname with SSL/TLS we also get failure - typically a refused connection or some such message, but once again, shift the client to anywhere else and access via WAN and all is well.
So I suspect it is related to the equivalent of a hairpin issue, but how can that be when the servers are on a different subnet to the client?
after trying to follow the connection tracking / packet processing it appears that the tcp connection is in fact not completing - hit wait state and then drop - never get to established.
The connection tracking also only shows the first part of the connection - from local source subnet address (src-address) to the external IP address (dst-address). However the packet counter shows both Orig and Repl bytes which would indicate the packet made it to the server and it sent back a response which the router has seen and counted.
Thus it seems that the final packet flow from the router to the local subnet address is not happening .... this should be the reverse flow of the src-nat ??
Would really appreciate someone that understands this area of routing much better than me having a look and letting me know what I have done wrong.
Or how do I trouble shoot this further?
Looking at /tool/sniffer/quick with either ip-address= or interface= does show both forward and backward packet flows so now I'm more confused.

TIA
Rob
 
robkampen
newbie
Topic Author
Posts: 32
Joined: Mon Aug 05, 2019 10:44 pm

Re: HTTPS server access from LAN fails, from WAN it works. [SOLVED]

Mon Jan 16, 2023 12:17 pm

Well it seems RouterOS <7.7 doesn't do the hairpin type connection across different subnets when you have multiple IP addresses and WAN connections.
So when you try accessing a local server, via its external public IP addresses, from one of the internal LAN subnets the packet never hits the local server. It works fine only where you have just a single server and a single IP address and put a hairpin NAT in place. Four months of chasing my tail and hundreds of wasted hours later.

So release of RouterOS 7.7 on Friday the 13th fixed some of the issues that have plagued the Mikrotik proxy DNS server. Now the proxy will query the parent DNS for records it does not hold and mostly deals with the TTL values OK.

So my [SOLVED] solution is to revert back to my original design and put DNS A records in place pointing to the internal server IP addresses. The really good thing is I only need to load the A records for all my domains and subdomains - everything else will come from the public DNS server.
This is what I had working some four or so months ago, but had to abandon as back then the Mikrotik DNS proxy would not correctly query the authoritative DNS server for records it does not hold. (say a TXT record) thus things like email servers could not get the TXT records needed for secure mail transport - and duplicating an entire set of DNS records for every domain was just too difficult (error prone) and time consuming.
FINALLY we now have a Mikrotik DNS proxy that behaves a little more correctly - certainly sufficient for my purposes.

Who is online

Users browsing this forum: Ahrefs [Bot], jvanhambelgium and 84 guests