Community discussions

MikroTik App
 
David00
just joined
Topic Author
Posts: 9
Joined: Tue Mar 29, 2022 1:06 am

Dual WAN - Sending dst-nat'd traffic back out the interface it came in on

Thu Jun 30, 2022 1:11 pm

Hello,

Network engineer here. I've been stuck on this for a few days now. I've made a little bit of progress but ultimately I am stuck. I'm running Router OS v6.48.6 on a hEX S.

I have two WANs at home (no load balancing or anything fancy, they just have different metrics). One WAN is for work, so I have a bunch of static routes to send traffic out that link. The other WAN is for normal home networking needs and is the best default route in my routing table.

I have a couple of servers at home, and some DNS records that point to my work WAN IP. So, I am attempting to dst-nat (port forward) HTTPS traffic that hits my work WAN (via the A record I've published). The traffic is getting dst-nat'd to my server just fine; I can see it show up on when using tcpdump on the server. The server is in one of my LANs with the rest of the clients that normally get routed out the personal WAN, not the work WAN. So, if traffic comes in on the work WAN, having it go back out the personal WAN won't work. To prevent this, I'm using the mangle feature to apply routing, packet, and connection marks on the incoming dst-nat'd traffic so that I can catch them on their way back out (assuming the "connection" mark will apply to both the incoming and outgoing traffic). Here are my mangle rules:

/ip firewall mangle
add action=mark-routing chain=prerouting comment="Tag packets going back to IPs in the NAS-Web-Access list AND in through the work circuit with the routing mark \"route-to-work-circuit\"" connection-mark=\
    nas-web-conn-inbound dst-address-list=NAS-Web-Access new-routing-mark=route-to-work-circuit passthrough=yes
add action=mark-connection chain=prerouting in-interface="ether4 (work)" new-connection-mark=nas-web-conn-inbound passthrough=yes src-address-list=NAS-Web-Access
add action=mark-packet chain=prerouting in-interface="ether4 (work)" new-packet-mark=nas-web-packet-inbound passthrough=yes src-address-list=NAS-Web-Access
add action=mark-routing chain=prerouting comment="Mark route - traffic destined to addresses in Work-Mgmt-Networks" dst-address-list=work-Mgmt-Networks new-routing-mark=route-to-work-circuit passthrough=yes

The IPs of remote clients that I want to allow to hit the server on my LAN are defined in the NAS-Web-Access address list. I am new to MikroTik's approach on route-maps, but my intent here was to tag both the "packet" and the "connection" coming in from the remote client to catch them on their way in, and then apply a route mark on the reply packets that would go back out that is based off the connection mark applied to the incoming packet.

Here are my routes. Note that the default route out the personal WAN is missing from this list because it's a dynamic route from a DHCP assigned interface. However, if I were to represent it statically, it would look like this:


add distance=1 gateway="Personal (VLAN 201)"

/ip route
add distance=1 gateway="ether4 (Work)" routing-mark=route-to-work-circuit
add distance=5 gateway="ether4 (Work)"


And my NAT rules... I am testing with port 80 right now, and then will switch over to 443 once I iron out the configs. So,

/ip firewall nat
add action=dst-nat chain=dstnat comment="Test - redo forwarding port 80 from Work WAN to 192.186.10.105" connection-mark=nas-web-conn-inbound dst-port=80 in-interface="ether4 (Work)" protocol=tcp to-addresses=\
    192.168.10.105 to-ports=80
add action=masquerade chain=srcnat comment="NAT packets that have been route tagged with \"route-to-work-circuit\" AND are going to an address from the \"NAS-Web-Access\"" out-interface="ether4 (Work)" routing-mark=\
    route-to-work-circuit
add action=masquerade chain=srcnat comment="Masquerade VPN traffic if outgoing interface is Work Circuit" out-interface="ether4 (Work)" src-address=192.168.175.0/27
add action=masquerade chain=srcnat comment="Masquerade VPN traffic to go out Personal Circuit" out-interface="Personal (VLAN 201)" src-address=192.168.175.0/27
add action=masquerade chain=srcnat comment="Masquerade VLAN 20 out Personal interface." out-interface="Personal (VLAN 201)" src-address=192.168.20.0/25
add action=masquerade chain=srcnat comment="Masquerade VLAN 5 if outgoing interface is Work Circuit" out-interface="ether4 (Work)" src-address=192.168.10.0/25
add action=masquerade chain=srcnat comment="NAT LAN networks out Personal if they are a part of the LAN_Networks_NAT address list." out-interface="Personal (VLAN 201)" \
    src-address-list=LAN_Networks_NAT

The problem is the return traffic back to the client. I have been capturing packets on the destination server in my LAN, on my Mikrotik router, and on the remote client that's trying to access the server.

On the server in my LAN at home, I see packets coming in and replies going back out.

On the remote client, I do not see ANY return traffic. I only see the requests going out with a destination address of my work WAN.

On the MikroTik, I see the reply packets coming from the server, hitting the MikroTik's LAN interface (assuming it's getting re-translated), and then I see a packet go out the work WAN interface destined to the remote client's external IP. The remote client is also behind NAT and I manage that end of the network too. However, many of the packets leaving the work WAN interface on the Mikrotik are retransmits for sequence 0, which confirms the remote client is not getting anything at all back.

So, I'm pretty sure this is happening because the replies from the server on my LAN are not being NAT'd back out the TCP port they came in on. In other words, I think the MikroTik is NAT'ing them out the correct interface, but not as part of an existing connection. But, in that case, I would at least expect to see the traffic hit the remote client (and subsequently ignore it), but it's not. Plus, I don't see any src-nat rules incrementing on the MikroTik interface.

Here is the packet capture from the Mikrotik that shows packets leaving my WAN interface and going to the remote client's external IP.

Image

Any ideas on what I can try? I hope I've shared enough of the details. Thank you for your time!
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Dual WAN - Sending dst-nat'd traffic back out the interface it came in on

Thu Jun 30, 2022 2:20 pm

First of all, have you disabled fasttracking, which is incompatible with mangling? There are ways to keep using fasttrack for connections that don't need to be mangled, but to make things clear, start by fully disabling it.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 19107
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Dual WAN - Sending dst-nat'd traffic back out the interface it came in on

Thu Jun 30, 2022 4:57 pm

Why dont you simply create one LAN dedicated to the work WAN. Since there is no network diagram i am probably missing the boat.
 
David00
just joined
Topic Author
Posts: 9
Joined: Tue Mar 29, 2022 1:06 am

Re: Dual WAN - Sending dst-nat'd traffic back out the interface it came in on

Thu Jun 30, 2022 7:36 pm

First of all, have you disabled fasttracking, which is incompatible with mangling? There are ways to keep using fasttrack for connections that don't need to be mangled, but to make things clear, start by fully disabling it.

The fastrack forward rule was near the bottom of my filter rules, right above a couple of drop statements. So, it wasn't fully disabled, but it was pretty close to being disabled. Only 3k packets out of 119M have hit the fastrack forward rule since the last counter reset. I have disabled the fasttrack forward rule per your troubleshooting recommendation. The issue is still happening with TCP re-transmissions going back to the remote client. Also, I've moved the remote client out from behind NAT and it's now on a public IP address to help reduce the complexity of the situation. The remote client is not receiving ANY packets back from the web server on my home network, as if my MikroTik is blackhole-ing the responses somehow.


Why dont you simply create one LAN dedicated to the work WAN. Since there is no network diagram i am probably missing the boat.

There are several reasons but it all boils down to the fact that I only want to use the work WAN for traffic that needs to go out of it (or come into it, in the case of accessing the server on my LAN).
- The work WAN connection has much lower bandwidth compared to my personal WAN (150 Mbps down, 10 up for work vs. the 1 Gbps bidirectional throughput I get on my personal WAN)
- The work WAN has high latency as it's on an overly congested DOCSIS HFC infrastructure
- I work for the ISP that provides the WAN connection, so I don't want my normal internet traffic to be snooped on
- Lastly, I work from home most of the time, so I don't want to have to switch my computer's network (or deal with routing on the computer itself) every time I need to access resources through the work WAN.



I don't think a network diagram will help too much in this case as the problem is clearly with the MikroTik router configuration, but I quickly put this one together in case it helps you think of any ideas.

Image
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Dual WAN - Sending dst-nat'd traffic back out the interface it came in on

Thu Jun 30, 2022 8:28 pm

Only 3k packets out of 119M have hit the fastrack forward rule since the last counter reset.
That doesn't matter much as a connection gets fasttracked as soon as any packet belonging to it (in any direction) hits the fasttracking rule, and once a connection gets fasttracked, most of its packets skip most of the firewall processing. So the rule only counts the number of connections it has fasttracked, not the number of packets belonging to those connections.

But that's just lyrics - as complete disabling of the fasttracking didn't help, it clearly is not the root cause of your issue, or at least not the only one.

I cannot spot an error in the mangle rules and routes you've posted. Do you have anything in the /ip route rule table? Does the rule translating the connection-mark to routing-mark count?

Regarding sniffing, Mikrotik keeps using .pcap format to store the data, so you cannot see the interface names in Wireshark. But there is the source MAC address of the packets, can you check whether the SYN,ACK responses are sent from ether4's MAC address or from another one, and whether their destination MAC address is the one of the ISP router to which ether4 is connected?

The thing is that you've configured just ether4 as a gateway, which makes RouterOS send ARP requests for the destination address, under an assumption that arp-proxy is enabled at the external router(s) connected to that interface. This is quite an unusual setup, so maybe something doesn't work as expected when such a route is in another routing table than main.
 
David00
just joined
Topic Author
Posts: 9
Joined: Tue Mar 29, 2022 1:06 am

Re: Dual WAN - Sending dst-nat'd traffic back out the interface it came in on

Thu Jun 30, 2022 9:06 pm

Thanks sindy. There are no rules in /ip route rule.

Yes, the rule that adds the routing mark to the already-marked connections is taking effect - I can see the counters increase in real time when I run a test.

I have been investigating the layer 2 flow of frames since my last post, and from what I can tell, everything looks right to me. The [SYN, ACK] responses that have a source IP address of my work WAN interface have a source MAC that matches the MAC on the Mikrotik's work WAN interface. I even see the destination MAC as the upstream layer 2 neighbor, which should mean that the frame is indeed going out the right interface to the next physical device on my work ISP infrastructure.

Question - do I need to have a specific src-nat rule to NAT the packets that come in and get port forwarded to the server in my LAN? The router should already have a mapping from the dst-nat translation to send the return traffic out via, right?
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Dual WAN - Sending dst-nat'd traffic back out the interface it came in on

Thu Jun 30, 2022 9:19 pm

Question - do I need to have a specific src-nat rule to NAT the packets that come in and get port forwarded to the server in my LAN? The router should already have a mapping from the dst-nat translation to send the return traffic out via, right?
Right. The dst-nat and src-nat rules only handle the initial packet of each connection. Their verdict is stored in the context of that connection in the conntrack module, so the same nat operations are then taken on all downstream packets, and reverse nat operations are taken on all upstream packets of that connection. I.e. the responses to dst-nated requests are "un-dst-nated" after getting routed and before getting actually sent via an interface; vice versa, the responses to src-nated requests are "un-src-nated" before getting routed.

But if you have checked that the response is sent from the correct MAC address (of ether4) to the correct MAC address (of the gateway), it doesn't sound like a Mikrotik issue to me any more. If your Mikrotik has a switch chip that supports traffic mirroring, you may want to mirror ether4's traffic to another port and capture there to be sure that it has indeed been sent by the CPU.

When you ping the public IP of ether4 from the outside, and allow the ICMP echo requests in the input chain, do you get the responses? I have seen many incredible behaviours with ISPs, so I would not be surprised if filtering TCP responses was one of these.
 
David00
just joined
Topic Author
Posts: 9
Joined: Tue Mar 29, 2022 1:06 am

Re: Dual WAN - Sending dst-nat'd traffic back out the interface it came in on

Thu Jun 30, 2022 10:16 pm

Right. The dst-nat and src-nat rules only handle the initial packet of each connection. Their verdict is stored in the context of that connection in the conntrack module, so the same nat operations are then taken on all downstream packets, and reverse nat operations are taken on all upstream packets of that connection. I.e. the responses to dst-nated requests are "un-dst-nated" after getting routed and before getting actually sent via an interface; vice versa, the responses to src-nated requests are "un-src-nated" before getting routed.
That's what I thought - just wanted to make sure because this is very strange. Thanks!

But if you have checked that the response is sent from the correct MAC address (of ether4) to the correct MAC address (of the gateway), it doesn't sound like a Mikrotik issue to me any more. If your Mikrotik has a switch chip that supports traffic mirroring, you may want to mirror ether4's traffic to another port and capture there to be sure that it has indeed been sent by the CPU.

When you ping the public IP of ether4 from the outside, and allow the ICMP echo requests in the input chain, do you get the responses? I have seen many incredible behaviours with ISPs, so I would not be surprised if filtering TCP responses was one of these.
I'm not sure if the hEX S supports traffic mirroring. I have a Cisco 3560 that I could put inbetween the MikroTik and the work WAN handoff to mirror the port, though.

Great idea on trying ICMP. I just added the rule to my firewall table and ran some pings from my remote client. I see the requests coming in and the replies going out, but I'm not getting responses back on the client. Oddly enough, I see two replies for every request:

... where the 7....35 is the remote client, and the 2...0 is my work WAN interface.


Image


The dest. mac on both replies is the same (the upstream layer 2 neighbor). It is strange that there are two replies going out within 1ms of each other for every request, but even with that behavior, I should still be seeing the packets arrive at the destination, and I'm not getting anything back at all (confirmed via tcpdump on the remote client)
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Dual WAN - Sending dst-nat'd traffic back out the interface it came in on  [SOLVED]

Thu Jun 30, 2022 10:32 pm

When you have a look at the replies, you can see that they differ in size by 4 bytes, which suggests that the larger one has a VLAN header (as the dissection pane will show you). That further suggests that these replies are sent via VLAN201 on ether1. So maybe it is time to post the complete configuration, anonymized as per my automatic signature below.

If you run /tool sniffer quick ip-protocol=icmp while pinging from the outside, you should see through which interface the requests and replies pass.
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Dual WAN - Sending dst-nat'd traffic back out the interface it came in on

Thu Jun 30, 2022 10:51 pm

Sorry - to make those ping replies take the correct route, you need yet another mangle rule translating the connection-mark to a routing-mark, in chain output, because that's where packets sent (not forwarded) by the router itself are handled.
 
David00
just joined
Topic Author
Posts: 9
Joined: Tue Mar 29, 2022 1:06 am

Re: Dual WAN - Sending dst-nat'd traffic back out the interface it came in on

Thu Jun 30, 2022 10:56 pm

Great catch - I didn't see the difference in size. And, thanks for the note on running the sniffer on the mikrotik CLI. That's so much easier!

Here is the output of the sniffer from just a couple of pings:
INTERFACE			TIME	NUM DIR SRC-MAC             DST-MAC     VLAN	SRC-ADDRESS     DST-ADDRESS     PROTOCOL	SIZE    CPU FP
ether4 (Work)		2.509	1 <-	28:52:61:FB:94:19   2C:C8:1B:03:EE:49   	7...5	        2...0			ip:icmp	    98	    3 no
Personal (VLAN 201)	2.509	2 ->	2C:C8:1B:03:EE:46   10:E8:78:4C:C3:EC   	2...0	        7...5			ip:icmp	    98	    3 no
ether1 (Personal)	 2.509	3 ->	2C:C8:1B:03:EE:46   10:E8:78:4C:C3:EC    201    2...0	        7...5			ip:icmp	    102	    3 no
ether4 (Work)		3.521	4 <-	28:52:61:FB:94:19   2C:C8:1B:03:EE:49   	7...5	        2...0			ip:icmp	    98	    3 no
Personal (VLAN 201)	3.521	5 ->	2C:C8:1B:03:EE:46   10:E8:78:4C:C3:EC   	2...0	        7...5			ip:icmp	    98	    3 no
ether1 (Personal)	 3.521	6 ->	2C:C8:1B:03:EE:46   10:E8:78:4C:C3:EC    201    2...0	        7...5			ip:icmp	    102	    3 no
ether4 (Work)		4.545	7 <-	28:52:61:FB:94:19   2C:C8:1B:03:EE:49   	7...5	        2...0			ip:icmp	    98	    3 no
Personal (VLAN 201)	4.545	8 ->	2C:C8:1B:03:EE:46   10:E8:78:4C:C3:EC   	2...0	        7...5			ip:icmp	    98	    3 no
ether1 (Personal)	 4.545	9 ->	2C:C8:1B:03:EE:46   10:E8:78:4C:C3:EC    201    2...0	        7...5			ip:icmp	    102	    3 no

Also, I've been continuing to try different things. I forwarded port 443 to my web server, and to my surprise, I can get to it from the remote client! I've also forwarded external port 50080 to internal port 80 as I randomly recalled most ISPs block port 80, and this works too. I was under the impression that my work WAN was unrestricted, but I guess I was wrong. So, I think my original issue is now working, probably since I cleaned up a lot of the configs that I started with. Testing with port 80 also screwed me.

But, I am definitely still interested in learning why ICMP is being redirected back out the personal WAN and not the work WAN. I will get started on exporting and sanitizing my config. Thank you for all your help!

On the subject of ICMP going back out the personal WAN and not the work WAN where it originated from... the personal WAN is provided tagged in VLAN 201, and that is what my best (metric=1) default route is set to use. So, there shouldn't be any untagged traffic traversing the link to the personal WAN. It's strange that the mikrotik is sending the replies out the personal WAN untagged.

Sanitized config coming shortly.
 
David00
just joined
Topic Author
Posts: 9
Joined: Tue Mar 29, 2022 1:06 am

Re: Dual WAN - Sending dst-nat'd traffic back out the interface it came in on

Thu Jun 30, 2022 10:57 pm

Sorry - to make those ping replies take the correct route, you need yet another mangle rule translating the connection-mark to a routing-mark, in chain output, because that's where packets sent (not forwarded) by the router itself are handled.

Oh, okay! I completely overlooked that. I'll try that now.
 
David00
just joined
Topic Author
Posts: 9
Joined: Tue Mar 29, 2022 1:06 am

Re: Dual WAN - Sending dst-nat'd traffic back out the interface it came in on

Thu Jun 30, 2022 11:03 pm

Sorry - to make those ping replies take the correct route, you need yet another mangle rule translating the connection-mark to a routing-mark, in chain output, because that's where packets sent (not forwarded) by the router itself are handled.
That did it (for the pings)! Fantastic - I think I am all set now. Thank you so much!

I do have another question regarding the fasttrack feature though. Should the fasttrack forward rule be immediately after the the fasttrack dummy/passthrough rule in order to best take advantage of the fasttrack feature? In other words, should fasttrack be the first couple of rules in my firewall filter, or should I do all my unique firewall processing first?

I don't quite understand the fasttrack feature and the docs don't really explain what it does or how it works.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 19107
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Dual WAN - Sending dst-nat'd traffic back out the interface it came in on

Thu Jun 30, 2022 11:09 pm

I disagree with your mixing of home and work traffic on one pc. I would suggest that you have a switch in between and switch ethernet ports so that there is a physical disconnect at all times.
I think your breaking some terms of agreement with your company but since you work for them that is your concern. :-) Glad you got it working.
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Dual WAN - Sending dst-nat'd traffic back out the interface it came in on

Thu Jun 30, 2022 11:20 pm

The whole stateful firewall idea is based on letting each packet be matched against as little firewall rules as possible. So the first rule in each chain of filter should normally be action=accept connection-state=established,related. Only packets that get past this one are processed individually, because they are either new (initial packets of connections not yet known to the connection tracking) or invalid (packets which match a known connection by addresses and ports but something about their state disqualifies them).

Fasttracking takes the idea one step further - if a packet belongs to a known connection flagged with fasttracking, it skips not only filter but also mangle and IPsec policy matching.

But in order that a connection could ever get fasttracked, the connection-state=established,related action=fasttrack-connection rule must be placed even before the connection-state=established,related action=accept one. And still before it, there must be other action=accept rules that prevent connections you don't want to get fasttracked from reaching the action=fasttrack-connection one.

In your case, it is enough to add connection-mark=no-mark to the action=fasttrack-connection rule, making it ignore packets belonging to connection-marked connections.
 
David00
just joined
Topic Author
Posts: 9
Joined: Tue Mar 29, 2022 1:06 am

Re: Dual WAN - Sending dst-nat'd traffic back out the interface it came in on

Fri Jul 01, 2022 7:30 pm

Thanks again for the help, sindy. All of your replies were helpful and thoughful, so it was hard to mark any one as a solution, but I picked the one around the time I noticed that inbound hosts on 443 were working.

Just as a follow up, I really wanted to figure out whether or not my work WAN was dropping traffic sourced from port 80, so I connected a Raspberry Pi directly to the WAN and tried to access it from my test client on a public IP. Sure enough, no luck. The ISP's side of my work WAN appears to be dropping traffic coming from my LAN that's sourced from port 80. That's not the case for my personal WAN though, which I also confirmed allows traffic I send sourced from port 80.

I learned a lot more about how MikroTiks work from this troubleshooting and your assistance, and I'm just as happy about learning as I am about solving the issue.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 19107
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Dual WAN - Sending dst-nat'd traffic back out the interface it came in on

Sat Jul 02, 2022 1:08 am

Your lucky, Sindy is the best, and that is not just the beer talking. Sob, mkx and rextended are not bad either ;-)
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11982
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Dual WAN - Sending dst-nat'd traffic back out the interface it came in on

Sat Jul 02, 2022 1:24 am

Speak of the devil? :lol:

Who is online

Users browsing this forum: anav, Bing [Bot], ernieball17, ret411, wsantos and 77 guests