Community discussions

MikroTik App
 
robkampen
newbie
Topic Author
Posts: 32
Joined: Mon Aug 05, 2019 10:44 pm

Routing between two subnets on different bridges

Thu Nov 10, 2022 6:50 am

Brief background.
We have two ISP connections - one fibre the other DSL and both provide IP4 and IPv6/56 - thus we are running dual stack.
We have four subnets on four bridges to keep LAN, WiFi, IOT and Management traffic separate and provide a degree of isolation.
We have six or so servers running, two of which are email servers that provide virtual mailboxes for multiple domains. One of these email servers .232 and ::232 runs off the VDSL connection, the other .242 and ::242 runs off the fibre.

Nearly everything is working as per design, the only issue remaining is trying to get clients in the WiFi subnet 192.168.131.0/24 and a printer in the IOT subnet 192.168.130.244 to be able to connect to the MX email servers on ports 993 IMAP and 587 SMTP.

I have set up filter rules for this traffic, and can see the traffic hitting the rules (via log messages and packet counts) but they never seem to reach the server. Not sure at which point the server logs postfix and dovecot connections when TCP is being established ... but no log messages happening in the MX server at all when attempts are being made.

So the question is what rules / routing needs to be in place to ensure that this specific connection type is allowed and functions.
The complete connection is in-house - i.e. we are using local dns to resolve to 192.168.xx.yy addresses - we are not using global or outside addresses.
I attach a sanitised copy of the export from the RB4011 doing all the work.

I have spent far too long looking at this and am probably missing something obvious, so looking forward to gaining some clarity and insight from y'all
TIA
Rob.
You do not have the required permissions to view the files attached to this post.
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11381
Joined: Thu Mar 03, 2016 10:23 pm

Re: Routing between two subnets on different bridges

Thu Nov 10, 2022 9:18 am

Your FW filter rule set is log, so I ceased to go through it. Just an idea: try to run tcptraceroute (pretty standard on all linux distributions) towards service ports on your mail servers and see where it breaks (tcptraceroute uses TCP probes targeting particular service port, so it resembles application as far as TCP SYN packets go). It might prove it's not firewall on router but some firewall on mail servers blocking connections.
 
robkampen
newbie
Topic Author
Posts: 32
Joined: Mon Aug 05, 2019 10:44 pm

Re: Routing between two subnets on different bridges

Fri Nov 11, 2022 10:04 am

Thanks for the suggestion. Tried it and found it only gets to the router, a single hop and then returns *** as it times out. Thus the router is not routing traffic from 192.168.131.x to the 192.168.128.232 MX server.
There is a dynamic routing rule in the IP route table for each subnet address that sets the gateway to the appropriate bridge - thus I cannot see why it does not route the traffic.
Not sure what else to check. If I use a source address from the same subnet (LAN) it works ok.
I feel like I'm not understanding how routing between subnets is meant to work, which entries in the various tables are designed to make this work.
Still stuck.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18959
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Routing between two subnets on different bridges

Fri Nov 11, 2022 2:35 pm

Your config is overly complex for what?
KISS: ONE BRIDGE, all vlans associated to the one bridge, the bridge does NO dhcp.

interface bridge ports and interface bridge vlans assign the vlans to ports as required, easy peasy.

Your config is a confused mess of things.

The only vlan not on the bridge is the one associated with your WAN and even that is suspect.
WHy would two ISPs use the same vlan for their ISP connection (vlan10), makes no sense or did you just invent using vlan10, which is totally unnecessary probably 99% of the time)
 
robkampen
newbie
Topic Author
Posts: 32
Joined: Mon Aug 05, 2019 10:44 pm

Re: Routing between two subnets on different bridges

Sat Nov 12, 2022 3:22 am

So if all on one bridge, how do I do DHCP? for different traffic types on different subnets?
The vlans to the two WANS are specified by the two different ISP companies - nothing to do with me. That is why we have a WAN interface created and then the vlan10 setup on that, otherwise the pppoe will not work.
All of this stuff is working just fine.
All the world can interact just fine with our various servers, it is just folk on our local Wifi network trying to connect with our local servers that are having the issues.
My question remains - how does traffic from subnet 192.168.131.x when resolving a DNS request, gets a public address that terminates on our router, goes through dstNAT and gets a 192.168.128.224/27 destination then tries to connect to the server and then the return path correctly gets back to the original 192.168.131.x
That is where hairpin NAT comes in .... ??
YES I HAVE READ THE EXTENSIVE dialog on this topic, and after trying to filter the flame wars between a number of "experts" am still unable to resolve my issue.
It works fine for traffic from 192.168.128.x via public IP to the servers also on 192.168.128.224/27
It just chokes on traffic from a different source subnet.
So how does one route traffic from one subnet on one bridge to another subnet on another bridge?
Is there a simple example somewhere for this scenario?
Thanks for your help and suggestions.
 
Sob
Forum Guru
Forum Guru
Posts: 9119
Joined: Mon Apr 20, 2009 9:11 pm

Re: Routing between two subnets on different bridges

Sat Nov 12, 2022 4:27 am

It's... erm... not nice. One thing is all the bridges, it's not recommended way (see e.g. this popular thread), but it should work too (I didn't examine all details).

As for firewall, it's not better. There's no problem with filter, e.g. 192.168.131.x connecting to 192.168.128.232:587 will jump to mx-web chain and will be allowed there. You even have more than one jump rule that will do it, either the first "Allow wifi/iot access to printer/servers" or following one with dst-address=192.168.128.232. And that's it, happy end. Well, it would be, if it wasn't for your unnecessary (for this) and wrong hairpin NAT rule:
/ip firewall nat
add action=masquerade chain=srcnat comment="hairpin NAT" dst-address=192.168.128.0/22 log-prefix=hairpin protocol=tcp src-address=192.168.128.0/22 to-addresses=192.168.128.240/30
This takes any traffic between any local subnets passing through router and changes source to 192.168.128.240-3. So server will see incoming connections from those addresses and will try to respond to local devices that have them, not to router. If you need any hairpin NAT at all, it would be one of (note the masks):
/ip firewall nat
add chain=srcnat dst-address=192.168.128.0/24 src-address=192.168.128.0/24 action=masquerade
add chain=srcnat dst-address=192.168.128.0/24 src-address=192.168.128.0/24 action=src-nat to-addresses=x.x.x.x
Where x.x.x.x can be pretty much anything except 192.168.128.x (192.168.128.1 would work, but then it's easier to use first rule with masquerade).

Edit: Actually, RouterOS got me again, it mistakenly keeps to-addresses from previous action=src-nat when you change it to action=masquerade. So it should work as is, only connections will show up as from 192.168.128.1 (you're still masquerading traffic that doesn't need to be masqueraded).

Edit2: Nah, on yet another look, it's mainly the routing rule where you say that anything from 192.168.128.232 should use toVDSL routing table.
 
robkampen
newbie
Topic Author
Posts: 32
Joined: Mon Aug 05, 2019 10:44 pm

Re: Routing between two subnets on different bridges

Sat Nov 12, 2022 10:59 am

@sob
thanks for looking and comments. I agree, I too think the problem is somewhere in the NAT hairpin or the mangle packet marking.
You will note that the mangle rule is src-address-list resolves to 192.168.128.232 but the dst-address-list = ! local where local is 192.168.128.0/22 - I tried adding log on this rule and it doesn't fire, thus for some reason the destination is being seen as something else? Do I need to add my router terminating public IPs to the local list?
unfortunately I cannot test this remotely, need to be on the local subnets, so will need to wait until I can visit site.
edit:
I have tried disable of hairpin - that doesn't fix the issue, this traffic does fire the hairpin rule, so it is incorrect, just not sure what actually does what with each of the parameters of this rule. All the extensive discussions on hairpin fail to actually describe which parameter is used on the initial masquerade and which on the return path.... colour me confused.
 
Sob
Forum Guru
Forum Guru
Posts: 9119
Joined: Mon Apr 20, 2009 9:11 pm

Re: Routing between two subnets on different bridges

Sat Nov 12, 2022 3:05 pm

It's this:
/routing rule
add action=lookup-only-in-table dst-address=0.0.0.0/0 src-address=192.168.128.232 table=toVDSL
If packet is from 192.168.128.232, then unless it has some other routing mark, it will use toVDSL table, which contains only single default route. So even packets with local destinations will be sent to internet. You do have following three routing rules that are supposed to exclude local subnets, but they can't do anything when they are after the first one. Order matters, the first matching rule is used. They have common src-address=192.168.128.232, but dst-address=0.0.0.0/0 in first rule matches everything, so it won't ever get further.

As for hairpin NAT, it's just another NAT, it deals only with initial packet and the rest of connection and all responses are handled automatically by connection tracking.
 
robkampen
newbie
Topic Author
Posts: 32
Joined: Mon Aug 05, 2019 10:44 pm

Re: Routing between two subnets on different bridges

Sun Nov 13, 2022 12:52 am

In IP route the rules are applied from most specific to least - thus 0.0.0.0/0 would be the last catch all.
So you are telling me that /routing is different - it is order important (like IP/firewall/filter)
I will investigate this, thanks for the pointer.
 
Sob
Forum Guru
Forum Guru
Posts: 9119
Joined: Mon Apr 20, 2009 9:11 pm

Re: Routing between two subnets on different bridges

Sun Nov 13, 2022 7:18 am

WinBox/WebFig have included hints, if the first column is "#", then order matters.
 
robkampen
newbie
Topic Author
Posts: 32
Joined: Mon Aug 05, 2019 10:44 pm

Re: Routing between two subnets on different bridges

Mon Nov 21, 2022 11:43 pm

@sob
thanks for the info about the # as first column - makes sense.
So I looked at the popular thread you posted - single bridge and multiple vlans and decided to give it a try.

So the network with the subnet routing issue has 4 x RB962UiGS-5HacT2HnT

I have reset one of them (MAIN_AP2) and done an AP config based upon viewtopic.php?t=143620
It took some doing as some items in the sample config don't play well on ROS 7.3.1 on this device.

It actually works with my original router setup!
Well mostly.

Each of the four wifi AP work - get their appropriate DHCP allocation. can connect to the internet etc.
Each of the four ethernet AP connections likewise.

Only issue is :-
I cannot see the device on Winbox,
and I cannot ping it,
and the device cannot ping the router
or use the router's DNS for itself to check for ROS updates.

I can maybe understand why winbox doesn't see it - the device running winbox is on the LAN subnet 192.168.128.0 and thus on vlan40.
I added the extra three /interface/vlan IOT_VLAN, LAN_VLAN, WIFI_VLAN so I could assign IP addresses in the hope of being able to ping the device from elsewhere in the local network - no joy.
Tried to update the OS to 7.6 - cannot resolve domain.

The new config for the AP is attached.

Once I can sort the above issues, I will make this the new config on all four AP and then begin on the router and hopefully get to fix my original problem.
I will buy a new RB4011 to do this development as I must keep everything working 24x7

This has been a great learning opportunity, and the new bridge vlan functionality seems to make sense on many fronts - really great that it works with the existing router setup.

Thanks for any insights you can bring.
You do not have the required permissions to view the files attached to this post.
 
Sob
Forum Guru
Forum Guru
Posts: 9119
Joined: Mon Apr 20, 2009 9:11 pm

Re: Routing between two subnets on different bridges

Tue Nov 22, 2022 2:38 am

If you want to communicate with router from VLANs, then bridge must be listed as tagged in "/interface bridge vlan" for those VLANs. You have it that way for VLAN 99, but not for others.
 
robkampen
newbie
Topic Author
Posts: 32
Joined: Mon Aug 05, 2019 10:44 pm

Re: Routing between two subnets on different bridges

Tue Nov 22, 2022 4:16 am

Awesome - that was it!
Thanks so much for your help - funny how one can miss the obvious.
Onwards....

Who is online

Users browsing this forum: No registered users and 86 guests