Dual WAN Dual LAN Setup

Hello Experts,

I’m trying to setup a 2 WAN 2 LAN setup

So far i tried marking connections to a route mark but i believe i am making a mistake somewhere because as soon as i am selecting route mark on route list client’s connection to Internet is failing.

I have CRS125-24G-1S routerboard.
with firmware 6.41.4
with bootloader 3.33 for ar9344

I have 2 public IPs available from the ISP (which they work as solo)

I have divided switch’s ports to 2 segments:

2 Bridges as LAN1, LAN2


WAN1 is on Port 0
Physical ports from 1 to 13 is for LAN1


WAN2 is on Port 23
Physical ports from 14 to 23 is for LAN2


My IP settings are


/ip address
add address=XX.0.XX.XX/30 comment=defconf interface="WAN1" network=XX.0.XX.XX
add address=192.168.1.1/24 interface=LAN1 network=192.168.1.0
add address=XX.0.XX.XX/30 interface="WAN2" network=XX.0.XX.XX
add address=192.168.2.1/24 interface=LAN2 network=192.168.2.0

/ip firewall mangle
add action=mark-routing chain=prerouting new-routing-mark=LAN1RouteMark passthrough=yes src-address=192.168.1.0/24
add action=mark-routing chain=prerouting new-routing-mark=LAN2RouteMark passthrough=yes src-address=192.168.2.0/24

/ip route
add check-gateway=ping disabled=yes distance=1 gateway=XX.0.XX.81 routing-mark=LAN2RouteMark
add check-gateway=ping distance=1 gateway=XX.0.XX.69

/ip firewall nat
add action=masquerade chain=srcnat comment=DEFAULT out-interface="WAN1"

As you can see LAN1 clients currently connects to internet through NAT without route mark and it works OK. But I want clients go through their own WAN addresses

Can i get a help here? Thank you.

Is bumping allowed here?

I don’t understand very clear your problem,but, if you want to get internet by Wan2, you need a NAT Masquerade of WAN2 interface.

I divided my physical network into two logical network.

I want to configure router for both logical networks, so far

  • I have created 2 bridges which includes physical LAN ports (1-13, 14-23) named them LAN1 and LAN2
  • Configured WAN interfaces (physical ports 0 and 24)
  • Configured WAN IP addresses given from ISP
  • Configured DHCP servers

LAN1 clients can access internet through WAN1 (NAT’ed masquerade)

also

LAN2 clients gets IP address from DHCP but cannot access Internet through WAN2 (even if I configure masquerade NAT on WAN2 interface)

tried several approaches. I believe there can only 1 route to 0.0.0.0/0 if there is no route mark available (i know it because it does not get active)

each LAN should access internet through their corresponding WAN interface, without load balancing (PCC, etc).

In mangle rules, change passthrough from yes to no

In routes, change distance for routing-mark=LAN2RouteMark from 1 to 2
Add routing mark LAN1RouteMark for the other route

Add NAT for LAN 2 with out-interface=WAN2

/ip address
add address=FFFFFFFFFFF/30 comment=defconf interface="WAN1" network=FFFFFFFFFFF
add address=192.168.1.1/24 interface=LAN1 network=192.168.1.0
add address=FFFFFFFFFFF/30 interface="WAN2" network=FFFFFFFFFFF
add address=192.168.2.1/24 interface=LAN2 network=192.168.2.0

/ip settings
set accept-source-route=yes allow-fast-path=no route-cache=no rp-filter=loose tcp-syncookies=yes

/ip firewall mangle
add action=mark-routing chain=prerouting new-routing-mark=LAN1RouteMark passthrough=no src-address=192.168.1.0/24
add action=mark-routing chain=prerouting new-routing-mark=LAN2RouteMark passthrough=no src-address=192.168.2.0/24

/ip firewall nat
add action=masquerade chain=srcnat dst-address=0.0.0.0/0 out-interface="WAN1"
add action=masquerade chain=srcnat dst-address=0.0.0.0/0 out-interface="WAN2"

/ip route
add check-gateway=ping distance=2 gateway=FFFFFFFFFFF routing-mark=LAN2RouteMark
add check-gateway=ping distance=1 gateway=FFFFFFFFFFF routing-mark=LAN1RouteMark

As soon as i apply these settings LAN1 clients can not connect to internet.

Please notice “/ip settings” some time ago i changed these settings in order to follow a tutorial that i do not remember what the default values are.

Rather place the full export here, there might be rules in firewall filter, etc that can also cause issues. Use export hide-sensitive in terminal window

Then also confirm if both WAN addresses are the same as I see you marked them the same in previous output, i.e. FFFFFFF

Hello again,

Here is the full export, i still made the obfuscation by hand because of hide-sensitive did not work which i do not know why.

# apr/24/2018 14:27:49 by RouterOS 6.41.4
# software id = -
#
# model = CRS125-24G-1S
# serial number = -

/interface bridge
add fast-forward=no name=COZUM
add admin-mac=---- auto-mac=no comment=defconf name=PITON

/interface ethernet
set [ find default-name=ether24 ] name="COZUM WAN"
set [ find default-name=ether10 ] name=DELLIDRAC
set [ find default-name=ether1 ] name="PITON WAN"
set [ find default-name=ether2 ] name=SRV1-DELL
set [ find default-name=ether3 ] name=SW1-MTDAR

/interface list
add name=pitonLanList
add name=cozumLanList

/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik

/ip pool
add name=PITONPOOL ranges=192.168.1.21-192.168.1.254
add name=COZUMPOOL ranges=192.168.2.21-192.168.2.254

/ip dhcp-server
add add-arp=yes address-pool=PITONPOOL disabled=no interface=PITON name=PITONDHCP
add add-arp=yes address-pool=COZUMPOOL disabled=no interface=COZUM name=COZUMDHCP

/interface bridge port
add bridge=PITON comment=defconf disabled=yes interface="PITON WAN"
add bridge=PITON comment=defconf interface=SRV1-DELL
add bridge=PITON comment=defconf interface=SW1-MTDAR
add bridge=PITON comment=defconf interface=ether4
add bridge=PITON comment=defconf interface=ether5
add bridge=PITON comment=defconf interface=ether6
add bridge=PITON comment=defconf interface=ether7
add bridge=PITON comment=defconf interface=ether8
add bridge=PITON comment=defconf interface=ether9
add bridge=PITON comment=defconf interface=DELLIDRAC
add bridge=PITON comment=defconf interface=ether11
add bridge=PITON comment=defconf interface=ether12
add bridge=PITON comment=defconf interface=ether13
add bridge=COZUM comment=defconf interface=ether14
add bridge=COZUM comment=defconf interface=ether15
add bridge=COZUM comment=defconf interface=ether16
add bridge=COZUM comment=defconf interface=ether17
add bridge=COZUM comment=defconf interface=ether18
add bridge=COZUM comment=defconf interface=ether19
add bridge=COZUM comment=defconf interface=ether20
add bridge=PITON comment=defconf interface=ether21
add bridge=COZUM comment=defconf interface=ether22
add bridge=COZUM comment=defconf interface=ether23
add bridge=COZUM comment=defconf disabled=yes interface="COZUM WAN"
add bridge=PITON comment=defconf interface=sfp1

/ip settings
set accept-source-route=yes allow-fast-path=no route-cache=no rp-filter=loose tcp-syncookies=yes

/interface list member
add interface=PITON list=pitonLanList
add interface=COZUM list=cozumLanList

/ip address
add address=XX.XX.XX.XX/30 comment=defconf interface="PITON WAN" network=XX.XX.XX.XX
add address=192.168.1.1/24 interface=PITON network=192.168.1.0
add address=XX.XX.XX.XX/30 interface="COZUM WAN" network=XX.XX.XX.XX
add address=192.168.2.1/24 interface=COZUM network=192.168.2.0

/ip dhcp-server network
add address=192.168.1.0/24 dns-server=192.168.1.1 gateway=192.168.1.1
add address=192.168.2.0/24 dns-server=192.168.2.1 gateway=192.168.2.1 netmask=24

/ip dns
set allow-remote-requests=yes servers=8.8.4.4,8.8.8.8

/ip firewall mangle
add action=mark-routing chain=prerouting new-routing-mark=LAN1RouteMark passthrough=no src-address=192.168.1.0/24
add action=mark-routing chain=prerouting new-routing-mark=LAN2RouteMark passthrough=no src-address=192.168.2.0/24

/ip firewall nat
add action=masquerade chain=srcnat comment=pitonWanNat dst-address=0.0.0.0/0 out-interface="PITON WAN"
add action=masquerade chain=srcnat comment=cozumWanNat dst-address=0.0.0.0/0 out-interface="COZUM WAN"

/ip route
add check-gateway=ping distance=2 gateway=XX.XX.XX.XX routing-mark=LAN2RouteMark
add check-gateway=ping distance=1 gateway=XX.XX.XX.XX

/ip service
set telnet disabled=yes
set ftp disabled=yes
set www port=81
set ssh disabled=yes
set api disabled=yes
set api-ssl disabled=yes

/system identity
set name=SW2-MTGEN

I think packages actually getting marked i guess

I’m afraid that there is simply no routing table for packets marked with routing mark ****

LAN1RouteMark

. So either do not route mark them at all and let them be handled by the default routing table, or do


/ip route add check-gateway=ping distance=1 gateway=XX.XX.XX.XX routing-mark=LAN1RouteMark

Remarks:

  • if two addresses you want to obfuscate differ, use different replacement patterns for them, otherwise information is lost
  • if the IP addresses of WAN 1 and WAN 2 are actualy from the same subnet and thus they really do use a common gateway IP, two routes with the same IP address as ****
gateway

will not cause any difference because both will use the same physical interface. If this is the case, you don’t need two routing tables (so no routing marks) and WAN interfaces; instead, you have to assign packet marks rather than route marks using

/ip firewall mangle

rules and replace your

action=masquerade

rules by the following ones:


/ip firewall nat
add chain=postrouting out-interface=WAN packet-mark=LAN2Mark action=src-nat to-addresses=ip.of.wan.2 
add chain=postrouting out-interface=WAN packet-mark=LAN1Mark action=src-nat to-addresses=ip.of.wan.1
  • the ****
distance

parameter of a route only makes a difference between routes with identical

dst-address

and

routing-mark

parameters.


hide-sensitive

removes passwords and alike from the export, but does not replace public IP addresses by distinctive patterns, so this does require handicraft

Thx sindy,

yes, I was waiting for confirmation also on the WAN gateway addresses.

To add, since there is so many topics re route marking, i just tested in a lab environment and found no issues, I used RoS 6.42.1 on all devices in the below config and got results I expected everytime without fail.

/---->Hap ac lite
PC—>Hap mini
----->951Ui-2HnD

Hello sindy and CZFan,

Silly me, I just pasted same XXs for both

They are simply different.

Sorry for the late reply, the two WAN interface are on separate subnet and both have different gateway for each.

/ip route add check-gateway=ping distance=1 gateway=XX.XX.XX.XX routing-mark=LAN1RouteMark

You can not see the “routing-mark=LAN1RouteMark” part on the dump because if I activate it LAN1 clients can no longer connect to Internet.

Thank you for clearing the hide-sensitive behavior.

Disabled Mangle rules are from my previous attempts. Should I reboot the router after activating “routing-mark=LAN1RouteMark”?

I really lost some of my hair during the setup. Thank you for your kind responses.

Reboot is definitely not necessary after any change. The effect of changes may become visible later (as an example, if an ssh session is already established and you add a rule preventing new ones from being established, the already established one doesn’t break but once you terminate it, a new one cannot be established).

So re-insert the configuration line which normally breaks it for the group of clients but with ****

disabled=yes

so that it wouldn’t break the clients, and post another export of the configuration with hide-sensitive but with distinctive patterns replacing the public addresses so that there is no ambiguity.

When you deal with several cases like this one every day, the context disappears from your head very quickly :slight_smile:

In your case in particular, when sessions have established via one WAN and then you activate routing via another one, all those sessions break because the remote end does not accept packets for existing session to start arriving from another address all of a sudden. So until the clients start new sessions, you cannot say whether it works or not.

piton@debian:~$ ping 192.168.2.1 -I eth1
PING 192.168.2.1 (192.168.2.1) from 192.168.2.250 eth1: 56(84) bytes of data.
^C
--- 192.168.2.1 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2050ms

piton@debian:~$ ping 192.168.2.1 -I eth0
PING 192.168.2.1 (192.168.2.1) from 192.168.1.57 eth0: 56(84) bytes of data.
64 bytes from 192.168.2.1: icmp_seq=1 ttl=64 time=0.673 ms
64 bytes from 192.168.2.1: icmp_seq=2 ttl=64 time=0.428 ms
64 bytes from 192.168.2.1: icmp_seq=3 ttl=64 time=0.399 ms
64 bytes from 192.168.2.1: icmp_seq=4 ttl=64 time=0.521 ms
^C
--- 192.168.2.1 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3074ms
rtt min/avg/max/mdev = 0.399/0.505/0.673/0.108 ms
piton@debian:~$



2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether e2:37:9c:2d:45:a6 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.57/24 brd 192.168.1.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::e037:9cff:fe2d:45a6/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether fa:6b:c4:5c:8d:6e brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.250/24 brd 192.168.2.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::f86b:c4ff:fe5c:8d6e/64 scope link
       valid_lft forever preferred_lft forever

I tested how ping will work and the result is confusing

Setup is like this

Linux Box with two NICs

eth0 is connected to LAN1
eth1 is connected to LAN2

both configured to get options from DHCP server

192.168.2.1 can not be pinged from eth1 interface (which its IP address is 192.168.2.250)
192.168.2.1 can be pinged from eth0 interface (which its IP address is 192.168.1.57)

weird isn’t it?

[admin@SW2-MTGEN] > export hide-sensitive  
# apr/26/2018 15:41:53 by RouterOS 6.41.4
# software id = ZZBI-83WG
#
# model = CRS125-24G-1S
# serial number = 787506C7C649

/interface bridge
add fast-forward=no name=COZUM
add admin-mac=6C:3B:6B:D8:C6:B0 auto-mac=no comment=defconf name=PITON


/interface ethernet
set [ find default-name=ether24 ] name="COZUM WAN"
set [ find default-name=ether10 ] name=DELLIDRAC
set [ find default-name=ether1 ] name="PITON WAN"
set [ find default-name=ether2 ] name=SRV1-DELL
set [ find default-name=ether3 ] name=SW1-MTDAR


/interface list
add name=pitonLanList
add name=cozumLanList


/ip pool
add name=PITONPOOL ranges=192.168.1.21-192.168.1.254
add name=COZUMPOOL ranges=192.168.2.21-192.168.2.254


/ip dhcp-server
add add-arp=yes address-pool=PITONPOOL disabled=no interface=PITON name=PITONDHCP
add add-arp=yes address-pool=COZUMPOOL disabled=no interface=COZUM name=COZUMDHCP


/interface bridge port
add bridge=PITON comment=defconf disabled=yes interface="PITON WAN"
add bridge=PITON comment=defconf interface=SRV1-DELL
add bridge=PITON comment=defconf interface=SW1-MTDAR
add bridge=PITON comment=defconf interface=ether4
add bridge=PITON comment=defconf interface=ether5
add bridge=PITON comment=defconf interface=ether6
add bridge=PITON comment=defconf interface=ether7
add bridge=PITON comment=defconf interface=ether8
add bridge=PITON comment=defconf interface=ether9
add bridge=PITON comment=defconf interface=DELLIDRAC
add bridge=PITON comment=defconf interface=ether11
add bridge=PITON comment=defconf interface=ether12
add bridge=PITON comment=defconf interface=ether13
add bridge=COZUM comment=defconf interface=ether14
add bridge=COZUM comment=defconf interface=ether15
add bridge=COZUM comment=defconf interface=ether16
add bridge=COZUM comment=defconf interface=ether17
add bridge=COZUM comment=defconf interface=ether18
add bridge=COZUM comment=defconf interface=ether19
add bridge=COZUM comment=defconf interface=ether20
add bridge=PITON comment=defconf interface=ether21
add bridge=COZUM comment=defconf interface=ether22
add bridge=COZUM comment=defconf interface=ether23
add bridge=COZUM comment=defconf disabled=yes interface="COZUM WAN"
add bridge=PITON comment=defconf interface=sfp1


/ip settings
set accept-source-route=yes allow-fast-path=no route-cache=no rp-filter=loose tcp-syncookies=yes


/interface list member
add interface=PITON list=pitonLanList
add interface=COZUM list=cozumLanList


/ip address
add address=XX.XX.XX.XX/30 comment=defconf interface="PITON WAN" network=XX.XX.XX.XZ
add address=192.168.1.1/24 interface=PITON network=192.168.1.0
add address=YY.YY.YY.YY/30 interface="COZUM WAN" network=YY.YY.YY.YZ
add address=192.168.2.1/24 interface=COZUM network=192.168.2.0


/ip dhcp-server network
add address=192.168.1.0/24 dns-server=192.168.1.1 gateway=192.168.1.1 netmask=24
add address=192.168.2.0/24 dns-server=192.168.2.1 gateway=192.168.2.1 netmask=24


/ip dns
set allow-remote-requests=yes servers=8.8.4.4,8.8.8.8


/ip firewall mangle
add action=mark-routing chain=prerouting new-routing-mark=LAN1RouteMark passthrough=no src-address=192.168.1.0/24
add action=mark-routing chain=prerouting new-routing-mark=LAN2RouteMark passthrough=no src-address=192.168.2.0/24


/ip firewall nat
add action=masquerade chain=srcnat comment=pitonWanNat dst-address=0.0.0.0/0 out-interface="PITON WAN"
add action=masquerade chain=srcnat comment=cozumWanNat dst-address=0.0.0.0/0 out-interface="COZUM WAN"


/ip route
add check-gateway=ping disabled=yes distance=1 gateway=XX.XX.XX.XA routing-mark=LAN1RouteMark
add check-gateway=ping distance=1 gateway=YY.YY.YY.YB routing-mark=LAN2RouteMark
add check-gateway=ping distance=1 gateway=XX.XX.XX.XA

With this setup at least LAN2 clients should access internet through their WAN interface right?

Also tested with Windows Client

Client gets IP address from DHCP server as follows

IP Address : 192.168.2.248
Gateway: 192.168.2.1
DNS: 192.168.2.1
Subnet: 255.255.255.0

but cannot ping 192.168.2.1

Weird is at first place to test the two LANs using two network cards of same machine because in such case the issues of response routing on the remote end complicate the situation even more, but that’s another point and is not relevant here.

What happens here is that if you use routing marks to choose a routing table, you affect also routing between local subnets. As soon as a routing mark is assigned, only routes with that routing mark are taken into account for that packet. So if you have three routing tables as below,


main: 0.0.0.0/0 -> gw.1.ip.addr
main: 192.168.1.0/24 -> lan1-interface-name (dynamically created route)
main: 192.168.2.0/24 -> lan2-interface-name (dynamically created route)

fromlan1: 0.0.0.0/0 -> gw.2.ip.addr

fromlan2: 0.0.0.0/0 -> gw3.ip.addr

then a packet marked with ****

fromlan1

to

192.168.2.0/24

is sent out via

gw.2.ip.addr

although the destination is on a local subnet, because no other route than

0.0.0.0/0

matches the destination address in routing table

fromlan1

.

So you have to either add local routes also to routing tables ****

fromlan1

and

fromlan2

, or not route-mark packets with local subnets as destinations.

So basically changing mangle rules to not use src-address but instead in-interface is sufficient to accomplish what i want to do, i am not certain but i may tried that out before.

C:\Users\Administrator>tracert google.com

Tracing route to google.com [216.58.212.46]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  192.168.2.1
  2     1 ms    <1 ms    <1 ms  XX . XX . XX . XX static.ttnet.com.tr [XX. XX XX XX]
  3    <1 ms    <1 ms    <1 ms  10.11.0.1
  4     5 ms     2 ms     3 ms  88.255.41.254.static.ttnet.com.tr [88.255.41.254]
  5     1 ms    <1 ms    <1 ms  212.175.136.1.static.ttnet.com.tr [212.175.136.1]
  6     9 ms     8 ms     9 ms  195.175.170.13.06-incesu-t2-2.26-tepebasi-t3-1.statik.turktelekom.com.tr [195.175.170.13]
  7    31 ms    31 ms    31 ms  212.156.104.110.static.turktelekom.com.tr [212.156.104.110]
  8    26 ms    26 ms    26 ms  74.125.52.6
  9    32 ms    32 ms    32 ms  108.170.250.161
 10    26 ms    26 ms    26 ms  216.239.54.5
 11    26 ms    26 ms    26 ms  sof02s18-in-f46.1e100.net [216.58.212.46]

Trace complete.

C:\Users\Administrator>

I quickly tested it and LAN2 client still can not ping 192.168.2.1 but can connect to Internet.

No, you haven’t got the point. It doesn’t matter what is the basis for assigning the routing mark, the trouble with local traffic is that there is nothing what would automatically route it properly despite the routing mark. So when we talk about traffic between LAN 1 and LAN 2, you must either prevent it from being route marked (and you cannot use out-interface for that because at the time of route marking, routing has not yet been done so the out-interface is not yet known, so you must use dst-address), or add routes for local traffic into the respective routing tables.

If you don’t mind that traffic between the LANs doesn’t pass through, you may do nothing at all, but then don’t be surprised that pings work in an unexpected way.

So connect the Windows PC to LAN1, the debian PC to LAN2, or vice versa, use route marking based on individual addresses of these two PCs as src-address in the marking rules so that you wouldn’t disturb the other customers’ work, and debug your route marking and routing tables on these two machines. Once it starts working the way you need, change the dst-address values in the route marking rules to subnet addresses and the new connections of customers’ PCs will work with route marking too.