Community discussions

MikroTik App
 
omersiar
just joined
Topic Author
Posts: 21
Joined: Mon Apr 16, 2018 2:34 pm

Dual WAN Dual LAN Setup

Wed Apr 18, 2018 9:14 am

Hello Experts,

I'm trying to setup a 2 WAN 2 LAN setup

So far i tried marking connections to a route mark but i believe i am making a mistake somewhere because as soon as i am selecting route mark on route list client's connection to Internet is failing.

I have CRS125-24G-1S routerboard.
with firmware 6.41.4
with bootloader 3.33 for ar9344

I have 2 public IPs available from the ISP (which they work as solo)

I have divided switch's ports to 2 segments:

2 Bridges as LAN1, LAN2

------------------------------------------------------------

WAN1 is on Port 0
Physical ports from 1 to 13 is for LAN1

------------------------------------------------------------

WAN2 is on Port 23
Physical ports from 14 to 23 is for LAN2

------------------------------------------------------------

My IP settings are

/ip address
add address=XX.0.XX.XX/30 comment=defconf interface="WAN1" network=XX.0.XX.XX
add address=192.168.1.1/24 interface=LAN1 network=192.168.1.0
add address=XX.0.XX.XX/30 interface="WAN2" network=XX.0.XX.XX
add address=192.168.2.1/24 interface=LAN2 network=192.168.2.0

/ip firewall mangle
add action=mark-routing chain=prerouting new-routing-mark=LAN1RouteMark passthrough=yes src-address=192.168.1.0/24
add action=mark-routing chain=prerouting new-routing-mark=LAN2RouteMark passthrough=yes src-address=192.168.2.0/24

/ip route
add check-gateway=ping disabled=yes distance=1 gateway=XX.0.XX.81 routing-mark=LAN2RouteMark
add check-gateway=ping distance=1 gateway=XX.0.XX.69

/ip firewall nat
add action=masquerade chain=srcnat comment=DEFAULT out-interface="WAN1"

As you can see LAN1 clients currently connects to internet through NAT without route mark and it works OK. But I want clients go through their own WAN addresses

Can i get a help here? Thank you.
 
omersiar
just joined
Topic Author
Posts: 21
Joined: Mon Apr 16, 2018 2:34 pm

Re: Dual WAN Dual LAN Setup

Mon Apr 23, 2018 8:53 am

Is bumping allowed here?
 
MasterXP
newbie
Posts: 25
Joined: Fri Jun 12, 2009 9:05 am

Re: Dual WAN Dual LAN Setup

Mon Apr 23, 2018 9:29 am

I don't understand very clear your problem,but, if you want to get internet by Wan2, you need a NAT Masquerade of WAN2 interface.
 
omersiar
just joined
Topic Author
Posts: 21
Joined: Mon Apr 16, 2018 2:34 pm

Re: Dual WAN Dual LAN Setup

Mon Apr 23, 2018 11:12 am

I divided my physical network into two logical network.

I want to configure router for both logical networks, so far

* I have created 2 bridges which includes physical LAN ports (1-13, 14-23) named them LAN1 and LAN2
* Configured WAN interfaces (physical ports 0 and 24)
* Configured WAN IP addresses given from ISP
* Configured DHCP servers

LAN1 clients can access internet through WAN1 (NAT'ed masquerade)

also

LAN2 clients gets IP address from DHCP but cannot access Internet through WAN2 (even if I configure masquerade NAT on WAN2 interface)

tried several approaches. I believe there can only 1 route to 0.0.0.0/0 if there is no route mark available (i know it because it does not get active)

each LAN should access internet through their corresponding WAN interface, without load balancing (PCC, etc).
 
User avatar
CZFan
Forum Guru
Forum Guru
Posts: 2098
Joined: Sun Oct 09, 2016 8:25 pm
Location: South Africa, Krugersdorp (Home town of Brad Binder)
Contact:

Re: Dual WAN Dual LAN Setup

Mon Apr 23, 2018 3:02 pm

In mangle rules, change passthrough from yes to no

In routes, change distance for routing-mark=LAN2RouteMark from 1 to 2
Add routing mark LAN1RouteMark for the other route

Add NAT for LAN 2 with out-interface=WAN2
 
omersiar
just joined
Topic Author
Posts: 21
Joined: Mon Apr 16, 2018 2:34 pm

Re: Dual WAN Dual LAN Setup

Mon Apr 23, 2018 4:12 pm

/ip address
add address=FFFFFFFFFFF/30 comment=defconf interface="WAN1" network=FFFFFFFFFFF
add address=192.168.1.1/24 interface=LAN1 network=192.168.1.0
add address=FFFFFFFFFFF/30 interface="WAN2" network=FFFFFFFFFFF
add address=192.168.2.1/24 interface=LAN2 network=192.168.2.0

/ip settings
set accept-source-route=yes allow-fast-path=no route-cache=no rp-filter=loose tcp-syncookies=yes

/ip firewall mangle
add action=mark-routing chain=prerouting new-routing-mark=LAN1RouteMark passthrough=no src-address=192.168.1.0/24
add action=mark-routing chain=prerouting new-routing-mark=LAN2RouteMark passthrough=no src-address=192.168.2.0/24

/ip firewall nat
add action=masquerade chain=srcnat dst-address=0.0.0.0/0 out-interface="WAN1"
add action=masquerade chain=srcnat dst-address=0.0.0.0/0 out-interface="WAN2"

/ip route
add check-gateway=ping distance=2 gateway=FFFFFFFFFFF routing-mark=LAN2RouteMark
add check-gateway=ping distance=1 gateway=FFFFFFFFFFF routing-mark=LAN1RouteMark
As soon as i apply these settings LAN1 clients can not connect to internet.

Please notice "/ip settings" some time ago i changed these settings in order to follow a tutorial that i do not remember what the default values are.
 
User avatar
CZFan
Forum Guru
Forum Guru
Posts: 2098
Joined: Sun Oct 09, 2016 8:25 pm
Location: South Africa, Krugersdorp (Home town of Brad Binder)
Contact:

Re: Dual WAN Dual LAN Setup

Mon Apr 23, 2018 6:16 pm

Rather place the full export here, there might be rules in firewall filter, etc that can also cause issues. Use export hide-sensitive in terminal window

Then also confirm if both WAN addresses are the same as I see you marked them the same in previous output, i.e. FFFFFFF
 
omersiar
just joined
Topic Author
Posts: 21
Joined: Mon Apr 16, 2018 2:34 pm

Re: Dual WAN Dual LAN Setup

Tue Apr 24, 2018 2:35 pm

Hello again,

Here is the full export, i still made the obfuscation by hand because of hide-sensitive did not work which i do not know why.
# apr/24/2018 14:27:49 by RouterOS 6.41.4
# software id = -
#
# model = CRS125-24G-1S
# serial number = -

/interface bridge
add fast-forward=no name=COZUM
add admin-mac=---- auto-mac=no comment=defconf name=PITON

/interface ethernet
set [ find default-name=ether24 ] name="COZUM WAN"
set [ find default-name=ether10 ] name=DELLIDRAC
set [ find default-name=ether1 ] name="PITON WAN"
set [ find default-name=ether2 ] name=SRV1-DELL
set [ find default-name=ether3 ] name=SW1-MTDAR

/interface list
add name=pitonLanList
add name=cozumLanList

/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik

/ip pool
add name=PITONPOOL ranges=192.168.1.21-192.168.1.254
add name=COZUMPOOL ranges=192.168.2.21-192.168.2.254

/ip dhcp-server
add add-arp=yes address-pool=PITONPOOL disabled=no interface=PITON name=PITONDHCP
add add-arp=yes address-pool=COZUMPOOL disabled=no interface=COZUM name=COZUMDHCP

/interface bridge port
add bridge=PITON comment=defconf disabled=yes interface="PITON WAN"
add bridge=PITON comment=defconf interface=SRV1-DELL
add bridge=PITON comment=defconf interface=SW1-MTDAR
add bridge=PITON comment=defconf interface=ether4
add bridge=PITON comment=defconf interface=ether5
add bridge=PITON comment=defconf interface=ether6
add bridge=PITON comment=defconf interface=ether7
add bridge=PITON comment=defconf interface=ether8
add bridge=PITON comment=defconf interface=ether9
add bridge=PITON comment=defconf interface=DELLIDRAC
add bridge=PITON comment=defconf interface=ether11
add bridge=PITON comment=defconf interface=ether12
add bridge=PITON comment=defconf interface=ether13
add bridge=COZUM comment=defconf interface=ether14
add bridge=COZUM comment=defconf interface=ether15
add bridge=COZUM comment=defconf interface=ether16
add bridge=COZUM comment=defconf interface=ether17
add bridge=COZUM comment=defconf interface=ether18
add bridge=COZUM comment=defconf interface=ether19
add bridge=COZUM comment=defconf interface=ether20
add bridge=PITON comment=defconf interface=ether21
add bridge=COZUM comment=defconf interface=ether22
add bridge=COZUM comment=defconf interface=ether23
add bridge=COZUM comment=defconf disabled=yes interface="COZUM WAN"
add bridge=PITON comment=defconf interface=sfp1

/ip settings
set accept-source-route=yes allow-fast-path=no route-cache=no rp-filter=loose tcp-syncookies=yes

/interface list member
add interface=PITON list=pitonLanList
add interface=COZUM list=cozumLanList

/ip address
add address=XX.XX.XX.XX/30 comment=defconf interface="PITON WAN" network=XX.XX.XX.XX
add address=192.168.1.1/24 interface=PITON network=192.168.1.0
add address=XX.XX.XX.XX/30 interface="COZUM WAN" network=XX.XX.XX.XX
add address=192.168.2.1/24 interface=COZUM network=192.168.2.0

/ip dhcp-server network
add address=192.168.1.0/24 dns-server=192.168.1.1 gateway=192.168.1.1
add address=192.168.2.0/24 dns-server=192.168.2.1 gateway=192.168.2.1 netmask=24

/ip dns
set allow-remote-requests=yes servers=8.8.4.4,8.8.8.8

/ip firewall mangle
add action=mark-routing chain=prerouting new-routing-mark=LAN1RouteMark passthrough=no src-address=192.168.1.0/24
add action=mark-routing chain=prerouting new-routing-mark=LAN2RouteMark passthrough=no src-address=192.168.2.0/24

/ip firewall nat
add action=masquerade chain=srcnat comment=pitonWanNat dst-address=0.0.0.0/0 out-interface="PITON WAN"
add action=masquerade chain=srcnat comment=cozumWanNat dst-address=0.0.0.0/0 out-interface="COZUM WAN"

/ip route
add check-gateway=ping distance=2 gateway=XX.XX.XX.XX routing-mark=LAN2RouteMark
add check-gateway=ping distance=1 gateway=XX.XX.XX.XX

/ip service
set telnet disabled=yes
set ftp disabled=yes
set www port=81
set ssh disabled=yes
set api disabled=yes
set api-ssl disabled=yes

/system identity
set name=SW2-MTGEN
 
omersiar
just joined
Topic Author
Posts: 21
Joined: Mon Apr 16, 2018 2:34 pm

Re: Dual WAN Dual LAN Setup

Tue Apr 24, 2018 2:46 pm

I think packages actually getting marked i guess

Image
 
sindy
Forum Guru
Forum Guru
Posts: 10206
Joined: Mon Dec 04, 2017 9:19 pm

Re: Dual WAN Dual LAN Setup

Tue Apr 24, 2018 4:26 pm

I'm afraid that there is simply no routing table for packets marked with routing mark
LAN1RouteMark
. So either do not route mark them at all and let them be handled by the default routing table, or do

/ip route add check-gateway=ping distance=1 gateway=XX.XX.XX.XX routing-mark=LAN1RouteMark


Remarks:
  • if two addresses you want to obfuscate differ, use different replacement patterns for them, otherwise information is lost
  • if the IP addresses of WAN 1 and WAN 2 are actualy from the same subnet and thus they really do use a common gateway IP, two routes with the same IP address as
    gateway
    will not cause any difference because both will use the same physical interface. If this is the case, you don't need two routing tables (so no routing marks) and WAN interfaces; instead, you have to assign packet marks rather than route marks using
    /ip firewall mangle
    rules and replace your
    action=masquerade
    rules by the following ones:
    /ip firewall nat
    add chain=postrouting out-interface=WAN packet-mark=LAN2Mark action=src-nat to-addresses=ip.of.wan.2 
    add chain=postrouting out-interface=WAN packet-mark=LAN1Mark action=src-nat to-addresses=ip.of.wan.1
    
  • the
    distance
    parameter of a route only makes a difference between routes with identical
    dst-address
    and
    routing-mark
    parameters.
  • hide-sensitive
    removes passwords and alike from the export, but does not replace public IP addresses by distinctive patterns, so this does require handicraft
 
User avatar
CZFan
Forum Guru
Forum Guru
Posts: 2098
Joined: Sun Oct 09, 2016 8:25 pm
Location: South Africa, Krugersdorp (Home town of Brad Binder)
Contact:

Re: Dual WAN Dual LAN Setup

Tue Apr 24, 2018 5:58 pm

Thx sindy,

yes, I was waiting for confirmation also on the WAN gateway addresses.

To add, since there is so many topics re route marking, i just tested in a lab environment and found no issues, I used RoS 6.42.1 on all devices in the below config and got results I expected everytime without fail.

/---->Hap ac lite
PC--->Hap mini
\----->951Ui-2HnD
 
omersiar
just joined
Topic Author
Posts: 21
Joined: Mon Apr 16, 2018 2:34 pm

Re: Dual WAN Dual LAN Setup

Thu Apr 26, 2018 2:53 pm

Hello sindy and CZFan,

Silly me, I just pasted same XXs for both

They are simply different.

Sorry for the late reply, the two WAN interface are on separate subnet and both have different gateway for each.
/ip route add check-gateway=ping distance=1 gateway=XX.XX.XX.XX routing-mark=LAN1RouteMark
You can not see the "routing-mark=LAN1RouteMark" part on the dump because if I activate it LAN1 clients can no longer connect to Internet.

Thank you for clearing the hide-sensitive behavior.

Disabled Mangle rules are from my previous attempts. Should I reboot the router after activating "routing-mark=LAN1RouteMark"?

I really lost some of my hair during the setup. Thank you for your kind responses.
 
sindy
Forum Guru
Forum Guru
Posts: 10206
Joined: Mon Dec 04, 2017 9:19 pm

Re: Dual WAN Dual LAN Setup

Thu Apr 26, 2018 3:10 pm

Reboot is definitely not necessary after any change. The effect of changes may become visible later (as an example, if an ssh session is already established and you add a rule preventing new ones from being established, the already established one doesn't break but once you terminate it, a new one cannot be established).

So re-insert the configuration line which normally breaks it for the group of clients but with
disabled=yes
so that it wouldn't break the clients, and post another export of the configuration with hide-sensitive but with distinctive patterns replacing the public addresses so that there is no ambiguity.

When you deal with several cases like this one every day, the context disappears from your head very quickly :-)

In your case in particular, when sessions have established via one WAN and then you activate routing via another one, all those sessions break because the remote end does not accept packets for existing session to start arriving from another address all of a sudden. So until the clients start new sessions, you cannot say whether it works or not.
 
omersiar
just joined
Topic Author
Posts: 21
Joined: Mon Apr 16, 2018 2:34 pm

Re: Dual WAN Dual LAN Setup

Thu Apr 26, 2018 3:38 pm

piton@debian:~$ ping 192.168.2.1 -I eth1
PING 192.168.2.1 (192.168.2.1) from 192.168.2.250 eth1: 56(84) bytes of data.
^C
--- 192.168.2.1 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2050ms

piton@debian:~$ ping 192.168.2.1 -I eth0
PING 192.168.2.1 (192.168.2.1) from 192.168.1.57 eth0: 56(84) bytes of data.
64 bytes from 192.168.2.1: icmp_seq=1 ttl=64 time=0.673 ms
64 bytes from 192.168.2.1: icmp_seq=2 ttl=64 time=0.428 ms
64 bytes from 192.168.2.1: icmp_seq=3 ttl=64 time=0.399 ms
64 bytes from 192.168.2.1: icmp_seq=4 ttl=64 time=0.521 ms
^C
--- 192.168.2.1 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3074ms
rtt min/avg/max/mdev = 0.399/0.505/0.673/0.108 ms
piton@debian:~$
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether e2:37:9c:2d:45:a6 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.57/24 brd 192.168.1.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::e037:9cff:fe2d:45a6/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether fa:6b:c4:5c:8d:6e brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.250/24 brd 192.168.2.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::f86b:c4ff:fe5c:8d6e/64 scope link
       valid_lft forever preferred_lft forever
I tested how ping will work and the result is confusing

Setup is like this

Linux Box with two NICs

eth0 is connected to LAN1
eth1 is connected to LAN2

both configured to get options from DHCP server

192.168.2.1 can not be pinged from eth1 interface (which its IP address is 192.168.2.250)
192.168.2.1 can be pinged from eth0 interface (which its IP address is 192.168.1.57)

weird isn't it?
 
omersiar
just joined
Topic Author
Posts: 21
Joined: Mon Apr 16, 2018 2:34 pm

Re: Dual WAN Dual LAN Setup

Thu Apr 26, 2018 3:50 pm

[admin@SW2-MTGEN] > export hide-sensitive  
# apr/26/2018 15:41:53 by RouterOS 6.41.4
# software id = ZZBI-83WG
#
# model = CRS125-24G-1S
# serial number = 787506C7C649

/interface bridge
add fast-forward=no name=COZUM
add admin-mac=6C:3B:6B:D8:C6:B0 auto-mac=no comment=defconf name=PITON


/interface ethernet
set [ find default-name=ether24 ] name="COZUM WAN"
set [ find default-name=ether10 ] name=DELLIDRAC
set [ find default-name=ether1 ] name="PITON WAN"
set [ find default-name=ether2 ] name=SRV1-DELL
set [ find default-name=ether3 ] name=SW1-MTDAR


/interface list
add name=pitonLanList
add name=cozumLanList


/ip pool
add name=PITONPOOL ranges=192.168.1.21-192.168.1.254
add name=COZUMPOOL ranges=192.168.2.21-192.168.2.254


/ip dhcp-server
add add-arp=yes address-pool=PITONPOOL disabled=no interface=PITON name=PITONDHCP
add add-arp=yes address-pool=COZUMPOOL disabled=no interface=COZUM name=COZUMDHCP


/interface bridge port
add bridge=PITON comment=defconf disabled=yes interface="PITON WAN"
add bridge=PITON comment=defconf interface=SRV1-DELL
add bridge=PITON comment=defconf interface=SW1-MTDAR
add bridge=PITON comment=defconf interface=ether4
add bridge=PITON comment=defconf interface=ether5
add bridge=PITON comment=defconf interface=ether6
add bridge=PITON comment=defconf interface=ether7
add bridge=PITON comment=defconf interface=ether8
add bridge=PITON comment=defconf interface=ether9
add bridge=PITON comment=defconf interface=DELLIDRAC
add bridge=PITON comment=defconf interface=ether11
add bridge=PITON comment=defconf interface=ether12
add bridge=PITON comment=defconf interface=ether13
add bridge=COZUM comment=defconf interface=ether14
add bridge=COZUM comment=defconf interface=ether15
add bridge=COZUM comment=defconf interface=ether16
add bridge=COZUM comment=defconf interface=ether17
add bridge=COZUM comment=defconf interface=ether18
add bridge=COZUM comment=defconf interface=ether19
add bridge=COZUM comment=defconf interface=ether20
add bridge=PITON comment=defconf interface=ether21
add bridge=COZUM comment=defconf interface=ether22
add bridge=COZUM comment=defconf interface=ether23
add bridge=COZUM comment=defconf disabled=yes interface="COZUM WAN"
add bridge=PITON comment=defconf interface=sfp1


/ip settings
set accept-source-route=yes allow-fast-path=no route-cache=no rp-filter=loose tcp-syncookies=yes


/interface list member
add interface=PITON list=pitonLanList
add interface=COZUM list=cozumLanList


/ip address
add address=XX.XX.XX.XX/30 comment=defconf interface="PITON WAN" network=XX.XX.XX.XZ
add address=192.168.1.1/24 interface=PITON network=192.168.1.0
add address=YY.YY.YY.YY/30 interface="COZUM WAN" network=YY.YY.YY.YZ
add address=192.168.2.1/24 interface=COZUM network=192.168.2.0


/ip dhcp-server network
add address=192.168.1.0/24 dns-server=192.168.1.1 gateway=192.168.1.1 netmask=24
add address=192.168.2.0/24 dns-server=192.168.2.1 gateway=192.168.2.1 netmask=24


/ip dns
set allow-remote-requests=yes servers=8.8.4.4,8.8.8.8


/ip firewall mangle
add action=mark-routing chain=prerouting new-routing-mark=LAN1RouteMark passthrough=no src-address=192.168.1.0/24
add action=mark-routing chain=prerouting new-routing-mark=LAN2RouteMark passthrough=no src-address=192.168.2.0/24


/ip firewall nat
add action=masquerade chain=srcnat comment=pitonWanNat dst-address=0.0.0.0/0 out-interface="PITON WAN"
add action=masquerade chain=srcnat comment=cozumWanNat dst-address=0.0.0.0/0 out-interface="COZUM WAN"


/ip route
add check-gateway=ping disabled=yes distance=1 gateway=XX.XX.XX.XA routing-mark=LAN1RouteMark
add check-gateway=ping distance=1 gateway=YY.YY.YY.YB routing-mark=LAN2RouteMark
add check-gateway=ping distance=1 gateway=XX.XX.XX.XA



With this setup at least LAN2 clients should access internet through their WAN interface right?
 
omersiar
just joined
Topic Author
Posts: 21
Joined: Mon Apr 16, 2018 2:34 pm

Re: Dual WAN Dual LAN Setup

Thu Apr 26, 2018 4:03 pm

Also tested with Windows Client

Client gets IP address from DHCP server as follows

IP Address : 192.168.2.248
Gateway: 192.168.2.1
DNS: 192.168.2.1
Subnet: 255.255.255.0

but cannot ping 192.168.2.1
 
sindy
Forum Guru
Forum Guru
Posts: 10206
Joined: Mon Dec 04, 2017 9:19 pm

Re: Dual WAN Dual LAN Setup

Thu Apr 26, 2018 4:22 pm

Weird is at first place to test the two LANs using two network cards of same machine because in such case the issues of response routing on the remote end complicate the situation even more, but that's another point and is not relevant here.

What happens here is that if you use routing marks to choose a routing table, you affect also routing between local subnets. As soon as a routing mark is assigned, only routes with that routing mark are taken into account for that packet. So if you have three routing tables as below,

main: 0.0.0.0/0 -> gw.1.ip.addr
main: 192.168.1.0/24 -> lan1-interface-name (dynamically created route)
main: 192.168.2.0/24 -> lan2-interface-name (dynamically created route)

fromlan1: 0.0.0.0/0 -> gw.2.ip.addr

fromlan2: 0.0.0.0/0 -> gw3.ip.addr


then a packet marked with
fromlan1
to
192.168.2.0/24
is sent out via
gw.2.ip.addr
although the destination is on a local subnet, because no other route than
0.0.0.0/0
matches the destination address in routing table
fromlan1
.

So you have to either add local routes also to routing tables
fromlan1
and
fromlan2
, or not route-mark packets with local subnets as destinations.
 
omersiar
just joined
Topic Author
Posts: 21
Joined: Mon Apr 16, 2018 2:34 pm

Re: Dual WAN Dual LAN Setup

Thu Apr 26, 2018 4:44 pm

So basically changing mangle rules to not use src-address but instead in-interface is sufficient to accomplish what i want to do, i am not certain but i may tried that out before.
 
omersiar
just joined
Topic Author
Posts: 21
Joined: Mon Apr 16, 2018 2:34 pm

Re: Dual WAN Dual LAN Setup

Thu Apr 26, 2018 4:49 pm

C:\Users\Administrator>tracert google.com

Tracing route to google.com [216.58.212.46]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  192.168.2.1
  2     1 ms    <1 ms    <1 ms  XX . XX . XX . XX static.ttnet.com.tr [XX. XX XX XX]
  3    <1 ms    <1 ms    <1 ms  10.11.0.1
  4     5 ms     2 ms     3 ms  88.255.41.254.static.ttnet.com.tr [88.255.41.254]
  5     1 ms    <1 ms    <1 ms  212.175.136.1.static.ttnet.com.tr [212.175.136.1]
  6     9 ms     8 ms     9 ms  195.175.170.13.06-incesu-t2-2.26-tepebasi-t3-1.statik.turktelekom.com.tr [195.175.170.13]
  7    31 ms    31 ms    31 ms  212.156.104.110.static.turktelekom.com.tr [212.156.104.110]
  8    26 ms    26 ms    26 ms  74.125.52.6
  9    32 ms    32 ms    32 ms  108.170.250.161
 10    26 ms    26 ms    26 ms  216.239.54.5
 11    26 ms    26 ms    26 ms  sof02s18-in-f46.1e100.net [216.58.212.46]

Trace complete.

C:\Users\Administrator>
I quickly tested it and LAN2 client still can not ping 192.168.2.1 but can connect to Internet.
Last edited by omersiar on Thu Apr 26, 2018 4:57 pm, edited 1 time in total.
 
sindy
Forum Guru
Forum Guru
Posts: 10206
Joined: Mon Dec 04, 2017 9:19 pm

Re: Dual WAN Dual LAN Setup

Thu Apr 26, 2018 4:57 pm

So basically changing mangle rules to not use src-address but instead in-interface is sufficient to accomplish what i want to do, i am not certain but i may tried that out before.
No, you haven't got the point. It doesn't matter what is the basis for assigning the routing mark, the trouble with local traffic is that there is nothing what would automatically route it properly despite the routing mark. So when we talk about traffic between LAN 1 and LAN 2, you must either prevent it from being route marked (and you cannot use out-interface for that because at the time of route marking, routing has not yet been done so the out-interface is not yet known, so you must use dst-address), or add routes for local traffic into the respective routing tables.

If you don't mind that traffic between the LANs doesn't pass through, you may do nothing at all, but then don't be surprised that pings work in an unexpected way.

So connect the Windows PC to LAN1, the debian PC to LAN2, or vice versa, use route marking based on individual addresses of these two PCs as src-address in the marking rules so that you wouldn't disturb the other customers' work, and debug your route marking and routing tables on these two machines. Once it starts working the way you need, change the dst-address values in the route marking rules to subnet addresses and the new connections of customers' PCs will work with route marking too.
 
omersiar
just joined
Topic Author
Posts: 21
Joined: Mon Apr 16, 2018 2:34 pm

Re: Dual WAN Dual LAN Setup

Thu Apr 26, 2018 5:10 pm

Now I am really lost.

But now i am close to what I want to get done.

Now both LAN clients can connect to internet trough their WAN interface.

No route marking done on LAN1 and they connect through WAN1 route
add action=mark-routing chain=prerouting in-interface=COZUM new-routing-mark=LAN2RouteMark passthrough=no
With this configuration LAN2 clients can connect through WAN2 route

Why LAN2 clients can not ping 192.168.2.1?
How LAN2 clients can connect Internet?
Why LAN1 clients can ping 192.168.2.1?
Why mangle rule for LAN1 does not work?

I just want to divide networks into two separate networks, like having two router board, but instead just using one router board?

Is it even feasible?
 
sindy
Forum Guru
Forum Guru
Posts: 10206
Joined: Mon Dec 04, 2017 9:19 pm

Re: Dual WAN Dual LAN Setup

Thu Apr 26, 2018 5:18 pm

Why LAN2 clients can not ping 192.168.2.1?
Because their packets are redirected to WAN2's gateway as you didn't exclude packets with local addresses as destinations from route-marking.
How LAN2 clients can connect Internet?
Because there is a route to internet (via WAN2's gateway) in the routing table for routing mark FromLAN2
Why LAN1 clients can ping 192.168.2.1?
Because you don't route-mark packets from LAN1 clients, so the default ("main") routing table is used, and 192.168.2.1 is one of addresses of the Mikrotik itself. It doesn't matter that it is not in LAN1's subnet.
Why mangle rule for LAN1 does not work?
This one I cannot answer without seeing the export of complete configuration while it does not work. Is it the one in post #15 or you've made some changes in the meantime?
I just want to divide networks into two separate networks, like having two router board, but instead just using one router board?
Is it even feasible?
Yes, it is.
 
omersiar
just joined
Topic Author
Posts: 21
Joined: Mon Apr 16, 2018 2:34 pm

Re: Dual WAN Dual LAN Setup

Thu Apr 26, 2018 5:45 pm

/interface bridge
add fast-forward=no name=COZUM
add admin-mac=6C:3B:6B:D8:C6:B0 auto-mac=no comment=defconf name=PITON

/interface ethernet
set [ find default-name=ether24 ] name="COZUM WAN"
set [ find default-name=ether10 ] name=DELLIDRAC
set [ find default-name=ether1 ] name="PITON WAN"
set [ find default-name=ether2 ] name=SRV1-DELL
set [ find default-name=ether3 ] name=SW1-MTDAR

/interface list
add name=pitonLanList
add name=cozumLanList

/ip pool
add name=PITONPOOL ranges=192.168.1.21-192.168.1.254
add name=COZUMPOOL ranges=192.168.2.21-192.168.2.254

/ip dhcp-server
add add-arp=yes address-pool=PITONPOOL disabled=no interface=PITON name=PITONDHCP
add add-arp=yes address-pool=COZUMPOOL disabled=no interface=COZUM name=COZUMDHCP

/interface bridge port
add bridge=PITON comment=defconf disabled=yes interface="PITON WAN"
add bridge=PITON comment=defconf interface=SRV1-DELL
add bridge=PITON comment=defconf interface=SW1-MTDAR
add bridge=PITON comment=defconf interface=ether4
add bridge=PITON comment=defconf interface=ether5
add bridge=PITON comment=defconf interface=ether6
add bridge=PITON comment=defconf interface=ether7
add bridge=PITON comment=defconf interface=ether8
add bridge=PITON comment=defconf interface=ether9
add bridge=PITON comment=defconf interface=DELLIDRAC
add bridge=PITON comment=defconf interface=ether11
add bridge=PITON comment=defconf interface=ether12
add bridge=PITON comment=defconf interface=ether13
add bridge=COZUM comment=defconf interface=ether14
add bridge=COZUM comment=defconf interface=ether15
add bridge=COZUM comment=defconf interface=ether16
add bridge=COZUM comment=defconf interface=ether17
add bridge=COZUM comment=defconf interface=ether18
add bridge=COZUM comment=defconf interface=ether19
add bridge=COZUM comment=defconf interface=ether20
add bridge=PITON comment=defconf interface=ether21
add bridge=COZUM comment=defconf interface=ether22
add bridge=COZUM comment=defconf interface=ether23
add bridge=COZUM comment=defconf disabled=yes interface="COZUM WAN"
add bridge=PITON comment=defconf interface=sfp1

/ip settings
set accept-source-route=yes allow-fast-path=no route-cache=no rp-filter=loose tcp-syncookies=yes

/interface list member
add interface=PITON list=pitonLanList
add interface=COZUM list=cozumLanList

/ip address
add address=XX.XX.XX.XX/30 comment=defconf interface="PITON WAN" network=XX.XX.XX.XX
add address=192.168.1.1/24 interface=PITON network=192.168.1.0
add address=YY.YY.YY.YY/30 interface="COZUM WAN" network=YY.YY.YY.YY
add address=192.168.2.1/24 interface=COZUM network=192.168.2.0

/ip dhcp-server network
add address=192.168.1.0/24 dns-server=192.168.1.1 gateway=192.168.1.1 netmask=24
add address=192.168.2.0/24 dns-server=192.168.2.1 gateway=192.168.2.1 netmask=24

/ip dns
set allow-remote-requests=yes servers=8.8.4.4,8.8.8.8

/ip firewall mangle
add action=mark-routing chain=prerouting in-interface=PITON new-routing-mark=LAN1RouteMark passthrough=no
add action=mark-routing chain=prerouting in-interface=COZUM new-routing-mark=LAN2RouteMark passthrough=no

/ip firewall nat
add action=masquerade chain=srcnat comment=pitonWanNat dst-address=0.0.0.0/0 out-interface="PITON WAN"
add action=masquerade chain=srcnat comment=cozumWanNat dst-address=0.0.0.0/0 out-interface="COZUM WAN"

/ip route
add check-gateway=ping distance=1 gateway=XX.XX.XX.XX routing-mark=LAN1RouteMark
add check-gateway=ping distance=1 gateway=YY.YY.YY.YY routing-mark=LAN2RouteMark
add check-gateway=ping disabled=yes distance=1 gateway=XX.XX.XX.XX

This is not work

unless i disable
add action=mark-routing chain=prerouting in-interface=PITON new-routing-mark=LAN1RouteMark passthrough=no
and
add check-gateway=ping distance=1 gateway=XX.XX.XX.XX routing-mark=LAN1RouteMark
and enable
add check-gateway=ping disabled=yes distance=1 gateway=XX.XX.XX.XX
 
sindy
Forum Guru
Forum Guru
Posts: 10206
Joined: Mon Dec 04, 2017 9:19 pm

Re: Dual WAN Dual LAN Setup

Thu Apr 26, 2018 6:09 pm

Okay. So now please replace, in
/ip firewall mangle
,
add action=mark-routing chain=prerouting in-interface=PITON new-routing-mark=LAN1RouteMark passthrough=no
by
add action=mark-routing chain=prerouting src-address=ip.of.test.pc new-routing-mark=LAN1RouteMark passthrough=no
and re-enable that rule and the route
gateway=XX.XX.XX.XX routing-mark=LAN1RouteMark
. Do not disable the route
gateway=XX.XX.XX.XX
(the one without any routing mark).

This way, LAN1 clients except the test PC should work as they do now (using the default routing table because they won't be route-marked), LAN2 clients should work as they do now, and the PC may or may not work.

Please confirm that LAN1 clients still work after the changes above, post the output of
/ip route print detail
(after obfuscating the addresses), and tell me whether the PC in LAN1 (which is the only one to be route-marked) can get to internet or not.
 
omersiar
just joined
Topic Author
Posts: 21
Joined: Mon Apr 16, 2018 2:34 pm

Re: Dual WAN Dual LAN Setup

Sat Apr 28, 2018 10:37 am

Hello sindy,

Thanks for not giving up on me.

I have done what you have asked for, result is as follows:

Test PC (192.168.1.10) can connect to Internet without an issue.
Test PC can not ping router's ip addresses (192.168.2.1, 192.168.1.1)
Test PC can ping 8.8.8.8 (as expected because it can connect to Internet)
Test PC can ping other LAN1 clients
Test PC's route trace is as expected
I can see packet count increasing on Test PC's mangle rule, so i assume it is working.
Other LAN1 clients can connect to internet.
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit 
 0 A S  dst-address=0.0.0.0/0 gateway=XX.XX.XX.XX gateway-status=XX.XX.XX.XX reachable via  PITON WAN check-gateway=ping distance=1 scope=30 target-scope=10 
        routing-mark=LAN1RouteMark 

 1 A S  dst-address=0.0.0.0/0 gateway=YY.YY.YY.YY gateway-status=YY.YY.YY.YY reachable via  COZUM WAN check-gateway=ping distance=1 scope=30 target-scope=10 
        routing-mark=LAN2RouteMark 

 2 A S  dst-address=0.0.0.0/0 gateway=XX.XX.XX.XX gateway-status=XX.XX.XX.XX reachable via  PITON WAN check-gateway=ping distance=1 scope=30 target-scope=10 

 3 ADC  dst-address=XX.XX.XX.XA/30 pref-src=XX.XX.XX.XX gateway=PITON WAN gateway-status=PITON WAN reachable distance=0 scope=10 

 4 ADC  dst-address=YY.YY.YY.YB/30 pref-src=YY.YY.YY.YY gateway=COZUM WAN gateway-status=COZUM WAN reachable distance=0 scope=10 

 5 ADC  dst-address=192.168.1.0/24 pref-src=192.168.1.1 gateway=PITON gateway-status=PITON reachable distance=0 scope=10 

 6 ADC  dst-address=192.168.2.0/24 pref-src=192.168.2.1 gateway=COZUM gateway-status=COZUM reachable distance=0 scope=10 
 
sindy
Forum Guru
Forum Guru
Posts: 10206
Joined: Mon Dec 04, 2017 9:19 pm

Re: Dual WAN Dual LAN Setup

Sat Apr 28, 2018 11:54 am

Test PC (192.168.1.10) can connect to Internet without an issue.
Test PC can ping 8.8.8.8 (as expected because it can connect to Internet)
Test PC can not ping router's ip addresses (192.168.2.1, 192.168.1.1)
I can see packet count increasing on Test PC's mangle rule, so i assume it is working.
So as you can see, routing mark is doing something, otherwise the test PC would behave exactly the same like other LAN1 clients.
The reason why the test PC can ping internet is because a default route in routing table LAN1RouteMark exists.
The reason why the test PC cannot ping Mikrotik's own IP addresses (192.168.2.1, 192.168.1.1) is that all packets from test PC's address are marked with LAN1RouteMark and no other route than the default one exists in routing table
LAN1RouteMark
. So even packets for 192.168.1.0/24 and 192.168.2.0/24 which reach Mikrotik's IP stack are sent out via the default gateway.
Test PC can ping other LAN1 clients
The reason is that packets from the test PC to other LAN1 clients are sent directly to their MAC addresses so they are not routed by Mikrotik, only bridged, so the mangle rules never see them.
Other LAN1 clients can connect to internet.
The reason is that you haven't removed the default route from the default routing table ("main").


Now we can proceed further. As to "access internet", the LAN1 clients not only need to be able to ping servers in the internet but also need to be able to resolve those servers' DNS names to IP addresses, the LAN1 clients must be able to talk to DNS server(s). If their configuration said that the DNS was somewhere outside, the DNS server would be a server like any other one so an available route to internet would be sufficient. However, your configuration shows that at least some of them get configuration from the Mikrotik via DHCP, and the Mikrotik tells them to use itself as a DNS server. So when a LAN device needs to know what is the IP address of
www.turkcell.com.tr
,
  1. the LAN device sends a DNS query to Mikrotik,
  2. if Mikrotik doesn't have a previous DNS answer cached, it sends a query in its own name (so it is not a forwarded packet from the client but a locally originated one) to its configured DNS server in the internet.
In your previous configuration, you've broken both the steps above:
  1. as the mangle rule was route marking all packets from LAN1, the client's DNS query to Mikrotik's local address was not received by Mikrotik but forwarded to the default gateway which doesn't know that it should forward packets to 192.168.1.1 back to Mikrotik,
  2. as you have removed the default route from the default routing table, the DNS query sent by Mikrotik (if it would be generated) could not be sent because locally originated packets do not pass
    chain=prerouting
    of
    /ip firewall mangle
    , so they were not marked with any of the two routing marks.
So take your
/ip firewall mangle
rule which route-marks packets coming from the test PC and add
dst-adddress=!192.168.1.1
ot its match conditions, and do not remove the default route from the default routing table so that Mikrotik could query the DNS (and maybe synchronize time etc.).

If after this modification the test PC starts working completely as you'd expect, you can modify the route-marking rule back so that it matches on packets with
in-interface=PITON
instead of
src-address=ip.of.test.pc
, keeping the
dst-adddress=!192.168.1.1
in place, and all clients in LAN1 will be treated the same way like the test PC.

Basically you should modify the same way (add
dst-adddress=!192.168.2.1
also the rule which route-marks the packets with
new-routing-mark=LAN2RouteMark
. In fact it's not really clear to me how LAN2 clients could have been treated better than LAN1 clients.
 
omersiar
just joined
Topic Author
Posts: 21
Joined: Mon Apr 16, 2018 2:34 pm

Re: Dual WAN Dual LAN Setup

Sat Apr 28, 2018 2:58 pm

Hello sindy,

Now all clients working as expected. What do you think of this setup in perspective of performance and being future proof? For example will this make firewall rules more complicated?

If you were in my shoos will you set it up like this or choose something else?

Thanks for the help
 
sindy
Forum Guru
Forum Guru
Posts: 10206
Joined: Mon Dec 04, 2017 9:19 pm

Re: Dual WAN Dual LAN Setup

Sat Apr 28, 2018 6:39 pm

If you were in my shoos will you set it up like this or choose something else?
From high level perspective the only "something else" is a second router, so I cannot see any reason to change the approach as long as the current one deals with the total required throughput. Far more complex configurations are used around the globe.

What do you think of this setup in perspective of performance and being future proof? For example will this make firewall rules more complicated?
I have little experience with throughput limits of these larger 'Tiks, and I don't know your traffic structure. So just the generic recommendations:
  • if a significant share of LAN traffic runs between LAN devices, it is much better to let the switch chip deal with it, leaving as much as possible CPU throughput for tasks it has to deal with. So in your case, it would mean to create one VLAN for Piton and another one for Çözüm using a single common bridge for both, thus allowing forwarding between ports to be handled by the switch chip (only one brigde can be "hardware accelerated" this way, for all eventual other ones the forwarding between ports of the same bridge loads the CPU). Internally, 192.168.1.1/24 would then be attached to one
    /interface vlan
    with this bridge as the carrying interface and one VLAN ID (like e.g. 11, get used to avoid using VLAN ID 1 as it sometimes has some implicit meaning), and 192.168.2.1/24 to another
    /interface vlan
    with the same bridge as the carrying interface and another VLAN ID, and ethernet ports which are now slaves of the two different bridges would be made access ports to one of the VLANs on that single bridge.
  • direct marking of packets with routing mark is fine while the matching rules are simple (which is currently your case) and while you don't need to place servers accessible from the internet to the LAN. As soon as one of these conditions is not met any more, it is better to use complex matching rules to mark connections when handling their initial packets, and for all other packets to just translate connection marks into routing marks.
  • to save some CPU, it is also good to avoid marking of the type of traffic which uses most bandwidth. Packet marking is incompatible with fasttracking, but if you only mark exceptions from the bulk, you can use the default routing table and thus also fasttracking for the bulk.
As for firewall rules, if you use interface lists and address lists, most of the rules remain the same regardless whether you use a single LAN or 30 LANs. As long as you handle packets between any LAN and the internet the same way and forbid direct routing between the LANs, which is often the case, a single common set of rules can handle all of them. The only rules you may need to add would be some exceptions permitting packets to be directly routed between two LANs if it is necessary, and some care needs to be taken if e.g. clients of one LAN should access servers forwarded from a public IP bound to another LAN, but that's nothing complex once you start understanding the firewall.

The configuration you've posted doesn't contain any firewall filter rules, so I hope your machine is not yet exposed to the internet, as otherwise it may have been already hacked if you don't run the latest RouterOS, see the recent vulnerabity-related topics on this forum. So if I wore your shoes, firewall rules would be the first thing to deal with now. See my turbo-introduction to the firewall and once you catch the principle, look for some more detailed tutorial, there are plenty of them.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 19352
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Dual WAN Dual LAN Setup

Sat Apr 28, 2018 6:58 pm

I too have a dual WAN and dual LAN setup.
The questions I needed to answer were:
1. Do I want to use One WAN and have the other for failover, or load balance(much more complicated to setup for the first time).
2. Do I want my LANs to be separated from seeing each other at layer 2 or layer 3.

For me:
1. The former, simple fail-over which makes Routing easier.
2. Yes at both layers so I defined two bridges and assigned (in address list) one lan to each bridge and thus they are blocked from each other at layer 2
(just putting one on a bridge and the other NOT, has the same effect).
3. For Layer 3 blocking I made Filter RUles Forward Chain, Lan1 to lan2 drop and lan 2 to lan 1 drop.
Ex.
add action=drop chain=forward comment="Block DMZ to LAN" dst-address=\
192.168.0.0/24 in-interface=DMZ_Bridge src-address=192.168.2.0/24
add action=drop chain=forward comment="Block LAN to DMZ" dst-address=\
192.168.2.0/24 in-interface=HomeBridge src-address=192.168.0.0/24

4. For Routing purposes I just ensured I had two default routes, one to the primary distance = 1 ping gateway and the
second, simply distance= 2 (destination in both 0.0.0.0/0 and gateway IP for each ISP (not the assigned wanip).
5. Since my email goes out through WAN2 I had to make a mangle rule to identify which traffic and a route rule to ensure such traffic went out WAN2
Ex.
/ip firewall mangle
add action=mark-routing chain=prerouting comment=Traffic_4_Email \
dst-port=25 in-interface-list=LAN new-routing-mark=email\
passthrough=no protocol=tcp src-address=0.0.0.0/0

(come to think of it using 0.0.0.0/0 for source address is probably not a good security practice but someone will hopefully tell me if so)

/ip route
add comment=Email_bypass distance=1 dst-address=actual IP of ISP email server gateway=\
actual gateway IP of my WAN2 ISP routing-mark=email
 
sindy
Forum Guru
Forum Guru
Posts: 10206
Joined: Mon Dec 04, 2017 9:19 pm

Re: Dual WAN Dual LAN Setup

Sat Apr 28, 2018 7:38 pm

4. For Routing purposes I just ensured I had two default routes, one to the primary distance = 1 ping gateway and the
second, simply distance= 2 (destination in both 0.0.0.0/0 and gateway IP for each ISP (not the assigned wanip).
This nice article explains why it is much better to
check
some IP address out there than the gateway and how to do it.

5. Since my email goes out through WAN2 I had to make a mangle rule to identify which traffic and a route rule to ensure such traffic went out WAN2
Ex.
/ip firewall mangle
add action=mark-routing chain=prerouting comment=Traffic_4_Email \
dst-port=25 in-interface-list=LAN new-routing-mark=email\
passthrough=no protocol=tcp src-address=0.0.0.0/0

(come to think of it using 0.0.0.0/0 for source address is probably not a good security practice but someone will hopefully tell me if so)

/ip route
add comment=Email_bypass distance=1 dst-address=actual IP of ISP email server gateway=\
actual gateway IP of my WAN2 ISP routing-mark=email
The above is actually an overkill.
/ip route add comment=Email_bypass distance=1 dst-address=actual.IP.of.ISP.email.server gateway=actual.gateway.IP.of.my.WAN2.ISP
(so staying in the default routing table) would do the same job without need to first route-mark the packets in mangle. The way you've set it up is only necessary if you only need to force SMTP packets towards that IP via WAN2 but any other packet towards the same IP may take the WAN1 way.

I mean, keeping the number of rules as low as possible saves CPU, and use of packet marking only where necessary saves CPU very much as not only that the marking rules need not exist but you may use fasttracking for non-marked packets.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 19352
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Dual WAN Dual LAN Setup

Sat Apr 28, 2018 9:38 pm

The above is actually an overkill.
/ip route add comment=Email_bypass distance=1 dst-address=actual.IP.of.ISP.email.server gateway=actual.gateway.IP.of.my.WAN2.ISP
(so staying in the default routing table) would do the same job without need to first route-mark the packets in mangle. The way you've set it up is only necessary if you only need to force SMTP packets towards that IP via WAN2 but any other packet towards the same IP may take the WAN1 way.

I mean, keeping the number of rules as low as possible saves CPU, and use of packet marking only where necessary saves CPU very much as not only that the marking rules need not exist but you may use fasttracking for non-marked packets.
...
Ahhhhhhhh I think I understand what you are saying!!! but I think you are missing something, or I do not understand how email works.
If I use the rule you stated below then lets follow the logic:
a. user sends email (port 25), other than that there is nothing special identifying email traffic!
b. router looks at traffic coming from LAN and applies the Route rules.
c. It attempts to send the email traffic out WAN1 (distance=1) but the traffic is not accepted.
d. The traffic is dropped.

I agree with you that if the email had a destination in the packets so that the router could then match it to the 3rd Route Rule, then I can understand your thinking.
But to my knowledge its just dummy traffic coming from the email program and that is why I have to tell it specifically to go out a certain WAN port.
My email program is not sending out email IP info or email ISP gateway IP Info for the router to inspect and thus match!!

Okay perhaps what you mean is that we order the distance of the rules ????
distance = 1 for email rule
distance = 2 ping gateway for WAN1 primary
ditance = 3 for WAN2 secondary

However you spoke of efficiency! In the case above every outgoing packet is checked against the first route which is very wasteful?? (IF routes works this way)
The Mangle is done in prerouting which is reasonably efficient and only affects a small miniscule amount of traffic!


PS. I read the article and it was interesting. I cannot think of why my ISPs gateway would be up but its connection to the internet down. Seems like a rare occurrence to me and thus not worth the hassle of the extra programming. By the way, it seems like the more complete methods still make use of distance and pings but are separated out with the pinging determining hosts availability and the distance strictly for routing order based on what is available via pinging.
However I have no ideas why the article created virtual hops and why the author selected the ISP gateways as the IP address for destination......... ?????
 
sindy
Forum Guru
Forum Guru
Posts: 10206
Joined: Mon Dec 04, 2017 9:19 pm

Re: Dual WAN Dual LAN Setup

Sat Apr 28, 2018 10:36 pm

If I use the rule you stated below then lets follow the logic:
a. user sends email (port 25), other than that there is nothing special identifying email traffic!
b. router looks at traffic coming from LAN and applies the Route rules.
c. It attempts to send the email traffic out WAN1 (distance=1) but the traffic is not accepted.
d. The traffic is dropped.

I agree with you that if the email had a destination in the packets so that the router could then match it to the 3rd Route Rule, then I can understand your thinking.

But to my knowledge its just dummy traffic coming from the email program and that is why I have to tell it specifically to go out a certain WAN port.
My email program is not sending out email IP info or email ISP gateway IP Info for the router to inspect and thus match!!
A destination TCP port 25 identifies SMTP (e-mail sending), that's true. Without combining it with a particular destination address, the rule matches SMTP packets being sent to any SMTP server in the world, not just your ISP's one.
Now think the other way round, how likely it is that you will need to contact your ISP's SMTP server for any other communication than sending e-mails? As you've created a special route for that server's IP, I guess you don't want any SMTP connection to be routed via WAN2 but only those towards that single server.

Actually your current rules drop any other SMTP traffic than to the ISP's server, because any SMTP traffic gets its own routing table and that routing table only contains a route to the ISP's server, so packets to any other address will be dropped.

Okay perhaps what you mean is that we order the distance of the rules ????
distance = 1 for email rule
distance = 2 ping gateway for WAN1 primary
distance = 3 for WAN2 secondary
That's a common misconception. The
distance
only decides between routes with identical
dst-address
prefix and
routing-mark
. If you have a route with longer (=more narrow) prefix and
distance=2
, it beats a route with shorter (=wider) prefix and
distance=1
, because the routes are first chosen by best match of the prefix and only if several routes with the same prefix legth match, the
distance
is taken into account to choose between them.

However you spoke of efficiency! In the case above every outgoing packet is checked against the first route which is very wasteful?? (IF routes works this way)
The Mangle is done in prerouting which is reasonably efficient and only affects a small miniscule amount of traffic!
This is a valid remark. Inspecting every packet for being a TCP one towards port 25 may be equally CPU consuming as checking one extra route for every packet. I'm not sure how exactly the route matching is done, but I know for sure that the routes once found are cached which makes the routing of second and later packets independent of the number of routes defined.

PS. I read the article and it was interesting. I cannot think of why my ISPs gateway would be up but its connection to the internet down.
Because shit happens :-) If your ISP uses redundant connections of devices to which subscriber lines are connected, congratulations, yet still if you spend the effort to make your uplink redundant, it seems strange to stay halfway there and only protect yourself against a failure of the last hop if you can do just a little bit more and be protected against ISP's network problems as well.

"Programming" would mean scripting to me in this context, while the recursive routes make scripting unnecessary for the task, configuration is enough.

However I have no ideas why the article created virtual hops and why the author selected the ISP gateways as the IP address for destination?
Because that's how recursive routing works in RouterOS. You must have a route to the monitored element via a physical gateway (the ISP's one), but if you then indicate that monitored element as a gateway in other routes, these other routes determine the address of the physical gateway recursively (so the packets are sent to the MAC address of the physical gateway although the IP address of the "virtual" one is specified in the route), but the availability of these routes is determined by availability of the monitored element.
 
Sob
Forum Guru
Forum Guru
Posts: 9121
Joined: Mon Apr 20, 2009 9:11 pm

Re: Dual WAN Dual LAN Setup

Sat Apr 28, 2018 10:36 pm

@anav: I got little lost in your post, but I think @sindy assumed that you need to send mail to one specific server. In that case, you could simply route all traffic to that server's address via WAN2. If you need to connect to different servers, your solution is ok.

Although maybe not, but for another reason. Port 25 is for mail servers (server to server communication), clients are supposed to use port 587. It's basically just smtp on different port, but the major difference is that it has mandatory authentication, so it can't be used for sending spam from anonymous users, and in turn ISPs don't have any reason to block access to this port.

And about routing, destination is looked up in routing table for all packets. That's what routers do, and they are very good at it. One extra rule will make no noticeable difference.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 19352
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Dual WAN Dual LAN Setup

Sun Apr 29, 2018 3:34 am

Sob Sindy,
I am not being obtuse I dont think but I fail to see how
email traffic is going to go out the route you created.
The route knows nothing about port 25 and the email traffic knows nothing about which IP address its supposed to go out of.
Thus my mangle route.

Where is my thinking wrong??
 
Sob
Forum Guru
Forum Guru
Posts: 9121
Joined: Mon Apr 20, 2009 9:11 pm

Re: Dual WAN Dual LAN Setup

Sun Apr 29, 2018 4:41 am

As already written, the route would work, if you'd need to connect to just one server. The route doesn't know anything about ports, that's right. It would work simply because all traffic to given address would use the route, and all traffic includes port 25.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 19352
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Dual WAN Dual LAN Setup

Sun Apr 29, 2018 5:59 am

As already written, the route would work, if you'd need to connect to just one server. The route doesn't know anything about ports, that's right. It would work simply because all traffic to given address would use the route, and all traffic includes port 25.
Yes, but I have two WANs and the primary WAN is ISP1, which is not my email ISP.
THe failover ISP is my WAN2.
So once again I fail to see how the rule provided by sindy would permit my email traffic to route properly.
 
Sob
Forum Guru
Forum Guru
Posts: 9121
Joined: Mon Apr 20, 2009 9:11 pm

Re: Dual WAN Dual LAN Setup

Sun Apr 29, 2018 6:19 am

I'd like to explain it, but I don't know how to do it better than the rule itself:
/ip route add dst-address=<mail server address> gateway=<WAN2 gateway>
I.e. if the target address is <mail server address>, then packets will be sent via <WAN2 gateway>.
 
sindy
Forum Guru
Forum Guru
Posts: 10206
Joined: Mon Dec 04, 2017 9:19 pm

Re: Dual WAN Dual LAN Setup

Sun Apr 29, 2018 9:40 am

I'd like to explain it, but I don't know how to do it better than the rule itself:
/ip route add dst-address=<mail server address> gateway=<WAN2 gateway>
I.e. if the target address is <mail server address>, then packets will be sent via <WAN2 gateway>.
The tiny bit missing may be that the
WAN2 gateway
is an IP address in the subnet of WAN2, and this is enough to send packets via that gateway from WAN2.

It's actually a definition of a gateway, it is a device in a network to which your own device is directly connected which is capable to forward packets to other networks. So when the router finds that packet for
x.x.x.x
has to be sent via gateway
y.y.y.y
, it first of all looks for a local interface whose IP address is in the same subnet as
y.y.y.y
. From this interface it sends an ARP broadcast packet asking "who has IP y.y.y.y? Tell me your MAC address". The gateway answers, the router remembers that MAC address for a while, and sends the actual packet still with
x.x.x.x
as destination IP address to the MAC address of the gateway. So the IP address of the gateway is not used for anything else than determining its MAC address.

Other than that, your existing rules do not redirect packets for TCP port 25 to the address of ISP2's SMTP server, so the address of the server must be in your e-mail client's configuration, maybe in the form of a DNS name (like
smtp.the-isp.ca
) which resolves to the same IP address you use as
dst-address
in your
route
. So if one day the ISP guys decide to shuffle their network a bit, your e-mail sending may break unexpectedly because the dns name will be translated to a different IP address but the exceptional route handles only packets for the old one, so packets to the new one will leave via WAN1 and thus be rejected by the SMTP server.

For ISP's SMTP servers this happens rarely; for globally present services, it is rather a rule that a DNS name resolves to a list of IP addresses and the list changes frequently, because this is how they distribute the load among servers or make some of them traffic-free for maintenance. And in these cases, if you need a specific treatment for that service, you can use mangle rules matching the destination address against an
address-list
which automatically keeps track of the DNS updates. In this situation a mangle rule with route marking and a separate routing table are necessary because neither a mere
route
nor a
route rule
can work with an
address-list
.
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11616
Joined: Thu Mar 03, 2016 10:23 pm

Re: Dual WAN Dual LAN Setup

Sun Apr 29, 2018 12:08 pm

Although maybe not, but for another reason. Port 25 is for mail servers (server to server communication), clients are supposed to use port 587.
What makes the difference between SMTP client and SMTP server? Is it software (e.g. Thunderbird VS postfix)? Or is it some kind of administrative demarcation (ISP VS client)? Or is it protocol variant (SMTP over port 25 has to talk plain text as well while SMTP over port 587 could decide to only talk SMTP over SSL which makes ISP hard to step in between)? Or is it valid reverse IP_address-to-domain_name mapping which is absent for so many ISP clients?
 
sindy
Forum Guru
Forum Guru
Posts: 10206
Joined: Mon Dec 04, 2017 9:19 pm

Re: Dual WAN Dual LAN Setup

Sun Apr 29, 2018 12:38 pm

Although maybe not, but for another reason. Port 25 is for mail servers (server to server communication), clients are supposed to use port 587.
What makes the difference between SMTP client and SMTP server? Is it software (e.g. Thunderbird VS postfix)? Or is it some kind of administrative demarcation (ISP VS client)? Or is it protocol variant (SMTP over port 25 has to talk plain text as well while SMTP over port 587 could decide to only talk SMTP over SSL which makes ISP hard to step in between)? Or is it valid reverse IP_address-to-domain_name mapping which is absent for so many ISP clients?
It is an administrative thing. The "e-mail client" applications (Outlook, Thunderbird, Eudora etc.) act as SMTP clients when sending e-mails, and use other protocols (POP3 or IMAP) to fetch messages from the mailbox. However, SMTP servers (as in "machines") can act as both SMTP clients (as in "TCP clients") or SMTP servers (as in "TCP servers") when forwarding the e-mail messages between each other, until the last one in the chain stores it into user's mailbox. There are no "CPE" and "core" dialects of the SMTP protocol itself.

Whether plain SMTP on port 25 or SMTP over TLS on port 587 is permitted for end user connections is a matter of security policy of the service provider. The client may request starttls on port 25 but it is not mandatory, while on 587 a plaintext SMTP is not accepted (or even expected) at all - TLS is running there by default, no starttls command is needed.

So instead of resolving the domain name of the e-mail address using a DNS MX record and using plaintext SMTP to deliver the message directly to that domain's mail server, as it used to work in the Good Old Days, we now use SMTPS to connect to our providers' SMTP servers which then forward the messages to the recipient domains' mail servers, often using plaintext SMTP but using some other spam prevention methods. It is somehow assumed that user's PCs are more likely to get infected by malware than ISP's servers, and it is also technically plausible for you as a mail server administrator to tell all your own clients "configure your e-mail clients for SMTPS", while you cannot tell the same to all the SMTP servers in the world as you don't know one even exists until it sends the First Message Ever to your one. And you have to deliver that message, otherwise your own client will be angry to you, and that's the better case because it indicates that he's heard from the potential customer by phone; if he hasn't and the customer did the business with someone else, you won't even know but the harm is bigger.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 19352
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Dual WAN Dual LAN Setup

Sun Apr 29, 2018 1:17 pm

So are you saying that outlook or live mail when I hit send, have knowledge of their ISPs mail server IP and gateway?

Okay I went to the email setup and yes the following is true.

POP3 -
pop.email_ISP.ca incoming (check server)
smtp.email.ISP.ca outgoing mail

From my experience when I lose connectivity on WAN2 (with my current router and specific routing rule) i can always get the incoming from other ISPs networks.
However I can never send email if there is no valid connection to the email ISP (not possible through the another internet connection).
I am talking email program,. There are ways to go to the ISP website to get email but thats another story.

So, based on the above information, without a route rule that specifically identifies email traffic and then identifies the ISP, I will not be able to send traffic!!
As I noted, unless there is something I am missing, nothing stated has convinced me otherwise.........
 
sindy
Forum Guru
Forum Guru
Posts: 10206
Joined: Mon Dec 04, 2017 9:19 pm

Re: Dual WAN Dual LAN Setup

Sun Apr 29, 2018 1:37 pm

So, based on the above information, without a route rule that specifically identifies email traffic and then identifies the ISP, I will not be able to send traffic!!
As I noted, unless there is something I am missing, nothing stated has convinced me otherwise.........
The route does not need to identify e-mail traffic in particular in order to be sufficient. It is enough that it is a route for any trafic, e-mail or not, to the smtp server. So if the SMTP server is colocated with the webmail server and the pop3 server on the same machine with the same IP address, with your rule, you will send e-mails via WAN2 but will access the webmail page and fetch incoming e-mails via WAN1, because your mangle rule only marks SMTP packets and so all the others will use the default route in the default routing table to access that IP address.

If you don't mind that also non-SMTP traffic towards that address is routed via WAN2, you don't need to mark the SMTP traffic using a mangle rule to make it use a dedicated routing table, and the individual route for the server's IP address may be placed to the default routing table.

But think about @Sob's suggestion as well - many ISPs accept your SMTP messages only from a particular IP address only if you use plaintext SMTP on port 25; if you send them using SMTPS on port 587 instead, they do not need to restrict the source IP because you're sufficiently authenticated using your credentials which are usually the same ones you use to fecth received e-mails.
 
Sob
Forum Guru
Forum Guru
Posts: 9121
Joined: Mon Apr 20, 2009 9:11 pm

Re: Dual WAN Dual LAN Setup

Sun Apr 29, 2018 5:11 pm

Whether plain SMTP on port 25 or SMTP over TLS on port 587 is permitted for end user connections is a matter of security policy of the service provider. The client may request starttls on port 25 but it is not mandatory, while on 587 a plaintext SMTP is not accepted (or even expected) at all - TLS is running there by default, no starttls command is needed.
It's slighly different. What you're describing is original SMTP over SSL (now TLS) with default port 465 and implicit encryption. Port 587 is used by Submission, and it uses STARTTLS, which I think isn't even required. The major difference is authentication. When sending mail to Submission port, you can't do it anonymously. Unlike with SMTP on port 25, where anonymous access must be allowed, otherwise whole mail system wouldn't work. And actually it looks like port 465 was recently repurposed to be used as Submission port with implicit encryption.

The whole idea behind Submission as special service with own port (even though it's the same protocol as SMTP) came from how things worked before. Anyone could send e-mail to anyone else using any server, even though it was not related to sender or recipicient. There was no verification of anything, you could connect to SMTP server on the other end of the world, and send prank emails to your friends from any faked sender (Bill Gates, ...). Typical "correct" scenario was to send outgoing mail to your ISP's SMTP server, even though your e-mail provider was someone else. In a way, it was nice friendly world where people had to trust each other. And it of course failed, because it allowed massive spam waves. Any infected home computer could send millions of mails with fake senders and nothing could be done about it.

As a reaction, e-mail started to get tighter rules, there's e.g. SPF that tells what servers can send mail from given domain. It forces users to use SMTP servers of their e-mail provider, instead of ISP's, because there's no way how e-mail provider could whitelist all possible ISP (and even if it was, the whole system would become useless again). At the same time, ISPs wanted to stop (or at least limit) amount of outgoing spam from their networks, so some of them started to block port 25. But it conflicts with the other plan to have users connecting to e-mail provider.

Submission with mandatory authentication solves this problem. You can connect to Microsoft's server as spammer, but it won't help you, because you don't know the password, so you can't use it to send mail to anyone else. And if you try to send mail with fake sender (it's still possible) using your own e-mail provider's server, it can be traced back to you, or at least to server. And that's not ISP's server, so they don't need to care about it, it's not their problem. And so they have no reason to block Submission port. It may look like happy end, and in a small way it is, but e-mail system as whole is still horrible nightmare, but it's another story.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 19352
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Dual WAN Dual LAN Setup

Sun Apr 29, 2018 5:28 pm

Okay, so if I see this right I only need the following route rules and NO mangle rule.

/IP route
add dst-address=0.0.0.0/0 gateway=<WAN1 gateway IP> ping gateway distance =1
add dst-address=0.0.0.0/0 gateway=<WAN2 gateway IP> ping distance =2
add dst-address=<mail server address> gateway=<WAN2 gateway>

I attempt to send an email, please trace what will happen?
All the email has is smpt.<name>.ca

Are you saying that the domain name in the email link "name".ca is enough??
How will the router figure out what the name resolves too? Can it do that?
Are you saying that when it resolves the name if it can on the fly, it will match the ISP IP address for mail or will it resolve to the gateway IP.
(in either case matching my route rule number 3) ??
 
sindy
Forum Guru
Forum Guru
Posts: 10206
Joined: Mon Dec 04, 2017 9:19 pm

Re: Dual WAN Dual LAN Setup

Sun Apr 29, 2018 5:52 pm

I attempt to send an email, please trace what will happen?
All the email client has is
smtp.<name>.ca
Are you saying that the domain name in the email link "name".ca is enough??
It is enough for the e-mail client application, as it asks the DNS to translate the domain name (
smtp.<name>.ca
) to an IP address, and the DNS answers "the address for that name is
<mail server address>
". The e-mail client application then sends the packets to
<mail server address>
. Most of, if not all, these packets do not contain
smtp.<name>.ca
even in their payload.

How will the router figure out what the name resolves too? Can it do that?
It could do that but it doesn't need to do that as you have done it instead, when deciding for what
dst-address
to configure the route via WAN2's gateway. The packet from the PC already has
<mail server address>
as destination IP address.

Are you saying that when it resolves the name if it can on the fly, it will match the ISP IP address for mail or will it resolve to the gateway IP.
(in either case matching my route rule number 3) ??
"Routing" means "choosing which way to send the packet depending on its attributes, above all (and in most cases only) its destination addres". So as the packet is for
<mail server address>
, a route matching best (i.e. most precisely) that address is chosen, which in your case is the third route in your list above. Once the route is found, the MAC address of the gateway device is determined the way I've described before, and the packet is sent to the MAC address of that device (which is the ISP's router), still with
<mail server address>
as destination IP address. The ISP router then repeats the same process, and finally, passing maybe several other routers, the packet arrives to the mail server.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 19352
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Dual WAN Dual LAN Setup

Sun Apr 29, 2018 9:17 pm

Okay I did no such thing
All I did was tell the router that traffic destined for the IP of the email server should go out this door.

The email application Im assuming hits the In-Interface of the router LAN, and the router says hmmm where do I send this.
Let me resolve this smpt.domainname.ca (using DNS I assume) and then says oh, this matches a route rule it needs to go out this door.

The packet does not have the IP address of the email server, it only has domain name.
The router has to intervene to figure out what the domain name means and I have to have provided the route in the route rules.

So what I have learned thus far is that the router can resolve these domain names on the fly and route them properly.
I was not aware that routers sent out dns QUERIES BEFORE ACTUALLY ROUTING TRAFFIC?

Or are you saying that the router blindly tried all three routes until one works and then remembers that in some table??
 
sindy
Forum Guru
Forum Guru
Posts: 10206
Joined: Mon Dec 04, 2017 9:19 pm

Re: Dual WAN Dual LAN Setup

Sun Apr 29, 2018 9:26 pm

Okay I did no such thing
All I did was tell the router that traffic destined for the IP of the email server should go out this door.

The email application Im assuming hits the In-Interface of the router LAN, and the router says hmmm where do I send this.
Let me resolve this smpt.domainname.ca (using DNS I assume) and then says oh, this matches a route rule it needs to go out this door.

The packet does not have the IP address of the email server, it only has domain name.
No. It is exactly the reverse. The packet coming from the PC does not have the domain name of the email server, it only has the IP address, because already the PC has done the DNS resolution, before sending the packet.

Install Wireshark, start packet sniffing into a file on the Mikrotik interface to which your PC is connected, send an e-mail, stop the packet sniffing, download the file to a PC and have a look how a packet looks like.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 19352
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Dual WAN Dual LAN Setup

Sun Apr 29, 2018 11:57 pm

Say what?
Now I am to believe that the PC is doing the work hmmmmmmmm That is the missing link that has been missing from my understanding!!!!

Okay so the things I didnt understand/.
a. email program has domain name in outgoing traffic (i forget about this and knew it)
b. the PC takes the domain name and attempts to resolve the name prior to moving the email.
c. The PC uses DNS services to do this as assigned by the router.
d. The DNS resolving is independently done from WANs?? Yes/NO or auto uses WAN1 - How does the PC do this and what route did it take?
e. The PC gets the IP info back and associated that with the email packets which it now sends to the Lan interface (for forwarding)
f. The router does an inspection of the routes and finds best match (not a blind send to WAN1) and discovers my rule 3 and sends out the traffic to WAN2

I dont need a mangle rule!!
 
Sob
Forum Guru
Forum Guru
Posts: 9121
Joined: Mon Apr 20, 2009 9:11 pm

Re: Dual WAN Dual LAN Setup

Mon Apr 30, 2018 12:42 am

c. PC sends query for mail server's hostname to configured DNS resolver (either router or some external server, it depends on what you configured). If it's external server, it's like any other outgoing connection from PC. If it's router, it will take the query from PC, will ask own resolver(s) configured in "/ip dns", and then return received response to PC.

d. It's decided by router, PC can't incluence it. Router uses whatever is the best route to resolver's address. In your case, if you have failover with WAN1 as default, and no extra routes to resolver's address, it will use WAN1.

e. I'd describe it as PC sending packets to resolved IP address, but we probably mean the same thing.

f. Yes, it's basic routing, receive packet, check its destination, find out where it should go based on routes in routing table, send it there, the end.
 
User avatar
CZFan
Forum Guru
Forum Guru
Posts: 2098
Joined: Sun Oct 09, 2016 8:25 pm
Location: South Africa, Krugersdorp (Home town of Brad Binder)
Contact:

Re: Dual WAN Dual LAN Setup

Mon Apr 30, 2018 1:54 am

e. PC gets IP info back, compares it with its own IP info to determine if it is local or external, if local, ARP request for MAC address and forwards there, if external, forwards to gateway IP for further forwarding / routing
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 19352
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Dual WAN Dual LAN Setup

Mon Apr 30, 2018 3:46 am

No worries, clear now.
What I didnt know was that the PC ensures the DNS function is used to resolve the domain name of the email send into an IP address, prior to sending to LAN interface.
That was the missing link.
In other words I was right, the router by itself would not have been able to forward the email traffic without the above step and that is why I was saying it was magic LOL.
Thus you can see why I was perplexed.
There was never an issue about needing a third route rule but I couldnt for the life of me figure out WITHOUT a mangle rule why the email would find that third route.

If only that explanation was giving on my first query LOL............ ;-P
Thanks for your patience!!

Who is online

Users browsing this forum: Strange0ne, svmk, vaio73, vetal12311, vikashdh and 125 guests