For each VRRP interface one is always active and the others are backup. There is no "both of them are active" with VRRP. Now each VRRP interface can live on different routers.I'm setting-up two RB5009 routers in VRRP active balanced configuration, so that means both of them are active at the same time.
VRRP interfaces should be ok. I used the /32 address. I confirm that from VRRP standpoint everything seems to be working ok. If I switch off one of the two routers the other takes over and when both are working they have 2 vlans running as master and 2 vlans as backup of the other router and viceversa.The quick suggestion is to make sure the VRRP interfaces are in the LAN and the VRRP ip address is a /32. As to what you think WAN/failover and VRRP will do with PPPoE and LTE, I'm not sure.
You should post your entire config as VRRP and ISP/LTE failover are exactly related.
I fully agree with you. When I say that both are active, I mean the both routers are running. Router 1 has two vrrp interfaces (vlans are attached to these) running as master (with their corresponding backup on router 2) and the other two vrrp interfaces are in backup mode, since their corresponding masters are running on router 2.For each VRRP interface one is always active and the others are backup. There is no "both of them are active" with VRRP. Now each VRRP interface can live on different routers.I'm setting-up two RB5009 routers in VRRP active balanced configuration, so that means both of them are active at the same time.
are you saying that you have 4 internet links, which are 2 internet for each router?router 1:
- ether1 pppoe ISP1 (2.5 gbps) (distance 1)
- ether2: backup LTE connection (dhcp client with an LTE modem) (distance 3)
- 4 VLANS (vlans 1 & 2 acting as VRRP master and vlans 3 & 4 as backup with router 2) all 4 have dhcp servers
- sfp-sfpplus1 trunk with the 4 VLANS -> mikrotik switch CRS305 (sfp-sfpplus1 in trunk)
router 2:
- ether1 pppoe ISP2 (2.5 gbps) (distance 1)
- ether2: backup LTE connection (dhcp client with the same LTE modem, as above) (distance 3)
- 4 VLANS (vlans 1 & 2 acting as VRRP backup and vlans 3 & 4 as master with router 1) all 4 have dhcp servers
- sfp-sfpplus1 trunk with the 4 VLANS -> mikrotik switch CRS305 (sfp-sfpplus2 in trunk
Can you kindly elaborate a bit more (I'm quite new to mikrotik and this is one of my first router setup) about this "WAN Routing" vlan? Should I create it in the same way I did for the 835 (required by the ISP), separate from this one(?), on which interface(?) and how can I use it on both routers and have 2 IPs on it to use on the other router as 2nd route (getting internet access from each of the pppoe connections)? You mention also to use the MGMT IP of the far-end router as the 2nd route, but this is exactly what I'm doing today:At a high level, I think I'd just use an additional VLAN, without VRRP, and separate from the PPPoE one, to use for WAN routing between the two routers. e.g. use the IP of the other router on that new VLAN as the destination for the backup 0.0.0.0 route. Let PPPoE get a WAN address on each router, and that it's for VLAN835. The new router-to-router VLAN can have recursive route on it to check it's valid (thus no need for VRRP on this particular VLAN). I also suppose you could use the MGMT IP of the far-end router as the 2nd route, but it likely just be cleaner to create a new "WAN routing" VLAN to keep it cleaner.
the @Sindy suggestion is the only one I've found on the forum that could address my issue having VRRP and having a 2nd route for ISP failover on a far-end router, that I suppose could avoid to have the invalid packets dropped, passing them to the WAN of the other router.Otherwise, re-using the PPPoE VLAN for traffic between the routers requires more extensive firewall treatment. Perhaps the src-nat as suggest by @sindy in the other thread, but that's not the entire story I suspect. Personally, I'd avoid additional NAT'ing since it really shouldn't be needed. If router1's internet is down, you want to route to router2 first, then get NAT'ed there. Still need a recursive route on it since if the other router was down, then you do want to go to the 3rd LTE route on the same router.
With regard to the ether2/LTE modem, I'm still in the configuration process, so it's simpler for me to test it with a dhcp client getting an IP from an ether cable with just internet access. The final situation will be a static address e.g. 192.168.1.2 with a GW 192.168.1.1 from the LAN side of the LTE modem. I created a static route just to be sure to have the highest distance and use it only as last resort (without risk to dragging the LTE connection without knowing it). I will use this LTE connection for both routers, so I'll put a dumb switch on the LAN LTE modem and I'll connect it with 2 ethernet cables to the ether2 of both routers.I'm not sure this is an issue yet. But Ether2/LTE modem, you have a /ip/dhcp-client and a static /ip/route for ether2... likely just need to set the default-route-distance and a default route will be created automatically... so no need for the static route for ether2 since dhcp-client should take care of it. As the "last chance" internet, you likely don't want any recursive routes and you're not doing this which seems right — both since there isn't another choice & also it adds more complexity (requires script on dhcp-client for LTE and additional "canary addresses" (e.g. 8.8.8.8, etc.) if you do want recursive routes).
To implement VRRP I google around the mikrotik forums and manuals (and I did not find too much honestly), so I understood how it should work and I tried on my own. I started to create the VLANs for each router and got them working, then I added the VRRP on top of them. Now, if you would suggest me a different way and I could avoid to have only 4 active dhcp servers instead of 8, for me it would be better and I'd be happier. Consider anyway that I split master and backup between the 2 routers and the only way I thought it would be possible in case of crash of one of the 2 routers is that the remaining one should have all the 4 dhcp servers, but this is the first VRRP experience for me, so the solution could be different. I believe you are suggesting that putting the VRRP (instead of VLAN) interface on the DHCP server this will have only 4 dhcp server active at any time. If you would suggest me how to implement this (maybe just change the interface reference in the dhcp server), I'd be very happy.One more thing, for the /ip/dhcp-server, I normally have them listen on the VRRP interface. But this is just preference since like seeing the leases for a VRRP'd VLAN on one router (e.g. the VRRP master). In theory, it doesn't matter who provides the DHCP address for a VRRP VLAN, but it does get annoying to trace down an lease down the road since you have to look in two places...
I've three: 1 iSP for each router (ISP1, ISP2) and a third 4G/LTE modem shared btw the 2 routers@ rickpal
...
are you saying that you have 4 internet links, which are 2 internet for each router?
I really appreciate your help, if you have any suggestion. pls let me know, I'm quite new to mikrotik, but I love it. I started with switches and I'm currently trying to place the router too.i am not saying that your vrrp setup is not doable. it's really nice to have everything in place - but i just thought that maybe you could make your homework easier by doing better design, better assessment? I'm sure that there are many other things to do with your network?
how about to put the lte modem in the drawer first, so that you can focus on the vrrp ?I've three: 1 iSP for each router (ISP1, ISP2) and a third 4G/LTE modem shared btw the 2 routers
well, it's not that easy to separate your 2 isp fail over with your vrrp active - active setup. since each vrrp router actually in active forwarding state.I'm currently having issues and struggling in managing failover of the 2 ISPs lines between the two routers.
Following this guide (viewtopic.php?t=182373), I setup dual wan recursive (using 2 recursive routes - flat) rules to manage the two ISP failover.
That means that if ISP1 goes down, router 1 should use (distance 2) the ISP2 from the router 2 and viceversa. See below.
ok. here is the thing - in every fail over schema there should be some kind of mechanism to move the failing routes to working one. be it static by script or dynamic using routing protocol.I would prefer avoiding scripts, whenever possible.
well, probably this vrrp discussion can't take a short answerfor failure modes: If one route is physically off, VRRP get the LANs to the working one, and recursive route would kill the route to the 2nd router since since 8.8.4.4 wouldn't be pingable. If ISP1 died (but still working locally), on one of the router (e.g. 8.8.8.8 primary recusive route fails ping check), then it route to the 2nd router.
It might be easy to try disabling the rp-filter, at least temporarily. That may be causing the invalid packets, but hard to know.I hooe I'll not have the same issue with the invalid packets being discarded.
Yes, you want "loose" – thought it be quick test to see if had an effect.With regard to the rpfilter it was originally set to none and i tried to set to loose following the router setup from anav (see link reference in my first post), that I used also for the recursive route definizione.
And that why I suggested a new VLAN – e.g. a "router-to-router" one. As this let you keep the PPPoE one as "WAN" in your firewall.That's why I started this thread mentioning the sindy thread that was talking about accessing the other router from the wan side instead of the lan side.
Anav, thank you answering too.Whats missing is a detailed network diagram, it may help!!
( besides that I am no good to do vrrf either, I can help with your failover and routes within a router only ) vrrf add another dimension that I would hope is separate from each routers own individual failover setup between the WANs on the router. AKA should be an isolated thing from VRRF takeover...........?????
I try to answer your question based on my understanding.Here is a question.
If ISP1 is for Router Master and ISP2 is for ROUTER slave, what happens when ISP1 goes down from a VRRP perspective?
In other words, does this constitute or mimic router failure and thus the Slave then becomes the Master?
Again, VRRP is not on the WAN side.Assuming the two routers are connected over ether2 lets say................
Both routers would still be up and running except there would be no internet access on Router Master
Is VRRP smart enough to recognize that Router Master has not working gateway and thus should no longer be the Master ??
Yes, I agree with you that this is the real question. recursive routes btw two different routers with independent connections.You asked about WANs and VRRP, that's more useful when you have a single upstream connection, and you want either router to be able to use it – active/passive on WAN. In your case, you have two independent connection, so recursive routes is for sure what you need on the WAN, not VRRP.
Or if you like it when a router reboots/upgrades/etc, it causes no service outages. In all honesty, VRRP has never kicked because of an actual hardware failure (sure VRRP works, just doesn't happen)...now frequent RouterOS v7 updates, different story.Nevertheless, I need VRRP for HW redundancy.
so, is this the right way to manage this?Thank you Amm0,
I name them invalid packets since their connection-state=invalid and they are dropped by the in the chain=forward, unless I accept them with these 2 rules in both routers before are being dropped as invalid:
/ip firewall filter
add action=accept chain=forward comment="need this rule to manage the ISP fail\
over on the other VRRP router, otherwise these packets will be discarded a\
s invalid by the next rule." out-interface=WAN_ROUTING_VLAN
add action=accept chain=forward comment="need this rule to manage the ISP fail\
over on the other VRRP router, otherwise these packets will be discarded a\
s invalid by the next rule." in-interface=WAN_ROUTING_VLAN
add action=drop chain=forward comment="defconf: drop invalid" \
connection-state=invalid log=yes log-prefix="*** invalid ***"
As you can see from the above I already have set the rule that drops invalids with log=yes and log-prefix="*** invalid ***"
and what I've seen is the if ISP2 goes down, then router 2 routes its connection towards ISP1.
In router 1 log (that is currently providing ISP1 failover connection to router 2) I see invalid forward packet where in-if=WAN_ROUTING_VLAN out-if=pppoe-out and mac address is the VLAN100 interface in router 1, then
In router 2 log I see invalid fwd packets where in-if=VRRP (the vlan connected with my laptop) out-if=WAN_ROUTING_VLAN and mac address is my laptop's mac address
Is this normal that is solved accepting the invalid packets (that should be normally dropped) but I accept them before being dropped?
tnx.
Could you pls elaborate more?Ahh, it's just a few invalids packets at failover... That's happens in the time when connection tracking is figuring it out the failover I think.
The is the newer "Sync Connection Tracking" option that keep the firewall connection tracking state on both routers. But you can't use this with your "dual active-active WAN" since the WAN IP changes, you do NOT want the connection tracking the same. So as TCP etc session figure out the new router, you'll get a few invalid.
If these invalid persist long after the failover, different story then...
do you mean that from a VRRP standpoint it works when both routers are connected to the same WAN IP? (https://wiki.mikrotik.com/wiki/File:Vrr ... haring.png)Both routers need to have one common public IP for VRRP on WAN with sync connections - you have two IP - which is why this cannot work.
One more thing, I understand from your suggestion that is better to move the interface for the DHCP Server from the VLAN to the VRRP. In this way, if I properly understand I'll have only 4 DHCP Servers actives at the same time, corresponding only to the masters ones will release DHCP IP address to clients. Am I right?One more thing, for the /ip/dhcp-server, I normally have them listen on the VRRP interface. But this is just preference since like seeing the leases for a VRRP'd VLAN on one router (e.g. the VRRP master). In theory, it doesn't matter who provides the DHCP address for a VRRP VLAN, but it does get annoying to trace down an lease down the road since you have to look in two places...
Correct, you still have 8 DHCP servers, but on the 4 on slave VRRP VLANs will be red/inactive until the slave becomes the master. If a VRRP switch happens, the clients remain on the old router until the lease expires, but upon renewal the lease will "move". Since client request their old address again, it should be seamless.if I properly understand I'll have only 4 DHCP Servers actives at the same time, corresponding only to the masters ones will release DHCP IP address to clients. Am I right?
That's why using recursive routes on the WAN make sense. And the VLAN's VRRP master selection control which routing table is used...so if you want a VLAN to prefer one WAN1 or WAN2, you ensure that VRRP priority get the master to the desire primary WAN.Of course, it does not help if you have two different ISP (e.g., ethernet + 4G). But with a single WAN connection and two routers, that is a way to do.
ok, let me better explain.I am not convinced LOL.
For example if you disable all routes except the one to LTE do you get internet
Since we have no information on this connection, No pppoe information etc, I cannot visualize what the gatewayIP address might be,
Does your LTE interface have an assigned name??
I wouldn't use the DHCP client, e.g. disable it. Add 192.168.1.2/24 as IP address on Mikrotik for ether2, and use your existing static route the ip. You already have a route for LTE, just include a higher target/target-scope*:I've a 4G TP-Link modem / router with a SIM inside.
On the LAN side of it, I've a DHCP server 192.168.3.1/24. So, the TP-Link GW is 192.168.3.1 and it is connected to the ether2 of the mikrotik router RB5009. The ether2 on the MT router has a DHCP client and it get assigned an address 192.168.3.2. ether2 is not part of the bridge and it is assigned to the WAN as the other pppoe (ether1) interface. I did not change ether2 name.
@Amm0 see my response. entering the interface, the dhcp client has specified distance=1 (I never know this) and I changed it to 15.I wouldn't use the DHCP client, e.g. disable it. Add 192.168.1.2/24 as IP address on Mikrotik for ether2, and use your existing static route the ip. You already have a route for LTE, just include a higher target/target-scope*:I've a 4G TP-Link modem / router with a SIM inside.
On the LAN side of it, I've a DHCP server 192.168.3.1/24. So, the TP-Link GW is 192.168.3.1 and it is connected to the ether2 of the mikrotik router RB5009. The ether2 on the MT router has a DHCP client and it get assigned an address 192.168.3.2. ether2 is not part of the bridge and it is assigned to the WAN as the other pppoe (ether1) interface. I did not change ether2 name.
/ip route add disabled=no distance=20 dst-address=0.0.0.0/0 gateway=ether2 scope=10 target-scope=35
The issue is the DHCP client in inserting a route with lower scope/target-scope than your recursive routes, so it become primary. In theory, you could lower scopes of the existing recursive routes so they're lower than DHCP client uses.
*I think there are right, but if not adjust target-scope
Likely true.AMMO is on to something maybe,
Dont worry about scope shit, its too in the weeds, and not required.
# may/09/2023 06:41:16 by RouterOS 7.8
# software id =
#
# model = RB5009UG+S+
# serial number =
/interface bridge
add admin-mac=XX:XX:XX:XX:XX:XX auto-mac=no comment=defconf frame-types=\
admit-only-vlan-tagged name=bridge protocol-mode=none pvid=20 \
vlan-filtering=yes
/interface vlan
add interface=bridge name=GUEST_VLAN vlan-id=3090
add interface=bridge name=MGMT_VLAN vlan-id=20
add interface=bridge name=VLAN5 vlan-id=5
add interface=bridge name=VLAN10 vlan-id=10
add interface=ether1 name=VLAN835 vlan-id=835
add comment="WAN Routing VLAN 100" interface=bridge name=WAN_ROUTING_VLAN \
vlan-id=100
/interface pppoe-client
add disabled=no interface=VLAN835 name=pppoe-out user=XXX
/interface vrrp
add interface=VLAN5 name=vrrp5 priority=200 vrid=5
add interface=VLAN5 name=vrrp6 vrid=6
add interface=VLAN10 name=vrrp10 priority=254 vrid=10
add interface=VLAN10 name=vrrp11 vrid=11
add interface=MGMT_VLAN name=vrrp20 priority=254 vrid=20
add interface=MGMT_VLAN name=vrrp21 vrid=21
add interface=GUEST_VLAN name=vrrp3090 priority=200 vrid=90
add interface=GUEST_VLAN name=vrrp3091 vrid=91
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
add name=VRRP
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=MGMT_POOL ranges=XX.YY.72.201-XX.YY.72.250
add name=VLAN5_POOL ranges=XX.YY.70.201-XX.YY.70.250
add name=VLAN10_POOL ranges=XX.YY.71.231-XX.YY.71.250
add name=GUEST_POOL ranges=192.KK.ZZ.101-192.KK.ZZ.150
/ip dhcp-server
add address-pool=MGMT_POOL comment="Secured / Management DHCP Server" \
interface=vrrp20 name=MGMT_DHCP
add address-pool=VLAN5_POOL comment="Standard DHCP Server" interface=vrrp6 \
name=VLAN5_DHCP
add address-pool=VLAN10_POOL comment="IoT DHCP Server" interface=vrrp10 name=\
VLAN10_DHCP
add address-pool=GUEST_POOL comment="Guest DHCP Server" interface=vrrp3091 \
name=GUEST_DHCP
/interface bridge port
add bridge=bridge comment=defconf frame-types=\
admit-only-untagged-and-priority-tagged interface=ether3 pvid=5
add bridge=bridge comment=defconf frame-types=\
admit-only-untagged-and-priority-tagged interface=ether4 pvid=10
add bridge=bridge comment=defconf frame-types=\
admit-only-untagged-and-priority-tagged interface=ether5 pvid=3090
add bridge=bridge comment=defconf frame-types=\
admit-only-untagged-and-priority-tagged interface=ether6 pvid=100
add bridge=bridge comment=defconf interface=ether7
add bridge=bridge comment=defconf frame-types=\
admit-only-untagged-and-priority-tagged interface=ether8 pvid=20
add bridge=bridge comment=defconf frame-types=admit-only-vlan-tagged \
interface=sfp-sfpplus1
/ip neighbor discovery-settings
set discover-interface-list=LAN
/ip settings
set rp-filter=loose
/ipv6 settings
set disable-ipv6=yes
/interface bridge vlan
add bridge=bridge comment="VLAN Standard" tagged=bridge,sfp-sfpplus1 \
untagged=ether3 vlan-ids=5
add bridge=bridge comment="VLAN IoT" tagged=bridge,sfp-sfpplus1 untagged=\
ether4 vlan-ids=10
add bridge=bridge comment="VLAN Secured / Management" tagged=\
bridge,sfp-sfpplus1 untagged=ether8 vlan-ids=20
add bridge=bridge comment="VLAN Guest" tagged=bridge,sfp-sfpplus1 untagged=\
ether5 vlan-ids=3090
add bridge=bridge comment="WAN Routing VLAN 100" tagged=bridge,sfp-sfpplus1 \
untagged=ether6 vlan-ids=100
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1 list=WAN
add interface=ether2 list=WAN
add interface=pppoe-out list=WAN
add interface=vrrp5 list=VRRP
add interface=vrrp6 list=VRRP
add interface=vrrp10 list=VRRP
add interface=vrrp11 list=VRRP
add interface=vrrp20 list=VRRP
add interface=vrrp21 list=VRRP
add interface=vrrp3090 list=VRRP
add interface=vrrp3091 list=VRRP
/ip address
add address=XX.YY.72.112/24 comment="Secure / Management Network Gateway" \
interface=MGMT_VLAN network=XX.YY.72.0
add address=XX.YY.70.112/24 comment="VLAN Standard Gateway" interface=VLAN5 \
network=XX.YY.70.0
add address=XX.YY.71.112/24 comment="VLAN IoT Gateway" interface=VLAN10 \
network=XX.YY.71.0
add address=192.KK.ZZ.254/24 comment="VLAN Guest Gateway" interface=\
GUEST_VLAN network=192.KK.ZZ.0
add address=XX.YY.72.115 interface=vrrp20 network=XX.YY.72.115
add address=XX.YY.72.116 interface=vrrp21 network=XX.YY.72.116
add address=XX.YY.70.115 interface=vrrp5 network=XX.YY.70.115
add address=XX.YY.70.116 interface=vrrp6 network=XX.YY.70.116
add address=XX.YY.71.115 interface=vrrp10 network=XX.YY.71.115
add address=XX.YY.71.116 interface=vrrp11 network=XX.YY.71.116
add address=192.KK.ZZ.1 interface=vrrp3090 network=192.KK.ZZ.1
add address=192.KK.ZZ.2 interface=vrrp3091 network=192.KK.ZZ.2
add address=172.22.1.2/24 comment="WAN Routing VLAN Gateway #02" interface=\
WAN_ROUTING_VLAN network=172.22.1.0
add address=192.168.3.4/24 comment=\
"ether2 fixed ip address to the 4G/LTE modem for the backup connection" \
interface=ether2 network=192.168.3.0
/ip dhcp-client
add default-route-distance=2 disabled=yes interface=ether2
/ip dhcp-server network
add address=XX.YY.70.0/24 dns-server=XX.YY.70.112,1.1.1.1,8.8.8.8 gateway=\
XX.YY.70.116
add address=XX.YY.71.0/24 dns-server=XX.YY.71.112,1.1.1.1,8.8.8.8 gateway=\
XX.YY.71.115
add address=XX.YY.72.0/24 dns-server=XX.YY.72.112,1.1.1.1,8.8.8.8 gateway=\
XX.YY.72.115
add address=192.KK.ZZ.0/24 dns-server=192.KK.ZZ.254,1.1.1.1,8.8.8.8 \
gateway=192.KK.ZZ.1
/ip dns
set allow-remote-requests=yes servers=1.1.1.1,8.8.8.8
/ip dns static
add address=XX.YY.72.116 comment="Secured / Management Network Gateway" name=\
router.lan
add address=159.148.172.226 name=upgrade.mikrotik.com
add address=173.194.76.108 name=smtp..gmail.com
/ip firewall address-list
add address=XX.YY.72.0/24 comment="Admin List Address" list=Admin
/ip firewall filter
add action=accept chain=input comment=\
"defconf: accept established,related,untracked" connection-state=\
established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
invalid
add action=accept chain=input comment="accept vrrp packets" protocol=vrrp
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment=\
"allow VLAN 5 only (inter-vlan is blocked)" dst-address=XX.YY.70.0/24 \
src-address=XX.YY.70.0/24
add action=accept chain=input comment=\
"allow VLAN 10 only (inter-vlan is blocked)" dst-address=XX.YY.71.0/24 \
src-address=XX.YY.71.0/24
add action=accept chain=input comment=\
"allow MANAGEMENT VLAN 20 only (inter-vlan is blocked)" dst-address=\
XX.YY.72.0/24 src-address=XX.YY.72.0/24
add action=accept chain=input comment=\
"allow GUEST VLAN 3090 only (inter-vlan is blocked)" disabled=yes \
dst-address=192.KK.ZZ.0/24 src-address=192.KK.ZZ.0/24
add action=accept chain=input comment="\"defconf: accept local loopback (for D\
ude, RADIUS, user-manager, CAPsMAN, Wireguard) (https://forum.mikrotik.com\
/viewtopic.php\?t=180838)" dst-address=127.0.0.1
add action=reject chain=input comment="*** TBC LOGGING *** optional --> useful\
\_but only if interested in tracking LAN issues (https://forum.mikrotik.co\
m/viewtopic.php\?t=180838) - The purpose of the action=reject rule is to p\
revent users in LAN from waiting for tens of seconds to get a timeout if t\
hey are trying to connect to forbidden destinations, and of course for the\
\_admin to be aware of traffic that has the potential to be a problem (aka\
\_pinpoint device with issues)." in-interface-list=LAN log=yes \
log-prefix="*** TRACKING LAN ISSUES ***" reject-with=\
icmp-admin-prohibited
add action=drop chain=input comment="block everything else - non presente in R\
B5009 default, aggiunto da CCR2216."
add action=accept chain=forward comment="defconf: accept in ipsec policy" \
ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" \
ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" \
connection-state=established,related hw-offload=yes
add action=accept chain=forward comment=\
"defconf: accept established,related, untracked" connection-state=\
established,related,untracked
add action=accept chain=forward comment="need this rule to manage the ISP2 fai\
lover (on this router), using the ISP1 from the other VRRP router, otherwi\
se these packets will be dropped as invalid by the 'drop invalid' rule." \
in-interface-list=VRRP out-interface=WAN_ROUTING_VLAN
add action=accept chain=forward comment="need this rule to manage the ISP fail\
over on the other VRRP router, otherwise these packets will be discarded a\
s invalid by the next rule." disabled=yes in-interface-list=VRRP \
out-interface=pppoe-out
add action=drop chain=forward comment="defconf: drop invalid" \
connection-state=invalid log=yes log-prefix="*** invalid ***"
add action=accept chain=forward comment=\
"allow internet traffic for VLAN 100 for WAN Routing for ISP failover." \
in-interface=WAN_ROUTING_VLAN out-interface-list=WAN
add action=accept chain=forward comment="allow internet traffic (all vrrp inte\
rfaces) - non presente in RB5009 default, aggiunto da xxx (che usava i\
nvece all-vlan)." in-interface-list=VRRP out-interface-list=WAN
add action=accept chain=forward comment="allow port forwarding - DA VERIFICARE\
\_CHE SERVA VERAMENTE (https://forum.mikrotik.com/viewtopic.php\?t=180838)\
" connection-nat-state=dstnat disabled=yes
add action=reject chain=forward comment="*** TBC LOGGING *** optional --> usef\
ul for tracking LAN issues - in most installations the rule doesn't have t\
o care about multicast traffic because it never sees it (https://forum.mik\
rotik.com/viewtopic.php\?t=180838) - The purpose of the action=reject rule\
\_is to prevent users in LAN from waiting for tens of seconds to get a tim\
eout if they are trying to connect to forbidden destinations, and of cours\
e for the admin to be aware of traffic that has the potential to be a prob\
lem (aka pinpoint device with issues)." dst-address=!0.0.0.0/0 \
in-interface-list=LAN log=yes log-prefix="*** TRACK LAN ISSUES ***" \
reject-with=icmp-admin-prohibited
add action=drop chain=forward comment="defconf: drop all from WAN not DSTNATed\
\_- drop access to clients behind NAT from WAN - drops all new connection \
attempts from the WAN port to our LAN network (unless DstNat is used). Wit\
hout this rule, if an attacker knows or guesses your local subnet, he/she \
can establish connections directly to local hosts and cause a security thr\
eat." connection-nat-state=!dstnat connection-state=new \
in-interface-list=WAN
add action=drop chain=forward comment="block everything else - non presente in\
\_RB5009 default." log-prefix=\
"*** blocked by fwd ***"
/ip firewall nat
add action=src-nat chain=srcnat ipsec-policy=out,none out-interface=pppoe-out \
to-addresses=xx.xx.xx.xx
add action=masquerade chain=srcnat ipsec-policy=out,none out-interface=ether2
add action=masquerade chain=srcnat comment="defconf: masquerade" disabled=yes \
ipsec-policy=out,none out-interface-list=WAN
/ip route
add check-gateway=ping distance=3 dst-address=0.0.0.0/0 gateway=9.9.9.9 \
scope=10 target-scope=13
add check-gateway=ping distance=3 dst-address=0.0.0.0/0 gateway=1.0.0.1 \
scope=10 target-scope=13
add check-gateway=ping disabled=no distance=10 dst-address=0.0.0.0/0 gateway=\
8.8.8.8 pref-src="" routing-table=main scope=10 suppress-hw-offload=no \
target-scope=13
add check-gateway=ping disabled=no distance=10 dst-address=0.0.0.0/0 gateway=\
208.67.222.222 pref-src="" routing-table=main scope=10 \
suppress-hw-offload=no target-scope=13
add check-gateway=ping distance=15 dst-address=0.0.0.0/0 gateway=8.8.4.4 \
scope=10 target-scope=13
add check-gateway=ping distance=15 dst-address=0.0.0.0/0 gateway=1.1.1.1 \
scope=10 target-scope=13
add comment="WAN1 ISP2 via PPPoE - ping Host 1" distance=3 dst-address=\
9.9.9.9/32 gateway=pppoe-out scope=10 target-scope=12
add comment="WAN1 ISP2 via PPPoE - ping Host 2" distance=3 dst-address=\
1.0.0.1/32 gateway=pppoe-out scope=10 target-scope=12
add comment="ISP1 via Backup Router - ping Host 1" disabled=no distance=10 \
dst-address=8.8.8.8/32 gateway=172.22.1.1 pref-src="" routing-table=main \
scope=10 suppress-hw-offload=no target-scope=12
add comment="ISP1 via Backup Router - ping Host 2" disabled=no distance=10 \
dst-address=208.67.222.222/32 gateway=172.22.1.1 pref-src="" \
routing-table=main scope=10 suppress-hw-offload=no target-scope=12
add comment="ether2 ISP3 via GW 4G/LTE - ping Host 1" distance=15 \
dst-address=8.8.4.4/32 gateway=192.168.3.1 scope=10 target-scope=12
add comment="ether2 ISP3 via GW 4G/LTE - ping Host 2" distance=15 \
dst-address=1.1.1.1/32 gateway=192.168.3.1 scope=10 target-scope=12
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www disabled=yes
set www-ssl certificate=Webfig disabled=no
set api disabled=yes
/system clock
set time-zone-name=XXX
/system identity
set name="MikroTik RB5009 #02"
/system ntp client
set enabled=yes
/system ntp client servers
add address=194.0.5.123
add address=216.239.32.15
/tool mac-server
set allowed-interface-list=none
/tool mac-server mac-winbox
set allowed-interface-list=LAN
OK, I'll do this change.I can only go with what I know works.
(1) From this
/interface bridge
add admin-mac=XX:XX:XX:XX:XX:XX auto-mac=no comment=defconf frame-types=\
admit-only-vlan-tagged name=bridge protocol-mode=none pvid=20 \
vlan-filtering=yes
TO
/interface bridge
add admin-mac=XX:XX:XX:XX:XX:XX auto-mac=no comment=defconf frame-types=admit all\
name=bridge protocol-mode=none vlan-filtering=yes
Ingress filtering is already yes on all ports. I believe it's the default.(2) Add ingress filtering=yes to all bridge ports listed
(3) The obfuscation of private IPs is not required, they are private not public, matters zilcho ( aka ridonkulous)
OK, these rules in the input chain are to keep separate VLANS (block inter-vlan traffic). How can I do this in the forward chain in the right way, since I was not able to make it working (shame on me ). Can you help me on this? Tnx.(4) The firewall rules are not efficient in the least and some are just wrong.......
For example these have no business in the input chain.
add action=accept chain=input comment=\
"allow VLAN 5 only (inter-vlan is blocked)" dst-address=XX.YY.70.0/24 \
src-address=XX.YY.70.0/24
add action=accept chain=input comment=\
"allow VLAN 10 only (inter-vlan is blocked)" dst-address=XX.YY.71.0/24 \
src-address=XX.YY.71.0/24
add action=accept chain=input comment=\
"allow MANAGEMENT VLAN 20 only (inter-vlan is blocked)" dst-address=\
XX.YY.72.0/24 src-address=XX.YY.72.0/24
add action=accept chain=input comment=\
"allow GUEST VLAN 3090 only (inter-vlan is blocked)" disabled=yes \
dst-address=192.KK.ZZ.0/24 src-address=192.KK.ZZ.0/24
Suggest you have no business making any firewall rules other than the defaults until you understand how to use them.
In other words, start the firewall rules from scratch and redo, with the intent to make it leaner.
I've a public fixed IP address.(5) Your source nat rules seem confused.
/ip firewall nat
add action=src-nat chain=srcnat ipsec-policy=out,none out-interface=pppoe-out \
to-addresses=xx.xx.xx.xx
If its a dynamic WANIP the to-address is not required and the action is masquerade.
If its a static WANIP then the format is correct but pppoe is dynamic is it not????
OK, clear, I'll follow your suggestions.The ether2 rule with a fixed WANIP should be
add action=src-nat chain=srcnat out-interface=ether2 to-address=192.168.3.4
The default (third rule you can get rid of).
That's clear, all what I was trying to say was that choosing a VLAN on the same bridge where your LANs are located as the WAN side interconnect has some implications you have to take into account.I already have a vlan for the routing created as suggested by Amm0 (VLAN 100).
When looking into this, I've noticed that the scope and target-scope parameters of your routes are not configured properly. The purpose of these parameters is to restrict the choice of "serving" routes for a "client" route - a client route with target-scope N can only use serving routes whose scope is smaller than N. All your routes have the same scope of 10, which means that if the route to 8.8.8.8 via 172.22.1.1 is inactive for some reason, the check-gateway process of the default route via 8.8.8.8 as gateway can use any of the other default routes. However, this should cause a different misbehaviour than the one you observe - it should mean that if the WAN 1 is unavailable (PPPoE down), the check-gateway processes of the default routes via 9.9.9.9 and 1.0.0.1 can still ping 9.9.9.9 and 1.0.0.1 using the other default routes, keeping these default routes active. So definitely fix that by setting the scope of all the default routes to something higher than 13, so that no default route could use another default route as its serving one, but I currently cannot see how this fix should resolve the issue that you cannot ping 8.8.8.8 via the WAN_ROUTING_VLAN. On the other hand, I cannot see anything else to be wrong - even with the firewall rules enabled, the rule chain=forward in-interface=WAN_ROUTING_VLAN out-interface-list=WAN action=accept is enough to let the check-gateway pings from the other router pass through to the primary WAN.My issue is that the routes I've defined the as per my post #65 (triple wan recursive), the second route via the other router is not working since the vlan100 (subnet 172.22.0.x) is not able to ping an host (8.8.8.8), so it is not reachable. I do not know why.
This shows why you have to take a lot of care when obfuscating - to avoid removing any information that is necessary for proper understanding.The 4 vlan that are in vrrp are not public, I was only trying to obfuscate my subnet numbers (maybe too much paranoid, sorry).
I'd say it is enough to add a rule chain=srcnat ipsec-policy=out,none out-interface=WAN_ROUTING_VLAN action=masquerade into /ip firewall nat. You can optimize the NAT rules a bit afterwards, but first we need to make it work, therefore I suggest only the minimum changes needed.I would prefer the solution src-nat you mention, when you say:
"the simplest possibility would be to use src-nat (or masquerade) when sending traffic to the other router via the WAN interconnect ..."
I believe this is what I'm trying to do with vlan 100, but it is not yet working.
It's not a matter of physical distance, it's a matter of how the whole redundancy concept is designed. As you bother to implement an active-active redundancy with two VRRP gateways per VLAN, I suppose that you want one group of devices in your LAN to prefer the uplink connected to WAN 1 of router A and another group of devices to prefer the uplink connected to WAN 1 of router B. But if the only interface connecting one of the routers to the LAN breaks, the outcome will be the same as if that router broke completely, because the WAN_ROUTING_VLAN goes via that same physical interface so all traffic of the LAN devices will use solely WAN 1 (or WAN 3) of the other router if that port breaks.I'm not really concerned about having a redundant connection btw the two routers, since they are not so far one from each other.
Try the above and the extra firewall rules should not be necessary any more.I'm also sure I'm having issues that the request and the response are not following the same path, since in the firewall I've invalid packets dropped in the forward chain (for which I've created rules to recover them before being dropped).
Ah, so you have routed 8.8.8.8 via WAN_ROUTING_VLAN at both routers, so the request packets were circulating between the routers until their TTL has finally expired.I solved the route host(s) pinging issue. It was due to the fact I was pinging the same hosts from the 2 different routers for the ISP failover route.
I assume you have written this before reading my post #79?@Sindy: it remains the issue about the VLAN 100 for ISP failover btw the 2 routers.
Sorry, I do not understand if I've to do anything else and make other changes to the routes based on this comment (apart from changing the scope from 10 to 13).You don't need to use different canary (virtual gateway) IPs at both routers but you must make sure that if a given canary address is routed via WAN_ROUTING_VLAN on one router, it is routed via the actual WAN 1 on the other. So e.g. 9.9.9.9 and 1.0.0.1 via WAN 1 on router A and via WAN_ROUTING_VLAN on router B, and 8.8.8.8 and 208.67.222.222 via WAN 1 on router B and via WAN_ROUTING_VLAN on router A.
Yes, tnxI assume you have written this before reading my post #79?@Sindy: it remains the issue about the VLAN 100 for ISP failover btw the 2 routers.
Not for all routes, only for those with dst-address=0.0.0.0/0.With regard to the scope parameter in the routes, if I properly understand, I just change all the scope=10 with scope=13? Am I right?
Correct, this way you won't need any additional routing tables and associated connection&route marking to ensure that the responses take the "proper" path.With regard to the src-nat rule, I'll try this today, and I'll let you know.
...
So, in this case, will not be necessary any marking and creating new routes tables?
You do not *have* to do anything else, but you *can* reduce the total number of canary IPs in use because you can use the same address as a canary one for both routers if you do it "properly".Sorry, I do not understand if I've to do anything else and make other changes to the routes based on this comment (apart from changing the scope from 10 to 13).You don't need to use different canary (virtual gateway) IPs at both routers...
OK cristal clear.Not for all routes, only for those with dst-address=0.0.0.0/0.With regard to the scope parameter in the routes, if I properly understand, I just change all the scope=10 with scope=13? Am I right?
OK, it should be simple and straightforward. I'll test the src-nat later on today and I'll let you know.Correct, this way you won't need any additional routing tables and associated connection&route marking to ensure that the responses take the "proper" path.With regard to the src-nat rule, I'll try this today, and I'll let you know.
...
So, in this case, will not be necessary any marking and creating new routes tables?
[/quote]You do not *have* to do anything else, but you *can* reduce the total number of canary IPs in use because you can use the same address as a canary one for both routers if you do it "properly".
Sorry, I do not understand if I've to do anything else and make other changes to the routes based on this comment (apart from changing the scope from 10 to 13).
Let's use just 4 rows per router (A, B) for simplicity of the illustration.Sorry, what do you mean, with "if you do it properly"?
do you mean something like this, if I proper understood what you mean?Let's use just 4 rows per router (A, B) for simplicity of the illustration.Sorry, what do you mean, with "if you do it properly"?
....
This "proper" way, router A sends the check ping packet to 1.1.1.1 from its own address interconnect.ip.of.A using the interconnect.ip.of.B as a gateway; once that ping packet arrives to router B, router B forwards it to the internet using gw.of.primary.wan.B as a gateway.
When you previously had the same routing on B as on A (the "wrong" way), the check ping from router A to 1.1.1.1 arrived to router B, but instead of forwarding it to internet via gw.of.primary.wan.B, router B forwarded it back to router A via interconnect.ip.of.A as a gateway because there was the route dst-address=1.1.1.1 gateway=interconnect.ip.of.A scope=10 target-scope=11.
# router #01
/ip route
add check-gateway=ping dst-address=0.0.0.0/0 gateway=9.9.9.9 scope=13 target-scope=13 distance=5
add check-gateway=ping dst-address=0.0.0.0/0 gateway=8.8.8.8 scope=13 target-scope=13 distance=10
add check-gateway=ping dst-address=0.0.0.0/0 gateway=1.0.0.1 scope=13 target-scope=13 distance=5
add check-gateway=ping dst-address=0.0.0.0/0 gateway=208.67.222.220 scope=13 target-scope=13 distance=10
add dst-address=9.9.9.9/32 gateway=pppoe-out scope=10 target-scope=12 comment="WAN1 ISP1 via PPPoE - ping Host 1" distance=5
add dst-address=8.8.8.8/32 gateway=172.22.1.2 scope=10 target-scope=12 comment="ISP2 via Backup Router - ping Host 1" distance=10
add dst-address=1.0.0.1/32 gateway=pppoe-out scope=10 target-scope=12 comment="WAN1 ISP1 via PPPoE - ping Host 2" distance=5
add dst-address=208.67.222.220/32 gateway=172.22.1.2 scope=10 target-scope=12 comment="ISP2 via Backup Router - ping Host 2" distance=10
add dst-address=0.0.0.0/0 gateway=192.168.4.1 scope=10 target-scope=12 comment="ether2 ISP3 via GW 4G/LTE" distance=15
# router #02
/ip route
add check-gateway=ping dst-address=0.0.0.0/0 gateway=8.8.8.8 scope=13 target-scope=13 distance=5
add check-gateway=ping dst-address=0.0.0.0/0 gateway=9.9.9.9 scope=13 target-scope=13 distance=10
add check-gateway=ping dst-address=0.0.0.0/0 gateway=208.67.222.220 scope=13 target-scope=13 distance=5
add check-gateway=ping dst-address=0.0.0.0/0 gateway=1.0.0.1 scope=13 target-scope=13 distance=10
add dst-address=8.8.8.8/32 gateway=pppoe-out scope=10 target-scope=12 comment="WAN1 ISP2 via PPPoE - ping Host 1" distance=5
add dst-address=9.9.9.9/32 gateway=172.22.1.1 scope=10 target-scope=12 comment="ISP1 via Backup Router - ping Host 1" distance=10
add dst-address=208.67.222.220/32 gateway=pppoe-out scope=10 target-scope=12 comment="WAN1 ISP2 via PPPoE - ping Host 2" distance=5
add dst-address=1.0.0.1/32 gateway=172.22.1.1 scope=10 target-scope=12 comment="ISP1 via Backup Router - ping Host 2" distance=10
add dst-address=0.0.0.0/0 gateway=192.168.4.1 scope=10 target-scope=12 comment="ether2 ISP3 via GW 4G/LTE" distance=15
do you mean that:Here, it can be done this "proper" way because if the primary WAN of router B fails, you do not want router A to use the LTE WAN of router B - instead, you want it to use its own LTE WAN. But in other situations the requirement may be different, so the setup would have to be adjusted accordingly, either by using unique canary addresses or by using not only the canary addresses of the primary uplink of router B but also those of router B's LTE uplink for the server routes at router A.
Yet another point for those who come here for inspiration - there is little point in using recursive routing for the WAN of the last resort. As you only ever use it if the other WANs have failed, there is nothing you can do if it fails as well, so there is no reason to monitor its state.
I've one TP-Link modem/router with 1 sim card 4GProgress.
Things that are bugging me
(1) How does one LTE backup become available on two routers?
How does LTE work (one sim card?) Chateau router or MT LTE device with two ports, feeding both routers?
I believe the IPs to be used are the same already specified in the previous routes (and reused in the new ones).(2) What do you mean by interconnect link.... or interconnect,ip of a or b ?????
I will scope your choices
Typically VRRF is
a. Decide on a subnet ( prefer vlans for everything, like bacon goes with everything)
b. One common IP is used as the VRRP address 172.22.0.1 interface VRRP (both routers)
c. One vrrp IP address on interface etherX for master Router1 - 172.22.0.2/24
d. One vrrp IP address on interface etherX for slave Router2 - 172.22.0.3/24
e. IP routes common to both add dst-address=0.0.0.0/0 gateway=172.22.0.1 table=main
Note: To check which Router is live, check Ip address the one which is master the 172.22.0.1 is live, the other router the floating IP is red.
(3) How do we deal with two sources of dhcp servers?
Yes, exactly.do you mean something like this, if I proper understood what you mean?
1) I am not sure I understand the question. I did not read the complete history of 70 posts in detail so I may have missed something - if you were concerned about how to prevent the backup via the other router from using LTE on that other router if its primary (PPPoE) WAN is down, then yes, this is the solution.do you mean that:
1) I can solve the issue that for the failover back it will check only for remote WAN working ISP1 / ISP2 and not LTE. Also if from the tests I did, I thought I already solved this issue ...
2) the route for ISP3 (LTE) is just what I proposed in this new code? Is this right?
WOW, very excited, let me test the src-nat and then change the routes as per your suggestionYes, exactly.do you mean something like this, if I proper understood what you mean?
1) I am not sure I understand the question. I did not read the complete history of 70 posts in detail so I may have missed something - if you were concerned about how to prevent the backup via the other router from using LTE on that other router if its primary (PPPoE) WAN is down, then yes, this is the solution.do you mean that:
1) I can solve the issue that for the failover back it will check only for remote WAN working ISP1 / ISP2 and not LTE. Also if from the tests I did, I thought I already solved this issue ...
2) the route for ISP3 (LTE) is just what I proposed in this new code? Is this right?
2) yes
add action=accept chain=forward comment="need this rule to manage the ISP2 fai\
lover (on this router), using the ISP1 from the other VRRP router, otherwi\
se these packets will be dropped as invalid by the 'drop invalid' rule." \
in-interface-list=VRRP out-interface=WAN_ROUTING_VLAN
This is an expected behaviour. Do you remember I was talking about simplification of the firewall rules once the basics starts working? This is one of such potential simplifications - in fact, on each of the two routers, WAN_ROUTING_VLAN basically plays a role of yet another WAN (and also a role of yet another LAN). But as it is currently not a member of interface list WAN, the rule action=accept chain=forward in-interface-list=VRRP out-interface-list=WAN ignores packets that arrive to one of the VRRP interfaces and get forwarded via WAN_ROUTING_VLAN, so you need a separate rule (the action=accept chain=forward in-interface-list=VRRP out-interface=WAN_ROUTING_VLAN one) to allow them to pass. If you add WAN_ROUTING_VLAN as a member of the interface list WAN, this extra rule in filter will not be necessary any more, and you will also be able to use a single common action=masquerade rule matching on out-interface-list=WAN instead of multiple src-nat/masquerade rules matching on individual out-interface names. Similarly, if you also add WAN_ROUTING_VLAN as a member of interface list VRRP (and indeed, doing so will make the name of the list slightly misleading), you will not need the separate filter rule action=accept chain=forward in-interface=WAN_ROUTING_VLAN out-interface-list=WAN any more too.I'm not sure why this rule is necessary ...
That's what I had in mind when mentioning readability. There are actually no "side effects", but until you understand perfectly what each of your firewall rules does, it may seem to you as if they wereI'm a little bit scared to include the WAN_ROUTING_VLAN in the WAN interfaces list,
...,
I already added to the rules, when Amm0 suggested to use a separate VLAN for routing).
Both action=srcnat (as well as action=netmap in the srcnat chain) and action=masquerade activate a src-nat behaviour for matching connections. The difference is that masquerade is designed specifically for dynamically assigned addresses, which requires two things:One more question about masquerade vs srcnat,
Yes, maybe for one of the two routers I'll not be able to pass another wan cable.I found the load sharing link on the updated docs site................
https://help.mikrotik.com/docs/display/ ... n+Examples
So you are adding another evil twist, no physical connection?
The purpose of VLANs is to separate the traffic in the individual VLANs from each other while it passes through the same cable. So unless you connect external equipment to trunk ports (where multiple VLANs are allowed) or if your firewall configuration permits routing between interfaces representing the individual VLANs on the router, there is no problem with having WAN traffic on the same cable like the LAN one, in distinct VLANs.1) this is a WAN signal, so it is unprotected (my current TP-Link 4G Modem is in DMZ and the firewall filtering is in the RB5009), so is it safe to transport this together with all the other LAN VLANs?
Nothing to be confused about, just another subnet in its own VLAN. Just think of it as of yet another four pairs in the same cable if that makes you feel better2) in terms of solution implementation, I imagine I'll have to create a VLAN as I normally do, on the 2 routers and all the switches will transport it. This VLAN200 will be added to the WAN interfaces list, but it will be a tagged interface associated to the switch. The fact is confusione me, is that VLAN200 will be a WAN but at the same time is defined within the switch as all other LAN VLANs to be transported and used across the LAN.
The routers themselves will access VLAN200 on the bridge by means of an /interface vlan with vlan-id=200 attached to the bridge. "Untrusted" external devices should be connected to access ports with ingress filtering enabled.3) in terms of utilization of VLAN200, I can use access ports where needed, or alternatively, I can define on the router an address on the subnet of VLAN200 and use this (as I currently do for VLAN100). Am I right?
The latter approach (an additional VLAN on the single common bridge) is the only possible one if VLAN 200 shares the same bridge with the LAN VLANs on the routers. Each physical interface must belong to at maximum one bridge; you could use a single bridge for each VLAN instead (where the physical interface would not be a member port of any bridge and the VLAN interfaces would be connected to it directly), but you cannot use a mixed approach where some VLANs would share the common bridge and some would have their own.4) unless the TP-Link already supports VLANs, I've to use a dedicated physical small managed switch that has in input an access port on VLAN200 connected to the ether cable from the TP-Link WAN side and output with a trunk port on VLAN200 to connect to the rest of my LAN. I believe, I cannot use the current mikrotik switch, since I'd create a new bridge for WAN for witch I'll not have HW offloading, unless I can use the current defined bridge for the LAN (having a port belonging to WAN within the bridge).
Yes same switch for the 2 routers, but the tplink wan could most probably connected l to a separate one. Does this matter?So is there a single switch to which both the routers are connnected, and you would connect the TP-Link LTE router to that switch?
In your instructions you are saying that I plug physically the cable coming from tp-link into an ether access port of the router (etherXY) and I use dhcp-client.A switch itself has no "WAN" or "LAN" interfaces as far as it is concerned - it only has trunk ports and access ports to particular VLANs. The roles of those VLANs are determined by the routers. So you choose a port of the switch, make it a an access port to VLAN 200, and connect the existing LTE router to it. That's it on the switch.
On each of the routers, an /interface vlan attached to the bridge represents the WAN 3 interface, to which you attach the DHCP client and for which you add a masquerade rule:
/interface bridge vlan
add vlan-ids=200 bridge=bridge tagged=bridge,etherXY
/interface vlan
add vlan-id=200 interface=bridge name=bridge.200.wan
/ip dhcp-client
add disabled=no interface=bridge.200.wan default-route-distance=20
/ip firewall nat
...
add chain=srcnat out-interface=bridge.wan.200 action=masquerade
Even on the routers, "LAN" interfaces differ from "WAN" ones only in how you treat and use them. And the interfaces connected using the interconnect link (WAN_ROUTING_VLAN) even behave both ways - as WANs for outgoing connections from the LAN but at the same time they accept those incoming conections that end up routed out via the primary WAN.
Nope, I'm not, read the instructions again. In my config excerpt, etherXY is the interface of the router you use to physically connect the router to the switch, acting as a trunk port of the bridge on the router. I did not provide a config excerpt from the switch because it is really simple and because I didn't even know whether it was a Mikrotik one, so I just gave a non-formalized text instruction on how to set it up.In your instructions you are saying that I plug physically the cable coming from tp-link into an ether access port of the router (etherXY)
I can't see why you should run VRRP atop the LTE WAN interfaces on the routers. Just let each Mikrotik get its own IP address from the TP-Link LTE. Or give them static addresses from TP-Link's LAN subnet, whatever you prefer.having chosen a VRRP config, I assume that one of the 2 could be off
You do not need any /interface vlan for vlan-id=200 on the switch - the switch itself doesn't need to access vlan200, it is enough that it forwards it between its ports. But you do need to tell the switch that the port to which the TP-Link is connected is an access port to VLAN 200, soSo, on the bridge:
/interface bridge vlan
add vlan-ids=200 bridge=bridge tagged=bridge,etherXY
/interface vlan
add vlan-id=200 interface=bridge name=bridge.200.wan
On the routers, you must add the physical interface that connects each router to the switch to the tagged list for vlan-ids=200 under /interface bridge vlan. Setting the IP address as a /32 one on an L2 interface is possible but unusual. And the to-addresses in the srcnat rule must be 192.168.3.x, not 192.168.3.1.and on each router (the only point of difference is the ip address 192.168.3.x):
/interface vlan add vlan-id=200 interface=bridge name=bridge.200.wan
/interface bridge vlan add vlan-ids=200 bridge=bridge tagged=bridge
/ip address add address=192.168.3.x interface=bridge.200.wan network=192.168.3.1
/ip firewall nat add chain=src-nat out-interface=bridge.wan.200 action=srcnat ipsec-policy=out,none to-addresses=192.168.3.1
- the routes will be the same I've already defined for ether2 (ISP3 distance=15), we agreed upon yesterday
I'm sorry, I misunderstood you.Nope, I'm not, read the instructions again. In my config excerpt, etherXY is the interface of the router you use to physically connect the router to the switch, acting as a trunk port of the bridge on the router. I did not provide a config excerpt from the switch because it is really simple and because I didn't even know whether it was a Mikrotik one, so I just gave a non-formalized text instruction on how to set it up.
No, no, here I was not clear. I'll not run VRRP on LTE WAN. Makes no sense. I just meant that I will connect the LAN LTE cable on a switch, and do not want to use the router, that's it.I can't see why you should run VRRP atop the LTE WAN interfaces on the routers. Just let each Mikrotik get its own IP address from the TP-Link LTE. Or give them static addresses from TP-Link's LAN subnet, whatever you prefer.
Yes my switches are mostly Mikrotik. Nevertheless, I believe that on the switch connected directly to the LTE WAN (since I've only 1 cable), I just need only 1 ether connection (access port) from the LTE LAN to the switch for creating VLAN 200, and in output, only 1 trunk (that will go to other switches to transport it) tagged=wan-vlan200 (not tagged=ether-to-R1,ether-to-R2), since when I'll connect from the router side, I'll assign a static ip addresses 192.168.3.x (one for each router). So I really do not understand why the purpose of the trunk tagged=ether-to-R1,ether-to-R2. But maybe I'm missing something, sorry.You do not need any /interface vlan for vlan-id=200 on the switch - the switch itself doesn't need to access vlan200, it is enough that it forwards it between its ports. But you do need to tell the switch that the port to which the TP-Link is connected is an access port to VLAN 200, so
/interface bridge vlan
add vlan-ids=200 bridge=bridge tagged=ether-to-R1,ether-to-R2
/interface bridge port
add bridge=bridge interface=ether-to-TP-Link pvid=200
if it is a Mikrotik switch.
Here, you are telling me that is better to exit from the switch using a physical interface. Apart from being unusual setting the IP address as a /32 one on an L2 interface, do you see issues or side effects for this? My idea is, I could just take the WAN3 from the VLAN 200 interface, otherwise, I've to create an access port on the same switch to which the router is connected and connect this to ether2 of the router. Nothing complicated (maybe more intuitive, more elegant, clear and manageable in case of issues for debugging - and to use the @anav's words is another 'evil twist') but I could save ports and cables. Nevertheless, if you tell me that is better to use physical interface, I'll follow your experience (mine is zero on this).On the routers, you must add the physical interface that connects each router to the switch to the tagged list for vlan-ids=200 under /interface bridge vlan. Setting the IP address as a /32 one on an L2 interface is possible but unusual.
Yes, sorry. You're right.And the to-addresses in the srcnat rule must be 192.168.3.x, not 192.168.3.1.
It's me who is missing something - namely, the network diagram From the previous posts my impression was that the network consisted of two routers connected to a single switch, hence I was assuming you need to make VLAN 200 available on two trunk ports of that switch, one for each router (along with the other VLANs used for the local subnets like IoT, guest etc.). Since there are multiple switches, of course the actual paths from each of the two routers to the access port for VLAN 200 on the switch next to the LTE router may look differently.So I really do not understand why the purpose of the trunk tagged=ether-to-R1,ether-to-R2. But maybe I'm missing something, sorry.
Something got lost in translation, so I'll try to re-word it. What I am only saying is that the VLAN 200 has to be permitted, in tagged mode, on the physical port of each router that connects that router to its adjacent switch, along with the VLANs carrying the various LAN subnets that you have already permitted on that physical port. I do not see any benefit in using a separate pair of ports to carry only the WAN VLAN between the router and the switch, except maybe bandwidth issues, but that's not relevant here because the bottleneck will be the LTE, not the Ethernet wire.Here, you are telling me that is better to exit from the switch using a physical interface.
No. The only IP address in VLAN 200 each router will be ever interested in is 192.168.3.1, and the LTE router will likey be configured with a usual /24 mask on its LAN interface, so it will send ARP requests for 192.168.3.x and 192.168.3.y as needed and the two Mikrotik routers will respond. I just cannot see any benefit in doing things in an unusual way if there is no special reason for that, because it adds headache when coming back to the configuration a few months later and looking for that (nonexistent) special reasonApart from being unusual setting the IP address as a /32 one on an L2 interface, do you see issues or side effects for this?
You're right Sindy. I posted a schematic showing both routers on the same switch, but this is the current configuration with the ongoing configuration under development and test, for which I can wire without problems wan3 cables straight to the routers (the schematic was requested by anav, and we are lucky it is very simple, that helped to debug the issue I was having with the routing and invalid packets dropped ...). Later on, in production, routers will be in two different rooms and one of them will not be reachable by the wan3 ether cable, so that's why I do need to virtualise it through vlans. Of course, in this situation, routers will be wired to different switches.It's me who is missing something - namely, the network diagram From the previous posts my impression was that the network consisted of two routers connected to a single switch, hence I was assuming you need to make VLAN 200 available on two trunk ports of that switch, one for each router (along with the other VLANs used for the local subnets like IoT, guest etc.). Since there are multiple switches, of course the actual paths from each of the two routers to the access port for VLAN 200 on the switch next to the LTE router may look differently.
ok, I see. It's my fault, your english is perfect. mine is not.Something got lost in translation, so I'll try to re-word it. What I am only saying is that the VLAN 200 has to be permitted, in tagged mode, on the physical port of each router that connects that router to its adjacent switch, along with the VLANs carrying the various LAN subnets that you have already permitted on that physical port. I do not see any benefit in using a separate pair of ports to carry only the WAN VLAN between the router and the switch, except maybe bandwidth issues, but that's not relevant here because the bottleneck will be the LTE, not the Ethernet wire.
I agree with you. Evil twisting will overcomplicate things. If tomorrow I need to come back and/or change config, I will unplug and re-plug cables. Hidden configurations will give unnecessary headaches.No. The only IP address in VLAN 200 each router will be ever interested in is 192.168.3.1, and the LTE router will likey be configured with a usual /24 mask on its LAN interface, so it will send ARP requests for 192.168.3.x and 192.168.3.y as needed and the two Mikrotik routers will respond. I just cannot see any benefit in doing things in an unusual way if there is no special reason for that, because it adds headache when coming back to the configuration a few months later and looking for that (nonexistent) special reason
ok anav,Hi rikpal.
I have a draft article going for VRRP.
I am now up to the point where I have two different ISPs, ISP1 on R1 and ISP2 on R2, with a backup LTE all configured.
They share one common pair of ports to
a. create the VRRP link ( vlan333 )
b. create the cross WAN traffic links ( vlan11 and vlan12 ).
My question now, is it worth it to explore the next step. Namely TWO VRRPs?
Why did you pursue this added complexity, as it makes no sense to me and need some direction to continue.
What does makes sense is that you actually want to use ISP2 (or ISP1 depending on which is master) at ALL times!
1. Want LAN subnet A to use ISP1 and LAN subnet B to use ISP2?
2. WAant all LANS to PCC ISP1 and ISP2 traffic?
3. Want specific user(s)/device(s) to use a specific ISP?
4. Have incoming external users coming to a server and they need to come in on a specific ISP?
A complete discussion of the requirements will assist.
It may very well be that there are better ways to achieve some of the above WITHOUT using dual VRRP.
I am not sure I understand the issue. A "DMZ" is usually a colloquial term for a 1:1 dst-nat of requests incoming to the WAN address of the firewall (in this case, the TP-link) to a single particular IP address in LAN no matter what the destination port is, and it only makes sense if the WAN IP is a public one reachable from the internet (leaving aside exotic scenarios where NAT is used in private networks).the TP-Link is connected on the Draytek 3910 with the address 192.168.3.2 in DMZ (only 1 single address is possible in DMZ). So now I'll need to have more destination addresses in DMZ (at least .3 and .4 for now).
/ip firewall filter
add action=accept chain=forward connection-state=established,related,untracked
add action=drop chain=forward connection-state=invalid
add action=accept chain=forward in-interface=LAN out-interface=WAN
add action=accept chain=forward connection-nat-state=dstnat
add action=drop chain=forward
OK, that explains the need for DMZ.My SIM has a public IP address.
If the only purpose of the Mikrotik LTE router was to forward the incoming connections landing at the LTE router's WAN further to the "big" Mikrotiks, there would be no point in such replacement - the right way is indeed to use VRRP on VLAN 200 and let the TP-link forward these incoming connections to the virtual address.And anyway, the TP-Link FW is quite basic and not so much flexible. I trust more the Mikrotik one.
...
I'm really thinking to replace the TP-Link modem (also for other more valid reasons), with a mikrotik 4G LTE router.
Port triggering is at best a weird and unreliable bandaid for old services that were designed before NAT came into wide use. The only advantage as compared to static port forwarding is that it can make a kind of time-based multiplexing of the same port, but if two devices on LAN need to connect to the same service at the same time and that service needs the same port on the same WAN IP address to be forwarded to both of them, it won't work anyway as the last LAN device to connect to the server will always redirect that port to its own address. All the cloud based services use a single TCP connection initiated from the LAN side so even if tens of devices connect to the same server, a normal src-nat handles that and no exotic port triggering is necessary.The Draytek router manages a special port forward feature that it's named 'Port Triggering'
I am confused. Your configuration shows that you have two VRRP interfaces in each LAN-side VLAN, and that the priorities of these two VRRP interfaces are set in such a way that when both routers are alive, each of these two VRRP interfaces is active on another router. The only purpose of such a setup is to have a possibility to make each host in the same VLAN prefer either ISP 1 or ISP 2 by setting one or the other VRRP address as the default gateway in the configuration of that host. But what you wrote now looks as if you actually wanted to place hosts that should prefer ISP 1 into a different VLAN (and subnet) than those that should prefer ISP 2. Both approaches are possible, both have their pros and cons, but you cannot mix them together. So what is the actual intention?3. TCP server on specific port (e.g. 3333) through cloudflare proxy servers on VLAN1 (router1)
4. TCP server reachable on specific port (e.g. 2222) on VLAN2 (router2)
If a server in LAN uses an IP address that is up on router 2 (no matter whether it is a "physical" one or a VRRP-controlled one) as a default gateway, it sends all packets via router 2. So if such packet is a response to a request packet that came to the server via router 1, you have the problem with firewall (non-symmetric routing) because router 2 has never seen the request packet so it will mishandle the response (drop it if it is a TCP packet, and treat it as an initial request of a new connection if it is a UDP packet).2. I access from outside using ISP2 a service provided by VLAN1 (router1) - will I've problems in this case? I expect routing will work ok and no headaches (it could be also viceversa ISP1->router2)
If you add a LTE-capable Mikrotik into the mix, you can simplify the overall setup a lot by removing the interconnections allowing Router 1 to use ISP 2 via Router 2 and vice versa. Instead, you can make all 3 routers use only their own ISP, and do all the traffic distribution by means of the VRRP addresses - you would use one VRRP address in each LAN VLAN to represent a distinct ISP. So as long as everything works, each host in the LAN sends its traffic through one of the three VRRP addresses in that LAN, which is up on the router that is a gateway to that ISP. If an ISP becomes unreachable, the gateway router to that ISP must lower the priority of the corresponding VRRP interfaces so that they would go to backup mode and the traffic from hosts that prefer that ISP would start flowing through another router. I don't know any way how to do this without scripting, though.With regard to ISP3, assuming to replace the 4G/LTE modem with a mikrotik one, I would like to address the incoming requests to the specific internal servers IP addresses using dstnat rules to the proper router1/2 (so I forget about DMZ).
That's what I have suggested above - if you know through which router the server in LAN will respond, you can use per-port dst-nat rules (aka port forwarding) rather than a port-agnostic dst-nat (aka DMZ).3. I access a service from outside using ISP3 and the service could be on VLAN1 (router1) or VLAN2 (router2) - I'll use specific dstnat rules in the mikrotikg LTE router, pointing to the right server.
You can indeed use the current address of the Draytek as one of the VRRP addresses, but for connections initiated from the outside, you'll still have the issue described above - how to make the response pass the same router(s) like the request.Al the above, assuming that I'm using the VRRPs virtual GWs, so for example if I'll continue to run my services with the same GW I've today on Draytek (e.g. 192.168.1.253), and I do not know if VRRP is referring to a physical GW on router1 (192.168.1.1) or on router2 (192.168.1.2). In this way, I'll replace the Draytek with the 2xRB5009 without changing anything in my network definitions.
There is no functional equivalent of Draytek's port triggering in RouterOS. You can use an address-list to ensure that the port-forwarding on the controlled port is only enabled for some time after some communication on the controlling one took place (which is what Draytek presents as a security improvement, but it's disputable), but there is no way to change the destination address in the port-forwarding rule except scripting, which may not be responsive enough.Finally, with regard to 'port triggering' my understanding is that moving to mikrotik, I do not have to do anything (no need to define any dstnat rule), since if the connection is originated from a nat address, it will be allowed. Am I right?
VRRP configuration is done in a way that 2 VLANs run completely on router1 and 2 VLANs completely on router2.I am confused. Your configuration shows that you have two VRRP interfaces in each LAN-side VLAN, and that the priorities of these two VRRP interfaces are set in such a way that when both routers are alive, each of these two VRRP interfaces is active on another router. The only purpose of such a setup is to have a possibility to make each host in the same VLAN prefer either ISP 1 or ISP 2 by setting one or the other VRRP address as the default gateway in the configuration of that host. But what you wrote now looks as if you actually wanted to place hosts that should prefer ISP 1 into a different VLAN (and subnet) than those that should prefer ISP 2. Both approaches are possible, both have their pros and cons, but you cannot mix them together. So what is the actual intention?
How can I mark connections initiated from ISP1 or ISP2?If a server in LAN uses an IP address that is up on router 2 (no matter whether it is a "physical" one or a VRRP-controlled one) as a default gateway, it sends all packets via router 2. So if such packet is a response to a request packet that came to the server via router 1, you have the problem with firewall (non-symmetric routing) because router 2 has never seen the request packet so it will mishandle the response (drop it if it is a TCP packet, and treat it as an initial request of a new connection if it is a UDP packet).
Are you saying that it would be better doing the following:If you add a LTE-capable Mikrotik into the mix, you can simplify the overall setup a lot by removing the interconnections allowing Router 1 to use ISP 2 via Router 2 and vice versa. Instead, you can make all 3 routers use only their own ISP, and do all the traffic distribution by means of the VRRP addresses - you would use one VRRP address in each LAN VLAN to represent a distinct ISP. So as long as everything works, each host in the LAN sends its traffic through one of the three VRRP addresses in that LAN, which is up on the router that is a gateway to that ISP. If an ISP becomes unreachable, the gateway router to that ISP must lower the priority of the corresponding VRRP interfaces so that they would go to backup mode and the traffic from hosts that prefer that ISP would start flowing through another router. I don't know any way how to do this without scripting, though.
But should I need anyway to define VRRP on VLAN 200 also if I do use port forwarding knowing on which router the server in lan will respond?That's what I have suggested above - if you know through which router the server in LAN will respond, you can use per-port dst-nat rules (aka port forwarding) rather than a port-agnostic dst-nat (aka DMZ).3. I access a service from outside using ISP3 and the service could be on VLAN1 (router1) or VLAN2 (router2) - I'll use specific dstnat rules in the mikrotik LTE router, pointing to the right server.
I'll do a test about this, tnx.I cannot answer the question whether you have to do anything as I don't know whether you actually need a dst-nat rule for the Ariston boiler or not. It just feel that NAT has been hanging around for 20+ years before boilers started having IP addresses, so there should be no need for the "port triggering". But experience suggests to never say something is impossible.
If you refer to 'inter-vlan' connections, I want to have all the VLANs completely segregated each one from the other. So no communications between to VLANs, expect in special cases, but I'll manage this in case (address lists?).A point we haven't mentioned throughout the whole conversation is whether you need hosts in different LAN VLANs to talk to each other and if yes, whether you want the firewall to control that communication (i.e. allowing your PC to initiate a connection to the boiler but not allowing the boiler to initiate a connection to the PC) as that puts additional requirements on the configuration.
If so, I don't understand why you have two VRRP interfaces per VLAN per router if you actually use only one of them.VRRP configuration is done in a way that 2 VLANs run completely on router1 and 2 VLANs completely on router2.
Virtual GWs are always the same for each VLAN/subnet and are exactly the same I currently have on the draytek, to avoid changing the clients definitions on my network. I don't use to refer to the backup virtual GWs to avoid confusion. For the sake of clearness, I just did several tests and when I power off one of the 2 routers all the 4 VLANs move on one single router and GWs remain the same.
You can only mark connections in mangle, nowhere else. Also translation of connection-mark to routing-mark only happens in mangle. The connection-mark value is not added to the actual packet, it is stored in the connection tracking "database" on the router and assigned to each packet belonging to the connection as it passes through the connection tracking module. So if the response goes via another router than the request, it will not get the connection mark (except if the connection tracking is synchronized between the routers, but that is not compatible with the load distribution mode, even with the one you actually use).How can I mark connections initiated from ISP1 or ISP2?
I believe I should just mark them and define new routing tables ad FIB, but not use any mangling ...
Worse than that, connection marking only works inside the same router. There is no way to add something to the packet being forwarded from WAN to the server that would tell the server "use another routing table to send responses to this packet". The server would have to have multiple VLAN interfaces with individual IP addresses and multiple routing tables so that it would respond via the same VLAN via which the request came in, which may not be possible with the "blackboxes" you have (NAS etc). If you cannot implement this on a server because it can only listen on a single address, you would have to override the use of default gateway by means of src-nat as I've suggested before.But doing in this way, for connections initiated from the outside, I believe that I will still have the issue described above and I've to mark incoming connections. E.g. in case I'll use ISP2 to access router1. Am I right?
This would help only if each server in LAN could only be contacted via one WAN. So in your "each server in its own VLAN" approach, this would mean you would have dedicated VLANs for servers that have to be contacted via WAN 3.But should I need anyway to define VRRP on VLAN 200 also if I do use port forwarding knowing on which router the server in lan will respond?
In case I've to use VRRP for VLAN 200, I do not understand how could help having the third (virtual) address for VLAN 200.
A single special case is enough to require the complete handling of inter-vlan traffic. If you want to use a stateful firewall, you get the same problem with non-symmetric routing like for the WAN<->LAN traffic. Each endpoint device will send the traffic for the other endpoint device through its default gateway; if these default gateways are on different routers, each direction will be handled by another router and connection tracking will get confused.So no communications between to VLANs, expect in special cases, but I'll manage this in case (address lists?).
OK, you're right, let me work on this.If so, I don't understand why you have two VRRP interfaces per VLAN per router if you actually use only one of them.
OK, can you help me with this config, pls? I found some partial examples but they are not for RoS7You can only mark connections in mangle, nowhere else. Also translation of connection-mark to routing-mark only happens in mangle. The connection-mark value is not added to the actual packet, it is stored in the connection tracking "database" on the router and assigned to each packet belonging to the connection as it passes through the connection tracking module. So if the response goes via another router than the request, it will not get the connection mark (except if the connection tracking is synchronized between the routers, but that is not compatible with the load distribution mode, even with the one you actually use).
This should not be my case, most important server, as well as, all NAS they have network cards that are connected to multiple subnets.Worse than that, connection marking only works inside the same router. There is no way to add something to the packet being forwarded from WAN to the server that would tell the server "use another routing table to send responses to this packet". The server would have to have multiple VLAN interfaces with individual IP addresses and multiple routing tables so that it would respond via the same VLAN via which the request came in, which may not be possible with the "blackboxes" you have (NAS etc). If you cannot implement this on a server because it can only listen on a single address, you would have to override the use of default gateway by means of src-nat as I've suggested before.
I'm sorry, but I do not understand this point. Can you elaborate a little bit more?This would help only if each server in LAN could only be contacted via one WAN. So in your "each server in its own VLAN" approach, this would mean you would have dedicated VLANs for servers that have to be contacted via WAN 3.
As of today, I do not need inter-vlan routing.A single special case is enough to require the complete handling of inter-vlan traffic. If you want to use a stateful firewall, you get the same problem with non-symmetric routing like for the WAN<->LAN traffic. Each endpoint device will send the traffic for the other endpoint device through its default gateway; if these default gateways are on different routers, each direction will be handled by another router and connection tracking will get confused.
#ROUTER #01
/tool netwatch
add comment="*** check ISP1 for failover ***" disabled=no down-script="/interf\
ace vrrp set [find name=vrrp5] priority=150\r\
\n/interface vrrp set [find name=vrrp10] priority=120\r\
\n/interface vrrp set [find name=vrrp20] priority=120\r\
\n/interface vrrp set [find name=vrrp3090] priority=150\r\
\n" host=<PUBLIC_IP_ISP2> http-codes="" interval=30s name=ISP1_status \
packet-count=3 startup-delay=1m test-script="" type=icmp up-script="/inter\
face vrrp set [find name=vrrp5] priority=250\r\
\n/interface vrrp set [find name=vrrp10] priority=200\r\
\n/interface vrrp set [find name=vrrp20] priority=200\r\
\n/interface vrrp set [find name=vrrp3090] priority=250\r\
\n"
# ROUTER #02
/tool netwatch
add comment="*** check ISP2 for failover ***" disabled=no down-script="/interf\
ace vrrp set [find name=vrrp5] priority=120\r\
\n/interface vrrp set [find name=vrrp10] priority=150\r\
\n/interface vrrp set [find name=vrrp20] priority=150\r\
\n/interface vrrp set [find name=vrrp3090] priority=120\r\
\n" host=<PUBLIC_IP_ISP2> http-codes="" interval=30s name=ISP2_status \
packet-count=3 startup-delay=1m test-script="" type=icmp up-script="/inter\
face vrrp set [find name=vrrp5] priority=200\r\
\n/interface vrrp set [find name=vrrp10] priority=250\r\
\n/interface vrrp set [find name=vrrp20] priority=250\r\
\n/interface vrrp set [find name=vrrp3090] priority=200\r\
\n"
# router #01
# YOU HAVE THESE ONLY IF YOU NEED TO MANAGE INCOMING CONNECTIONS incoming WANx -> out WANx
/routing table
add name=ISP1_route fib
# add name=ISP2_route fib
add name=ISP3_route fib
/ip route
add check-gateway=ping dst-address=0.0.0.0/0 gateway=9.9.9.9 scope=13 target-scope=13 distance=5 routing-table=main
# add check-gateway=ping dst-address=0.0.0.0/0 gateway=8.8.8.8 scope=13 target-scope=13 distance=10 routing-table=main
add check-gateway=ping dst-address=0.0.0.0/0 gateway=1.0.0.1 scope=13 target-scope=13 distance=5 routing-table=main
# add check-gateway=ping dst-address=0.0.0.0/0 gateway=208.67.222.220 scope=13 target-scope=13 distance=10 routing-table=main
# YOU HAVE THESE ONLY IF YOU NEED TO MANAGE INCOMING CONNECTIONS incoming WANx -> out WANx
add check-gateway=ping dst-address=0.0.0.0/0 gateway=9.9.9.9 scope=13 target-scope=13 distance=5 routing-table=ISP1_route
# add check-gateway=ping dst-address=0.0.0.0/0 gateway=8.8.8.8 scope=13 target-scope=13 distance=10 routing-table=ISP2_route
add check-gateway=ping dst-address=0.0.0.0/0 gateway=1.0.0.1 scope=13 target-scope=13 distance=5 routing-table=ISP1_route
# add check-gateway=ping dst-address=0.0.0.0/0 gateway=208.67.222.220 scope=13 target-scope=13 distance=10 routing-table=ISP2_route
add dst-address=9.9.9.9/32 gateway=pppoe-out scope=10 target-scope=12 comment="WAN1 ISP1 via PPPoE - ping Host 1" distance=5
# add dst-address=8.8.8.8/32 gateway=172.22.1.2 scope=10 target-scope=12 comment="ISP2 via Backup Router - ping Host 1" distance=10
add dst-address=1.0.0.1/32 gateway=pppoe-out scope=10 target-scope=12 comment="WAN1 ISP1 via PPPoE - ping Host 2" distance=5
# add dst-address=208.67.222.220/32 gateway=172.22.1.2 scope=10 target-scope=12 comment="ISP2 via Backup Router - ping Host 2" distance=10
add dst-address=0.0.0.0/0 gateway=192.168.4.1 scope=10 target-scope=12 comment="ether2 ISP3 via GW 4G/LTE" distance=15 routing-table=main
# YOU HAVE THIS ONLY IF YOU NEED TO MANAGE INCOMING CONNECTIONS incoming WAN3 -> out WAN3
add dst-address=0.0.0.0/0 gateway=192.168.4.1 scope=10 target-scope=12 comment="incoming ISP3 via GW 4G/LTE" distance=15 routing-table=ISP3_route
# YOU HAVE THESE ONLY IF YOU NEED TO MANAGE INCOMING CONNECTIONS incoming WANx -> out WANx
# MANGLING TO MANAGE incoming WANx -> out WANx
/ip firewall mangle
add chain=prerouting in-interface=pppoe-out connection-state=new action=mark-connection new-connection-mark=ISP1_conn
add chain=prerouting in-interface-list=VRRP connection-mark=ISP1_conn action=mark-routing new-routing-mark=ISP1_route
# add chain=prerouting in-interface=WAN_ROUTING_VLAN connection-state=new action=mark-connection new-connection-mark=ISP2_conn
# add chain=prerouting in-interface-list=VRRP connection-mark=ISP2_conn action=mark-routing new-routing-mark=ISP2_route
add chain=prerouting in-interface=WAN_ISP3_VLAN connection-state=new action=mark-connection new-connection-mark=ISP3_conn
add chain=prerouting in-interface-list=VRRP connection-mark=ISP3_conn action=mark-routing new-routing-mark=ISP3_route
# router #02
# YOU HAVE THESE ONLY IF YOU NEED TO MANAGE INCOMING CONNECTIONS incoming WANx -> out WANx
/routing table
# add name=ISP1_route fib
add name=ISP2_route fib
add name=ISP3_route fib
/ip route
add check-gateway=ping dst-address=0.0.0.0/0 gateway=8.8.8.8 scope=13 target-scope=13 distance=5 routing-table=main
# add check-gateway=ping dst-address=0.0.0.0/0 gateway=9.9.9.9 scope=13 target-scope=13 distance=10 routing-table=main
add check-gateway=ping dst-address=0.0.0.0/0 gateway=208.67.222.220 scope=13 target-scope=13 distance=5 routing-table=main
# add check-gateway=ping dst-address=0.0.0.0/0 gateway=1.0.0.1 scope=13 target-scope=13 distance=10 routing-table=main
# YOU HAVE THESE ONLY IF YOU NEED TO MANAGE INCOMING CONNECTIONS incoming WANx -> out WANx
add check-gateway=ping dst-address=0.0.0.0/0 gateway=8.8.8.8 scope=13 target-scope=13 distance=5 routing-table=ISP2_route
# add check-gateway=ping dst-address=0.0.0.0/0 gateway=9.9.9.9 scope=13 target-scope=13 distance=10 routing-table=ISP1_route
add check-gateway=ping dst-address=0.0.0.0/0 gateway=208.67.222.220 scope=13 target-scope=13 distance=5 routing-table=ISP2_route
# add check-gateway=ping dst-address=0.0.0.0/0 gateway=1.0.0.1 scope=13 target-scope=13 distance=10 routing-table=ISP1_route
add dst-address=8.8.8.8/32 gateway=pppoe-out scope=10 target-scope=12 comment="WAN1 ISP2 via PPPoE - ping Host 1" distance=5
# add dst-address=9.9.9.9/32 gateway=172.22.1.1 scope=10 target-scope=12 comment="ISP1 via Backup Router - ping Host 1" distance=10
add dst-address=208.67.222.220/32 gateway=pppoe-out scope=10 target-scope=12 comment="WAN1 ISP2 via PPPoE - ping Host 2" distance=5
# add dst-address=1.0.0.1/32 gateway=172.22.1.1 scope=10 target-scope=12 comment="ISP1 via Backup Router - ping Host 2" distance=10
add dst-address=0.0.0.0/0 gateway=192.168.4.1 scope=10 target-scope=12 comment="ether2 ISP3 via GW 4G/LTE" distance=15 routing-table=main
# YOU HAVE THIS ONLY IF YOU NEED TO MANAGE INCOMING CONNECTIONS incoming WAN3 -> out WAN3
add dst-address=0.0.0.0/0 gateway=192.168.4.1 scope=10 target-scope=12 comment="incoming ISP3 via GW 4G/LTE" distance=15 routing-table=ISP3_route
# YOU HAVE THESE ONLY IF YOU NEED TO MANAGE INCOMING CONNECTIONS incoming WANx -> out WANx
# MANGLING TO MANAGE incoming WANx -> out WANx
/ip firewall mangle
add chain=prerouting in-interface=pppoe-out connection-state=new action=mark-connection new-connection-mark=ISP2_conn
add chain=prerouting in-interface-list=VRRP connection-mark=ISP2_conn action=mark-routing new-routing-mark=ISP2_route
# add chain=prerouting in-interface=WAN_ROUTING_VLAN connection-state=new action=mark-connection new-connection-mark=ISP1_conn
# add chain=prerouting in-interface-list=VRRP connection-mark=ISP1_conn action=mark-routing new-routing-mark=ISP1_route
add chain=prerouting in-interface=WAN_ISP3_VLAN connection-state=new action=mark-connection new-connection-mark=ISP3_conn
add chain=prerouting in-interface-list=VRRP connection-mark=ISP3_conn action=mark-routing new-routing-mark=ISP3_route
Since I do not really need to distribute a single VLAN load on 2 routers, for me it's simpler to use only 1 gateway for my clients.i have just read couple of your last post, and found that you have abandoned your vrrp master-master setup? why?
yes, you're right but I implemented it to be sure the incoming traffic for a WAN will go out for the same WAN, having multiple WANs.about the mangle and mark etc. i don't think they are vrrp related.
can you better explain this to me? Is there any way for me that I can access from ISP1/router1 servers running on router2? I understand this from your post, but I do not know how it could be feasible (VRRP master-master? DNS?)your master -master setup is doable, to gain benefits from those isp links. but you should compensate vrrp for the nat sessions.
as for internet inbound to your network, you can have your dns to have both isp1 isp2 ip to any service you have on the network as usual.
you should choose which ip is the most reliable to have the first option. for example smtp mx 10, mx 20 which 10 is the most reliable ip.
dst nat those ip to your server respectively. no problem except for vrrp taking failover action - it will break existing session, but not for new sessions.
ok. try to observe this example.Since I do not really need to distribute a single VLAN load on 2 routers, for me it's simpler to use only 1 gateway for my clients.
isp1 --- r1/vrrp10,11,12 --- server10
|
isp2 --- r2/vrrp13,14,15 --- server13