With ether1 disconnnected?
Is it the same with ether1 connected but internet not available because dsl line is disconnected from router?
So:
- Both 8.8.4.4 and 8.8.8.8 works, got back all that packages
- Same as number 1. everything came back
- so as my previous post, ISP still disconnected from Mikrotik (no hard connection between them ethernet off) until ‘DOWN’ and LTE is on both 4.4 and 8.8 comes through, if in ‘UP’ LTE goes off and none of the pings goes through, so like nothing happens, if LTE on both 8.8.4.4 and 8.8.8.8 got pinged correctly so the netwatch turns off the LTE but than nothing comes through because not connection to the internet, because of that LTE goes ON again and this repeating.
- Waited 10 min with ISP off Netwatch enabled, mangle is still disabled, and a flipping just continues
- yeah If I reconnect ISP everything works as should be no flapping.
IF I keep the the ISP modem connected to Mikrotik but internet cable before the ISP modem (DSL cable, coax cable disconnected):
- (Netwatch off, mangle off, ISP modem connected to mikrotik ) 8.8.4.4 time out, 8.8.8.8 good ping
- (netwatch off, mangle off, ISP modem from mikrotik physically disconnected from mikrotik) 8.8.4.4 coming through and is ok , 8.8.8.8 not working until some time than flipping starts again,
AND if I connect back the ISP ethernet cable to the mikrotik a number 1 happens 4.4 time out 8.8. ok, no flipping - (netwatch on mangle off, number 2 config) Flipping
- (netwatch on mangle off, number 1 config) no flipping after some time internet comes back and LTE turns off
What I figured out… IF I physically disconnect ISP from Mikrotik, the flipping starts again, if I just disconnect the main internet cable from the ISP modem, and I put back the connection between the modem and Mikrotik flipping stops and works as should… and I think is the reason is, If I physically disconnect the ISP from Mikrotik the connection between them are disappears from the route list, if I just cut the internet in the ISP (not between ISP and Mikrotik) the 192.168.0.1 will be there
Mangle was off in the whole time! I am not sure what is its purpose overall in this case.
Sorry, it was cross-post, because I tried to manage the stuff that you suggested, so yes there is a different behavior if I cut the internet or cut the connection between ISP modem and Mikrotik.
Ok, it starts to make sense, so, for some reasons the "narrow" route becomes invalid if the ether1 cable is disconnected. (I suspect it to be due to the ether1 still having the dhcp client and not a static address, but maybe there are other reasons) so the 8.8.4.4 ping takes the "LTE" route and works fine.
Try adding a blackhole route with distance 2. (a blackhole route is something that fails at pinging as it goes nowhere)
Right now you should have in main the "narrow" rule:
/ip route
...
add dst-address=8.8.4.4/32 gateway=192.168.0.1%ether1
add a second route to the same address but with type blackhole and distance 2:
/ip route
add dst-address=8.8.4.4/32 distance=2 type=blackhole
add blackhole dst-address=8.8.4.4 distance=2
The idea is that:
- when the DSL modem is connected to ether1 AND there is internet on the DSL, the first rule is active (distance=1) and the pings of the netwatch script go through it (and succeed) thus nothing happens
- when the DSL modem is connected to ether1 BUT there is not internet on the DSL (simulated by disconnecting coax cable) the rule is active (distance=1) and the pings of the netwatch script go through it (but fail) thus switching to LTE
- when the DSL modem is NOT connected to ether1 AND thus there is not internet on the DSL AND that rule (for whatever reasons) disappears/becomes inactive, the second route will become active (distance=2) and the pings of the netwatch script go through it (but fail) thus doing nothing
- when the internet is restored on DSL the first rule returns active and the netwatch script disables the LTE
Let's see if this approach works.
EDIT: corrected wrong command line
There are some problem with the code, there is a red indication in type row, how should I proceed with this, because I don’t get the theory behind.
I tested the system, and it works nice, if I simulate an electricity blackout, it first connects to LTE then when the ISP boots up and Netwatch startup 5 min delay elapses connects to the ISP and deactivates LTE, pretty stable, no issues so far if I not disconnect ISP/Mikrotik ethernet cable. Thank you for reaching this setup 
I reenabled the mangle rule as well.
In the afternoon I made a factory reset for the whole Mikrotik router and set up everything from the basics without adding the tried rules and experiments.
# 2024-10-07 18:36:25 by RouterOS 7.16
# software id = xxxx
#
# model = C52iG-5HaxD2HaxD
# serial number = xxxx
/interface bridge
add admin-mac=xxxxxx auto-mac=no comment=defconf name=bridge
/interface wifi
set [ find default-name=wifi1 ] channel.band=5ghz-ax .skip-dfs-channels=\
10min-cac .width=20/40/80mhz configuration.mode=ap .ssid=HomeSSid_5Ghz \
disabled=no security.authentication-types=wpa2-psk .ft=yes .ft-over-ds=\
yes
set [ find default-name=wifi2 ] channel.band=2ghz-ax .skip-dfs-channels=\
10min-cac .width=20mhz configuration.mode=ap .ssid=HomeSSid disabled=no \
security.authentication-types=wpa2-psk .ft=yes .ft-over-ds=yes
add configuration.mode=ap .ssid=SSidGuest mac-address=4xxxxxx \
master-interface=wifi1 name=wifi3
add configuration.mode=ap .ssid=SSidGuest disabled=no mac-address=\
xxxxxxxx master-interface=wifi2 name=wifi4 \
security.authentication-types=wpa2-psk
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/ip pool
add name=default-dhcp ranges=192.168.88.10-192.168.88.254
/ip dhcp-server
add address-pool=default-dhcp interface=bridge name=defconf
/routing table
add disabled=no fib name=DSL
/disk settings
set auto-media-interface=bridge auto-media-sharing=yes auto-smb-sharing=yes
/interface bridge filter
# wifi3 not ready
# in/out-bridge-port matcher not possible when interface (wifi3) is not slave
add action=drop chain=forward in-interface=wifi3
# wifi3 not ready
# in/out-bridge-port matcher not possible when interface (wifi3) is not slave
add action=drop chain=forward out-interface=wifi3
add action=drop chain=forward in-interface=wifi4
add action=drop chain=forward out-interface=wifi4
/interface bridge port
add bridge=bridge comment=defconf interface=ether3
add bridge=bridge comment=defconf interface=ether4
add bridge=bridge comment=defconf interface=ether5
add bridge=bridge comment=defconf interface=wifi1
add bridge=bridge comment=defconf interface=wifi2
add bridge=bridge interface=wifi3
add bridge=bridge interface=wifi4
/ip neighbor discovery-settings
set discover-interface-list=LAN
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1 list=WAN
add interface=ether2 list=WAN
/ip address
add address=192.168.88.1/24 comment=defconf interface=bridge network=\
192.168.88.0
/ip dhcp-client
add comment=defconf default-route-distance=2 interface=ether1
add interface=ether2
/ip dhcp-server network
add address=192.168.88.0/24 comment=defconf dns-server=192.168.88.1 gateway=\
192.168.88.1
/ip dns
set allow-remote-requests=yes
/ip dns static
add address=192.168.88.1 comment=defconf name=router.lan type=A
/ip firewall filter
add action=accept chain=input comment=\
"defconf: accept established,related,untracked" connection-state=\
established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment=\
"defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=drop chain=input comment="defconf: drop all not coming from LAN" \
in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" \
ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" \
ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" \
connection-state=established,related hw-offload=yes
add action=accept chain=forward comment=\
"defconf: accept established,related, untracked" connection-state=\
established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
connection-state=invalid
add action=drop chain=forward comment=\
"defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat \
connection-state=new in-interface-list=WAN
/ip firewall mangle
add action=mark-routing chain=output dst-address=8.8.4.4 new-routing-mark=DSL \
passthrough=yes protocol=icmp
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" \
ipsec-policy=out,none out-interface-list=WAN
/ip route
add disabled=no dst-address=0.0.0.0/0 gateway=192.168.0.1 routing-table=DSL \
suppress-hw-offload=no
add dst-address=8.8.4.4/32 gateway=192.168.0.1%ether1
/ipv6 firewall address-list
add address=::/128 comment="defconf: unspecified address" list=bad_ipv6
add address=::1/128 comment="defconf: lo" list=bad_ipv6
add address=fec0::/10 comment="defconf: site-local" list=bad_ipv6
add address=::ffff:0.0.0.0/96 comment="defconf: ipv4-mapped" list=bad_ipv6
add address=::/96 comment="defconf: ipv4 compat" list=bad_ipv6
add address=100::/64 comment="defconf: discard only " list=bad_ipv6
add address=2001:db8::/32 comment="defconf: documentation" list=bad_ipv6
add address=2001:10::/28 comment="defconf: ORCHID" list=bad_ipv6
add address=3ffe::/16 comment="defconf: 6bone" list=bad_ipv6
/ipv6 firewall filter
add action=accept chain=input comment=\
"defconf: accept established,related,untracked" connection-state=\
established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
invalid
add action=accept chain=input comment="defconf: accept ICMPv6" protocol=\
icmpv6
add action=accept chain=input comment="defconf: accept UDP traceroute" \
dst-port=33434-33534 protocol=udp
add action=accept chain=input comment=\
"defconf: accept DHCPv6-Client prefix delegation." dst-port=546 protocol=\
udp src-address=fe80::/10
add action=accept chain=input comment="defconf: accept IKE" dst-port=500,4500 \
protocol=udp
add action=accept chain=input comment="defconf: accept ipsec AH" protocol=\
ipsec-ah
add action=accept chain=input comment="defconf: accept ipsec ESP" protocol=\
ipsec-esp
add action=accept chain=input comment=\
"defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=input comment=\
"defconf: drop everything else not coming from LAN" in-interface-list=\
!LAN
add action=accept chain=forward comment=\
"defconf: accept established,related,untracked" connection-state=\
established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
connection-state=invalid
add action=drop chain=forward comment=\
"defconf: drop packets with bad src ipv6" src-address-list=bad_ipv6
add action=drop chain=forward comment=\
"defconf: drop packets with bad dst ipv6" dst-address-list=bad_ipv6
add action=drop chain=forward comment="defconf: rfc4890 drop hop-limit=1" \
hop-limit=equal:1 protocol=icmpv6
add action=accept chain=forward comment="defconf: accept ICMPv6" protocol=\
icmpv6
add action=accept chain=forward comment="defconf: accept HIP" protocol=139
add action=accept chain=forward comment="defconf: accept IKE" dst-port=\
500,4500 protocol=udp
add action=accept chain=forward comment="defconf: accept ipsec AH" protocol=\
ipsec-ah
add action=accept chain=forward comment="defconf: accept ipsec ESP" protocol=\
ipsec-esp
add action=accept chain=forward comment=\
"defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=forward comment=\
"defconf: drop everything else not coming from LAN" in-interface-list=\
!LAN
/system clock
set time-zone-name=Exxxxxx
/system note
set show-at-login=no
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN
/tool netwatch
add disabled=no down-script=\
"/ip route enable [find dst-address=0.0.0.0/0 and gateway=10.0.0.1]" \
host=8.8.4.4 http-codes="" interval=10s test-script="" timeout=5s type=\
simple up-script=\
"/ip route disable [find dst-address=0.0.0.0/0 and gateway=10.0.0.1]"
My bad, should have been:
/ip route
add blackhole dst-address=8.8.4.4 distance=2
Let’s see if I can help you grasp the concepts, the more you get them the more you will be able to correct/fix/tune your setup.
Normally there are two “all internet” rules, the first going through the LTE (disabled) and the second going through the DSL.
We carve an exception to the above EITHER by using a mangle rule and a separate routing table OR by using a very narrow route in the main table to “force” connection to 8.8.4.4 to go ONLY through the DSL, no matter if the DSL or the LTE general rules are active.
The netwatch pings the 8.8.4.4 to switch the LTE on or off.
When you are (more properly) simulating the loss of DSL connection by removing the coax cable) everything works, but when you simulate the the loss of connection by disconnecting the cable from ether1, for some reasons the routes pointing to ether1/192.168.0.1, both in the added “DSL” routing table and in the main one are not anymore valid, so the 8.8.4.4 is reached through the “general” rule via the LTE connection.
When the netwatch script tries again to ping 8.8.4.4 can reach it alright, so it disables the LTE route, but at the following next ping (since there is no valid route as the one going through LTE is disabled and the main one through DSL is not active because the link to the DSL router is disconnected) it cannot reach 8.8.4.4 so it enables again the LTE.
Rinse and repeat and you get the flapping.
The idea of the blackhole route (with bigger distance of 2) is that normally this route is not active as there is already another route with the same dst-address but with the lower distance of 1.
When the ether1 is disconnected and the routes through ether1 become invalid, this one becomes active.
Netwatch tries to ping the 8.8.4.4 but since it goes to this blackhole it fails and does not disable the LTE.
When you reconnect the ether1, the 8.8.4.4 route via ether1 becomes again active, the blackhole one becomes inactive and netwatch can ping 8.8.4.4 successfully, so it disables the LTE route.
If you keep the mangle rule enabled, the “other” routing table will be used, and that one has no blackhole, so you will still have the flapping when disconnecting ether1.
It is EITHER mangle rule and separate routing table OR two routes to 8.8.4.4 (one through ether1, one blackhole) in main.
You should try the added blackhole route approach, disabling the mangle rule, as (if it works) it should cover also the possibility of an ethernet cable going bad (ether1 disconnected) or a failure of the ISP modem power supply.
BTW with a static address assigned to ether1 the original approach with the mangle rule or the one with narrow route in main should work also in case of ether1 disconnection, what triggers the mess must be the loss of address of ether1 from the DHCP or however the interrupted connection between ether1 and the DHCP server of the ISP router.
Ahaa, I started to get the concept, clearer step by step, thank you for your patience!
So, your method is working perfectly already tried the following tests:
- ISP modem to Mikrotik modem UTP cable disconnect - no flipping anymore, works as should!
- Internet coax cable disconnect from ISP modem, ISP modem and Mikrotik connected - no flipping, perfect switch over
- Blackout simulator (ISP, Mikrotik, LTE modem start simultaneously) - works perfectly after 5 min netwatch delay, ISP takes over, LTE feeds the system with internet until
Thank you very much for this and the education!
Any good advice for my setup?
- I saw there is a bridge drop filter for wifi3/4 guest wifi network, do you think that is enough? I usually connect for that wifi the not trusted stuff for example air purifier, air condition (I don’t feel I can trust Chinese-made home stuff, don’t want them to reach the home network)
- I want to implement a NAS drive for movies, is it really hard to manage for home use?
- I just want to be more or less safe in my network environment from outside attacks, myself is self-conscious about the usage of the internet so that won’t be a problem hopefully.
- Do you think if I just keep my ISP modem in router mode won’t make a big difference? What are the pros and contras between the bridge and router mode of the ISP modem?
Not that I am actually qualified for giving an expert opinion on bridging vs. routing, so take my advice with a grain or two of salt.
In theory the bridge mode is more direct, so It should be faster, in practice in your case It Is slower, and not a little slower, some 30 percent slower is a lot.
Of course there must be some reasons why this happens, but I cannot find any in your rather simple configuration and - if you think about it - the Mikrotik always works as a router, whether it routes between LAN 192.168.88.x and WAN 5.x.y.z or WAN 192.168.0.x should not affect the speed of the connection, so the issue must be in either the ISP router configuration or in some settings in the ISP source but I have no idea where to look for the cause .
Still in theory the double NAT (one in the Mikrotik and one in the ISP router) might increase the latency, but if the increase is from “very, very little” to “very little” it could be unnoticeable and only measurable by specific tests.
I am not expert enough on wireless setup and related increased security practices to give you valid advice.
The NAS seems to me a non-problem, it is a network connected device like any other, what kind of doubts/needs do you have?
Separating it from some other devices?
In this case you might want to implement VLANs, which Is a rather advanced topic.
The default firewall rules are usually considered “good enough”, though they can be made “stricter” following the principle of only explicitly allow what is wanted/needed and then, as last rule of each chain put a “drop all else” one.
Before attempting to modify the default rules one must understand why they are there and what they do, as it is easy to drill big holes in the firewall accidentally (Chesterton’s fence principle) or - still accidentally - make everything so “strict” that for one reason or the other you end up being locked out of the router.
Check this recent thread that revolves around these themes:
http://forum.mikrotik.com/t/router-settings-from-beginners/177370/1
Thank you for everything
You are welcome .
In the meantime there is yet another very interesting ongoing thread about firewall,and its rules, JFYI:
http://forum.mikrotik.com/t/whats-the-point-of-this-default-fw-rule/179262/1
Hey!
After one month test I found one problem which is not a big deal, just to let you know.
If I restart the LTE modem (ISP ON for the whole time) and LTE modem boots up, routing changes to LTE and and do not want to change to ISP modem anymore, only if I make some changes in netwatch to make the netwatch refresh. (Netwatch shows UP but do not disable LTE modem, just after I force netwatch to refresh)
Just keep in mind that Netwatch might be pretty unreliable on LTE when using Carrier Aggregation (CA), which is the default mode for most connections.
Yeah jaclaz that was way simpler than just using two vlans. ;-PP
Rethinking about it, I think this is “normal” (and had to be expected) due to the dynamic nature of the setup.
It should be tested/checked, but this is what I believe it is happening:
-
Since the route to the LTE is created by the LTE modem DHCP, when you turn it off it simply disappears (poof).
-
The connection continues to go happily through the ISP, Netwatch checks continuously that one, and finds that nothing has changed, so it does nothing.
-
Then you reconnect/switch on the LTE modem, and its DHCP adds a new dynamic route (which is active, with lower distance - 1 - and since the ISP one has 2, the LTE one is used).
-
But netwatch still can ping happily 8.8.4.4 through the ISP one, so it does not do anything.
It would probably make sense to add a static route through 10.0.0.1 and disable the dynamic one (add-default-route=no).
Or use a scheduler script to periodically check that if 8.8.4.4 is reachable through the ISP, then the LTE route (if existing) should be disabled (but this would cause a double switch from ISP to LTE and then from LTE to ISP when you switch off and then on the LTE).
It have been watching this thread with interest. Thanks to the OP and jaclaz for bringing it up.
I have one question though. Is it better for the LTE device to be in passthrough mode so as not to duplicate settings between devices, merely passing the LTE signal only to the router and let the router take care of the firewall, dhcp settings, etc by itself?
Cheers.
@KiwiBloke
I believe the question is up for debate, and possible answers to it can be different depending on use and hardware involved, some devices (ISP router and LTE modem router) might have very good built-in firewalls or may have not any.
If we take for granted that the Mikrotik firewall is the best of the best (if properly configured) the more “direct” and “dumb” are the devices upstream connecting to the internet (bridge mode, etc.) the better.
This will BTW avoid the double NAT issue, provided it is an issue in practice. (In theory it slows down the connection, in practice it depends, if the change is from very, very, very fast to only very, very fast, it is not an issue).
But as we have seen in bbmikrotik’s case, his ISP router set in bridge mode is actually giving half the speed it does in router mode, so you never really know.
And also on personal preferences.
Only as an example (and this shows how old I am) I personally like to have as much as possible things “static”, I mean you have a “central router” hooked to a ISP modem/router and to a LTE modem/router, all three devices are bolted/screwed to the wall, can you imagine anything more “static” than that?
Every time the ISP router boots, the Mikrotik has to ask a DHCP IP address (and a route) to it (and every time it gets the SAME IP address and route), every time the LTE modem/router boots the Mikrotik has to ask a DHCP IP address (and a route) to it (and every time it gets the SAME IP address and route). It looks a bit vain (besides repetitive), and periodically the DHCP leases end and the questions/answers are re-asked and re-answered.
But on the other hand another scenario would be if you have a LTE portable modem/router that you normally bring with you when you travel but that you hook to the Mikrotik when you are at home (for failover use), it is much more convenient to have it fully configured so that it works when used “standalone” when traveling and when connected to the home router without needing to change anything in its configuration (except maybe turning the wi-fi radio on/off).
The onus of the ISP provider is to ensure the difference of their device in bridged mode or Router mode is crystal clear.
Using doublenat, should have no significant slow down in traffic so there is no downside other than a slightly more complicated setup on the MT and possibly the need to forward ports on the ISP device acting as a router. Most here would rather terminate the public IP on the MT device but both ways work almost equally well.
Thanks for the reply guys. So, what if the LTE device is a Mikrotik along with a Mikrotik router? Would passthrough then be the best option? It would make sense to me to have only the main router do all the heavy lifting, as it were.
I will try to set static IP addresses for ISP modem and LTE modem, but I struggling with it a little bit, can you please tell me how can I set them up properly? If I set up only in the ISP or LTE modem they disappear from mikrotiks eyes, I would appreciate the help!
Of course It depends, but usually DHCP servers (in any device) are pre- configured to not cover the whole address space of the network.
As an example, Mikrotik devices are usually configured (defconf) for addresses 192.168.88.10 to 192.168.88.254, while keeping for itself 192.168.88.1. (the wireless wire set Is usually set as192.168.88.2 and 192.168.88.3).
So you have 192.168.88.4 up to 192.168.88.9 that will never be served to any client and won’t also be used by default by any device.
Your ISP modem/router could be (say) set to give addresses in the 192.168.1.100 to 192.168.1.250, keeping for itself the 192.168.1.1.
So you need to add to the Mikrotik ether1 a static address, let’s say 192.168.1.42, that Is not in the DHCP range but Is still in the 192.168.1.0/24 network.
Rinse and repeat for the LTE router.
Then you disable the DHCP client on ether1 and ether2.
If the router DHCP server leases instead from 192.168.1.2 up to 192.168.1.254, you need to reduce the range to carve out at least one address for the Mikrotik.
This approach has the advantage that you can anytime just connect a laptop to one of the unused ports of the ISP router to configure it (or connect a device bypassing the Mikrotik).
Otherwise, you can disable the DHCP server on the ISP router, set a /30 network, so that there are only two addresses available, one for the ISP router itself and one for the ether1 of the Mikrotik.
So, I made it work!
What I did just for confirmation:
IP address of Mikrotik from ISP: 192.168.0.11
IP address of Mikrotik from LTE: 10.0.0.100
IP address of ISP: 192.168.0.1
IP address of LTE: 10.0.0.1
- I kept DHCP setting in LTE and ISP modem settings just reduced the DHCP range from 10.0.0.100-10.0.0.100 for LTE and 192.168.0.11-192.168.0.12 for ISP (In ISP and LTE modem settings)
In Mikrotik router winbox: disabled both DHCP client and in IP-Address in Address line I set up:
for ISP modem: 192.168.0.11/24
for LTE modem: 10.0.0.100/24
Then I went to routes-add: set up Gateway:
ISP: 192.168.0.1
LTE: 10.0.0.1
Then went to DNS settings and Set up 8.8.8.8 and 8.8.4.4
And now it seems everything works fine, even the failover setup now works properly, if I restart LTE modem keeps the ISP connected! I will test today if it is stable!