LHGs - repeatedly losing Winbox connection

LeaUK · Fri Nov 03, 2023 11:56 am

Hi all

New(ish) with Mikrotik and experimenting with several LHGs and a LHGG. The LHGs (latest Router OS and LTE firmware) randomly seem to lose connection to Winbox (ironically still ping and route to internet on the same client device), then don't even appear in the Neighbourhood scan. This can be resolved by resetting to defaults and re-config but I must be doing something wrong. It's like the firewall at some point decides to take a disliking to my local machine. No web console access either

Disabled Windows Firewall on the client just to see, no access
Forced a different client static IP, no access.

Any thoughts?

Cheers
Lea

Update:

All LHGs are second hand, so I held down reset before power up and waited until LEDs flashed then let go, now I've read there are differences between holding reset down before power up and after power up.

From the manual:

RouterBOARD reset button
RouterBOOT reset button has three functions:

Hold this button during boot time until the LED light starts flashing, release the button to reset the RouterOS configuration (total 5 seconds)
Keep holding for 5 more seconds, LED turns solid, release now to turn on CAPs mode (total 10 seconds)
Or Keep holding the button for 5 more seconds until LED turns off, then release it to make the RouterBOARD look for Netinstall servers

If you hold the button before applying power, backup RouterBOOT will be used in addition to all the above actions. To do the above actions without loading the backup loader, push the button right after applying power to the device.

So perhaps all three LHGs are in backup RouterBoot and this isn't normal operation mode and effects neibourhood discovery and Winbox connection after a period of time?

Update:

Reset Mikrotik again ensuring reset button held down after initial power up until LEDs blink, connect with Winbox fine, then I note when I disconnect the client ethernet lead (only have one windows client and one LHG on network, no other devices) and reconnect I lose Winbox and neighbourhood detection.

What on earth am I doing wrong?

LeaUK · Fri Nov 03, 2023 1:58 pm

OK, from testing I can confirm Winbox fails to connect directly after I set the APN and the WAN (LTE) retrieves an IP, WInbox continues to function UNTIL I try and reconnect. PIng is fine, Internet passthrough still functioning, BUT no Winbox or Neighbourhood. Only way to re-establish Winbox connection is holding down the rest and power up the LHG.

This is weird!

Router OS v 7.11.2
LTE6 version: R11e-LTE6_V036

HELP!

mkx · Fri Nov 03, 2023 3:06 pm

Two suggestions (not sure if any if them will help):

disable "Detect internet" ... winbox is only allowed through LAN interfaces and if "detect internet" somehow misdetects and "proclaims" LAN interface as WAN, then you loose connectivity. IMO "Detect internet" is in general troublesome and for most users not worth its money
manually set bridge MAC ... by default it assumes MAC address of first wired member interface. If that one gets removed from bridge, bridge MAC address changes and that can break connectivity. But be careful about address being used, it has to be unique in your LAN.

Another suggestion: do a netinstall on devices. In the past there were reports of weird behaviour which could not be explained by (visible) device configuration and netinstall fixed it. Also there are indications that it is possible to hide some malware inside ROS which can only be eradicated by netinstall-ing the device. Since your devices are second-hand, you can never be sure about their history.

LeaUK · Fri Nov 03, 2023 6:17 pm

Hi mkx

Thanks for your suggestions. I finally tracked it down (after 5 hours of going a bit crazy), it's the onboard Firewall! To reproduce, simply reset to defaults, connect via Winbox, disable firewall through Quick Set menu, try reconnecting through Winbox (should all be ok), now re-enable firewall, restart WInbox, it will not be able to establish a new connection and no Neighbourhood functionality.

Even being of limited experience, I'm surprised RouterOS appears so buggy in this 'stable' release, but I'm still sure it's me, it can't be as simple as a buggy FW can it?

It may also be my mind playing tricks, but I'm also sure at some point the FW disabled itself when adding APNs, so of course that would tie in to my original findings, linking the issue to APNs and LTE WAN IP, but no WAN IP is required to reproduce the issue.

So maybe it is indeed 'detect internet' causing havok in the FW, where do I disable this?

Ah ha, another thought, the IP received from the LTE is 10.xx.xx.xx.xx is the FW becoming confused and assuming this is local, or linking Winbox/Neighbourhood to this WAN IP? When enabled, 4 additional rules are added, one being a drop - input - !LAN How does it define LAN, by IP?

I took a look at the Interface list tab, it lists LAN BUT the interface was 'unknown' odd, the drop down has ether1 or LTE1, so have set ether1. Maybe this is the route cause (forgive the pun) - time to test and check across all LHGs I have.

Lea

Amm0 · Fri Nov 03, 2023 6:27 pm

In internet detect set it all the options to "none".

You mention "passthrough". If the upstream is a CGNAT (which the 10.x.x.x indicates), there isn't much value using passthrough IMO. Just normal L3 routing to the LHG would be fine since NAT is going to be applied by carrier anyway.

But it's hard to have any clue without some config and/or diagram of what the setup is.

LeaUK · Fri Nov 03, 2023 7:00 pm

Sorry, can't find Internet Detect in any menu

So it's nothing to do with the WAN IP as the failure to connect to Winbox can be reproduced by simply:

1. Reset LHG to defaults
2. Connect to Winbox OK
3. No WAN IP at this stage (0.0.0.0)
4. Disable Firewall in Quick set
5. Connect to Winbox OK
6. Enable firewall
7. Restart Winbox
8. Fail to connect, no neighbour discovery

I've noted the FW rules are not re-created equally between the initial default rules and when the FW button is disabled then re-enabled.

If I could export the rules in txt I'd post them, but unsure how this can be achieved (cant add images easily to this forum).

Moreover, the rules between the initial default config and when the FW is disabled and enabled are remarkably different. This is my next avenue of investigation.

Update:

Found how to export FW rules.

These are from default config:

# 2023-11-03 17:04:57 by RouterOS 7.11.2
# software id = 
#
# model = RBLHGR
# serial number = 
/ip firewall filter
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment=\
    "defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=drop chain=input comment="defconf: drop all not coming from LAN" \
    in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" \
    ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" \
    ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" \
    connection-state=established,related hw-offload=yes
add action=accept chain=forward comment=\
    "defconf: accept established,related, untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid
add action=drop chain=forward comment=\
    "defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat \
    connection-state=new in-interface-list=WAN
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" \
    ipsec-policy=out,none out-interface-list=WAN

And these after the FW feature is disabled, then re-enabled from Quick set:

# 2023-11-03 17:10:29 by RouterOS 7.11.2
# software id = 
#
# model = RBLHGR
# serial number = 
/ip firewall filter
add action=accept chain=forward comment="defconf: accept in ipsec policy" \
    ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" \
    ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" \
    connection-state=established,related hw-offload=yes
add action=accept chain=forward comment=\
    "defconf: accept established,related, untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid
add action=drop chain=forward comment=\
    "defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat \
    connection-state=new in-interface-list=WAN
add action=accept chain=input protocol=icmp
add action=accept chain=input connection-state=established
add action=accept chain=input connection-state=related
add action=drop chain=input in-interface-list=!LAN
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" \
    ipsec-policy=out,none out-interface-list=WAN

Amm0 · Fri Nov 03, 2023 7:22 pm

Ideally you'd only run QuickSet once – it NOT always safe to come back and make changes without breaking things. In particular, enabled/disabling firewall using QuickSet could very well leave you in an odd state.

The command in V7 to export a config is:

:export file=configforforum

And then in files, you can download "configforforum.rsc" to post. It QuickSet that I think is messing you up. You likely need to make changes "manual" in firewall and/or address-lists.

So I'd reset to defaults ("/system/reset-configuration keep-users=yes" at CLI). Once reset, in QuickSet, leave firewall etc enabled, and change on LAN IP, password, and/or APN as needed. And then NEVER run QuickSet again. If more changes are needed, it really best to do those "by hand". QuickSet does a job of laying out a firewall, so I'd leave it there – if you put all interface into LAN interface list, the firewall doesn't do much. And if you need the firewall, the config is still there.

Basically checking/un-checking the firewall box in QuickSet would strike me a very bad idea.

LeaUK · Fri Nov 03, 2023 7:34 pm

Thanks Amm0

Possibly posted at similar times as I added the chaos between FW rules.

Whilst I agree to a point, the menu is named Quick Set, not Quick Start (and don't ever use again) so how on earth would anyone know which options are ok to set in Quick Set and which are not, tbf the option to enable/disable the firewall should NOT be present if it causes complete failure of the most basic features and breaks even itself. I haven't added any custom rules which arguably one might expect to break, but it's own operation I would expect to function.

This experience has left rather a sour taste sadly, but we live and learn, it's all part of the 'fun'.

@Mikrotik - do not allow options in Quick Set to be toggled if they break everything, or at least provide warnings.

Amm0 · Fri Nov 03, 2023 7:47 pm

Well, I tend agree Mikrotik doesn't put enough into making QuickSet more useful for beginners. I'm not the QuickSet complaint department

. They do have a video on this topic:
"When not to use QuickSet in MikroTik devices": https://www.youtube.com/watch?v=hQgGzgPOFFY
It actually has a lot good info about this topic. TL;DW it okay if you ONLY QuickSet, but the minute you change something, you risk breaking stuff since it might conflict with QuickSet.

As you can tell the firewall use the "interface list". So if the interface that winbox is using is NOT in the LAN interface, you'd lose a Winbox connection, period. And, what I suspect is going here is QuickSet does muck with the interface list e.g. why you see "unknown" there – which is the problem. If an interface isn't LAN interface list, no winbox. Or if LTE isn't in WAN interface list, you wouldn't have internet via LTE.

Amm0 · Fri Nov 03, 2023 7:51 pm

You might want to look into RoMON, which another way to connect to router using winbox: https://help.mikrotik.com/docs/display/ROS/RoMON

They do have a video on that too: https://www.youtube.com/watch?v=Peg6UcSJ_eA

This would protect you again QuickSet messing stuff up since it doesn't change RoMON configuration AFAIK.

LeaUK · Fri Nov 03, 2023 7:53 pm

I do appreciate your wise words and help, nor am I the Quick Set police, but hey lesson learned, trouble is if I can't even use Quick set without getting into issues, I have limited chance of setting up a load balancer, 30 minute full on tutorials on YouTube, with so many ways of configuring!!! But not everyone can afford the luxury of a point and click Kemp.

In summary, DONT USE the Firewall toggle in Quick Set - solved!

Cheers again,

Amm0 · Fri Nov 03, 2023 8:06 pm

QuickSet can be useful. Keep in mind Mikrotik has a lot of different kind of customers. So in our case, we use it for endusers to have some "dashboard"

The big thing to understand is that there are "config profiles" (IDK the official name) in QuickSet. And, what "config profile" is determined based on items in the configuration – that profile isn't "saved". Further, how it updates the config DOES vary depending on what's selected in the top left corner. And this is really how things can go wrong when you modify a config, especially in the bridge.

For example, by adding/remove "ether1" to a bridge manualyl is likely to going to change the QuickSet profile it uses, since QuickSet now thinks you might a switch configuration screen in QuickSet, not a router configuration one. Thus, when you go back to QuickSet it will then apply whatever else based on the select profile – which may be different than when you first used QuickSet...

Anyway, the subtle little dropdown in the top left corner is actually pretty important.

Amm0 · Fri Nov 03, 2023 8:12 pm

I have limited chance of setting up a load balancer

Typically you want failover & load-balancing, but there is no QuickSet for this

If you have multiple WANs. You can follow the discussion here, there are some examples and links that might help:
viewtopic.php?t=192736

LeaUK · Sat Nov 04, 2023 10:58 am

Thanks Amm0 for getting a nuub up to speed, appreciated.

The LHG only has one drop down in the Quickset menu but I hear what you're saying. What's odd to me as a newcomer (although familiar with the likes of FWs and LBs such as Watchguard/Kemp/Azure and alike, all a bit point and shoot granted) is that the FW 'enable' toggle in Quickset doesn't actually simply enable/disable the FW, it actually re-writes the rules under the hood in order to achieve this, that's one of the things that's thrown me.

But I feel achievement in finding the root cause and of course tech is always a learning experience, typically the best when things don't function as expected as we can leverage deeper learning.

In terms of different customers, many companies understand this and provide templates/scripts for complex setups, for example I'd like to bond (aggregate with failover) multiple LTE/WANs, so instead of wading through multiple YouTube videos (including Mikrotik's) are there template/scripts to help configure such? This would be a valuable tool to increase efficiency and reach different markets. The complexity of RouterOS (whilst exceptionally powerful and inexpensive in $$$ (beyond basic router/bridge scenarios) will discourage users I'm sure, hence I tended to go with Kemp due to reliability, templates, web guides and support, even over the huge costs.

One thing that did strike me with multiple WANs thus multiple egress IPs, is how the end point reassembles packets from such a distribution, especially considering low latency MS Teams (video/audio) requirements. I understand the LB can be configured in several ways, one of which can ensure all packets from a client remain associated to the same WAN channel thus ensuring the same exit IP, but to me isn't aggregation per user, that's more like LB over multiple users.

Anyway I digress. Thanks for the link, I'll take a read.

Cheers
Lea

LeaUK · Thu Nov 16, 2023 4:26 pm

Resolved in RouterOS v7.12 viewtopic.php?p=1036592#p1036592

Kamlon · Fri Dec 15, 2023 6:26 am

Resolved in RouterOS v7.12 viewtopic.php?p=1036592#p1036592

I reviewed the firewall rules on the MikroTik LHG but am still having trouble maintaining a stable connection to Winbox and scanning the Neighborhood

LHGs - repeatedly losing Winbox connection

LHGs - repeatedly losing Winbox connection

Re: LHGs - repeatedly losing Winbox connection

Re: LHGs - repeatedly losing Winbox connection

Re: LHGs - repeatedly losing Winbox connection

Re: LHGs - repeatedly losing Winbox connection

Re: LHGs - repeatedly losing Winbox connection

Re: LHGs - repeatedly losing Winbox connection

Re: LHGs - repeatedly losing Winbox connection

Re: LHGs - repeatedly losing Winbox connection

Re: LHGs - repeatedly losing Winbox connection

Re: LHGs - repeatedly losing Winbox connection

Re: LHGs - repeatedly losing Winbox connection

Re: LHGs - repeatedly losing Winbox connection

Re: LHGs - repeatedly losing Winbox connection

Re: LHGs - repeatedly losing Winbox connection

Re: LHGs - repeatedly losing Winbox connection

Who is online