On the reasons for Winbox not connecting to MAC address (a.k.a. "MAC telnet not working") from a Windows 10 host

RouterOS General
1

Hi,

I am writing this post in an attempt to gather useful information and shed some light on the issue mentioned in the subject. My apologies for the length, but I am trying to be as comprehensive as possible, including an investigation of the state of the art as well as a description of my own contributions (sorry, no solutions so far).

Problem Statement
On some Windows 10 hosts, connecting using Winbox to the MAC address of a MikroTik router (also known as MAC telnet) rarely works or does not work altogether, resulting in a

Could not connect to XX:XX:XX:XX:XX:XX

error. This happens regardless of whether the router is listed by Winbox among the neighbors (as it usually is) or not. This type of connection is called “MAC telnet” in the following of this post, referring to the underlying protocol used to talk to the router using its MAC address only and, yet, still assuming that Winbox is used as MAC telnet client.

State of the Art
Besides affecting me, this issue has been discussed multiple times in the forum without apparently coming to a true explanation and, consequently, a widely applicable solution:

terminal.exe

utility, but may be applicable to Winbox as well) may attempt to connect using the wrong host interface. The author resorts to porting a custom MAC telnet client to Windows, but that is unofficial and has likely been broken by (relatively) recent protocol changes

So far MikroTik staff has suggested the following tests and/or mitigating measures (I am filtering out those that propose access from another MikroTik router, as that would be a “less focused” workaround):

Finally, in a third-party forum, disabling certain protocol bindings (filter drivers) on the network interface used to establish the MAC telnet connection has also been suggested
https://www.reddit.com/r/mikrotik/comments/ect8ol/cant_access_routeros_via_mac_address/fbdxh1h?utm_source=share&utm_medium=web2x&context=3

My Own Observations
Software Setup
I am in an advantageous condition of experiencing the issue only on a specific Windows 10 host in my inventory. So far the problem for me is reproduceable with the following target routers:

My Winbox release on all hosts is 3.27 64 bit (latest at the time of writing).
I am pretty sure the same issue occurred with earlier RouterOS releases as well. Switching to an earlier or 32-bit Winbox release never helped either.

The software setup on my hosts is pretty much aligned in terms of installed tools (at least the most relevant ones for networking) and releases. In particular, I am using:

  • Windows 10 Pro 64 bit 20H2 build 19042.867, with Feature Experience Pack 120.2212.551.0
  • AVG AntiVirus Free edition, version 21.1.3164 (build 21.1.5968.643)
  • Oracle VM VirtualBox 6.1.18 r142142
  • OpenVPN 2.5.0 x86_64-w64-mingw32
  • Pulse Secure VPN client 9.1.10 (5655) (only on the host where MAC telnet is failing, but please read further before concluding that this is the cause)

Operating System and IP Addressing
First of all, the issue appears to be unique to Windows hosts. Despite Winbox not being available as a native application on Linux or macOS, my experience with MAC telnet when wrapping Winbox in Wine (or any of its derivatives like CrossOver or PlayOnMac) on any of these two operating systems is flawless: either Winbox to MAC address worked out of the box or little tweaks along an easy consequential path were required (e.g., adjustments of firewall policies).
Usage of Wine was also helpful because output from its internal libraries suggested that Winbox is indeed only able to “talk MAC telnet” on network interfaces that have an IP address assigned. While this was already known by observing that MAC telnet is transported by UDP, it is further confirmed by Wine emitting a message similar to the following for each interface that has an IP address assigned:

connecting to 127.255.255.255
connecting to 192.168.10.255
connecting to 172.17.255.255

This requirement is valid for any OS: also on Linux I had to assign an IP address to the interface facing the MikroTik router before being able to MAC telnet. On Windows this specifically means that the IPv4 protocol must be bound to the network interface of interest (I always had hard times working even with Ethernet frames alone if this binding was not enabled) and an IP address (preferably static) must be assigned to it.

Network Interfaces
It is widely enough known that having multiple network interfaces may disturb Winbox’s ability to send MAC telnet traffic on the correct one. Indeed, I have experienced this condition a few times, but I am pretty confident in demoting it as a cause for persistent MAC telnet failure for at least four reasons:

  1. My Windows 10 host where MAC telnet succeeds is a laptop with 21 (yes, twenty-one) network interfaces: 2 physical ones (RJ45 + Wi-Fi) and a mixture of OpenVPN, ZeroTier, Forticlient, VirtualBox, Bluetooth and WAN Miniport (L2TP, PPTP) virtual ones. During normal operation around 10 of them are enabled and working. Admittedly, sometimes Winbox “does not know where to look at” (it lists neighbors seen only on a subset of these interfaces) but, once a router is listed as neighbor, MAC telnet works straight.
    The Windows 10 host where MAC telnet fails is a desktop with 7 network interfaces: 2 physical ones (RJ45) and a few among VirtualBox, OpenVPN, WAN Miniport (L2TP, PPTP) virtual ones. MAC telnet always fails here.
  2. On the failing host, I have tried to disable all network interfaces except the one facing the MikroTik router (since it is the secondary NIC on my desktop, it has a direct connection to the router alone, with no switches in between): MAC telnet kept always failing.
  3. Using Process Monitor I could observe that Winbox enumerates all network interfaces at any time when it is important to know which NIC to use: namely, at least when IP neighbors are refreshed and when a MAC telnet connection is attempted. This is confirmed in the following screenshot:
    PM_interface_enumeration.png
    This means that Winbox does a good job at attempting to detect all applicable network interfaces and picking the correct one whenever needed.
  4. On the failing host, I have used Wireshark to inspect network traffic on the NIC where I expected to see MAC telnet packets, and that traffic was indeed there:
    MACtelnet_Wireshark.png
    So, the issue is not with Winbox picking the wrong network interface.

Network interface drivers have been suggested as a potential source of problems. As with all drivers, this is very reasonable. However, also in this case I have arguments against this possibility, specifically applying to my host where MAC telnet fails:

  1. The two physical NICs on this host are pretty different: one is an Intel I219-V built into the motherboard and the other is a PCI-E Realtek GBE. The Intel is directly connected to the hAP ac lite alone, whereas the Realtek is connected to the hEX via a switch. Both routers are detected as neighbors by Winbox but MAC telnet does not work to either one.
  2. Admittedly, drivers for the Intel NIC were outdated. Even when installing them afresh from an official Intel package, MAC telnet continued to fail.
  3. The Intel NIC has a number of tunable advanced parameters like hardware offloading, energy saving, speed/duplex settings, compatibility with legacy switches. I have tried to play with them a number of times, both from the OS hardware settings panel and from Intel’s native utility, searching for a more conservative combination of these parameters, but MAC telnet continued to fail.

Antivirus
This is pretty easy to handle. Of course, I am only considering the case of the host where MAC telnet fails.
Adding an exception to prevent AVG from “watching” winbox.exe did not improve the situation. MAC telnet continued to fail even after disabling the antivirus altogether. I am not willing to uninstall it.
As a general observation, the antivirus has never reported any detections related to winbox.exe even while it was active. In addition, it has always been active on the host where MAC telnet has always worked.

Firewall
Even easier, as I am only running the native Windows Defender Firewall service on any hosts. On the failing one I have added an exception for any protocols used by process winbox.exe and double checked that this was listed among the input traffic policies before any other policy denying the same traffic. MAC telnet still failed.
I have disabled the firewall altogether, but MAC telnet still failed.
Even though the above attempts make this observation irrelevant, the Realtek NIC wass marked as “Private” whereas the Intel NIC is “Unidentified”. Once again, MAC telnet does not work through either NIC. More interestingly, the firewall has always been active on the host where MAC telnet works.

VirtualBox
Related to the fact that too many network interfaces may confuse Winbox when MAC telnet is used, I have tried to first disable and then remove (from the VirtualBox GUI) the only VirtualBox network adapter I have on my host. MAC telnet still failed. On the other hand, there is no need to remove any virtual adapters on the host where MAC telnet has always been working.
Here comes a very interesting observation. If I restore the VirtualBox network adapter (it is unused, but just to set things to their original state) and launch Winbox from a VirtualBox virtual machine with a network interface bridged to the physical Intel NIC, then MAC telnet works flawlessly! This definitely settles the scope of the issue to the software domain.

Additional Tests
For the sake of completeness, the following attempts did not help either:

  • Enabling Legacy mode in Winbox
  • Running Winbox as administrator
  • Resetting (one of the) MikroTik routers to factory defaults (indeed, the hAP ac lite - connected to the Intel NIC and successfully accessed from a VirtualBox guest - is already factory reset)

So, I have tried to deepen the investigation a little bit using additional tools, and found the following:

  • MAC telnet uses UDP, which is a connection-less protocol, as its transport layer. Therefore, to establish a “telnet-like” session based on UDP, the connection-wise semantic must be reimplemented from scratch. For the case of MAC telnet this means that Winbox first emits a UDP packet from a random port number P targeting a broadcast IP address and the standard MAC telnet port (20561). Immediately after, Winbox listens on UDP port P for replies to this packets that confirm session establishment with the router. The fact that Winbox temporarily listens to this random port P is clearly visible by using utilities like Process Explorer or TCPView. The time for which Winbox listens on this port is quite short, but should be enough for the router to reply. In any case, multiple attempts are made (by Winbox to start the session and by the router to send acknowledgments) before giving up, so that Winbox always has enough chances to establish the session.
  • On my host where MAC telnet fails, Wireshark and Process Monitor confirm that UDP packets are indeed emitted and the router replies, sending its packets from
0.0.0.0

to

255.255.255.255

. So, it is not a network adapter problem: there must be some piece of software interfering with the ability of Winbox to process these response packets from the router.

  • Further investigation by using Process Monitor during a MAC telnet attempt revealed an important behavioral difference between the working host and the failing one. Soon after UDP packets are emitted by the working host, Winbox looks for standard system-wide cryptographic libraries in order to prepare session establishment:
    PM_working_host.png
    This behavior is not observed on the failing host, where “UDP Send” events are just followed by… nothing (possibly just some late “UDP Receives” which are not processed anyway).

(Temporary) Conclusion
All the above considered, I am prone to think that MAC telnet is simply failing because of either of the following reasons:

  • Winbox not attemping to find system-wide cryptographic libraries (they are not missing: it’s just that Winbox never reaches a phase when it needs to look for them)
  • Some software layer preventing Winbox from being able to process MAC telnet acknowledgments sent back by the router (which are actually received by the host, as confirmed by Wireshark)

Any further advancements or sound hints are of course more than welcome.

2

A few updates and a working solution (for me) follow.

Additional Observations
Pushed by a true intent to address this issue once and for all, I have performed a few slightly more intrusive tests on the Windows 10 host where MAC telnet (from Winbox) is not working (the test outcome is marked by a colored circle):

  • :red_circle: (temporarily) Uninstalling the AVG antivirus by using the standard uninstaller and, additionally, the AVG Clear tool, and rebooting afterwards. MAC telnet continued to fail.
  • :green_circle: Rebooting in “safe mode with networking”. This time MAC telnet worked immediately. Interestingly, the Windows Defender firewall was regularly reported as activated, thus confirming that this is not the disturbing factor.
  • :red_circle: Using Process Monitor again to monitor system-wide interactions with UDP sockets, in an attempt to detect the application that might be intercepting MAC telnet acknowledgments returned by MikroTik devices. Unfortunately, nothing useful was reported.
  • :red_circle: Selectively killing a number of boot-time and login-time automatically started applications. Even after terminating most of them, MAC telnet continued to fail.

Final Solution
The aforementioned observations induced me to conclude that the cause could be searched among services or drivers that are not loaded when Windows is executed in safe mode. Supported by the evidence that networking worked perfectly in that mode, I excluded the related (NIC) drivers as a cause, and focused on services instead.
So, after trying to selectively terminate a number of system services, MAC telnet suddenly started to work! After a few more tests I could focus on the sole and only service that indeed blocked operation of MAC telnet: it’s called AsusGameFirstService and is part of a network traffic monitor and prioritization utility bundled with my motherboard. As useful as it can be, as a far-from-habitual gamer I can definitely live without it.

MAC telnet is back! :grinning_face:

Lessons Learned
After this thorough investigation I can definitely confirm that there are a number of factors that might affect ability to access MikroTik devices by their MAC address using Winbox from a Windows 10 host. I try to summarize the most important ones below, in the form of a checklist for use by anyone else that might experience this issue. I am mentioning mostly Windows-side checks, in a suggested order of application:

  1. Software Updates: needless to say, always make sure that you are using the most up-to-date version, at least of Winbox.
  2. MAC Server Status: double check that the MikroTik router is ready to be accessed via MAC telnet/MAC Winbox. Default configuration allows this on most devices, but may restrict the router’s physical ports from which such connections are accepted: you may want to try plugging the cable into a router port numbered no lower than 2 (1 is usually meant for Internet access in the default configuration, thus MAC telnet/MAC Winbox may be disabled on it).
  3. Firewall: check if a proper exception is added to allow Winbox to listen for incoming UDP replies from MikroTik routers (there is no specific UDP port, as this is determined dynamically). Normally Windows asks you to do so when Winbox is first used for this kind of connection, so make sure to answer positively and that there are no other deny firewall rules applied to the Winbox process, which normally would take precedence. Also make sure that the firewall exception applies to the correct network zone (private/public/domain), or just apply it to all zones.
  4. Network interface IP address: make sure a static IP address (whatever it is), with a netmask shorter than
/30

(so that a broadcast address exists), is assigned to the NIC that is used to contact the router.
5. Network interface selection: in case multiple NICs exist, Winbox should already be able to pick the correct one. It may help to quickly disable and enable the NIC used to contact the router to refresh Winbox’s internal knowledge of which router is seen from which NIC. As a more drastic measure, try to keep all NICs disabled except the one that is used to contact the router.
6. Virtualization software & hypervisors, Java Virtual Machines, etc.: their presence may matter as long as these pieces of software create a virtual network adapter to accomplish their functons. In case some of them does, please refer to the previous suggestion.
7. Safe Mode: to rule out some hardly solvable issues (e.g., incompatibility with vendor-specific drivers), try to run Winbox while Windows is running in “safe mode with networking”. If Winbox still fails, then the problem may be more serious and not addressed by this forum thread.
8. System Software & Services: third-party software may interfere with Winbox’s ability to receive response MAC telnet packets from routers (which actually are UDP packets with a broadcast destination IP address). You are on your own here, as an undefined number of software tools may improperly catch these packets before they can be processed by Winbox. Please reason based on your usage experience (e.g., which network-related pieces of software you may have recently installed) or try to selectively disable system services as I have described above.
9. Antivirus: in my experience this has never been a real problem. In case you really want to rule this out, you can add an exception for the Winbox.exe process. Also check whether the antivirus software is bundled with an advanced firewall that enhances (or supersedes) the native Windows Defender Firewall (most free editions do not have one), in which case it may also need to be tuned.

:light_bulb: If nothing else works, there is a zero-day alternative: connecting to the IPv6 link-local address of the router (which, if you are lucky, should show up in the Neighbors list of Winbox). As in most cases the IPv6 stack should already be enabled in the default router configuration, this may work nearly as out-of-the-box as MAC telnet, and is also robust to loss of IPv4 connection to the router.