Device-Mode Enabled on Majority of Newly Received Routers – Causing Major Deployment Delays

RouterOS General
1

I’ve posted about this before, but it needs to be done again. Over the last several months I’ve been running into an increasingly serious problem with newly purchased MikroTik routers, and today was the worst example yet. Other people have complained with no response, time, and time, and time again…

This week I received a shipment of 300 routers, and approximately 60% of them arrived with device-mode enabled straight from the factory. That means about 180 routers cannot be configured or deployed out of the box until device-mode is manually disabled. I also have another 360 devices currently en route, and if the pattern holds that will be another ~220 routers requiring additional handling before they can even accept a basic configuration, Not to mention another 200 routers currently on back order for when the next container arrives. That equates to up to 560 routers that I’m expecting to deal with disabling device mode between now and year end, which means approximately 14 hours of extra work!!!

For integrators and ISPs doing large-scale provisioning, this is a significant operational issue. Device-mode adds multiple extra steps, introduces avoidable delays, and increases the configuration time per router by 2–3x compared to units manufactured as standard US versions without device-mode restrictions. Multiply that overhead across hundreds of devices and the impact becomes substantial. In fact the concept of Device-Mode is most attractive to ISPs and people that do large scale provisioning, yet it’s design absolutely takes none of their needs into account, and abjectly fails to deliver any meaningful usefulness to them.

To be clear:
This is not a reseller configuration problem or a shipping issue. These devices are arriving from the factory already locked down, often with a US label applied over an international model or handwritten “US” markings. These cases (20 devices to a case) consistently require roughly 30 minutes of extra work to disable device-mode before they can be brought into our automation pipeline.

I do have a process to work though the device-mode settings, but with hundreds of units arriving at once, it adds hours of additional time to deal with it, time that I shouldn’t be losing.

I am posting here in hopes that someone at MikroTik can escalate this internally. If the manufacturing or labeling process is resulting in units being converted to US versions after production—rather than being built as US versions from the start—it would explain why so many are arriving with device-mode enabled. But the downstream effect on integrators is real and increasingly problematic.

I (and I’m sure many others) would strongly prefer not to receive any more of these rebadged INTL→US units with device-mode locked on. They add unnecessary manual workload, slow down deployments, and make large-scale provisioning far more time-consuming than it needs to be.

If MikroTik can address this at the manufacturing or distribution level, it would make a meaningful difference for anyone deploying routers in volume.

For context, I have many thousands of routers I manage that are deployed coast to coast across the US in dozens of different states, as an ISP they are located inside customer’s homes, even if it were possible to get to the location they are installed at, walking in the door to push a reset button to change a device-mode setting would constitute a crime (breaking and entering) and send me to jail.

There ABSOLUTLY, UNEVQUIVOCALLY, UNDEBATABLY, MUST be a change from MikroTik with regard to device mode handling, and without exception you need to stop shipping devices with it pre-enabled to resellers and distributors.

I would LOVE to have a discussion and debate on the specifics with the community here, followed quickly with real, meaningful, changes.

2

It is useless to post that here, we (the community) already know it is a nuisance, and we cannot do anything about it.

About the only new fact you bring up here is that there apparently are devices that are rebadged as US models, which probably means they were turned on at a distributor and the “secret locking” was applied that makes them operate within US regulations only. And maybe that changes the behavior on the initial powerup? Do you use “flashfig” and observe issues with that when you deploy the rebadged devices?

3

I’ve confirmed several times that these re-labeled cases are arriving from MikroTik directly this way, it’s not being done by the distributor.

So far 100% of the cases I’ve encountered like this were initially Intl labeled, some had a small US sticker applied over the Intl text, others were quite literally sharpie crossed out as Intl and marked US.

These have all been device-mode locked to Home or Basic, whatever that default is called, and one of the restrictions is that you cannot use fetch. My deployment system doesn’t use flashfig, because I found it was unreliable, and only worked on some devices sometimes (out of a dozen devices I tried it on, only a couple actually worked). Instead what we found works 99.99% of the time is the API, we have a system that logs into the router, then uses fetch to download the necessary files to configure it, and then reboots when done. The only time that ever failed was the rare circumstance where a router was received without any default config at all, so it wasn’t reachable via the API. Those received a quick reset and then we were good to go. The introduction of default passwords was a little tricky to figure out, but since the distributor is able to get us those passwords in a CSV format, our system is able to easily lookup the default password for login via the API

4

Of there has been. It was probably the loudest issue in the "testing" threads that I recall. And gotten worse, not better. I was just burned by it a few months ago since I forgot some units were from newer batch post-device mode.

But many, many solutions were proposed - all rejected. @rextended has suggested setting /system/device-mode/update duration=1d at end of netinstall script, and then unplugging units after a netinstall. Others have proposed using microcontroller/"robotic" solutions to press the button. So it's a sad state of affairs.

So totally agree between the password and device-mode... they've really f*¢k up a lot of use cases where "home" units are used professionally. Historically they were great at this...now it basically a very specific, carefully considered, and potentially changing workflow to mange deployment of CPEs.

5

Unfortunately the duration=1d doesn’t help me any because I can’t actually load my config until AFTER device-mode is turned off.

BTW, I’m dealing with the power off process using some very creative thinking with a CRS328-24P-4S+ and PoE power cycle commands. The trick is figuring out which port to power cycle and when…. I’ll give one hint, the distributor supplied csv data for the default passwords also provides the key needed to accomplish this step.

6

Yup, if you a "branding kit strategy" then @rextended approach won't work. And which I've proposed many times that the device-mode should be settable by branding kit.

You're kinda the poster child of the problem. I recall you're using MikroTik as CPE as part of a managed service offering. And, already changed your workflow because of the passwords. And while apparently late to the party on device-mode, now left changing your workflow again to adapt this change. But welcome to the party anyway.

You/mine/others use case for "home" units as CPEs used be a MikroTik strength... but they now focus on protecting against end-users using their products incorrectly, really messing up the flexibility that was once offered for professionals..

7

You are correct, I do use branding maker to load the configs and defaults. It works exceptionally well, except for dealing with Device-Mode. And also correct that they are deployed as part of a managed ISP service type offering for apartment complexes and other multi-tenant environments.

I’m late to the party because I stayed on 7.12.2 until only a short while ago when I started getting equipment that had minimum versions of 7.15+. I’ve got so many deployed now that I had to setup a throttling system to limit updates to 125 devices per hour, partially to not overwhelm MikroTik’s own update content distribution servers, partially to limit the number of potential complaints and support calls that could be triggered in a given day. I am fairly certain that some corrupt package files were downloaded as a result of commanding several thousand devices to download routeros-7.19.6-arm.npk near simultaneously resulting in ~20 or so devices bricking on the update. Now I’m spreading updates out over a ~2 week period instead of all in one evening.

8

There are ~100 threads containing device-mode:
https://forum.mikrotik.com/search?q=device-mode%20order%3Aviews

So I don't think it an awareness issue... perhaps you can have your distributor put pressure on MikroTik is about all I got, and presume you've already tried that.

Or, yet again, adapt your provisioning scheme to use netinstall and PoE to cause reboot, in a multi-step process. But this does get complex and specialized. And certainly add/waste time... Given we used be able to have plug-in, copy config/branding, and reboot using "admin"/no-password (and/or open Wi-Fi) to setup new routers.

So really all I got is sympathy.

9

Out of curiosity - what function in these devices you need that are blocked in device-mode?

10

fetch being the biggest one. We have a fully custom command and control API that’s built around fetch calls, and is entirely CPE side script driven (aka the device checks into the control system, so it works even when the device is behind NAT), we push config changes and firmware updates to devices this way.

11

Just posting a follow-up in case anyone reading this in the future is curious. I shipped >1000 AC2 routers last month, over 600 of them arrived with device-mode set to home. My automation workflow detects this at the beginning and sets it to advanced with a 10m timeout, and then power cycles the PoE port on the switch that the device is plugged into, then after the router is back up it proceeds with the config loading. After a lot of process optimization and doing cases of 20 routers at a time, I’ve gotten it down to an average of 2 minutes 49 seconds per router (for the AC2 model). The process before device-mode is right about 2 minutes 15 seconds per router, so I’ve been able to get it down to roughly 11 minutes of additional time added per case when dealing with Device-Mode on from the factory. That added up to an additional 9 hours of labor I paid for in December just to turn off device mode in bulk.

For the record, we do not use netinstall at all, it’s 100% controlled through the API and branding maker loads all configs. Netinstall is way too slow.

12

If I may ask:
Aren't you installing a CERTAIN version of RouterOS?
Are you happy with the one it came with?

If you upgrade later, it's better to do a netinstall to align the versions with the ones you've tested and found to be working, rather than (auto)upgrade/update and hoping everything works fine on the times...

Also because in v7 the commands often change between two revisions of the same RouterOS version...

13

Perhaps the solution to your device-mode troubles might a better flashfig that handled device-mode, passwords, and full-automated applying a branding kit.

While I tend to agree with @rextended that I'd want to control the version before they go out & figured out other optimization around netinstall. But MikroTik offers flashfig to deal with your case. I take the point that a simple enough defconf is going to work on any version, so perhaps me being anal.... And, assuming branding add some auto-update scheduled script to do the upgrade when installed...you end up at latest version eventually.

EXCEPT*...and why we are here... flashfig would leave you screwed since you need the upgrade script to work & the device-mode block the relative innocuous schedule/fetch needed to do your version checks. Also, given there is no RouterOS auto-update setting you could use in a defconf... you're forced to use scheduler to check for updates, which is then blocked by device-mode.

FWIW, when I think of your use case... I always think of a scene in WarDogs with a timer while repackaging Albanian ammunition...

https://youtu.be/nozIkRy0v-M?t=61

14

If that question about versions was directed at me, absolutely yes we tightly control the RouterOS version and perform extensive testing before we chose to deploy each new version, and update our default configuration scripts every single build that we deploy. Every new router shipped runs the same version of RouterOS that every other device deployed in the field is running. We also update the default config scripts on equipment deployed in the field by downloading new branding packages to the router along with the new package files. We don’t ever use the built in update tool at all.

Routers I shipped ~8 years ago on v6.Twenty something are now running v7.19.6 with the same up to date default config script loaded as every other router in the field. If you factory reset that router, it’ll boot back up and redeploy itself for the next subscriber it’s configured for.

15

:rofl: :rofl: :rofl: :rofl: that scene is perfect.

If you got the impression that I wasn’t updating / controlling the RouterOS version at the same time as applying the configuration, then I didn’t explain the process well enough. In that time 2:15 or 2:49 with device-mode set, we open the case of 20 routers, unbox the individual routers and plug them into the PoE switch, our system looks up and finds the default credentials from the list our distributor supplies and logs into the router, checks for device-mode setting, if found changes mode to advanced and power cycles the PoE port, otherwise determines the current RouterOS version, if it’s less than the current version we support (7.19.6 at the time of writing this) it downloads the correct package file for 7.19.6, including correct wireless or wifiqcom package, downloads the current branding package, and then commands a system/reset-configuration to wipe any existing config and reboot. Upon reboot the router installs the new RouterOS packages, Installs the branding package with our default config script and other files, which auto-executes and applies our configuration. We then launch a time delayed reboot of the router. Once all 20 routers from the case have rebooted and checked into our backend system the dashboard that the installer is monitoring tells them everything is complete and they unplug the routers, slap a label on the box, and package everything back up. The time to do all of that for a full case of 20 routers averages out to 2 min 49s per device, or a little under 1 hour, if it has to wait for the device-mode reboot to unlock the config. If there is no device mode lock to disable then it’s right about 45 minutes.

Thousands upon thousands of devices deployed this way and I’ve never netinstall’d a single one (I’ve ran netinstall on plenty of devices over the years, just not for this purpose).

FYI just as an interesting fact, our latest default config script is 2,478 lines long, though much of that is the embedded scripts that it loads onto the routers.

edit to add: I forgot, in the event a device is shipped to us with a newer version than we currently support, the system will also automatically downgrade the version to the latest one we support, it won’t leave it running a newer version.

16

Just reading the forum shows that upgrading and downgrading often produces unexpected results.

It's better to do a netinstall: a little time "wasted", more invested in reliability...

17

That is nog guaranteed to work, because MikroTik sometimes change hardware without mentioning it, and modify RouterOS accordingly (there sometimes are “factory only” versions for that) and you cannot dowgrade past the version that already was on the device.

The minimal version of RouterOS and RouterBOOT are indicated in the System→Routerboard and System→Resources pages (“Factory version”)

18

In my experience I buy, for example hAP ac² that are coming with 7.19.x, and firmware 7.x
but upgrading to 6.49.18 with netinstall do not cause problems, and can be doed without hack.

19

I thought it was not possible to downgrade upgrade install a "before factory version"? :astonished_face:

20

@pe1chl has a point that the factory-version may limited downgrades via packages... I guess on hAPac2 they still use an older "factory-version" on them, even if shipping with V7 but dunno...

Now even device-mode with its install-all-versions=no would allow @rextended downgrade and @BrianHiggins 7.19.6, even using packages, as currently allowed-versions: 7.13+,6.49.8+.

But I can see install-all-versions=yes might be stickler in any future device-mode changes from MikroTik... The thing is that allowed-versions could changes in future... And while possible MikroTik might be more flexible with some device-modes for features like fetch/scheduler/etc. under certain conditions/keys/whatever. Installing older versions is whole different animal, that could remain restricted... e.g. the new MikroTik website does not have the download archive, only the current channels. So possible new devices could be set with a allowed-versions= more restricted.