I was approached in mid-April of this year by a customer who was seeking to purchase preinstalled hEX devices in large quantities (~1500/yr).
The client had initially approached me, nine months prior, because they were looking to replace their current cpe brand and had included Mikrotik as a potential candidate to their evaluation process. I consequently loaned them a hEX and an RB4011iGS+RM.
The immediate hurdle for me, with regards to responding to their tender, was Flashfig’s limitation: up until then, Flashfig could only inject a “running” configuration to devices, and wasn’t able to replace the “default” configuration.
The other potential obstacle appeared in the form of the random pre-configured admin password that Mikrotik was beginning to apply to recently updated Routerboards.
Having conferred with Mikrotik support and others, I came to the conclusion that as a ‘one man band’ I wouldn’t be able to respond favourably to this client’s request; conclusion which I shared with my customer.
Two weeks later Druvis appeared on one of Mikrotik’s wonderfully presented Youtube videos explaining how Flashfig could be used to configure a large number of new (out-of-the-box) devices, allowing one to replace the default configuration in the process. Bingo !
With this new information in hand, I contacted my client hoping that they hadn’t yet finalised their decision. I was in luck : the shortlisting process was in its ultimate stage and I was afforded 24hrs to provide my own offer. As luck would have it, I won the tender.
Now I had to setup my installation workshop because the customer was expecting to receive the first (test) batch of fifty units within a fortnight. It’s one thing to know in your mind which bricks to use to obtain a result and another to actually put them together in order to produce the desired result.
The customer’s needs were three-fold: install a specific RouterOS version to each device, apply the client’s default configuration and finally label each device box with a sticker displaying a barcode of the device’s ether1 MAC address.
The second part could be accomplished using Flashfig while the RouterOS installation would require another tool altogether: Ansible. Let’s be honest though, I had never used this rich and powerful automation platform before.
A little Googling around however led me to Zach Biles’ well-structured Ansible playbook (“mikrotik_upgrade”) aimed at automating the distribution of RouterOS to Routerboards. Just what the doctor ordered ! I have to admit that it felt as if the stars were pretty well aligned. Now for the practical side of things; it was time to get my hands dirty.
My company occupies a room within our flat. In fact, grouped together within this room I have my stock, my work desk and my orders preparation table. Not your garden-variety ‘startup garage’ per se, but close enough and, as far as I’m concerned, a lot more comfortable.
So, I laid out fifty hEX devices upon our dining room table and hooked them up to a CRS354-48P-4S+2Q+RM.
Inspired by my childhood game of ‘battleships’ I devised an imaginary grid so as to assign each device its own specific position on the grid. Once the installation was over, I would box the devices and make sure that I returned them to their shipping boxes in the same grid position. When time came to apply the barcode labels, I would simply follow the order instated by the grid.
Let me state in passing that I wasn’t very proud of my initial approach concerning the barcode requirement. I thought that this would simply require using the proper barcode font. I finally ended up using ‘TBarCode Office’ created by Tec-it.com, which allowed me, in a cinch, to use Word’s merge function to produce the barcode labels. Moving on.
My first order of business was to test Flashfig. I feel the need to mention that I often have trouble controlling my impatience and the ‘eager beaver’ I am likes to dig into a task so as to get things moving along without further delay.
The one situation however that I wanted to avoid was having to manually reset each device in case Flashfig didn’t work as expected. Therefore, I tested my planned approach only on a handful of devices.
Regardless of this forethought I ended up at different stages having to manually reset the devices as well as having to re-enable the Flashboot setting on each individual device. ‘Live and learn’ as the saying goes, or as I’ll admit to in my case : live, learn and then potentially repeat silly mistakes.
My first unexpected realization was that the CRS354 powers up its ports before RouterOS is actually fully up and running. What this implies is that the connected devices would power up before Flashfig, which was running on my laptop, was able to contact them. Darn !
I therefore had to initially disable the power output to the ports to ensure that Flashfig would be visible and hence be available to flash the devices once I had re-enabled the power to the ports from the CLI : :for x from 0 to 47 do={/interface ethernet poe set $x poe-out=auto-on}.
The next part of this endeavour was to update RouterOS on the devices using Ansible. Having viewed a bunch of Youtube based tutorials I began to play around with Zach’s playbook. Upgrading then downgrading a couple of devices to get a feel, not only for the playbook, but also for Ansible itself.
Obviously I wasn’t expecting to become an Ansible expert within the allotted time but I still needed to get familiar with the way this platform worked. I very quickly noticed that some issues were going to require my attention.
Since Ansible communicates with the Routerboards via SSH I sorted the authentication requirement by adding the following line to my inventory file : ansible_ssh_common_args=‘-o StrictHostKeyChecking=no’ It was now time to tackle the reboot issue.
Even though Zach’s playbook included the required reboots following the RouterOS and Routerboot upgrades (/downgrades). The unfortunate issue was that the script would end in error following the execution of the reboot command; because it couldn’t communicate with the device anymore (‘duh !’). The simplest solution here was to ignore the error : ignore_errors: true, allowing the script to run its course.
I also had to adjust the ‘Push’ task used to handle the RouterOS installation. One of the reboot conditions that checked whether the file had been successfully copied to the router wasn’t working. I therefore added a wee task of my own to check for the presence and the version of the copied file, as a replacement to the original condition.
I also added a command in the Firmware update task so as to re-enable the Flashboot setting : commands: /system routerboard settings set boot-device=flash-boot-once-then-nand
I then tinkered around a little with the playbook’s timeout values in an attempt to accelerate certain phases and I added a: meta: end_play, so as to skip the final validation phase, which I handled manually.
The time had come to string all the processes together into an executable procedure which I wanted to test and document. The first phase required removing each device’s running configuration and the setting of a common password. This would allow Ansible to access the devices via ‘ether1’ and provide admin with a ‘required’ password. The task was easily managed using Flashfig and a two-line script.
I then executed a RouterOS script to register the devices’ MAC addresses and their corresponding connection ports. To barcode a device’s box with its ether1 MAC address, I needed to collect the port’s address and the CRS354 interface to which it was connected; interface number that corresponded to the device’s position within the grid.
Next I executed Zach’s playbook. The fifty units were upgraded in approximately forty minutes. Sometimes I had to execute the gather_facts task twice in a row. I then activated the client’s configuration file in Flashfig and power cycled the devices so as to apply it: :for x from 0 to 47 do={/interface ethernet poe power-cycle $x}. What a sweet sound it is to hear fifty devices tweeting away together in harmony.
I then disconnected the devices one by one, which allowed me to verify whether they had been successfully flashed and to ensure that none would accidently be rebooted. The client’s configuration was programmed to execute an initial startup sequence allowing it to adapt the device’s configuration to the type of Internet access available to the router.
Once boxed I returned the devices to their shipping boxes, in the proper order, and simply applied their corresponding barcode labels. Voilà ! Finito !
