Good morning forum,
my company manages around 50 mikrotik devices for a customer, each one in a different remote site.
Each of these devices is a “all-in-one” Router/Firewall:
Wan links are directly connected to these Mikrotiks (95% of cases the Wans are made with PPPoE interfaces)
Mikrotiks are the default gateway of multiple internal Vlans
There are L2TP-IPSec tunnels towards HQ
OSPF is running
Configuration changes are made on daily bases (NAT/Firewall and Mangle)
After some hardware failures due to storms/electrical problems in remote branches (more than 3h of car from the HQ), the customer asked us to write a procedure/strategy that could be universal in this environment, with the purpose to replace the unit, in case of failure, as fast as possible (1 - 2 h would be perfect)
I have the possibility to keep a spare unit in the same rack, or even totally unplugged from the network.
A local employee could move the cables between units if necessary.
Note: in the past I used VRRP but it’s not suitable for two reasons:
It’s necessary to make a double configuration change (One on main and one on spare unit). Configuration changes are very common, so also the possibility to make errors.
VRRP and PPPoE is not suggested due to VRRP mac addresses standards.
Yes it’s possible to have different mac addresses, but it’s not suitable to manage it on a such medium/large environment.
How can I achieve this?
Do you have any suggestion?
It is not easy.
For most solutions you would need to make double configuration change anyway…
When you don’t, you will not be able to plug the new router into the network and have it work immediately.
In our network I have a spare router (the previuous one used for production) and I regularly export the current router config, run it through a Perl script I wrote to overcome the differences between the devices (the fallback router is a CCR1009 and the current one is a CCR2004), and do a reset configuration and import on the spare router to align its configuration.
That is only for one router so I can do it manually after significant config changes are made. In your case you would need to automate it.
The spare router is only plugged in on the LAN and gets a different IP address (this is one of the changes my Perl script makes).
I have a RouterOS script on each router that changes everything required to get into production status.
In the case of storms, everything will always get burned, keeping two devices synchronized is idiotic (this prevent other type of failures).
If you cannot protect the electrical system upstream with a double electrical insulator and a permanent generator
(always from the batteries, never from the line, the protected line recharges the battery), and have only optic data lines to the core, you fail at the start…
So the “best thing to do” if you can’t protect the power and data lines is to leave a duplicate of all the machines unplugged, and move the cables in place.
Then just reload latest backups on identical machines, not caring about the MACs they replace on the switched off/faulty machines.
Concur, if the buildings electrical is so flaky ( no business backups ) and no local UPS and everything gets fried…
Move to another location, the problem isnt MT
I infer from your description that there are not many locations outside a 1-2 hr window, so the two principal suggestions (have an unplugged spare with a locally tested procedure to replace, or harden the power system) may be within scope on cost and complexity. Hardening power may cost more but delivers greater benefits in uptime and avoided risks in executing a procedure.
Another possibility is faster transport. If events are relatively rare, quite a high delivery cost might be borne within the ceiling of the above solutions.
Thanks to everyone,
I recieved more details (which I didn’t know): In the most of the sites, everything in the rack is behind professional UPS.
Actually the last outages happened to old Mikrotik routerboards (older than 7 years).
The failures happened after storms but because there was a power failure, the UPS was overloaded and the Mikrotiks rebooted.
The switches didn’t have any problem, then I suppose a problem related to the board or memory failure after the power loss.
The point is avoiding a SPOF and being as fast as possible in the substitution.
Let me say, in case of storms, or improper current return: yes, both device will burn together with the network switches, but there will be more critical damages on the industrial production devices, probably the customer should stop the production with huge damage in business. In that case, Network infrastructure outage would be a small problem compared to the whole situation.
Based on these last details, could you please tell me if there is any flaw in this hypothetical procedure:
Both Mikrotiks (Primary and spare) are turned on.
All the cables are connected to the Primary. Spare unit has only a lan cable, in this way I can give a secondary IP to it and monitor it via ICMP or SNMP. In case of spare unit failure before the primary, the monitor server will send me a notification and I can organize the replacement
Configuration synchronization : I am not very into Mikrotik scripts. It would be perfect using the backup and restore functions inside a script, but I am aware that I have to manage some exceptions: for example, mac addresses are inherited, I would lose the secondary IP address used to manage and monitoring the spare unit. I don’t even know if it’s better pushing the configuration from Primary to slave or vice versa.
How would you handle it?
Thanks to everyone for the precious cooperation!
It’s not that easy, especially without the basic details. (For example: Is there a modem first? How do you know if the modem or MikroTik device broke, etc.)
I have been using MikroTik since 2007 and there has never been a fault that did not involve external factors.
The peripherals themselves have never broken (except the storm of green capacitors in the CPE with the RB411, just replace the capacitors…).
Also consider that in production all peripherals are regularly maintained(**) and cycled every two years for nodes and business customers,
instead the CPEs are replaced at every technology change (from a little still to “ac”).
(**) replaced with an already checked one, dismantled, cleaned, checked, reassembled, netinstalled, configured, replaced another one with this, etc…
You CANNOT use backup and restore to transfer a configuration from one MikroTik device to another!!!
You may be able to get away with it in some cases, like between two routers FROM THE SAME PRODUCTION BATCH and even in that case you cannot connect both of them to the same LAN.
What I sketched above is possible. Transfer config via export/import. But it still requires some manual massaging of the export file before it is imported.
In your case my first recommendation would be to replace the routers and configure the new routers manually.
At the very least, replace the plug power supplies. Those are probably the cause of current problem (router not coming back after power outage).
Sure I did that!
On our more complex installations, I am using a bridge for every “functional” connection, which contains only a single port as a member and has fast-forward enabled. Like one for LAN, one or more for fiber internet connections, etc.
All configuration is done on those bridges, and this makes it easy to import the config on another router and then put a suitable port in that bridge.
It looks like such a bridge has very little impact on performance, it does not maintain a host table and just forwards everything to the other side (CPU/port).
This avoids the problem of the port name occurring in many places in the config.
It would be possible to use interface lists for approximately the same purpose, but you cannot use it as the entire solution as some config cannot be applied to an interface list.