What changed with SSH on 6.49?

RouterOS General
1

I’ve just discovered a problem with some MikroTik devices where the config poller (Solarwinds NCM in this case) is having difficulty logging in and running commands
It can successfully connect, however its not recognising commands being echoed back properly

I have narrowed this down to version 6.49 (and all versions up to 6.49.15)
If I downgrade to 6.48.x or upgrade to 7.xx it works fine

One solution is to downgrade to 6.48.7 due to 7 breaking certain things on some devices (scripts especially) but its a manual process
But I would like to potentially fix this so I don’t have to touch production equipment that is running 6.49+
I can’t find anything in the changelogs mentioning what might have changed. Any assistance?

2

How is your ssh config of your Mikrotik and how is that client software configured regarding its ssh-client?

3

It’s not a config issue, its something inherent to the 6.49.x firmware
If I take a problematic router and downgrade it to 6.48.x or upgrade it to 7.x with no changes to the config at all, it works fine

Every single device regardless of what it is or what config is in place it does not work with 6.49.x
And same result as above, upgrade/downgrade fixes it

As for the Solarwinds client side of things, the short answer is I don’t know, it’s their own client. It seems I have quite limited ability to change parameters and they are higher up the chain. Things like RegEx settings for prompt detection. This is all fine

4

I remember having an issue with rancid (https://github.com/haussli/rancid) ssh a few years back. I found this in the forums:

http://forum.mikrotik.com/t/has-mikrotik-changed-something-with-ssh-output/153376/1

I’ve since been using oxidized (https://github.com/ytti/oxidized) to backup routers and it doesn’t have an issue.

5

So start by posting the config to find out if you got a malfunction there or some default hats been changed.

Regarding taking backup of the running-config I would prefer to do it as this instead:

  1. Connect to the device and run: /export terse show-sensitive file=flash/customr.rsc

  2. Download the file using scp on the host: scp username@192.0.2.1:flash/custom.rsc ./custom_192.0.2.1.rsc

6

Much easier if you combine those two steps into one:


ssh username@router-address /export terse show-sensitive > backup.rsc

Bonus 1: You don’t add any wear to the device’s flash.

Bonus 2: It’s immune to the terminal I/O strangeness that the OP’s send/expect method is tripping over.

Bonus 3: No race condition if there are other hosts trying to back routers up in parallel.

Guess how my backup script does it? Yep.

7

I assure you posting the config doesn’t help. I can take any router and wipe its config, do the absolute bare minimum which is just give it an IP address, and it’ll fail on 6.49.x, yet succeed on any other version




lol its my own thread. So 6.49 has been out that long and the problem must have been there the entire time, i’ve just not specifically checked the log files for routers running that version (might have only had 1 or 2 back then, and I typically only update when there is good reason to do so)
Thing is this is a different problem. That original thread was about garbage being prepended into the config, which can be filtered out in other ways with a custom Device Template
This issue is not that. The issue is it failing to recognize the prompt and/or commands being echoed back (or something along those lines)
I have this topic open with MikroTik support and provided all the log files, am having they can identify it and push a 6.49.16 hotfix
The issue is with SSH communication, something has inherently changed in 6.49 onwards, possibly an update to the SSH server they are running. But does not affect the 7.x branch