I've got a ROS 4.5 RB750 that stopped responding to API login attempts overnight. Winbox responded fine, and rebooting the router resolved the issue. Since the rb only logged to memory, that was cleared when it was rebooted so I don't have any errors logged by the router available for troubleshooting. However, from the API library logs, the login was being sent, the process tried reading the initial len encoding from the socket, and timed out.

Yes, I do know 4.5 is somewhat old but it hasn't been upgraded due to the API bug in newer versions where an API user does not get logged out and the router stops responding to logins entirely until being rebooted, as I really don't want to be doing scheduled reboots of core routing infrastructure.

Is this simply another manifestation of this bug? Has anyone else seen something like this when using the API?

It is responding with something! Any idea what character is being read?

2011/02/23 00:50:05 INFO [pid=24239] MtikAPI->raw_talk (544) Reading reply from socket
2011/02/23 00:50:05 DEBUG [pid=24239] MtikAPI->_read_str (1230) Reading 1 chars from socket

I had problems with this if I did not complete the last read from the router. Mine ended up being the last character of the previous response.

ADD: What I found is the API will not take another request (ignores input) until you read the entire previous response. This is the case if you do not use separate Tag names on the requests.

It is responding with something! Any idea what character is being read?

2011/02/23 00:50:05 INFO [pid=24239] MtikAPI->raw_talk (544) Reading reply from socket
2011/02/23 00:50:05 DEBUG [pid=24239] MtikAPI->_read_str (1230) Reading 1 chars from socket

I had problems with this if I did not complete the last read from the router. Mine ended up being the last character of the previous response.

ADD: What I found is the API will not take another request (ignores input) until you read the entire previous response. This is the case if you do not use separate Tag names on the requests.

No, it's not -- that's logging the entry/exit from the _read_str method, and is basically:

{
$log->"reading chars";
###Set up sig alarm to timeout if read has problems, which logs as a no response recieved (which is seen in the logs)
$socket->recv num chars;
$log->"num chars read";
}

Where it's dying means that it's failing to read anything whatsoever from the socket, not even garbage from a previous response. You can see that _write_str() is doing the same thing, logging the entry to the method (writing num chars...) and the exit (wrote num chars...).

Edit: It's basically a perl port of the python official library -- you can see the method names are similar and the program operation is exactly the same at the communication level; i.e. the read/write methods for sentences, words, lengths and strings.

maybe Perl does something different from Python?

My suggestion would be to try known to work client and sniff the traffic and then sniff traffic of your Perl client to see where is the differences. in actual network traffic.

maybe Perl does something different from Python?

My suggestion would be to try known to work client and sniff the traffic and then sniff traffic of your Perl client to see where is the differences. in actual network traffic.

They're both effectively a thin layer over the C socket libraries, really. This is rearing its head extremely intermittently and I'm unable to reproduce the issue when testing on a local network, even over a interface set to perform terribly (20% dropped packets, 150 ms latency, etc etc) via netem to see if network issues were causing it. I can't, however, sniff and log every single packet I send to the routers for weeks on end in an attempt to catch what I'm seeing.

Effectively:

I open a new socket to the router
I send chr(6).'/login'.chr(0) over the socket.
I log my entry into the recieve method.
I do an explicitly blocking recv() (via MSG_WAITALL - c/f man recv(2)) trying to get a single char on the socket I just opened and sent '/login' on, with a sig alarm set up to error out after 5 sec.
The error fires and the object kills the socket and waits 5 secs, retrying the process.

Logins to this router fail continuously for ~6 hours. At the same time the library still had no problem accessing other routers on the network. I then log into the router and reboot it. The failing router starts working.

If it happens again I can try another client to confirm an entirely different codebase sees the same thing, but I'd like to spend /some/ effort in trying to figure out what's going on first. I can see really only two potential situations here in terms of actual network traffic:

* My '/login' never reached the router so it had nothing to reply to. I check the return value of send() for errors and none was triggered. In addition, rebooting the router really shouldn't have done anything to change this. I do admit, however, there are some exceptional cases that this may be possible. However, I can't do anything with troubleshooting this unless I see a router doing the same thing again, at which point I can use a different codebase or a different host to try API logins and see if I get a response.

* The router never responded to the '/login' for some reason. I have no control over that which is why I'm exploring that option on here.

on router you have /tool sniffer (aka Packet sniffer) where you can see exactly what is received by router, you can sniff for API traffic exclusively by setting up filters.

Make sure you are running latest RouterOS and when you cannot connect, please check what you see in sniffer (maybe can save pcap file for us) and create supout.rif file. If possible (and you are not already running) try 5.0rc10

on router you have /tool sniffer (aka Packet sniffer) where you can see exactly what is received by router, you can sniff for API traffic exclusively by setting up filters.

Make sure you are running latest RouterOS and when you cannot connect, please check what you see in sniffer (maybe can save pcap file for us) and create supout.rif file. If possible (and you are not already running) try 5.0rc10

Yeah, I know. I'll give it a shot. However, I'm not going to install beta RouterOSes on production hardware and I'm likely going to be sticking with 4.05 as outlined in my first post. I don't see anything in the relase notes through 4.16 since it that should affect API operations. Are they incorrect?

Happened again. Did testing on it myself. Router authentication seems to be the problem; I couldn't get in via winbox/ssh/api/webbox/telnet.

Telnet gave: but with a ~30 sec pause between password entry and error, instead of the 2-3 sec pause like it does when an invalid pass is actually entered.

ssh never actually connected.

winbox gave an authentication failure logging in.

webbox worked for about a half minute after POSTing the login credentials then gave: API just did not respond anything to /login request. My API never got a response from the router (confirmed no incoming packets w/ pcap), and the delphi studio app errored out logging in as well.

Power cycling the router resolved all issues.

I looked through the changelog and can't find any mention of fixing anything that looks like this. I've mentioned previously I was holding off on updating to recent versions due to the api logoouts never removing accounts from the active user list and requiring scheduled reboots of routers. However, I'll live with that if this is a known issue that will be resolved by updating. Any ideas?


Edit: it's possible that this is the same issue as previous -- though I said winbox was functioning last time, I was working off of what I had been told; it's not something I saw for myself.

for support - update to 4.17 and see if anything changes, if not, create supout.rif file and send it to support after you have tried to connect.