why is it that system health works in ros 4.17 but not in ros 5
was something removed?

which routerboard model? do you have the routerboard package installed?

forgot to say x86

There are many different motherboards, it's hard to keep an eye on all of them. Probably a kernel driver was changed for your model.

would that change be on the linux side or on mikrotik side

kernel is on the linux side. it's hard so say why it stopped working for you, mayb try sending a supout.rif file to support@mikrotik.com

Came in this morning after monitoring v5.2 all weekend with no obvious problems to find as soon as the users started routing traffic through it, it started jumping up and around >95% CPU load and started dropping packets.

I've dropped back to v5.0 to see if I can establish if it is a configuration change or the firmware change to 5.2 that has had the effect as we have a few mods over the past couple of weeks that might possibly be causing the issues.

I'll run with v5.0 through today and post some information later.

do you use dynamic routing? OSPF?

Goto /ip route cache
and print the output. See if your used memory is almost full during having high cpu load and packet losts.

anyone having issues in v5.2, please generate supout.rif file at the moment of the problem, write a description, make a screenshot, and send it all to support.

Yes its a month I have submitted many files but yet no answer!

#2011041066000044 and
#2011032666000151


This memory leak in /ip route cache, has been present since 5. I remember having the issue even with RCs but don no which RC.

anyone having issues in v5.2, please generate supout.rif file at the moment of the problem, write a description, make a screenshot, and send it all to support.

Yes its a month I have submitted many files but yet no answer!

#2011041066000044 and
#2011032666000151


This memory leak in /ip route cache, has been present since 5. I remember having the issue even with RCs but don no which RC.

you were instructed to adjust your EoIP configuration because it caused a loop. You didn't respond about that.

I disabled them! Cleared them!
It didn't fix the issue

I then downgraded to v4 cause I couldn't reboot my router twice a day. It started with 7 days, 2 days and 1 , then twice a day...
....................
But I still have 2 of my routers on this version. One of them has 2 EOIPs which gets problem every 2 days.
The other 1 EOIP (even I disabled it but no fix) which runs into problem every 7 days.

Thank GOD there is a good counter! I check /ip route cache. When the usage reaches max ram size, it's time to reboot the router. If I do not reboot, the router CPU load gets into more than 70%, packet lost apears with "No buffer available" in ping command in winbox. All router functionality starts fainting untill router fully hangs.

I have reported all of these with corresponding supout file. These are long tickets. But I have not received any update recently.

you were instructed to adjust your EoIP configuration because it caused a loop. You didn't respond about that.

In addition, the EOIP disconnection was a symptom of router mulfunction and not the root cause! When router stops normal functionality, everything bad happens. EOIP interfaces disconnect/connect... bridges won't work. Ping times out... packet forwarding decreases.... pppoe interfaces disconnect...

As I said, there is something with route memory cache. Why should it increase until getting full? Why should it increase at a fixed rate and during a fixed period of time? have you seen my RAM usage graphs? Isn't having a pattern for a problem occurance unsuall?

so why didn't you respond to the support ticket? if there is no response, support engineer assumes the suggestion fixed the problem. there is no use to complain here, email support

Normis ... and when you said that you were looking into my ticket which is about this same issue a few days ago, and forgot about it ... ? Even when others asked about the status - we received no answer.

I don't see what the added value is in disregarding a serious problem in ROS 5.X . Not talking about it will not make us just forget and stop bringing it up.

This problem is not just an inconvenience. It truly effects ROS features and stability and questions its usefulness in real life situations.

Ticket#2011041766000095

... after a day I have half my cache full (no P2P and no DDOS)

[admin@MikroTik] > system resource print
uptime: 1d2h24m17s
version: "5.2"
free-memory: 232320KiB
total-memory: 270224KiB
cpu: "Intel(R)"
cpu-count: 1
cpu-frequency: 3024MHz
cpu-load: 40%
free-hdd-space: 218790KiB
total-hdd-space: 253803KiB
write-sect-since-reboot: 5596
write-sect-total: 5596
architecture-name: "x86"
board-name: "x86"
platform: "MikroTik"
[admin@MikroTik] > ip route cache print
cache-size: 29487
max-cache-size: 65536


The router will stop responding in another 16-20 hours ... should it not lock up until then.



Here is another one after just 1 day 15 hours

[admin@MikroTik] > system resource print
uptime: 1d15h17m24s
version: "5.2"
free-memory: 219892KiB
total-memory: 256280KiB
cpu: "Intel(R)"
cpu-count: 1
cpu-frequency: 2394MHz
cpu-load: 0%
free-hdd-space: 30781468KiB
total-hdd-space: 30945632KiB
write-sect-since-reboot: 9777
write-sect-total: 9777
architecture-name: "x86"
board-name: "x86"
platform: "MikroTik"
[admin@MikroTik] > ip route cache print
cache-size: 32767
max-cache-size: 32768
[admin@MikroTik] >

I already said above, instead of complaining here, email this information to support, where our support engineer will look at it. Other people on this forum have no benefit if you just complain and don't inform MikroTik

Other people on this forum have no benefit if you just complain and don't inform MikroTik

In the same way that nobody benefits by you complaining about a customer complaining about bug in your software. Sometimes it seems you spend more time insulting people for the way they asked a question than actually trying to help them solve it. It is pretty obvious that things can't be communicated nearly as easily through the "support" system, so why not just offer a little official support on the forums.

I'm not trying to be rude here, but the easier you make it for your customers to communicate with you and help troubleshoot problems like these, the easier it will be to solve them and the happier everyone will be.

glucz here is complaining that he didn't receive an answer to his ticket. In fact, the latest mail in the ticket was a question to him, which he didn't answer. That's why there is no progress on the issue in his specific case. I am only talking about him and his ticket right now.

I understand.. I wasn't necessarily speaking about his ticket specifically, but just an observation over the last few years I have been on this forum. It is obvious that quite a few people are having similar issues with 5.x leaking memory or something like that, so it surprises me that you aren't doing more to reach out to troubleshoot with them. I understand the importance of being able to track issues, but there's got to be an easier way to support 10 people in the forum having the same problem by getting them to work together, rather than simply having all 10 work separately with MT support.

Sorry, don't mean to be dragging this off topic. That was my 2 cents... I'll be quiet now.

It is too often that people post angry messages like "when is this going to be fixed" because they assumed somebody reported this somewhere, but actually all of them are just waiting, and nothing is posted to the forum or to support. I am simply encouraging everyone to make those supout.rif files and participate to get problems fixed.

glucz here is complaining that he didn't receive an answer to his ticket. In fact, the latest mail in the ticket was a question to him, which he didn't answer. That's why there is no progress on the issue in his specific case. I am only talking about him and his ticket right now.

normis, we all do agree we should not complain here.
As I said, I have updated ticket and since didn't get reply, I posted to forum too. I am glad to see I am not alone on this. Also, new people get familiar with possible problems when upgrading.
........................................
Now, I appreciate if you provide me with your email address or contact my profile email so I send you bounch of ticket replies that I sent and didn't get answer back. May be you have not received them. Or you may ask your tech co-workers to simply update the ticket and ask me what they need to know se we together help to solve the problem.

please list the ticket numbers in question, and I will make sure you get an answer

Hm. It seems like that my previous post got deleted. I hope it was because of the size and not the content ... so I will just phrase it simply:

Please send me the question you refer to, because I received no questions from support.

Thank you
GL

...
[admin@MikroTik] > system resource print
uptime: 1d2h24m17s
version: "5.2"
free-memory: 232320KiB
total-memory: 270224KiB
architecture-name: "x86"
board-name: "x86"
platform: "MikroTik"
[admin@MikroTik] > ip route cache print
cache-size: 29487
max-cache-size: 65536

...
Here is another one after just 1 day 15 hours
[admin@MikroTik] > system resource print
uptime: 1d15h17m24s
version: "5.2"
free-memory: 219892KiB
total-memory: 256280KiB
architecture-name: "x86"
board-name: "x86"
platform: "MikroTik"
[admin@MikroTik] > ip route cache print
cache-size: 32767
max-cache-size: 32768
[admin@MikroTik] >

I'm just interested Is there a way to increase "max-cache-size" ?
Is there any dependency between RAM/model and max-cache-size?

It seems like that the max cache size depends on the RAM in the router.

The minimum size seems to be 16384
If you have over 128M RAM, the size is 32768
If you have over 256M RAM, the size is 65535
etc ...

The point is that it will fill up regardless of the cache size. The added bonus is that 5.2 is not stable enough (at least for me) to give enough time to fill up 65535. It would lock up with 100% CPU. I made a post in the forum about that with different screenshots.

GL


ps: I just received the question from support. The question really is not technical in nature and requires no true answer. I answered it however to speed up the process.

the last email from us says "thank you for the report, we are investigating this issue".

It seems like that the max cache size depends on the RAM in the router.
..

As I understand you have the same RAM size on both routers , but max-cache-size for routers is different.

No. They are all different. Whatever was available at the time.

Whoever else is also having route cache problems: Are you using tarpit actions in firewall rules?
I picked 2 MT750's off the shelf and am tring to build the minimum required rules and services to replicate the problem.

Thanks
GL

No. They are all different. Whatever was available at the time.

Whoever else is also having route cache problems: Are you using tarpit actions in firewall rules?
I picked 2 MT750's off the shelf and am tring to build the minimum required rules and services to replicate the problem.

Thanks
GL

I have currently 4 routers running v5 which 2 of them have this route cache memory problem.
I confirm the two having problem, have TARPIT rule in firewall while the other 2 which are working fine do not have any tarpit rule. For test, I am going to disable tarpit rule in one problem routers.

But I know another company utilizing MK with the same problem and no tarpit rule.

No. They are all different.
GL

I' realy sorry ( ), but what is the difference between
total-memory: 270224KiB and total-memory: 256280KiB ???
Looks like both routers has 256Mb RAM.

This is total RAM-shared video ram, so the RAM available might be slighly different for each configuration.

GL

ospf+nbma+mpls - bug in nbma mode -not send hello packet
im have around 1K routes distributed by OSPF nbma method, neighbor stil active but MPLS is lost. in Debug mode in not freeze router im dont see any hello packet to lost node. After reboot freeze router all working done day or two. When im going on freeze rb and change network type ptmp all is standing up without reboot.

i've upgaraded from 4.13 to 5.1 and then to 5.2 on a x86 system;
now i can't access via ssh with admin privilege using password but using ssh-key still works ok.
i tryed by creating other users and i've no problem to login with passwords.

its got bugs 5.2 has route problems

ask support for 5.3 test

it works well for me
much better then 5.2

Care to explain the "route bugs"

5.2 is without doubt the best of the rest.
Still problems with Usermanager, but definitely the most stable IMHO.
I am not using any of the wireless functions of the RouterOS.
Simply a hotspot/router/authenticator with UBNT radios.

No. They are all different.
GL

I' realy sorry ( ), but what is the difference between
total-memory: 270224KiB and total-memory: 256280KiB ???
Looks like both routers has 256Mb RAM.

Glucz,
It seems my route cache momory ram has stopped increasing when I disabled tarpit rules... It's now 14 hours since I did it. I will keep posting if it's fixed.
Of course I have slight total memory usage icrease but route cache memory increase has stopped.

Yes. I can too confirm after 16 hours that route cache is now stable. So tarpit seems to leak into route cache.

GL

i have a eoip tunnel network setup over ospf routes with bridge on radius

keeps crashing over 5-6M of data over bridge

Yes. I can too confirm after 16 hours that route cache is now stable. So tarpit seems to leak into route cache.

GL

My memory usage graph always showed slight memory increase. Even on the routers without this tarpit rule and of course these routers were stable in functionality.
But from this tarpit rule experience and also day to day monitoring of graph I can coclude that root cause for all memory leaks is that, MK does not release used memories. Another example is when I "make supout file". Just when I click, about 5 mbytes of memory is used up instantly and at once and never get released. Of course I have provided all evidences to Janis Megis who is working great with me to follow up the issue in ticket #2011041066000044.

Attached you can see the memory graph I am talking about. See spikes in weekly section. Spikes happened when I tried to generate supout file. Slight increase is also notable.
....................
We noticed Tarpit rule problem cause it was using up route cache mem which is small and very vital to day to day fine operation of router. In contrast, total memory consumption with slight pace may be noticed in 2 or 3 month based on the total memory you install on the router. For most we have to install new releases so we never hit this long period of uptime
I have a RB1000 box running 3.28 with 100mbit/s of traffic. Its sooooooooo stable and even if memory is used more for a few hours, you have it free too when not needed.

what all you running through your box as i dont even use half of that you use


do you use proxy on your server?

no proxy... but your graph also shows memory usage increases... I beleive that should not happen unless you have increase of traffic or load on the router or puting new configuration that require more ram. For me, everything is the same during a month but still memory usage increases...

this is correct it also the firewall setup example i found that

user one uses internet the hole month for facebook http traffic 512k
then next month user finds out about torrents then download a hole month of p2p also at 512k

but the p2p has more rules in the firewall so it uses more load on the server even if its the same amount of data

It seems some of you on this topic are familiar with memory leaking:

I have this rb1000 with rosv4.11 that is doing simple queus for all my clients and some basic firewall filtereing.
The routing it performs is no more than a general route back into my network (next unit) and some policy routing with src addr list classifier in the mangle for the two gateway with failover.

My memory usage goes sligthy up for some weeks and then when it comes near 50% it ´hangs´. Meaning the internet ports stop functioning. This is going on for more then a year now, sometimes worse, sometimes better.
I see no relation in the total traffic running over this router nor do I see it with the amount of client in the simple queus. (approx. 225)

Normally, after the crash, we have to power cycle the unit to bring it back and all works fine than for another xx weeks.
Yesterday I saw the memory usage graph near a level to where the rb crashed 2 weeks ago and decided just to reboot it in the middle of the night. As you can see in the graph, the memory usage goes down considerably after the reboot.

Any ideas?

Forgot to say, I have ROS 5.2 now implemented on my whole network to satisfaction (routers and AP`s and CPE`s) but this main gateway of my network not yet. I am a bit afraid to do so because if this router goes down my network is down. So first I wanted to know if my problem (which seems to me a sort of a ´memory leakage´) is solved in ros5.x

Hence my previous post.

We saw ROS 5.x deployed for a few weeks before crashes. If you have it on busy AP's i suggest downgrading to 4.16 before you have issues...

I installed 5.2 on a pppoe (bras) mk router.

Till now I see a bug. in PPP, active connection you can see new connecting pppoe client uptime starts from 39seconds and starts decreasing till reaching 0, then starts increasing! So funny

So it's user "yyy" connect. you can see it's alive for 39, 38, 37, 36, .... , 0 , 1, 2 ,.... uptime!

It seems it's like starting timer from -40seconds

It is too often that people post angry messages like "when is this going to be fixed" because they assumed somebody reported this somewhere, but actually all of them are just waiting, and nothing is posted to the forum or to support. I am simply encouraging everyone to make those supout.rif files and participate to get problems fixed.

So I mention this again, what about a public facing bug-tracker? This would solve all these problems mentioned. We (the paying customers) will be able to keep track of bugs / features on our own time, and always have a place to reference when we ask a question about a bug/feature. This would solve the endless forum postings about 'ticket xyz, what is status?'.

Also my 2 cents. I know RouterOS is a complex software, and I feel we (the paying / prospective customers) could benefit greatly from such as system as mentioned above.

Can you give an example public bug tracking system for a commercial closed software product?

Both Cisco and Juniper have bug trackers that are open for customers with appropriate credentials

Cisco, Juniper, Microsoft, Novell just to name a few, Admittedly you need to sign NDA's for most access but as I said to MT at MUM AU 11 the larger consumers of ROS will sign this in a heart beat to be able to see issue's and history of fix's because as we all know the current MT changlog is only useful for toilet paper.

The fact is even a NDA backed bug tracker would be incredibly useful to medium to big WISP's, At the moment it takes 4-8 weeks for us to vet a new release simply because we can't trust MT to release accurate changelog's or MT support to provide current known issues

It can start simple, nothing too complex, just a way for 'trusted' users to submit/modify features / bugs / fixes, maybe this forum could be used as a starting point for vetting the users' allowed to 'post' data to the bugtracker. Then maybe read-only access for general users.

All I would like to see really is just a central place specifically for bugs/features/fixes (or temp workarounds).

Another issue with v5.2 running OSPF. After some time of uptime, my single instance of OSPF resets to default config like router id becomes 0.0.0.0 and other parameters becomes default.

I installed 5.2 on a pppoe (bras) mk router.

Till now I see a bug. in PPP, active connection you can see new connecting pppoe client uptime starts from 39seconds and starts decreasing till reaching 0, then starts increasing! So funny

So it's user "yyy" connect. you can see it's alive for 39, 38, 37, 36, .... , 0 , 1, 2 ,.... uptime!

It seems it's like starting timer from -40seconds

I was not able to repeat it. What architecture is the PPPoE server and client? Any specific configuration?

Support confirmed our memory leak is caused by SNMP, just an FYI for anyone out there suffering from the same

I installed 5.2 on a pppoe (bras) mk router.

Till now I see a bug. in PPP, active connection you can see new connecting pppoe client uptime starts from 39seconds and starts decreasing till reaching 0, then starts increasing! So funny

So it's user "yyy" connect. you can see it's alive for 39, 38, 37, 36, .... , 0 , 1, 2 ,.... uptime!

It seems it's like starting timer from -40seconds

I was not able to repeat it. What architecture is the PPPoE server and client? Any specific configuration?

I have seen a similar thing with OVPN uptime. I was able to make a screenshot and send it to support with supout. The issue is that the counter doesn't start from 0, but instead it will count from the current uptime value of ROS ... So In my case it said that the uptime was 19hrs ... I guess if the uptime value is large enough and the counter's underlying variable is unsigned, I can see this causing a backward count, but the initial value would be something random.

There is some serious issue with pppoe connectivity in 5.2 & 4.17.

I downgraded to 5.0rc4 all issue solved but rc4 UM is not stable

There is some serious issue with pppoe connectivity in 5.2 & 4.17.

I downgraded to 5.0rc4 all issue solved but rc4 UM is not stable

Please describe this problem in more details.

anyone running 5.2 is getting stuck bgp routes after peer down?

i have a two main bgp routers and thei are interconected via ibgp. on the first router i just lost a peer to an exchange point. so a lost about 5k routes from them. but the lost routes are stuck in both routers, the routing table didnt got updated with the lost routes.

Anyone with that problem too?

bad blocks 4.x,5.2> 0% ........ install RouterOS 5.3 (distib. 17.05.) bad blocks 4,2% ... RB433UAH

bad blocks 4.x,5.2> 0% ........ install RouterOS 5.3 (distib. 17.05.) bad blocks 4,2% ... RB433UAH

BTW, I netinstalled 5.2 on a RB433 and at two RB450 because they had 0.3 ... 0.5% badblocks with the built in ROS (4.11 I guess). Even formatted NAND from boot menu (via RS232 console) to "zero" this readings.

But they still there.

Regards;

I installed 5.2 on a pppoe (bras) mk router.

Till now I see a bug. in PPP, active connection you can see new connecting pppoe client uptime starts from 39seconds and starts decreasing till reaching 0, then starts increasing! So funny

So it's user "yyy" connect. you can see it's alive for 39, 38, 37, 36, .... , 0 , 1, 2 ,.... uptime!

It seems it's like starting timer from -40seconds

I was not able to repeat it. What architecture is the PPPoE server and client? Any specific configuration?

I have seen a similar thing with OVPN uptime. I was able to make a screenshot and send it to support with supout. The issue is that the counter doesn't start from 0, but instead it will count from the current uptime value of ROS ... So In my case it said that the uptime was 19hrs ... I guess if the uptime value is large enough and the counter's underlying variable is unsigned, I can see this causing a backward count, but the initial value would be something random.

You are absolutely correct. When the router reached a higher uptime no more of this issue.

I have tried ROS 5.2 on multiple boards: RB800, RB600, RB433, RB433AH. In each case I am no longer able to right click a registered client and use telnet. The terminal window opens and displays the following:

telnet: inet_pton failed for 00:00:00:00:00:00/telnet: Success
Trying 00:00:00:00:00:00...
Connected to 00:00:00:00:00:00.
Escape character is '^]'.

MikroTik v5.2
Login:


The login prompt is for the AP itself vs the remote client. I am able to telnet if I use the Telnet applet under tools.

Loses its default gateway through pptp provider after 5 minutes at full load - 98%. Only helps to reboot. Downgrade to version 4.17 solved the problem.
ps: 450G and 750G