Community discussions

 
AndreyS
just joined
Topic Author
Posts: 11
Joined: Tue Mar 13, 2018 2:19 pm

CCR1009-7G-1C No buffer space available

Tue Mar 13, 2018 2:38 pm

I have CCR1009-7G-1C:
Version 6.41.2
Current Firmware 6.41.2

Last few monthes I have a trouble - router became unavailable from lan and wan (every 2 5 days). I can't connect to router in case of trouble and make supout.rif (router even no pinging from lan).
Before last reboot, I configed to send all log to external log server. Before reboot router (watchdog pinging lan ip and reboot when it's unavailable) a find few messages:
"ipsec sendfromto failed: No buffer space available"

I have about 20 active openvpn and l2tp+ipsec connections and nothing special (no bgp/ospf, no mpls)

May be buffer it's not a reason, but I don't find another error messages in logs. How to fix this trouble?
 
pe1chl
Forum Guru
Forum Guru
Posts: 4328
Joined: Mon Jun 08, 2015 12:09 pm

Re: CCR1009-7G-1C No buffer space available

Tue Mar 13, 2018 3:57 pm

I would say, before it can be fixed it first has to be diagnosed.
Maybe you can watch system->resources and see if the free memory has a tendency to drop to zero.
You could also use tools->graphing and graph the resources then watch the memory graph.
(I would advise not to store it on disk to save flash writes, except for a debug session and then later remove it)
Maybe you can find a relation between activity you do and the occurance of the problem.
But I should say I never saw this problem on two CCR1009 I manage and that have quite some activity.
 
tippenring
Frequent Visitor
Frequent Visitor
Posts: 88
Joined: Thu Oct 02, 2014 8:54 pm

Re: CCR1009-7G-1C No buffer space available

Tue Mar 13, 2018 4:36 pm

This is a real shot in the dark, but do you happen to have a default route pointing at a broadcast interface? Something like
add distance=1 gateway=ether1
could conceivably fill up the ARP table.
 
AndreyS
just joined
Topic Author
Posts: 11
Joined: Tue Mar 13, 2018 2:19 pm

Re: CCR1009-7G-1C No buffer space available

Tue Mar 13, 2018 5:07 pm

I would say, before it can be fixed it first has to be diagnosed.
Maybe you can watch system->resources and see if the free memory has a tendency to drop to zero.
You could also use tools->graphing and graph the resources then watch the memory graph.
(I would advise not to store it on disk to save flash writes, except for a debug session and then later remove it)
Maybe you can find a relation between activity you do and the occurance of the problem.
But I should say I never saw this problem on two CCR1009 I manage and that have quite some activity.
I get info from Zabbix (get by snmp):
RAM: Before reboot it was used 270 MB, after 202. (from 933)
Disk: Used around 47 MB (from 128) and just little changes after reboot.

I try to find the relation, but I can't. I see many reconnections of l2tp and ovpn clients (many of them work on 3g), but it's typical and router can work 5 days with no problems...
 
bulero77
just joined
Posts: 5
Joined: Thu Nov 12, 2015 11:12 pm
Location: valencia

Re: CCR1009-7G-1C No buffer space available

Tue Mar 13, 2018 5:16 pm

I have a similar problem, it is this Running out of disk space, when minimum 500kB is reached UM will be stopped! someone knows the solution
 
pe1chl
Forum Guru
Forum Guru
Posts: 4328
Joined: Mon Jun 08, 2015 12:09 pm

Re: CCR1009-7G-1C No buffer space available

Tue Mar 13, 2018 5:30 pm

When running things like UM or Dude you always need to add external storage!
(SD card, USB stick)
 
AndreyS
just joined
Topic Author
Posts: 11
Joined: Tue Mar 13, 2018 2:19 pm

Re: CCR1009-7G-1C No buffer space available

Tue Mar 13, 2018 6:31 pm

This is a real shot in the dark, but do you happen to have a default route pointing at a broadcast interface? Something like
add distance=1 gateway=ether1
could conceivably fill up the ARP table.
I know, that it's difficult to troubleshoot ... I have 6 default route with different distances and routing marks (i have 2 ISP on this router and few on another routers). I understand that i can lost default route, but when the problem begin I can't ping local adress from lan (I have 192.168.2.1/24 on bridge). It begin from lost of packets and after 5-10 minutes totally lost from wan and lan (but watchdog work).
 
AndreyS
just joined
Topic Author
Posts: 11
Joined: Tue Mar 13, 2018 2:19 pm

Re: CCR1009-7G-1C No buffer space available

Tue Mar 13, 2018 6:33 pm

I have a similar problem, it is this Running out of disk space, when minimum 500kB is reached UM will be stopped! someone knows the solution
I think we have different problems - my disk utilization around 35-40%
 
bulero77
just joined
Posts: 5
Joined: Thu Nov 12, 2015 11:12 pm
Location: valencia

Re: CCR1009-7G-1C No buffer space available

Tue Mar 13, 2018 8:33 pm

IF THE PROBLEM IS SEEMING DIFFERENT, ALTHOUGH MY DISC ALSO MARK 35 OR 40 AND I LEAVE THE ERROR OF LACK OF SPACE
 
bulero77
just joined
Posts: 5
Joined: Thu Nov 12, 2015 11:12 pm
Location: valencia

Re: CCR1009-7G-1C No buffer space available

Tue Mar 13, 2018 8:43 pm

THE DUDE, I DO NOT USE IT, UM I DO NOT KNOW WHAT IS SOMEBODY TO SAY ME PLEASE
 
pe1chl
Forum Guru
Forum Guru
Posts: 4328
Joined: Mon Jun 08, 2015 12:09 pm

Re: CCR1009-7G-1C No buffer space available

Tue Mar 13, 2018 9:18 pm

I know, that it's difficult to troubleshoot ... I have 6 default route with different distances and routing marks (i have 2 ISP on this router and few on another routers). I understand that i can lost default route, but when the problem begin I can't ping local adress from lan (I have 192.168.2.1/24 on bridge). It begin from lost of packets and after 5-10 minutes totally lost from wan and lan (but watchdog work).
Ok that is a situation similar to what I have on one of my two routers and it runs absolutely stable.
Of course the routes have a gateway specified, not only an interface name.
However, it appears that "no buffer space" is different from running out of memory. Would be difficult to imagine anyway on a CCR.

bulero77 - please don't take the topic away from the original poster and use search to answer your questions.
 
AndreyS
just joined
Topic Author
Posts: 11
Joined: Tue Mar 13, 2018 2:19 pm

Re: CCR1009-7G-1C No buffer space available

Wed Mar 14, 2018 8:24 am

I know, that it's difficult to troubleshoot ... I have 6 default route with different distances and routing marks (i have 2 ISP on this router and few on another routers). I understand that i can lost default route, but when the problem begin I can't ping local adress from lan (I have 192.168.2.1/24 on bridge). It begin from lost of packets and after 5-10 minutes totally lost from wan and lan (but watchdog work).
Ok that is a situation similar to what I have on one of my two routers and it runs absolutely stable.
Of course the routes have a gateway specified, not only an interface name.
However, it appears that "no buffer space" is different from running out of memory. Would be difficult to imagine anyway on a CCR.

bulero77 - please don't take the topic away from the original poster and use search to answer your questions.
Few monthes ago, before upgrade the router, I used 951G with same config and without any problems, but we upgraded it in cause of perfomance.
All my routes (about 60 prefixes) have just interface like gateway. Can it be the reason? When I found a message about buffer space yesterday, I found few post, when disabling ip route cache was a decision of similar problem and I disabled it. Now uptime 1d 09:04:59. Waiting ...
 
pe1chl
Forum Guru
Forum Guru
Posts: 4328
Joined: Mon Jun 08, 2015 12:09 pm

Re: CCR1009-7G-1C No buffer space available

Wed Mar 14, 2018 11:46 am

Check in IP->ARP if your ARP table has only local addresses and the address(es) of gateways.
When it contains all internet addresses you have visited, all with the same MAC address, you have found a problem.
You need to fix that by putting the gateway address in your default route.
 
AndreyS
just joined
Topic Author
Posts: 11
Joined: Tue Mar 13, 2018 2:19 pm

Re: CCR1009-7G-1C No buffer space available

Wed Mar 14, 2018 12:52 pm

Check in IP->ARP if your ARP table has only local addresses and the address(es) of gateways.
When it contains all internet addresses you have visited, all with the same MAC address, you have found a problem.
You need to fix that by putting the gateway address in your default route.
Thank for advice! Fixed it. Really, my arp table was huge and kept all ip with mac of my gw.
Keep waiting for reboot
 
tippenring
Frequent Visitor
Frequent Visitor
Posts: 88
Joined: Thu Oct 02, 2014 8:54 pm

Re: CCR1009-7G-1C No buffer space available

Wed Mar 14, 2018 4:49 pm

Did I seriously get that right based on the vague description in the OP? Wow. lol.
 
AndreyS
just joined
Topic Author
Posts: 11
Joined: Tue Mar 13, 2018 2:19 pm

Re: CCR1009-7G-1C No buffer space available

Thu Mar 15, 2018 9:02 am

Rebooted again this night. After that it was updated firmware to 6.41.3
 
AndreyS
just joined
Topic Author
Posts: 11
Joined: Tue Mar 13, 2018 2:19 pm

Re: CCR1009-7G-1C No buffer space available

Mon Mar 19, 2018 10:10 pm

I found the reason, but i don't understand how to fix it.
[admin@MikroTik] > /ip route cache print
cache-size: 131054
max-cache-size: 131072

When cache-size = max-cache-size router stop working - loses about 70%.
Sometimes I connected to it and saw:
[admin@MikroTik] > /ip route cache print
cache-size: 131087
max-cache-size: 131072

About 5 minutes it works with a huge loses, and after watchdog reboot I saw:
[admin@MikroTik] > /ip route cache print
cache-size: 85
max-cache-size: 131072

A collected data by zabbix - cache-size increase on 20k every 24 hours.
 
tippenring
Frequent Visitor
Frequent Visitor
Posts: 88
Joined: Thu Oct 02, 2014 8:54 pm

Re: CCR1009-7G-1C No buffer space available

Tue Mar 20, 2018 12:29 am

That makes sense I suppose. I'm not sure why your route cache is filling up.

Can you post an "ip route export"?
 
AndreyS
just joined
Topic Author
Posts: 11
Joined: Tue Mar 13, 2018 2:19 pm

Re: CCR1009-7G-1C No buffer space available

Tue Mar 20, 2018 3:22 pm

That makes sense I suppose. I'm not sure why your route cache is filling up.

Can you post an "ip route export"?
This comand doesn't show all prefixes. I've changed sn and gw ip to xxx:
[admin@MikroTik] > ip route export
# mar/20/2018 16:17:49 by RouterOS 6.41.3
# software id = 47QI-ZBNN
#
# model = CCR1009-7G-1C
# serial number = xxx
/ip route
add distance=10 gateway=ZF_Server routing-mark=ZF
add check-gateway=ping distance=2 gateway=pppoe-MTS routing-mark=ISP-MTS-route
add check-gateway=ping distance=2 gateway=xxx.xxx.xxx.xxx routing-mark=ISP-Spark-route
add check-gateway=ping distance=2 gateway=xxx.xxx.xxx.xxx
add check-gateway=ping distance=3 gateway=pppoe-MTS


I also have about 20 static routes. Now I changing static routes to ospf.
 
tippenring
Frequent Visitor
Frequent Visitor
Posts: 88
Joined: Thu Oct 02, 2014 8:54 pm

Re: CCR1009-7G-1C No buffer space available

Tue Mar 20, 2018 4:17 pm

/ip route
add distance=10 gateway=ZF_Server routing-mark=ZF
add check-gateway=ping distance=2 gateway=pppoe-MTS routing-mark=ISP-MTS-route
add check-gateway=ping distance=2 gateway=xxx.xxx.xxx.xxx routing-mark=ISP-Spark-route
add check-gateway=ping distance=2 gateway=xxx.xxx.xxx.xxx
add check-gateway=ping distance=3 gateway=pppoe-MTS


I also have about 20 static routes. Now I changing static routes to ospf.
That's what I thought. I'm pretty sure your gateway=pppoe-MTS is the problem. The router has to ARP for every destination IP, and also adds the entry to the route cache. Gateway should be an IP address.
 
User avatar
mrz
MikroTik Support
MikroTik Support
Posts: 5520
Joined: Wed Feb 07, 2007 12:45 pm
Location: Latvia
Contact:

Re: CCR1009-7G-1C No buffer space available

Tue Mar 20, 2018 4:26 pm

That is not the case. PPPoE does not use ARP, route cache can be leaked from something else. Generate supout file and send to support.
 
tippenring
Frequent Visitor
Frequent Visitor
Posts: 88
Joined: Thu Oct 02, 2014 8:54 pm

Re: CCR1009-7G-1C No buffer space available

Tue Mar 20, 2018 5:11 pm

That is not the case. PPPoE does not use ARP, route cache can be leaked from something else. Generate supout file and send to support.
Good point. ARP only occurs on broadcast interfaces. PPPoE would not be a broadcast. Thanks for the correction.
 
AndreyS
just joined
Topic Author
Posts: 11
Joined: Tue Mar 13, 2018 2:19 pm

Re: CCR1009-7G-1C No buffer space available

Tue Mar 20, 2018 6:20 pm

That is not the case. PPPoE does not use ARP, route cache can be leaked from something else. Generate supout file and send to support.
Generate at any time, or when route cache almost full?
 
AndreyS
just joined
Topic Author
Posts: 11
Joined: Tue Mar 13, 2018 2:19 pm

Re: CCR1009-7G-1C No buffer space available

Wed Mar 21, 2018 4:30 pm

I think, that i'm found the reason (thanks tech support!!!): for my ovpn clients I chose profile with option "Only One : yes". When I changed to "no", some of my clients made second connection. I changed the parameters on clients and they don't try to make second connection now.
After changing option "only one" to no, ip route cache stoped growth, that's mean that if valid client try to connect, router add "something" to "ip route cache" and if router discard connection in cause of "only one", router didn't remove "something" from "ip route cache"

Who is online

Users browsing this forum: No registered users and 16 guests