Please prove me wrong.

I found this old thread. It looks like RouterOS is only using 2GB RAM regardless the platform. Even the official hardware from Mikrotik cannot utilise more than 2GB of RAM. Does that mean RouterOS is not meant to be used in large scale? Do I miss anything here?

Interesting post, I am looking for same answer, How to increase max-entries: I have recently bought CCR-1036 Hardware. I am doing lots of firewall stuff and i badly need that setting bump. Document lying about it, that it will increase base on available RAM. We have 16GB RAM and 14GB Free. What i should do now???

Code: Select all

[admin@MikroTik] > /ip firewall connection tracking print
                   enabled: auto
      tcp-syn-sent-timeout: 5s
  tcp-syn-received-timeout: 5s
   tcp-established-timeout: 1d
      tcp-fin-wait-timeout: 10s
    tcp-close-wait-timeout: 10s
      tcp-last-ack-timeout: 10s
     tcp-time-wait-timeout: 10s
         tcp-close-timeout: 10s
   tcp-max-retrans-timeout: 5m
       tcp-unacked-timeout: 5m
               udp-timeout: 10s
        udp-stream-timeout: 3m
              icmp-timeout: 10s
           generic-timeout: 10m
               max-entries: 524288
             total-entries: 524316

Free Memory:

Code: Select all

[admin@MikroTik] > /system resource print
             uptime: 4h6m43s
            version: 6.35rc3 (testing)
         build-time: Feb/03/2016 07:32:45
        free-memory: 14.6GiB
       total-memory: 15.9GiB
                cpu: tilegx
          cpu-count: 36
      cpu-frequency: 1200MHz
           cpu-load: 100%
     free-hdd-space: 883.3MiB
    total-hdd-space: 1024.0MiB
  architecture-name: tile
         board-name: CCR1036-8G-2S+
           platform: MikroTik

This number in "max-entries" will increase only when needed. When you will have 60'000 entries, the max entries will increase to accommodate that. I know it may be confusing, but it works that way. It will not reserve RAM for connections that don't exist. It will increase when you will hit the limit for some period of time. It will use 16GB, there is no scam

What does it have to do with RAM??? It looks like amount of connections in conntrack is simply limited to half a million, and together with RAW table (firewall filter that happens before connection tracking) added to latest versions, i do not see the reason why there should be more.

memory is utilized quite well especially when you using BGP, so there are no limit of RAM usage.

This number in "max-entries" will increase only when needed. When you will have 60'000 entries, the max entries will increase to accommodate that. I know it may be confusing, but it works that way. It will not reserve RAM for connections that don't exist. It will increase when you will hit the limit for some period of time. It will use 16GB, there is no scam

Hi normis. Thank you for the quick reply. But please read carefully. The total-entries is already greater than max-entries. I don't see max-entries increased anyhow. The original post of the thread even kept it running for hours on x86 platform and didn't see the value changed.
max-entries: 524288
total-entries: 524316

Original post artificially increased connections for experiment sake. Do you really need half a million connections? In what real-life scenario?

What does it have to do with RAM??? It looks like amount of connections in conntrack is simply limited to half a million, and together with RAW table (firewall filter that happens before connection tracking) added to latest versions, i do not see the reason why there should be more.

memory is utilized quite well especially when you using BGP, so there are no limit of RAM usage.

Thank you for your reply. But the original post already quoted on normis. normis said the following in this thread so I assume connection tracking has something to do with RAM.

and what about conntrack max-entries?

this number doesn't automatically increase just by adding RAM, it increases based on remaining RAM, when you use most of the entries given in the number. So if your default max is 500000 with 4GB, it will be also 500000 with 16GB until you use 500000, then it will increase based on free RAM

Original post artificially increased connections for experiment sake. Do you really need half a million connections? In what real-life scenario?

Well... What is wrong with "artificially inscresed connections for experiment sake"? People need to test the limit of the hardware/software. How are people suppose to test without artificially generating data? We all need to test the equipment before we deploy to production environment. At least I would've done my feasibility study first which is what I am doing now.

Well... I plan to start a small ISP for my neighbourhood. I happen to live around college campus. Here has entire village of college students. As we all know, college students do whatever they want. Let's say for example they all use BitTorrent. It's very easy to reach 10000 connections for 1 PC with BitTorrent running. Let's say 1 house has 10 PCs and I would start small with just 10 houses. That's already 1 million connections.

Based on the math, 1 CCR can only server 5 clients max...

Sorry, but what planet are you from?

i have /22 network full with clients, and only time i run into problems with amount of connections was when i was under DDoS attack(to be more precise - some of the clients were infected to be a part of DDoS attack to somewhere).

it is 500k, it is 500 connections per /22 client. atm they use 83 connections on average.

Sorry, but what planet are you living on?

i have /22 network full with clients, and only time i run into problems with amount of connections was when i was under DDoS attack(to be more precise - some of the clients were infected to be a part of DDoS attack to somewhere).

Haha. I'm living on planet Earth. I believe it's the same planet as yours.

Anyway, joking aside.

Thank you very much for your valuable experience. Do you use connection tracking for your firewall? If so, how many connections do you normally get for presumably 1000 clients? Do you have SNMP set up to track your connections? Do you ever see your max-entries greater than 500k?

Thank you very much in advance if you can share your more about your real life situation.


Edit: Thank you. I see you added your comment. How do you manage to have only 83 connections on average? Do your clients not use BitTorrent? Don't they download anything from Internet? Can you please share a bit more. Thank you.

First of all putting a state-full firewall on such large network is stupid. on almost all my routers that work with 100+ clients i have connection tracking off.

But after i got a call that i'm source of DDoS attack, i had to turn it on for some "special" clients.

Now with new RAW table - especially action=no-track, all problems are solved, i can leave firewall on form some clients, and rest of the clients go "no-track" mode. So i can't give you number for whole /22 network, i can just say that those "special" clients are using 83 connections on average right now.

First of all putting a state-full firewall on such large network is stupid. on almost all my routers that work with 100+ clients i have connection tracking off.

But after i got a call that i'm source of DDoS attack, i had to turn it on for some "special" clients.

Now with new RAW table - especially action=no-track, all problems are solved, i can leave firewall on form some clients, and rest of the clients go "no-track" mode. So i can't give you number for whole /22 network, i can just say that those "special" clients are using 83 connections on average right now.

Thank you for your reply once again.

I don't really want to use connection tracking, but I see an example from wiki that marks connections to handle load balancing. Is there a better way to load balance without using connection tracking? How do you manage your WAN connections in such scale? This is my bottleneck right now.

normis!!! Please don't disappear on me.

Can you please please elaborate on how max-entries for connection tracking works? Is it NOT properly implemented?

Is it just like what macgaiver said that we shouldn't use connection tracking in large scale?

BGP is managing it for me

PCC is fine for client's network if you use NAT.

BGP is managing it for me

PCC is fine for client's network if you use NAT.

OTZ...

Thank you, but I guess BGP doesn't work for me. I honestly have no clue what BGP is besides it's a routing protocol like OSPF. I have never ever had my hands on BGP in my life. I don't think my local ISPs would ever let me use BGP or any other routing protocol.

Like I said, I would like to start small to subscribe a better ADSL/Fibre line and resale it.

I just had a quick look at my local ISP (Bell Canada). It's Business level doesn't say anything about BGP and I don't think I can afford Enterprise level of Internet. In other words, I need connection tracking with more than 500k.

I'm 99,999% sure you will be fine with 500k

I'm 99,999% sure you will be fine with 500k

Hahaha!!! Thank you very much. It's nice to have someone in the field saying so.



normis! Can you please confirm if it's a bug that can be fixed regarding to the 500k? Although I may not (0.001%) need it, it's still good to know it's there when I need it.

we use a hash function in our connection tracking table, bigger the table, bigger the size of the hash necessary to map all connections. Bigger hash size, slower the searches, more resources used etc.

So there is no point having bigger hash than you can possibly need. From our experience in support, number of connections is rarely the bottleneck, but all those cases were fixed with firewall raw table and "no-track" action.

I strongy suggest to test everything out before jumping to theoretical conclusions.

we use a hash function in our connection tracking table, bigger the table, bigger the size of the hash necessary to map all connections. Bigger hash size, slower the searches, more resources used etc.

So there is no point having bigger hash than you can possibly need. From our experience in support, number of connections is rarely the bottleneck, but all those cases were fixed with firewall raw table and "no-track" action.

I strongy suggest to test everything out before jumping to theoretical conclusions.

normis! Thank you very VERY MUCH. You answer has been very helpful.

My idea is just to prepare for the very worst case scenario. I will try my best to come up the most efficient firewall rules for my needs, and play with RAW table.

I'm 99,999% sure you will be fine with 500k

originally it was something like "I'm 99,999% sure you will be fine with 640k"

Well, now it's official. There's hard limit on the number of entries in connection tracking table: http://forum.mikrotik.com/viewtopic.php ... 10#p562410

So, no 16G can be used =)

Well, now it's official. There's hard limit on the number of entries in connection tracking table: http://forum.mikrotik.com/viewtopic.php ... 10#p562410

So, no 16G can be used =)

1M connections with conntrack on - amount of used RAM will be the less of your problems

there are many other ways, to use RAM, not only conntrack.... I still don't understand point of this topic...

I still don't understand point of this topic...

well, I know people who add new routers just because they approach 500k limit, not because the router is overloaded by CPU or bandwidth

well, I know people who add new routers just because they approach 500k limit, not because the router is overloaded by CPU or bandwidth

What is the source of such amount? DDoS?

popular p2p, I think

popular p2p, I think

You have to really try to get even in 5 digit number of connections per client using any current p2p.

popular p2p, I think

You have to really try to get even in 5 digit number of connections per client using any current p2p.

I have never reached 4 digits, even when trying hard.

Small home networks with one u torrent running machine easily reach 3-5 thousands connections. You definitely were not trying so hard.

Small home networks with one u torrent running machine easily reach 3-5 thousands connections. You definitely were not trying so hard.

OK, I have no idea how, but anyway, if you reach 1 million connections with connection tracking ON, you have other problems before the connection limit. If this is torrent, your bandwidth ran out long time ago anyway

That's true of course. But it is another topic... And anyway, normis, isn't here a rule that forbids repeating of the last post by citation?

Is there? If there are multiple posts above mine, I like to quote to be more specific, who I am responding to. Never know if somebody will post, while I am writing.

And anyway, normis, isn't here a rule that forbids repeating of the last post by citation?

I didn't know that was a rule - it does annoy me whenever someone quotes an entire long post over just a sentence or two....

Your typical posts tend to be one liners anyway, so Normis had no choice but to quote your entire post anyway.

You know that tapping in tapatalk on the phone is not so convenient therefore I am trying to be as short as possible. And I believe I saw a rule saying do not citate the last post. Well, maybe it was somewhere else, if it is not good behavior here, then never mind. But like zerobyte, I don't like it too.