Hi,
we have installed a CHR realease of rouuteros on a vmware VM on a dedicated host phisical machine in our datacenter.
It acts as pppoe server on our network, 1850 subscribers active.
On peak hours, subscribers have packet loss when they ping hosts on the internet (se when they pass through the pppoe server). In theese moments we observe saturation of some of the virtual CPUs from the CPU usage table on “Resources” section of routeros.
We have excluded other kind of transport issues and the proof of this is that customers ping perfectly the pppoe server lan interface and the server (from terminal) ping smoothly the internet.
It seems CHR isn’t able to manage efficently the available cores (it apparently spreads the load but in un unbalanced way, leading to a saturation of cpu0 and cpu1). This way some of the vCPU remains partially offloaded. Performance graphs from vsphere show just 6000mhz of 27000mhz, used at most.
Can you find any incorrect settings in our configuration? We attach supout and exported config and others screens for your convenience

Thanks in advance and best regards

Some thoughts to lower the CPU load on your CHR. This is what I do on my CHR hosted on VMware ESXi.

1)- consider swapping out both of your Intel Xeon E5-2637 processors to an Xeon E5-2690.
I have found that CPU built-in cache is more important than CPU clock speed - especially when running tiny hosts which mostly run in CPU cache.

2)- It looks like you have hyper-threading enabled. Disable it. Hyper-threading cuts your build-in Xeon CPU cache in half and also slightly slows down the system because you have your Xeon processor emulating twice as many processors. This costs CPU throughput. Never use hyper-threading. Hyper-threading is only good for systems that can not be hardware upgraded where you need more CPUs (at the cost of slowing down the entire system to make it appear that it has more CPUs).

3)- There may be some issues with your 5 (five) network cards.
Each network card should use a unique CPU interrupt. If I remember correctly, you can go up to 4 networks card with unique interrupts. Five (5) network cards and higher will start using shared interrupts. Shared interrupts slow down the system because the operating system now has the added job of determining which device triggered an interrupt.

4)- Configure your VMware ESXi so that the next time you boot/power-on your CHR - that you end up going into the BIOS.
In the BIOS, disable/remote the following
serial ports
floppy drive
CD drive
Each of these in-necessary in-used devices has an interrupt associated to the device. There is no point in having the CHR check these devices to see if they generated an interrupt.

5)- network cards for CHR via VMware ESXi
--- If drop down to a max total of only 4 network cards !!!!!
--- If can use 802.1q vlans on your CHR - it works pretty well

6)- After all of the above ... Perform a btest ( UDP send ) to 127.0.0.1
With your older/slower Xeon processor, I would guess you should hit 2-gig to 11-gig.
My CHR best to 127.0.0.1 reports 17 to 19 Gig.
FYI - 127.0.0.1 is an internal IP address in your CHR. Kinda like a built-in internal virtual network card.
If you tweak and settings to improve your CHR, get a base-line btest to 127.0.0.1 prior to making any changes. The faster the btest to 127.0.0.1, the faster your CHR is running.

7)- If you have other virtual hosted machines on your VMware ESXi server ... then as a test , stop/shutdown the other servers then run a btest to 127.0.0.1 and see if there is any difference.

8 ) - You should always get much better/faster network throughput if you Intel 10-gig network cards

9)- On your VMware ESXi server, go into the BIOS on your physical machine. Delete/remove any devices you do not need (you want to keep the physical interrupts down so that your VMware ESXi physical server does not spend a lot of time processing interrupts - which in turn slows down everything it hosts). Disable all power management and set BIOS for maximum performance.

10)- VMware ESXi, edit your CHR settings (console - edit - Options - General ... then disable logging)

11)- If you know how to, convert any hosted machines on your VMware ESXi box "Hard disk #" from "thick" to "thin". Do this for all guest hosted machines on your physical VMware ESXi server.

EDIT:
12)- Every hosted guest machine and including the CHR should be using VMXNET-3 network cards. This is a paravirtual optimized network card to enhance network throughput and lower the CPU cost to operate the network interface.


Post something if anything here helps

North Idaho Tom Jones

Hi,
after your post I changed the ethernet driver to VMXNET-3 and a I can see a slight improvement. The strange thing is that I need to keep RPS enabled to be able to have the load spread over all the vCPUs. Without it only CPU0 and CPU1 are loaded, others remain idle. I’ve read it would be better to disable RPS when using vmxnet-3.

I’ve also phisically removed two of five ethernet cards.

Throughput tests to the loopback address are around 12gbps.

If I increase vcpu assigned to vm from 2 to 4 to 8 to 16 cpu inside ESXi, CHR performance get worst.
Mikrotik cumulative CPU usage (from Resources) remains abaut 95% while the value in the cpu summary of ESXi a slow as 20%. This is the most evident issue.
I would like to be able to see the usage of the phisical CPUs of the host macchine occupied a lot more than 20%

I guess that if I would be able to install routeros x86 over the phisical machine I’ll get all cpu cycles used but I have problem with raid support (it’s not recognized).

Thanks for your suggestion and tips. This way the load has been lowered of about 10% but the aim of my experimentation is take advantage of the whole power of the phisical machine.

Maybe you should consider a CCR1036-8G-2S+EM ?

I tried CCR1072-1G-8S +, 100% cpu with various continuous ppp disconnects, removed after 10 min For network instability

That sounds like the "known" problem of having so much processing associated with disconnects that the system overloads when users disconnect, causing users to disconnect due to the overload, resulting in an avalanche.
A couple of known causes:
- using of MASQUERADE instead of SRC-NAT
- using a routing protocol that creates routes for each individual IP (e.g. for fail-over)

scara -- install routeros x86 over the phisical machine
ROS x86 is OK , however in my environment , I found ROS x86 was prone to locking up.

I run the Mikrotik public accessible btest server ( 207.32.195.2 ).
In the beginning when I set it up, I was running x86 ROS. My physical VMware ESXi servers had/have an Intel 2-port 10-Gig SFP+ ports. My Internet feed is also connected via a 10-gig port and my Cisco switches are also 10-Gig. My 207.32.195.2 ROS x86 btest server could hit up to around 6-Gig to/from the Internet on btests. I experienced a problem almost on day-1 where my x86 ROS btest server would just lock up. All networking would halt - and I could not even ping 127.0.0.1 and-however the x86 ROS console was still talking. It hit the point that I would have to reboot the x86 ROS btest server almost every other day. (I am guessing the problem may be related somewhat that there were no paravirtulized Ethernet drivers).

Months later, I changed from x86 ROS (32-Bit) to CHR 64-bit. The CHR has never locked up as a btest server.

In conclusion, I found the following:
- For almost everything, x86 ROS (32-Bit) and CHR (64-Bit) have almost the same throughput ability
- As a btest server, x86 ROS is prone to random crashes
- As a btest server, CHR has proven 100 percent stable.

Now - re your possible decision to try/run x86 ROS, I would say "If it is not critical core, then give x86 ROS a try".

North Idaho Tom Jones

How does the cpu of the chr have an average of 90% and the cpu of the physical machine is at 20% when it's all devoted to chr?

Hi,
we have installed a CHR realease of rouuteros on a vmware VM on a dedicated host phisical machine in our datacenter.
It acts as pppoe server on our network, 1850 subscribers active.
On peak hours, subscribers have packet loss when they ping hosts on the internet (se when they pass through the pppoe server). In theese moments we observe saturation of some of the virtual CPUs from the CPU usage table on “Resources” section of routeros.
We have excluded other kind of transport issues and the proof of this is that customers ping perfectly the pppoe server lan interface and the server (from terminal) ping smoothly the internet.
It seems CHR isn’t able to manage efficently the available cores (it apparently spreads the load but in un unbalanced way, leading to a saturation of cpu0 and cpu1). This way some of the vCPU remains partially offloaded. Performance graphs from vsphere show just 6000mhz of 27000mhz, used at most.
Can you find any incorrect settings in our configuration? We attach supout and exported config and others screens for your convenience

Thanks in advance and best regards

Have you tried setting Latency Sensitivity: High on VM settings ?

Hi,
we have installed a CHR realease of rouuteros on a vmware VM on a dedicated host phisical machine in our datacenter.
It acts as pppoe server on our network, 1850 subscribers active.
On peak hours, subscribers have packet loss when they ping hosts on the internet (se when they pass through the pppoe server). In theese moments we observe saturation of some of the virtual CPUs from the CPU usage table on “Resources” section of routeros.
We have excluded other kind of transport issues and the proof of this is that customers ping perfectly the pppoe server lan interface and the server (from terminal) ping smoothly the internet.
It seems CHR isn’t able to manage efficently the available cores (it apparently spreads the load but in un unbalanced way, leading to a saturation of cpu0 and cpu1). This way some of the vCPU remains partially offloaded. Performance graphs from vsphere show just 6000mhz of 27000mhz, used at most.
Can you find any incorrect settings in our configuration? We attach supout and exported config and others screens for your convenience

Thanks in advance and best regards

Also, what NIC cards do you have ?
Maybe you should change nic driver kernel parameter in esxi so it should have more tx/rx queues.
Rps should be disabled, and multi queue int default should be set at nic interface.
Besides of that, check resources->irq (how many tx/rx queues of vmxnet3 avaiable).
I think 35% ethernet usage is too much...

I can help you with that.

To the original poster - did you ever try disabling Hyper-Threading ?

FYI: Hyper-Threading is a built-in function of many newer Intel CPUs which is a semi-software/semi-firmware/semi-hardware feature which makes a single CPU behave as if it were two CPUs. If you have a multi-core processor, then Hyper-Threading makes the CPU count behave as if it had twice as many CPUs.

Hyper-Threading uses internal times CPU interrupts which act like this:
#1 run Virtual CPU #1 (of 2)
#2 a few moments later , stop Virtual CPU #1 (of 2)
#3 copy the contents of the CPU registers used for Virtual CPU #1 (of 2) to a 1st memory location (aka Pop the stack)
#4 copy the contents of a 2nd memory location to the CPU registers to be used by Virtual CPU #2 (of 2) (aka Push the stack)
#5 run Virtual CPU #2 (of 2)
#6 a few moments later , stop Virtual CPU #2 (of 2)
#7 copy the contents of the CPU registers used for Virtual CPU #2 (of 2) to a the2nd memory location (aka Pop the stack)
#8 copy the contents of the 1st memory location to the CPU registers to be used by Virtual CPU #1 (of 2) (aka Push the stack)
#9 Jump to #1 (and continue looping #1 -through #9)

The issue here is that both Virtual CPUs have a combined total processing power of just slightly less than a single non Hyper-Threading CPU. This is because of the following:
- The CPU has to Pop and Push the stack non-stop. This takes away CPU processing time that could of been used doing something wanted.
- The CPU built-in Cache is constantly re-learning & re-populating the Cache memory. The memory contents being cached is now being shared and re-written. With a non Hyper-Threading CPU, the CPU Cache has a better chance for Cache Hits. CPU Cache runs at CPU speed. When there is a Cache Hit, the processor can process at CPU speed. When there is a Cache Miss, the CPU has to slow down to memory speed (and any possible wait-states imposed on memory). So ... By having as many Cache Hits as possible, you actually can run much faster. If the operating system memory and/or program used is small enough, it is possible to achieve a near 100 percent Cache Hit system. This can really make a system go much much faster.

There are Pros and Cons to Hyper-Threading. If you plan out your hardware and software/programs correctly, a non Hyper-Threading system can be measurably much faster.


FYI - One of the reasons I prefer newer Xeon processors with the largest amount of CPU Cache is this:
A - A slower Xeon processor with lots of CPU Cache can give you a near 100 percent Cache-Hit system running almost always at CPU clock speed.
B - A faster Xeon processor with a small amount of CPU Cache might give you an almost always Cache-Miss system which results in the CPU processing memory at memory + memory-wait-state speeds.

In general, when selecting components for a bad-ass computer, I always prefer the Xeon processor with the largest amount of CPU Cache then clock speed 2nd.

North Idaho Tom Jones

We are having the same problem where users are getting slow service and drops over 3000 customers and huge drops over 3000 plus users.
CHR using 40% CPU and host CPU utilization not exceeding more than 65% cpu if 2000 plus customers are connected results are fine but over 3000 users CPU and DATA utilization on CHR will be remain same and got slowness in services and huge ping drops.

We are having the same problem where users are getting slow service and drops over 3000 customers and huge drops over 3000 plus users.
CHR using 40% CPU and host CPU utilization not exceeding more than 65% cpu if 2000 plus customers are connected results are fine but over 3000 users CPU and DATA utilization on CHR will be remain same and got slowness in services and huge ping drops.

Along with all of the above items I have posted here ...
If you CHR is running on a Hyper-Visor (aka something like VMware ESXi), there are some additional tricks you can do:
- On the physical BOX, remove all devices not needed. (CDROM, Serial Ports, Parallel ports, Floppy drives ...)
- On the physical BOX, in the BIOS setup, disable all non-necessary devices (CDROM, Serial Ports, Parallel ports, Floppy drives ...)
- On the physical BOX, in the BIOS setup, set all settings to performance instead of the default slower-power-save settings.
- On the physical BOX, when ever possible, try not to have any two devices use the same CPU interrupt. Example, two network cards using the same interrupt forces the host operating system to spend time trying to figure out what device needs attention - this takes away from CPU processing speed to do other things.
- On the physical BOX HyperVisor, convert all virtual drives to THIN for all virtual machines.
- On the physical BOX HyperVisor, disable logging.
- On the physical BOX HyperVisor, try operate two physical boxes. One box for your essential fast virtual stuff and a second physical box for all other non-essential guest operating systems that do not need to be top-priority.
Also ...
- On the Virtual box, remove all virtual devices not needed. (CDROM, Serial Ports, Parallel ports, Floppy drives ...)
- On the Virtual BOX, in the virtual BIOS setup, disable all non-necessary devices (CDROM, Serial Ports, Parallel ports, Floppy drives ...)
- When possible, use ParaVirtual device (device-derivers). ParaVirtual devices are virtual devices where the drivers are optimized to perform faster/better when running in a HyperVisor environment. An examples of a ParaVirtual devices on VMware ESXi are; "VMXNET-3" network cards - and "VMware Paravirtual" SCSI Controllers
- When possible, if you have a 10-Core Xeon CPU (with Hyper-Threading disabled), the number of assigned CPUs to all guest hosted operating systems should be 9 CPUs or less. Avoid over-subscribing the CPUs and Memory. Over subscription of CPUs and Memory causes the physical HyperVisor to start swapping.

One thing I think is possible to make a really fast CHR would be to try and do this:

Convert virtual CHR system to a physical box (no hypervisor - the CHR is the only booted operating system).

also - somewhat related
In a ROS or CHR-ROS, avoid the use of bridges when ever possible. Software bridges take away from the CPU to do other things. If you need to bridge a network, use a physical Ethernet switch. If possible, put IP address directly on an interface and not on a bridge. Also, the more fire-wall filters you have, the more it slows everything down.

Also - a decent CHR or x86 ROS system should be able to perform a "UDP send" tools-btest to 127.0.0.1 and achieve and hold a rate faster than 17-Gig !!! If you can't hit that mark, then you have a slow physical box.

And also - if possible , avoid using 1-Gig Ethernet interfaces. A high-throughput data center should be using 10-Gig switches and 10-Gig Ethernet interfaces on everything.

Dear Tom,

Yes i knew about Vm tweaking infarct we have used specialized high end computing network card but still no luck.
one of my teak hits my Host CPU 100% with 2G but condition was same timely more users are get connected and data did not cross more than 2G.
one more issue i observe that i got message that 1522 can not be send to over 1500 MTU on my host as CHR L2 is 0 and Mikrotik disabled edit option in this section.
I dont know why they did not set 10G L2 MTU over 10G sync interface they just hard coded it to 0 which is completely unpredictable.

i have used btest with three of CCR's and got easily 3G over udp but with user load its sucks as i mention above.

After second tweaks i got low CPU usage on host but its down CHR throughput as well after my last tweaks CHR even drop data over 1400 to 1500 Mbps.
I have also CCR1072 which is also similar issues and that also reported by other users as well in production environment.
I was hope that CHR might not such issue as its claimed by Mikrotik that its virtualization aware/tweak appliance.

After second tweaks i got low CPU usage on host but its down CHR throughput as well after my last tweaks CHR even drop data over 1400 to 1500 Mbps.
I have also CCR1072 which is also similar issues and that also reported by other users as well in production environment.
I was hope that CHR might not such issue as its claimed by Mikrotik that its virtualization aware/tweak appliance.

This is somewhat interesting.
I have been able to achieve faster than 8 Gig sustained when routing between a virtual CHR hosted on one VMware ESXi machine talking to a virtual x86 ROS hosted on a totally different VMware ESXi machine --- with two other x86 virtual servers performing a btest where the btest sessions were layer 3 routed.
VMware ESXi server #1 btest server (connected to 10-Gig Ethernet to #2
VMware ESXi server #2 CHR routing non-natted LAN from server 1 to server 4 (via #2)
VMware ESXi server #3 x86 ROS routing non-natted LAN from server 4 to 1 (via #2)
VMware ESXi server #4 x86 ROS system performing the btest to x86 ROS on server #1

Everything is 10-gig
All x86 ROS and CHR systems hosted on four different VMware ESXi physical systems.
No NAT - No Firewalls

The btest sessions are routed from #4 to #1 (no btest sessions running on the same network - thus forcing routing)
I was able to hold a sustained UDP send or receive faster than 8 gig.

North Idaho Tom Jones

Sheezzzeee

I am totally convinced that VMware running on a XEON E5410 @2.33 GHz (2 physical CPUs with 4-Cores with Hyper-Threading disabled) on a Dell PowerEdge 2950 with 16 gig of RAM is the slowest thing I have ever encountered. It takes forever to do anything.

My SuperMicros with XEON E5-2960 V2 processors running at 3.00 GHz totally scream crazy fast compared to the Dell with the XEON E5410 .

I don't think I have a mis-configuration. I would suggest anybody using a XEON E5410 (version 1), to dump the physical box and use a faster box for their HyperVisor to run on. Especially if you are running a virtual router such as CHR or x86 ROS.

North Idaho Tom Jones

Dear Tom/ Experts,

You may achieved that throughput on L3 traffic with Btest which i can also get approx 3G plus with our Three CCR as client and CHR as server but in real production environment CHR is very disappointed me over tunnel traffic.
I have tested PPTP as well now and pppoe as i said earlier CHR drops packets in both tunnel wih PCQ and Dynamic Simple Queue as well
i have checked hypervisor software where we do not observe a single packet drop on main interface observed on Esxi HOST.
Furthermore i am also using hp g8 series with dual octa core processors.

Dear Tom,

Kindly share your VMX config file if its possible for you.

Dear Tom,

Kindly share your VMX config file if its possible for you.

I assume, that on my SuperMicro box running VMware ESXi (version 6.00) you want my
-> local datastore1 (sas internal raid drive array)
--> CHR directory
---> CHR.vmx file

It contains the following information:

.encoding = "UTF-8"
config.version = "8"
virtualHW.version = "11"
nvram = "CHR.nvram"
pciBridge0.present = "TRUE"
svga.present = "TRUE"
pciBridge4.present = "TRUE"
pciBridge4.virtualDev = "pcieRootPort"
pciBridge4.functions = "8"
pciBridge5.present = "TRUE"
pciBridge5.virtualDev = "pcieRootPort"
pciBridge5.functions = "8"
pciBridge6.present = "TRUE"
pciBridge6.virtualDev = "pcieRootPort"
pciBridge6.functions = "8"
pciBridge7.present = "TRUE"
pciBridge7.virtualDev = "pcieRootPort"
pciBridge7.functions = "8"
vmci0.present = "TRUE"
hpet0.present = "TRUE"
memSize = "2048"
sched.cpu.units = "mhz"
powerType.powerOff = "soft"
powerType.suspend = "hard"
powerType.reset = "soft"
ethernet0.virtualDev = "vmxnet3"
ethernet0.networkName = "Vlan810-Mikrotik-Public-BTest"
ethernet0.addressType = "generated"
ethernet0.uptCompatibility = "TRUE"
ethernet0.present = "TRUE"
displayName = "CHR"
guestOS = "other-64"
guestOSAltName = "CHR"
toolScripts.afterPowerOn = "TRUE"
toolScripts.afterResume = "TRUE"
toolScripts.beforeSuspend = "TRUE"
toolScripts.beforePowerOff = "TRUE"
uuid.bios = "56 4d ec 25 a3 3b a7 b5-66 73 e8 d3 97 34 d9 62"
uuid.location = "56 4d ec 25 a3 3b a7 b5-66 73 e8 d3 97 34 d9 62"
vc.uuid = "52 5a 39 14 02 7d 52 70-ab 99 0f 8c e3 bd 55 e3"
chipset.onlineStandby = "FALSE"
sched.cpu.min = "0"
sched.cpu.shares = "normal"
sched.mem.min = "0"
sched.mem.minSize = "0"
sched.mem.shares = "normal"
floppy0.present = "FALSE"
logging = "FALSE"
bios.forceSetupOnce = "FALSE"
virtualHW.productCompatibility = "hosted"
sched.swap.derivedName = "/vmfs/volumes/5515bf3c-885863e6-0ddf-90e2ba7d01e0/CHR/CHR-e277234d.vswp"
replay.supported = "FALSE"
replay.filename = ""
migrate.hostlog = "./CHR-e277234d.hlog"
pciBridge0.pciSlotNumber = "17"
pciBridge4.pciSlotNumber = "21"
pciBridge5.pciSlotNumber = "22"
pciBridge6.pciSlotNumber = "23"
pciBridge7.pciSlotNumber = "24"
ethernet0.pciSlotNumber = "160"
vmci0.pciSlotNumber = "32"
ethernet0.generatedAddress = "00:0c:29:34:d9:62"
ethernet0.generatedAddressOffset = "0"
vmci0.id = "-1758144158"
monitor.phys_bits_used = "42"
vmotion.checkpointFBSize = "4194304"
vmotion.checkpointSVGAPrimarySize = "4194304"
cleanShutdown = "TRUE"
softPowerOff = "TRUE"
ide0:0.fileName = "hdd-chr-6.37.3.vmdk"
ide0:0.present = "TRUE"
ide0:0.redo = ""

I just thought of something else I do on my VMware ESXi servers - which I don't think I have ever posted here.

I also change/use "delayed ACK" on my VMware ESXi hypervisor servers.
You may want to read about it.

I change the delayed ACK default settings on my all of my VMware ESXi hypervisors, and on all of my FreeNAS servers and on all of my PfSense servers.

Example; for me - changing the delayed ACK on my VMware ESXi servers and on my FreeNAS servers (I use NFS mounted datastores and local datastores), I am able to sustain and hold 8-Gig datastore copy speed between my local VMware ESXi datastore and my NFS mounted datastore on my FreeNAS server.

My CHR virtual system is running on a VMware ESXi server that has the modified delayed ACK.

I think you can find some information here:
http://www.stuartcheshire.org/papers/NagleDelayedAck/

delayed_ack=0 responds after every packet (OFF)
delayed_ack=1 always employs delayed ack, 6 packets can get 1 ack
delayed_ack=2 immediate ack after 2nd packet, 2 packets per ack (Compatibility Mode)
delayed_ack=3 should auto detect when to employ delayed ack, 4packets per ack. (DEFAULT)

Here is a command you can use try on your VMware ESXi server that does not stick with a reboot.

vsish -e set /net/tcpip/instances/defaultTcpipStack/sysctl/_net_inet_tcp_delayed_ack 0

The "0" changes the system so that it does not ACK after every packet. It instead will sent ACKs after multiple packets.

There are some huge speed gain advantages and some not-so-good things that can and will happen when you change the default delayed ACK settings.

Most systems use a default delayed ACK best suited for multi-multi-multi Internet wide hops. The default delayed ACK is designed to work better with remote and/or more problem prone possible packet dropping networks. It ACKs ever packet. This can slow high speed links because you are relying on the remote device to ACK every packet and forcing you to wait on every packet send while the ACK gets sent back to you via every hop it has to go through. This can be desired if you want to ensure and verify every packet makes it to/from the remote destination prior to sending the next packet.

With a modified delayed ACK, the system can send several packets and later receive a summary ACK that everything made it to the remote location OK. With delayed ACK, more packets get to the remote destination faster because you are not waiting around to receive an ACK prior to sending the next packet. A potential problem (which I have never encountered) is that it is more possible to get data corruption.

Example:
If you want the most reliable NAS system you can build is mounted on a VMware ESXi server, then use ISCSI and do not change the default delayed ACK settings. (((FYI - ISCSI also uses ACK on everything - so you end up with ISCSI protocol ACKs on everything to/from your remote datastores - which slooooowwwwwsss you down but gives you a much higher assurance the data made it correctly.
If you want a very fast (still reliable) NAS system, then use delayed ACK "0" on both servers and use NFS instead. ((The NFS protocol does not ACK like the ISCSI protocol))

If this turns on any interesting lights and proves to offer something you are interested in - then I would welcome some Mikrotik Reputation points


Anyways - I hope any and all of the information I have posted here and prior on this subject helps.

North Idaho Tom Jones

Additional note re delayed ACK setting of 0 (zero)

Every server on my entire ISP server network has been modified to use delayed ACK (zero).

I have found it improves throughput for all server to server and server to client communications.
It keeps traffic flowing much much much faster.
And - as far as I know of - I have never encountered a single problem using this setting in the last 20 years of doing so.

EDIT : If you use VMware ESXi, or and derivative BSD or Unix or Linux, I highly recommend you give it some testing and see how much you network can improve.
The default delayed ACK setting was designed back when the Internet was much much much slower and far less reliable.
With todays modern/faster networks (dial-up, 10 meg, 100 meg, now operating at 1-gig and 10-Gig), the default delayed ACK may be almost an obsolete & network hindering settings.
On some data transfers through the network (huge file copies between servers) , I used to get as slow as 300 meg sustained - now I can hit and sustain 8-Gig.
Huge terra-byte plus data transfers that used to take longer than a week for a single copy now normally take me the time to go get a cup of coffee and have a brief conversation with somebody.
The processing speed of the server & client computers are the only things that keep the network from not going faster.

EDIT-again: Thinking about it after my post, I now wonder what the ROS delayed ACK settings are configured for in their derivative of Linux used to build the Mikrotik ROS ???
( and - Does Mikrotik ROS use John Nagle's "Nagle algorithm" with ACK ? (Congestion Control in IP/TCP Internetworks ))

EDIT-again: I suspect that if delayed ACK 0 (zero), was or could be used in the transmission of TCP in/on an AP and all wireless clients, it could cut/reduce the number of packets sent & received over the wireless network by almost half !!! This could then possibly result in more packets containing data to get through a wireless network faster - resulting in a faster wireless network !!!

North Idaho Tom Jones

Here are some additional tips/tricks/procedures to use to help speed up the throughput of a ROS CHR (or any Mikrotik ROS device).

When ever possible - do the following:

- Avoid the use of software bridges
..Software bridges consume CPU resources to software connect two different interfaces
..Software bridges force the ROS system to stop and copy a packet from one interface to another interface
..When there is a large network bandwidth load, software bridges will slow down everything and drive up the CPU load

- Use hardware switches
.. Most/many Mikrotik ROS devices now have a physical Ethernet switch chip on the motherboards
.. If you need to connect two connect two or more networks together, use the Mikrotik switch chip or an external physical switch
.. Hardware switching networks (using the on-board switch chip -or- physical external Ethernet switch) places 0 ( zero ) CPU load on the ROS operating system
.. Hardware switching will speed up your network and dramatically lower the CPU load

- Put IP address directly on a physical device instead of a virtual device (a bridge is a virtual device).
.. When you put an IP address on a bridge, the ROS CPU has to software copy the packets to the other devices on the bridge. This slows down the system, drive up CPU load.

- Reduce the number of firewall rules to the bare essentials needed and nothing more
.. When you have a large number of firewall rules, data packets are processed by the ROS system one rule at a time to the next rule until it finds a PASS rule.
.. Excessive use of firewall rules slows down throughput and drives up the CPU load

- Move customer firewall rules and customer NAT functions away from your core ROS system and put those firewall rules and NAT configurations at each client ROS device
.. The combined ROS processing power of thousands of client ROS devices is far greater/faster than any single core router
.. When a single core router is performing the all firewall rules and all NAT functions for your clients, all clients run slower because the core ROS system is CPU busy

- Watch and monitor your CPU load on you core equipment
.. Higher CPU loads will always result in slower throughput
.. I consider a CPU load of 20 percent as --> needs attention - fix me or replace/upgrade me
.. I consider a CPU load of 30 percent as --> broken

- Turn off all non essential services that are not actually being used or needed
.. Services place an additional CPU load on any system - which results in the system running slower
.. Remote scans to your core and remote login attempts of services to your core and dos attacks to your core will place a higher CPU load on your core and slow down all processing and slow down system throughput

- If a service can be moved off of your core ROS system to another system , then move that service to a 2nd system
.. Example, if you have a core ROS system that is also a DHCP server, then move the DHCP server to a 2nd system and free up core system for faster throughput
.. Example, if you have a core ROS system that is also a captive-portal / walled-garden , then move that service to a 2nd system and free up the core system for faster throughput
.. Example, if you have a core ROS system that is also performing bandwidth rate-limiting to your clients, then move your bandwidth management system to a 2nd system and free up the core system for faster throughput
.. Example, if you have a DNS service or DNS forwarder or NTP service or NTP forwarder on your core ROS system, then disable those services and have all client devices communicate to a 2nd device which will free up the core system and also give you faster throughput
.. Example, if you have a client authentication system/service on your core system, move those systems/services to another 2nd system which will free up the core system and also give you faster throughput
.. Example, turn off the ROS graphing service. This slows down your system and drives up the CPU load. If you want bandwidth traffic graphs then use a 2nd system to SNMP graph your switches.
.. Example, turn off the ROS BTest Server. This is just another service that requires some CPU time to keep the service running. If you need to btest, temporarily turn it on and use it then turn it back off
.. Example, look at your ROS IP-Services and disable anything you do not need or use. Un-needed and un-used services still require some CPU time to keep the service running - and these IP services could be used by remote automated attempted logins/attacks - which drive up the CPU and slow down throughput
.. Example, consider disabeling/turning-off all ROS logging. This is a service which drives up the CPU load and slows down throughput
.. Example, If have a VPN service or any types of tunneling service, move those services to a 2nd server and free up your core router network for faster throughput
.. Example, do not use a MetaROUTER on your core network. This will drive up the CPU load and slow down throughput
.. Example, disable any ROS System Packages that are not needed or being used.
.. Example, if your core ROS router does not need/use wireless devices, then disable the System Package wireless system.
.. Example, disable any physical interfaces that are not being used
.. Example, move all Queues (simple queues) to a 2nd system or client ROS devices and free up your primary ROS core system so that it will have faster throughput and a lower CPU load
.. Example, if you use BGP or OSPF, consider moving those services to other systems so that you can free up your core ROS system for faster throughput and have a lower CPU load
.. Example, in IP-->Neighbors-->Discovery_Interfaces, disable everything (do this system wide on everything everywhere. This drives up CPU load and adds to network traffic. Only use this when needed then again disable it when not needed


- Use 10-gig network cards instead of 1-gig
- Use 1-gig network cards instead of 100 meg
- Use 100 meg network cards instead of 10 meg
.. Faster network cards hugely speed up propagation delay and will always give you faster throughput
.. Even if you only have a bandwidth load that is only 10 percent of what the Ethernet network card operates at, the next faster speed of network card will reduce the in-transit propagation time-delay for the packet to get from one network device to another network device

- Use switches instead of old hubs
.. hubs are a decades old device. All network hubs should be thrown in the trash and replaces with modern switches
.. hubs are prone to data collision
.. switches are a hardware solution which reduces or totally stops network collisions
.. hubs slow down a network
.. switches speed up a network

- From time to time, disable and turn off all bandwidth services and let the entire network run as fast as possible. Then look for the bottle necks and fix those issues.
.. Every month, I turn off all my client bandwidth systems and look to see where/what the problems are
.. Fix/upgrade/repair all bottle necks in you network

- Disable STP if it is not needed
.. STP and RSTP place a greater CPU load on the system and slows down system throughput
.. STP and RSTP can and will inject pauses in your network when something on/in your network is added or changed. These pauses in network traffic can last from seconds to minutes
.. If you have a multi-path (two paths or more) connection between any two points in your network, then use STP or RSTP only for that part of your network


I can think of hundreds of additional items to speed up a core network - but I think the above topics are some of the most important items that should always be addressed in any network.

I hope this information helps at least somebody ...

North Idaho Tom Jones

Dear Tom Jones,

Can you please tell me the same command for esxi 5.xx as your provided command is not working on esx 5.1
vsish -e set /net/tcpip/instances/defaultTcpipStack/sysctl/_net_inet_tcp_delayed_ack 0
error
VSISHPath_Form():Extraneous 'instances' in path.

Dear Tom Jones,

i have found this option for specific iscsi adapter as i am not using any iscsi storage i am only using internal storage of server.
Furthermore we have only one aim to reach maximum throughput with maximum pppoe tunnels along with dynamic simple queues.
Do you really think that this delay ack helps in throughput of CHR with lower CPU usage ...???? OR do you have any suggestion to achieve my aim

Dear Tom Jones,

sorry to mention that we have 10 Gb enabled network devices and using also 10 g Nic's over CHR.

Dear Tom Jones,

Can you please tell me the same command for esxi 5.xx as your provided command is not working on esx 5.1
vsish -e set /net/tcpip/instances/defaultTcpipStack/sysctl/_net_inet_tcp_delayed_ack 0
error
VSISHPath_Form():Extraneous 'instances' in path.

I have not worked with 5.x for a long time now and can't find any old notes on it.
Take a look at this URL I had in some of my notes:
http://eksafha.blogspot.com/2014/06/nfs ... d-ack.html
Although it refers to NFS storage and ESCi 5.5 TCP delayed ack , I think the procedure on the VMware ESXi server remains the same to set a delayed ACK 0 ( zero )

clip from the web site:

NFS storage and ESXi 5.5 TCP delayed ack
By default the behavior of ESXi 5.5 is to have TCP delayed_ack set to 1
It caused some performance issues related to latency between the ESXi host and our NFS storage.
We need to change the behavior by setting the value to 0 , so that latency issues are solved.
vsish -e set /net/tcpip/instances/defaultTcpipStack/sysctl/_net_inet_tcp_delayed_ack 0
In order too ensure that delayed _ack is disabled across reboots we need to edit /etc/rc.local.d/local.sh and add the following line in there.
vsish -e set /net/tcpip/instances/defaultTcpipStack/sysctl/_net_inet_tcp_delayed_ack 0

North Idaho Tom Jones

Dear Tom Jones,

i have found this option for specific iscsi adapter as i am not using any iscsi storage i am only using internal storage of server.
Furthermore we have only one aim to reach maximum throughput with maximum pppoe tunnels along with dynamic simple queues.
Do you really think that this delay ack helps in throughput of CHR with lower CPU usage ...???? OR do you have any suggestion to achieve my aim

To be quite honest - in my almost 30 years of IP networking, I have never used or configured pppoe tunnels.

However ...
A few years ago ... , I was an Upstream Internet provider to two other ISPs who both used Mikrotiks for their pppoe tunnels. We had a conversation about my system-wide Mikrotik network configuration and their system-wide Mikrotik (pppoe) configuration.
Everything on my network had low CPU usage.
Everything on their network had high CPU usage.
My network was about 2x or 3x larger.
My network was using (and still is using) PfSense for bandwidth throttling , dhcp , captive-portal & client authentication
Their network was pure Mikrotik.
My network was much faster to/from clients than my clients ISP pppoe network was to their customers.

I wish I could be of more help re pppoe but I just don't know it and don't see any valid reason to use it for anything in my ISP network.

North Idaho Tom Jones

re: Do you really think that this delay ack helps in throughput of CHR with lower CPU usage ...????

For TCP/IP traffic through the CHR, I don't think delayed ACK 0 ( zero ) will help. I also don't think it would hurt.

For TCP/IP traffic to your CHR that terminates on the CHR -and- for TCP/IP traffic from your CHR to a remote destination, I think it helps. ((( That's my point of view )))

Also ...

Re all of the postings I recently made
Here is another tip re VMware ESXi and hosted virtual machines.
1st - I always run "thin" hdd drives to my virtual machines. This appears to greatly speed up all disk I/O transfers from the Virtual hosted machine to hdd storage.
2nd - If a host had a "thick" hdd, I always convert it to "thin"
3rd - If the virtual host supports it , I choose the following:
- 3A , do not use IDE virtual drives. IDE is old school hdd technology
- 3B , For SCSI virtual drives on a virtual host, I always change from the default "SCSI controller" "LSI Logic xx" to a "VMware Paravirtual" SCSI controller. This is an optimized software/firmware/driver made specifically to reduce CPU loads and provide improved throughput. Using as many Paravirtual devices as possible (on all virtual hosts) on VMware ESXi speeds up all hosts.

Some of this was not easy to learn or do the first time. You may want to google it some.

North Idaho Tom Jones

By-The-Way ...

If anybody here thinks I am an expert on this stuff - I have news for you...

- My 40+ years ago college was for electronics not computers. Computers were the size of many rooms back then.
- I am NOT certified in anything Mikrotik or Cisco or Unix/Linux or Windows. However I have worked on several very-very-very large computer networks for almost 40 years now - and - I read & research & try stuff all of the time.

So I am not speaking/posting about stuff I am formerly trained in, I pretty much go with experience and what I have learned on my own. And , If I don't know something then I read & learn all about it.

So , please don't hold me/my-postings to be always 100 percent correct. I make mistakes just like everybody else. However - I do my best, learn from my mistakes and always try to help others.

None of us here reading these postings would be here if we did not totally love what we do and like what we work with

North Idaho Tom Jones

I tried CCR1072-1G-8S +, 100% cpu with various continuous ppp disconnects, removed after 10 min For network instability

Are you updating your igp or even worse egp with every pppoe connect/disconnect? If so this is where all your problem lie.
filter out the samll ppp prefixes and just anounce a aggregated one to your backbone and the routingdeamon's will be quiet and not hog one cpu.

If you make e CCR1072 to it's knees there is something in your config that is suboptimal.

Dear JimmyNyholm,

Nops we dont its stub network no IGP , EGP used as we have isolated core and distribution network and on core we use EGP only nothing else.
Furthermore pppoe is installed on edge and pppoe wan side is our distribution layer none of server is directly connected to core and one of our mate experienced CCR 1072 more less performance in production and he immediately shifted back to his clients to CCR-1036 -4S and thats the main reason to look forward to CHR.

Dear Tom Jones / Jimmy
Please inbox me your emails i will share more interesting results with you which i am sure you will never ever seen with Mikrotik devices.

Dear Tom Jones,

Can you please tell me the same command for esxi 5.xx as your provided command is not working on esx 5.1

ESXi 5.0/5.1:
vsish -e set /net/tcpip/sysctl set _net_inet_tcp_delayed_ack=0

ESXi 5.5:
vsish -e set /net/tcpip/instances/defaultTcpipStack/sysctl/_net_inet_tcp_delayed_ack 0

Another tip for max throughput on almost any system.

This is likely not even close to being Mikrotik endorsed or Mikrotik recommended. But they give you the tools to do such a thing.

This applies to your core Mikrotik and all client Mikrotik devices everyhere...

- If the clock speed of the Mikrotik is already at the fastest speed available - then leave it there.
- If it is NOT at the fastest speed available, then increase it to the 2nd to fastest speed available.(not the fastest).

-And the obvious , all Mikrotiks should be running a newer version of ROS. From time to time, when you upgrade your core Mikrotik, also upgrade your entire client base of Mikrotiks.

---> A fast core also depends of fast clients and everything between <----

I am well aware I have posted an almost overload of things to speed up a system. I practice all of it on all my networks all of the time.

North Idaho Tom Jones

Dear JimmyNyholm,

Nops we dont its stub network no IGP , EGP used as we have isolated core and distribution network and on core we use EGP only nothing else.
Furthermore pppoe is installed on edge and pppoe wan side is our distribution layer none of server is directly connected to core and one of our mate experienced CCR 1072 more less performance in production and he immediately shifted back to his clients to CCR-1036 -4S and thats the main reason to look forward to CHR.

Dear Tom Jones / Jimmy
Please inbox me your emails i will share more interesting results with you which i am sure you will never ever seen with Mikrotik devices.

OK - here ya go - I made this email just for this < NorthIdahoTomJones-WannaBeGodOfMikrotikNetworks-LOL-ThisIsBig@NorthWestUSA.com >
And yes - this is a real email address, I just created it on my mail server.

Dear thanks for this email address i will send you some stuff by tomorrow.
Lolz great email address