have you tried pci passthrough of the NICs to the CHR? (if the ESXi host CPU supports it)Is it any perfomance tests for CHR working on vmware? I am trying to get 10gbps with simple routing + nat load balancing (per connection classifier feature) and I cannot get more than 5gbps. I use 8 core CHR ver 7.6-7.8 with vmxnet3 network cards at esxi 7. I don't see cpu bottleneck (no CHR cores with 100% usage) but when traffic is getting to 5gbps I see huge traffic drops down to 100-200mbit/s every 2-3sec. What is wrong?
I have several VmWare ESXi ( with physical 10-Gig network interfaces ) servers running several Mikrotik CHR routers ( with vmxnet3 network interfaces ). When I perform a Mikrotik CHR btest between my CHRs from one VmWare ESXi server to a different VmWare ESXi server across the 10-Gig physical network , I am always able to hit near 10-GIg with my btest(s).Is it any perfomance tests for CHR working on vmware? I am trying to get 10gbps with simple routing + nat load balancing (per connection classifier feature) and I cannot get more than 5gbps. I use 8 core CHR ver 7.6-7.8 with vmxnet3 network cards at esxi 7. I don't see cpu bottleneck (no CHR cores with 100% usage) but when traffic is getting to 5gbps I see huge traffic drops down to 100-200mbit/s every 2-3sec. What is wrong?
No. How can I configure it?have you tried pci passthrough of the NICs to the CHR? (if the ESXi host CPU supports it)
I have almost the same results when I perfom CHR btest between my esxi servers. That's why I think the issue may be related to nat or routing.have several VmWare ESXi ( with physical 10-Gig network interfaces ) servers running several Mikrotik CHR routers ( with vmxnet3 network interfaces ). When I perform a Mikrotik CHR btest between my CHRs from one VmWare ESXi server to a different VmWare ESXi server across the 10-Gig physical network , I am always able to hit near 10-GIg with my btest(s).
I use P10 license.- On your CHRs ; are you running the P unlimited license ?
Where and how can I configure it?- If you have more than one VmWare ESXi server , you might want to consider setting delayed_ack = 1
I tried it and didn't notice any significant changes.- Consider disabling hyper-threading.
I use Xeon CPUs with basic freq 2.1-2.8Ghz and turbo boost freq up to 3.9Ghz.I use Intel Xeon CPUs running at 3-GHz or faster on my VmWare ESXi servers to get my CHRs to run their fastest.
I am getting 250-300Gbit/s for such tests.On your CHR , perform a btest to 127.0.0.1
127.0.0.1 is a local interface on the CHR server.
On one of my slowest VmWare ESXi servers ( Xeon 2-something GHz with Hyper-Threading ) running a CHR doing a btest to 127.0.0.1 , I get over 200-GIg.
/ip firewall filter add action=reject chain=forward connection-state=new dst-port=25,2525,465,587,139,445 log-prefix=smtp-block protocol=tcp reject-with=icmp-admin-prohibited
/ip firewall mangle add action=mark-connection chain=prerouting connection-state=new dst-address=!XXX.XXX.XXX.168 in-interface=ether1 new-connection-mark=PCC-CA1 passthrough=no per-connection-
classifier=src-address:3/0 src-address=!XXX.XXX.XXX.0/24
/ip firewall mangle add action=mark-connection chain=prerouting connection-state=new dst-address=!XXX.XXX.XXX.168 in-interface=ether1 new-connection-mark=PCC-CA2 passthrough=no per-connection-
classifier=src-address:3/1 src-address=!XXX.XXX.XXX.0/24
/ip firewall mangle add action=mark-connection chain=prerouting connection-state=new dst-address=!XXX.XXX.XXX.168 in-interface=ether1 new-connection-mark=PCC-CA3 passthrough=no per-connection-
classifier=src-address:3/2 src-address=!XXX.XXX.XXX.0/24
/ip firewall nat add action=dst-nat chain=dstnat comment="PCC dst-nat -> nl5-v50-ca1" connection-mark=PCC-CA1 dst-address=!XXX.XXX.XXX.168 in-interface=ether1 to-addresses=XXX.XXX.XXX.148
/ip firewall nat add action=dst-nat chain=dstnat comment="PCC dst-nat -> nl5-v50-ca2" connection-mark=PCC-CA2 dst-address=!XXX.XXX.XXX.168 in-interface=ether1 to-addresses=XXX.XXX.XXX.145
/ip firewall nat add action=dst-nat chain=dstnat comment="PCC dst-nat -> nl14-v50-ca3" connection-mark=PCC-CA3 dst-address=!XXX.XXX.XXX.168 in-interface=ether1 to-addresses=YYY.YYY.YYY.18
/ip firewall nat add action=dst-nat chain=dstnat comment="dst-nat -> nl5-v50-ca1" dst-address=!XXX.XXX.XXX.168 in-interface=ether1 to-addresses=XXX.XXX.XXX.148
/ip firewall nat add action=masquerade chain=srcnat out-interface=ether1
When I perform btest between two CHR I get nearly 10gbps results. So I think there is no problem at vmware level. In those tests no nat is in use, that's why I think the isssue may be in nat. I am going to make tests over nat, but it requires to a build special test environment.To rule out that your local vmware environment is not the culprit, create a couple of test instances and make sure sr-iov is in place. Apply for a couple of free 60 days p10 test licenses and run carefully engineered tests. Before you start, verify the test equipment is working correctly by hooking them up directly to each other. If you still get poor throughput during the tests, it's time to systematically troubleshoot the root cause.
By the way, 70-90% of our prod traffic is udp. We are running DTLS protocol at our target virtual machines which is udp based protocol. And CHRs are used as nat load balancers. As I know, delayed_ack is related to TCP only.re: - If you have more than one VmWare ESXi server , you might want to consider setting delayed_ack = 1
re: Where and how can I configure it?
to enable delayed_ack = 1
/ip firewall mangle
add action=mark-connection chain=prerouting connection-mark=no-mark connection-state=new dst-address=XXX.XXX.XXX.58 in-interface=ether1 new-connection-mark=pcc1 passthrough=no per-connection-classifier=src-address-and-port:2/0
add action=mark-connection chain=prerouting connection-mark=no-mark connection-state=new dst-address=XXX.XXX.XXX.58 in-interface=ether1 new-connection-mark=pcc2 passthrough=no per-connection-classifier=src-address-and-port:2/1
/ip firewall nat
add action=dst-nat chain=dstnat comment=pcc1 connection-mark=pcc1 dst-address=XXX.XXX.XXX.58 in-interface=ether1 to-addresses=XXX.XXX.XXX.177
add action=dst-nat chain=dstnat comment=pcc2 connection-mark=pcc2 dst-address=XXX.XXX.XXX.58 in-interface=ether1 to-addresses=XXX.XXX.XXX.178
add action=dst-nat chain=dstnat comment="no pcc" dst-address=XXX.XXX.XXX.58 in-interface=ether1 to-addresses=XXX.XXX.XXX.177
add action=src-nat chain=srcnat out-interface=ether1 to-addresses=XXX.XXX.XXX.165
if you mean fast-tracking yes... all QoS relevant connections/packets must be fasttrack disabledCan I disable connection tracking feature if I use it for per-connection classifier with nat load-balancing? I thought I can not.
Also I have thousands of connections in my real user traffic and the last screen is for real traffic, not any perfomance tests like btest or iperf.