Page 1 of 1

ROS v7 - SERVER with 2 interfaces 40G - Melanox CX354A

Posted: Mon Jul 25, 2022 2:00 am
by lbenalcazar
Hello everyone, I would like to know if anyone has ever been able to install RouterOS on a server with 40G cards for CG-NAT since the 1072 team is not enough when performing CG-NAT and interface bonding.

I think that if the processor does not need to bond, it will work without too much load.

Thank you very much for your help.

Re: ROS v7 - SERVER with 2 interfaces 40G - Melanox CX354A

Posted: Mon Jul 25, 2022 2:42 am
by chechito
with ccr1072 have you tried fast-track?

i think ccr1072 can do al least 20g of cg-nat with fast-track, ccr1036 can do 10g of cg-nat with fast-track without problem

if with fast-track ccr1072 still not able to do cg-nat i think is better to divide the task of 40g of cgnat in 4 10g cg-nat on 4 x ccr1036

Re: ROS v7 - SERVER with 2 interfaces 40G - Melanox CX354A

Posted: Mon Jul 25, 2022 2:57 am
by jspool
ROS v7 bare metal install on Dell R730 with 40G & 50G Mellanox ConnectX-4 series works great. I have not tried any ConnectX-3 series.

Re: ROS v7 - SERVER with 2 interfaces 40G - Melanox CX354A

Posted: Mon Jul 25, 2022 5:49 am
by lbenalcazar
with ccr1072 have you tried fast-track?

i think ccr1072 can do al least 20g of cg-nat with fast-track, ccr1036 can do 10g of cg-nat with fast-track without problem

if with fast-track ccr1072 still not able to do cg-nat i think is better to divide the task of 40g of cgnat in 4 10g cg-nat on 4 x ccr1036
Splitting is a good option, but I wanted to do it on one team.

On the other hand I think that 1072 has the potential for 20G CGNAT but the problem is its 10G interfaces, to join the traffic I would do it through bonding but bonding uses more CPU resources which makes it inefficient.

I think the 1072 should have been designed with 25G or 40G interfaces.

Re: ROS v7 - SERVER with 2 interfaces 40G - Melanox CX354A

Posted: Mon Jul 25, 2022 5:52 am
by lbenalcazar
ROS v7 bare metal install on Dell R730 with 40G & 50G Mellanox ConnectX-4 series works great. I have not tried any ConnectX-3 series.
Thank you very much, I think I should try it, I guess you put two CPUs on the server, and how many threads was each one?

Re: ROS v7 - SERVER with 2 interfaces 40G - Melanox CX354A

Posted: Mon Jul 25, 2022 6:24 am
by jspool
ROS v7 bare metal install on Dell R730 with 40G & 50G Mellanox ConnectX-4 series works great. I have not tried any ConnectX-3 series.
Thank you very much, I think I should try it, I guess you put two CPUs on the server, and how many threads was each one?
The one I am testing has 2 x E5-2680 v3 for a total of 24 cores.

Re: ROS v7 - SERVER with 2 interfaces 40G - Melanox CX354A

Posted: Fri Nov 10, 2023 1:21 am
by PortalNET
Hi guys

Sorry to bump on a old thread.. but i am having the following issue...

we have setup 2 dell servers R620 and R420..

setup R420 2 E5-2470v2 cpus.. for full capacity + mellanox MXC354-FCBT dual port 40GB (with firmwware upgradede and switched to ethernet mode)
setup2 R620 with 2 E5-2697v2 cpus + Mellanox MCX455-ECAT single port 100GB (with firmware upgraded latest available on nvidia site, and also in ethernet mode.

So i have tested with DAC 40GB cable.. with autonegotiation off.. it will not link 40GBps full-duplex.. so i removed the dac and placed 40gb gbic with cable and it linked full duplex 40gbps on both servers..

so i did some testing with bandwith test server.. and TCP running both sides.. i can only get 15Gbps aggregate traffic.. cpu wise running average 20% to 30% on both machines.. it seems to be something on the network side so i tried out.. jumbo frame set 9800 on both ethernet adapters as shown on pictures below.. but i cannot get past 15gbps..

if i do internal core test local on the server i can get nearly over 60gbps internal on the cpus TCP packets.. but on the network interfaces cannot get past 15gbps aggregate.. wondering if its software wise or hardware wise on the servers?

also multi x86_64 is enabled on both servers..


Anyideas on what else i could try out? as both servers are PCIe 3.0 slot cards.. so i am wondering if its hardware limitation.

Re: ROS v7 - SERVER with 2 interfaces 40G - Melanox CX354A

Posted: Fri Nov 10, 2023 5:05 pm
by PortalNET
Hi guys

continuing testing.. it seems that advertise on Mikrotik running the MCX354A-FCBT in ethernet mode switch... advertises ports as 5G FULL..


strange.. even disabling auto-negotiation and seting 40Gbps... it wont go past 5gbps in TCP.. in UDP it goes both tx and rx in 9.8GB forward test..


i am testing packets forward and backwards on bttest between the 2 servers.. any ideas if this could be a limitation on the PCIe ? or its a license issue on mikrotik? i am running evaluation test firmware latest 7.12.xx stable.. with 24h trial with no license.

Re: ROS v7 - SERVER with 2 interfaces 40G - Melanox CX354A

Posted: Fri Nov 10, 2023 5:13 pm
by PortalNET
Hi

ok guys so after some more searching it looks like its a hardware limitation on the Dell R620 and R420 sides... based on the speeds of the PCI-e 3.0 slots.. of max 16G aparently there are some tweaks available and tunning but it seems to be base on the R630 and R730.. so next stepp will try to get hold of 2 units for testing.

Re: ROS v7 - SERVER with 2 interfaces 40G - Melanox CX354A

Posted: Tue Nov 14, 2023 1:31 am
by ahmdzaki18
Networking with 2 CPU(s) is bad. Because the pcb lane on pcie are directly to 1 CPU socket. If all cpu plugged but only 1 pcie, the traffic will flow to another cpu and go back again through a lane called QPI. QPI is natural enemy for networking, latency and jitter. Please read about Networking with NUMA. Or you can take out the cpu that not direct to pci lane you, you will got better performance except your cpu is too old.


I used R620 with FRRouting + dpdk on 2690 v2.
Got 8mpps performance on single core, and got 16mpps on 2 core in same cpu socket. But when i pin 1 core on cross cpu socket its only 12mpps, thats it.