Community discussions

MikroTik App
 
lbenalcazar
just joined
Topic Author
Posts: 8
Joined: Wed Dec 01, 2021 4:36 pm
Location: Ecuador
Contact:

ROS v7 - SERVER with 2 interfaces 40G - Melanox CX354A

Mon Jul 25, 2022 2:00 am

Hello everyone, I would like to know if anyone has ever been able to install RouterOS on a server with 40G cards for CG-NAT since the 1072 team is not enough when performing CG-NAT and interface bonding.

I think that if the processor does not need to bond, it will work without too much load.

Thank you very much for your help.
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 2989
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: ROS v7 - SERVER with 2 interfaces 40G - Melanox CX354A

Mon Jul 25, 2022 2:42 am

with ccr1072 have you tried fast-track?

i think ccr1072 can do al least 20g of cg-nat with fast-track, ccr1036 can do 10g of cg-nat with fast-track without problem

if with fast-track ccr1072 still not able to do cg-nat i think is better to divide the task of 40g of cgnat in 4 10g cg-nat on 4 x ccr1036
 
User avatar
jspool
Member
Member
Posts: 468
Joined: Sun Oct 04, 2009 4:06 am
Location: Oregon

Re: ROS v7 - SERVER with 2 interfaces 40G - Melanox CX354A

Mon Jul 25, 2022 2:57 am

ROS v7 bare metal install on Dell R730 with 40G & 50G Mellanox ConnectX-4 series works great. I have not tried any ConnectX-3 series.
 
lbenalcazar
just joined
Topic Author
Posts: 8
Joined: Wed Dec 01, 2021 4:36 pm
Location: Ecuador
Contact:

Re: ROS v7 - SERVER with 2 interfaces 40G - Melanox CX354A

Mon Jul 25, 2022 5:49 am

with ccr1072 have you tried fast-track?

i think ccr1072 can do al least 20g of cg-nat with fast-track, ccr1036 can do 10g of cg-nat with fast-track without problem

if with fast-track ccr1072 still not able to do cg-nat i think is better to divide the task of 40g of cgnat in 4 10g cg-nat on 4 x ccr1036
Splitting is a good option, but I wanted to do it on one team.

On the other hand I think that 1072 has the potential for 20G CGNAT but the problem is its 10G interfaces, to join the traffic I would do it through bonding but bonding uses more CPU resources which makes it inefficient.

I think the 1072 should have been designed with 25G or 40G interfaces.
 
lbenalcazar
just joined
Topic Author
Posts: 8
Joined: Wed Dec 01, 2021 4:36 pm
Location: Ecuador
Contact:

Re: ROS v7 - SERVER with 2 interfaces 40G - Melanox CX354A

Mon Jul 25, 2022 5:52 am

ROS v7 bare metal install on Dell R730 with 40G & 50G Mellanox ConnectX-4 series works great. I have not tried any ConnectX-3 series.
Thank you very much, I think I should try it, I guess you put two CPUs on the server, and how many threads was each one?
 
User avatar
jspool
Member
Member
Posts: 468
Joined: Sun Oct 04, 2009 4:06 am
Location: Oregon

Re: ROS v7 - SERVER with 2 interfaces 40G - Melanox CX354A

Mon Jul 25, 2022 6:24 am

ROS v7 bare metal install on Dell R730 with 40G & 50G Mellanox ConnectX-4 series works great. I have not tried any ConnectX-3 series.
Thank you very much, I think I should try it, I guess you put two CPUs on the server, and how many threads was each one?
The one I am testing has 2 x E5-2680 v3 for a total of 24 cores.
 
PortalNET
Member Candidate
Member Candidate
Posts: 126
Joined: Sun Apr 02, 2017 7:24 pm

Re: ROS v7 - SERVER with 2 interfaces 40G - Melanox CX354A

Fri Nov 10, 2023 1:21 am

Hi guys

Sorry to bump on a old thread.. but i am having the following issue...

we have setup 2 dell servers R620 and R420..

setup R420 2 E5-2470v2 cpus.. for full capacity + mellanox MXC354-FCBT dual port 40GB (with firmwware upgradede and switched to ethernet mode)
setup2 R620 with 2 E5-2697v2 cpus + Mellanox MCX455-ECAT single port 100GB (with firmware upgraded latest available on nvidia site, and also in ethernet mode.

So i have tested with DAC 40GB cable.. with autonegotiation off.. it will not link 40GBps full-duplex.. so i removed the dac and placed 40gb gbic with cable and it linked full duplex 40gbps on both servers..

so i did some testing with bandwith test server.. and TCP running both sides.. i can only get 15Gbps aggregate traffic.. cpu wise running average 20% to 30% on both machines.. it seems to be something on the network side so i tried out.. jumbo frame set 9800 on both ethernet adapters as shown on pictures below.. but i cannot get past 15gbps..

if i do internal core test local on the server i can get nearly over 60gbps internal on the cpus TCP packets.. but on the network interfaces cannot get past 15gbps aggregate.. wondering if its software wise or hardware wise on the servers?

also multi x86_64 is enabled on both servers..


Anyideas on what else i could try out? as both servers are PCIe 3.0 slot cards.. so i am wondering if its hardware limitation.
You do not have the required permissions to view the files attached to this post.
 
PortalNET
Member Candidate
Member Candidate
Posts: 126
Joined: Sun Apr 02, 2017 7:24 pm

Re: ROS v7 - SERVER with 2 interfaces 40G - Melanox CX354A

Fri Nov 10, 2023 5:05 pm

Hi guys

continuing testing.. it seems that advertise on Mikrotik running the MCX354A-FCBT in ethernet mode switch... advertises ports as 5G FULL..


strange.. even disabling auto-negotiation and seting 40Gbps... it wont go past 5gbps in TCP.. in UDP it goes both tx and rx in 9.8GB forward test..


i am testing packets forward and backwards on bttest between the 2 servers.. any ideas if this could be a limitation on the PCIe ? or its a license issue on mikrotik? i am running evaluation test firmware latest 7.12.xx stable.. with 24h trial with no license.
You do not have the required permissions to view the files attached to this post.
 
PortalNET
Member Candidate
Member Candidate
Posts: 126
Joined: Sun Apr 02, 2017 7:24 pm

Re: ROS v7 - SERVER with 2 interfaces 40G - Melanox CX354A

Fri Nov 10, 2023 5:13 pm

Hi

ok guys so after some more searching it looks like its a hardware limitation on the Dell R620 and R420 sides... based on the speeds of the PCI-e 3.0 slots.. of max 16G aparently there are some tweaks available and tunning but it seems to be base on the R630 and R730.. so next stepp will try to get hold of 2 units for testing.
 
User avatar
ahmdzaki18
just joined
Posts: 11
Joined: Fri Oct 06, 2023 7:52 pm
Location: Jakarta, Indonesia
Contact:

Re: ROS v7 - SERVER with 2 interfaces 40G - Melanox CX354A

Tue Nov 14, 2023 1:31 am

Networking with 2 CPU(s) is bad. Because the pcb lane on pcie are directly to 1 CPU socket. If all cpu plugged but only 1 pcie, the traffic will flow to another cpu and go back again through a lane called QPI. QPI is natural enemy for networking, latency and jitter. Please read about Networking with NUMA. Or you can take out the cpu that not direct to pci lane you, you will got better performance except your cpu is too old.


I used R620 with FRRouting + dpdk on 2690 v2.
Got 8mpps performance on single core, and got 16mpps on 2 core in same cpu socket. But when i pin 1 core on cross cpu socket its only 12mpps, thats it.

Who is online

Users browsing this forum: No registered users and 16 guests