Community discussions

MikroTik App
 
sadude
just joined
Topic Author
Posts: 3
Joined: Sun Oct 20, 2019 9:38 am

CRS305-1G-4S+ Actual MTU, L2MTU

Sun Oct 20, 2019 10:17 am

Hi!

I am trying to achieve expected large sequential read speeds over 10g from my FreeNAS to my Windows box (Mellanox ConnectX-3 NICs on both ends). My expectation with a 6x3TB WD Red RAID-Z2 pool is 350-500MB/s read speeds from the NAS to my windows box.
At the moment I have just about eliminated my network setup as the bottleneck but the "Actual MTU" and "L2MTU" settings under "Interfaces" in the RouterOS of my CRS305-1G-4S+ are confusing me.

My current settings

CRS305-1G-4S+: Actual MTU: 9014; L2MTU: 9014
Windows Box: MTU 9014
FreeNAS: MTU 9014

IPerf benchmarking results

Sender (server) = FreeNAS: 9.87 Gb/s
Receiver (client) = FreeNAS: 8.62 Gb/s

Windows SMB transfer speed testing

Whilst testing I create a RAM disk on my Windows machine. I also never use the same movie file (ca. 8GB) twice to avoid the FreeNAS machine from using what is cached in RAM. All testing was done on a FreeNAS dataset with compression turned off to get the most accurate results, that are not skewed in any way.
Results:
160-200MB/s read
400MB/s write
As far as I understand RAID-Z2 my write speeds are perfectly adequate seeing as the data has to be written to multiple drives due to the 2 disk parity.
The read speeds however should be SIGNIFICANTLY higher because the data can be read from multiple drives at the same time.

Questions

1.) The Windows MTU and FreeNAS MTU values should correspond to which MTU value (Actual or L2?) on the CRS305-1G-4S+ Interface?

2.) According to this website https://wiki.mikrotik.com/wiki/Manual:M ... uterBoards the max L2MTU is 10218.
What does this mean for the MTU values on all the aforementioned devices?

Thanks in advance for any help! :)
 
Zacharias
Forum Guru
Forum Guru
Posts: 2309
Joined: Tue Dec 12, 2017 12:58 am
Location: Greece

Re: CRS305-1G-4S+ Actual MTU, L2MTU

Mon Oct 21, 2019 7:35 pm

L2MTU is used in packet forwarding, wiki:
For a device that is only supposed to forward packets, there is no need to increase the MTU size, it is only required to increase the L2MTU size

So since you forward packets from your NAS to your Windows box you dont care about MTU... L2MTU is what you should worry about...

Since we talk about it, MTU is used in routing.. wiki:
If you require the packet to be received on the interface and the device needs to process this packet rather than just forwarding it, for example, in case of routing, then it is required to increase the L2MTU and the MTU size

Also, MTU size can not be larger than the L2MTU...
Finally if packet fragmentation is supported you can leave MTU to its default size and not change it...

https://wiki.mikrotik.com/wiki/Manual:L ... _interface
 
sadude
just joined
Topic Author
Posts: 3
Joined: Sun Oct 20, 2019 9:38 am

Re: CRS305-1G-4S+ Actual MTU, L2MTU

Tue Oct 22, 2019 8:24 am

Thanks for your response, Zacharias.
Understood... L2MTU is what would interest me in my use case since the mikrotik switch is only forwarding and not routing.

If I were to change the L2MTU on the CRS305-1G-4S+ SFP+ ethernet interfaces to 10218, what exactly would that mean for my FreeNAS and Windows machines? Should I change their respective MTUs to 10218 as well or is that MTU value possibly not supported by the Mellanox ConnectX-3 adapters?
 
mkx
Forum Guru
Forum Guru
Posts: 4338
Joined: Thu Mar 03, 2016 10:23 pm

Re: CRS305-1G-4S+ Actual MTU, L2MTU

Tue Oct 22, 2019 9:26 am

MTU (the L3 one) must be set to same value within same L2 (broadcast) domain ... or else some connections get broken and some not (depending on individual host settings). If you're sure all of your LAN devices support jumbo (large) frames (this refers to l2mtu), then go ahead and set it to larger value. Beware that different devices have different maximum values and you have to select the lowest value.
And beware that router will have to fragment packets when routing between networks with different MTU (and internet generally works with MTU 1500) possibly making it suffer when currently it works just fine.

One also has to put use of jumbo frames into perspective ... in the times when 100Mbps ethernet came around, CPUs were much slower than today and became overwhelmed by the number of frames (not necessarily by bandwidth) ... so smart guys invented jumbo frames and reduced number of frames to around 1/3 of original amount, which helped to reduce CPU processing effort to some 1/2 of previous one.
Throughput gain because of using jumbo frames is not that great. Consider jumbo frames with MTU of 9000 bytes which presumably accomodate 6-times the payload compared to standard 1500 byte frames: TCP header size is 20 bytes, IP header size is another 20 bytes, base ethernet header size is 18 bytes and ethernet inter-frame gap is 12 bytes, overall overhead is thus 70 bytes and net payload is 1460 bytes (4.58% of overhead). If we take 6 times larger frame (and keep total payload size sum of 6 packets to make calculation simpler), it's a packet with net size of 8760 bytes and overhead of 70 bytes (0.79% overhead). So over 1Gbps link we can get 954.3Mbps net TCP throughput using standard 1500 byte MTU or 992.1Mbps net TCP throughput when using 9000 byte MTU. That's around 4% of increase ... if there are no retransmissions which take more bandwidth due to larger packets to retransmit. And modern devices generally don't have problems dealing with 100k packets per second (venerable RB951G can do 270kpps when switching and 60-200kpps when routing, more modern RBD52G can route 250kpps or more ... and both are actually tiny devices by all means).

So one has to wonder why bother with large frames ...
BR,
Metod
 
sadude
just joined
Topic Author
Posts: 3
Joined: Sun Oct 20, 2019 9:38 am

Re: CRS305-1G-4S+ Actual MTU, L2MTU

Tue Oct 22, 2019 9:52 am

Very interesting points!
I will go ahead and test the SMB transfer speeds again with the mtu values set to default for the SFP+ interfaces. These are "MTU = 1500" and "L2MTU = 1592", if my memory serves me correctly?
And obviously remove the "mtu = 9000" FreeNAS parameter and default back to "Jumbo Packet Value = 1500" on my Windows machine.
Thank you all for your timely responses. Greatly appreciated.
 
storp
Frequent Visitor
Frequent Visitor
Posts: 54
Joined: Tue Nov 24, 2015 2:53 pm

Re: CRS305-1G-4S+ Actual MTU, L2MTU

Tue Oct 22, 2019 10:51 am

I've done a lot of tinkering with jumbo frames but always ended up reverting back to standard. The small increase in performance is not worth the extra hazzle and problems which one might encounter, at least not for me. Currently I only use jumbo frames when directly connecting devices (for example freenas box with vmware server through directly connected DAC cable). But of course if maximum performance is a must then jumbo frames is needed. I think you should look more into SMB3 if you need to maximize performance.
 
Zacharias
Forum Guru
Forum Guru
Posts: 2309
Joined: Tue Dec 12, 2017 12:58 am
Location: Greece

Re: CRS305-1G-4S+ Actual MTU, L2MTU

Tue Oct 22, 2019 11:08 am

MTU (the L3 one) must be set to same value within same L2 (broadcast) domain ...
Equal or less than the L2MTU so it can fit without fragmentation...
 
mkx
Forum Guru
Forum Guru
Posts: 4338
Joined: Thu Mar 03, 2016 10:23 pm

Re: CRS305-1G-4S+ Actual MTU, L2MTU

Tue Oct 22, 2019 1:21 pm

Equal or less than the L2MTU so it can fit without fragmentation...
If device somehow ends up with MTU set larger than l2mtu, then packets with size equal or less than MTU but larger than l2mtu won't get fragmented, they will get dropped. Fragmentation is L3 function and works based on MTU setting (l2mtu is not directly taken onto account).
The same is the reason why switches don't fragment pakets.
BR,
Metod
 
Zacharias
Forum Guru
Forum Guru
Posts: 2309
Joined: Tue Dec 12, 2017 12:58 am
Location: Greece

Re: CRS305-1G-4S+ Actual MTU, L2MTU

Tue Oct 22, 2019 1:37 pm

Equal or less than the L2MTU so it can fit without fragmentation...
If device somehow ends up with MTU set larger than l2mtu, ...
In ROS you can not set an MTU value higher than than the L2MTU, you will get an error...

Who is online

Users browsing this forum: akakua, amala, Bing [Bot], mrz, Ogwuche and 127 guests