Thanks for all the replies everyone!
Okay, some more info about the iperf3 test I did to measure throughput between WAN and LAN:
Test Setup WITHOUT router:
10G_MacbookPro* ↔ 10G_Switch** ↔ 10G_NAS***
*8 x i7 CPU, 16GB ram, SONNET SOLO 10G Thunderbolt 3 dongle
**Mikrotik 10G switch (CRS306-1G-4S+IN)
***Linux PC (CentOS 7), 8 x XEON W-2123 @3.6GHz, 128GB ram, 12 x 12 TB SATA RAID, PCIe nic with 4 x 10G ports
MacbookPro iperf3 command: iperf3 -s
- The -s flag is “server” mode
NAS iperf3 command (for LAN–>WAN test): iperf3 -c -i 1 -t 30 -P 20
- The -c flag is “client” mode.
- The -i 1 argument is “interval” to output bandwidth reports every “1” second.
- The -t 30 argument is “time” to run test for “30” seconds.
- The -P 20 argument is “parallel” to run “20” iperf3 client streams in parallel.
Test result (LAN → WAN, no router): 9.40 gbps sustained
Now, run same test again but in reverse direction (WAN–>LAN):
NAS iperf3 command (for WAN–>LAN test): iperf3 -c -i 1 -t 30 -P 20 -R
- The -R flag is to do the test in the opposite direction (WAN–>LAN in this case).
Test result (WAN → LAN, no router): 9.40 gbps sustained
To this point, with no router, throughput is a “essentially” 10G (good enough for me).
Test Setup WITH router (ccr1036-8g-2s+):
10G_MacbookPro* ↔ 10G_WAN(sfp-sfpplus1) ↔ 10G_LAN(sfp-sfpplus2) ↔ 10G_Switch** ↔ 10G_NAS***
Testing is performed in the exact same way except the IP of the Macbook Pro had to be changed to work properly on the WAN side of the router.
Test result (LAN → WAN, WITH router): 8.40 gbps sustained
Test result (WAN → LAN, WITH router): 4.82 gbps sustained
NOTE: When I do this exact same test (through the router) out to the internet to an AWS EC2 Linux vm, I get very similar results:
Test result (LAN → cloud_vm, WITH router): 8.78 gbps sustained
Test result (cloud_vm → LAN, WITH router): 4.82 gbps sustained
Also, when I increase the parallel clients to 50 (“-P 50” arg) I STILL get 4.82 gbps for WAN → LAN and the CPU usage is about the same with some CPUs idle.
As for CPU usage, here is for the WAN → LAN test (sorted by CPU Load %):
(I don’t have an easy way to post screen captures so I’ll just type the data from my WebFig screen capture - grrrr!)
cpu10: 42%
cpu34: 40%
cpu30: 36%
cpu33: 24%
cpu7: 23%
cpu26: 22%
cpu11: 21%
cpu22: 21%
cpu9: 18%
cpu15: 11%
cpu21: 11%
cpu4: 2%
cpu6: 10%
cpu29: 10%
cpu3: 7%
cpu31: 6%
cpu32: 6%
cpu25: 5%
cpu35: 1%
…remaining 16 cpus: 0%
Here is for the LAN → WAN test (sorted by CPU Load %):
cpu35: 30%
cpu31: 26%
cpu4: 25%
cpu28: 12%
cpu9: 11%
cpu26: 10%
cpu5: 9%
cpu6: 9%
cpu8: 9%
cpu22: 9%
cpu2: 5%
cpu10: 5%
cpu3: 4%
cpu16: 4%
cpu21: 4%
cpu7: 3%
cpu11: 1%
cpu15: 3%
cpu29: 3%
cpu23: 2%
…remaining 16 cpus: 0%
Here is my exported config (thanks erlinden for showing me how to export it!) with the sensitive parts obfuscated with “XXXXXXXX”:
[admin@XXXXXXXX] > /export
# jul/10/2020 14:13:24 by RouterOS 6.47
# software id = XFYP-96JK
#
# model = CCR1036-8G-2S+
# serial number = XXXXXXXX
/interface bridge
add name=bridge1
/interface list
add name=WAN
add name=LAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/interface bridge port
add bridge=bridge1 interface=ether1
add bridge=bridge1 interface=ether2
add bridge=bridge1 interface=ether3
add bridge=bridge1 interface=ether4
add bridge=bridge1 interface=ether5
add bridge=bridge1 interface=ether6
add bridge=bridge1 interface=ether7
add bridge=bridge1 interface=ether8
add bridge=bridge1 interface=sfp-sfpplus2
/interface list member
add interface=sfp-sfpplus1 list=WAN
add interface=bridge1 list=LAN
/ip address
add address=10.3.0.254/24 interface=ether1 network=10.3.0.0
/ip dhcp-client
add disabled=no interface=sfp-sfpplus1
/ip firewall nat
add action=masquerade chain=srcnat out-interface-list=WAN
add action=dst-nat chain=dstnat dst-address=XXXXXXXX dst-port=XXXXXXXX in-interface=sfp-sfpplus1 log=yes log-prefix=XXXXXXXX protocol=tcp to-addresses=XXXXXXXX to-ports=XXXXXXXX
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www address=10.3.0.0/24
set ssh address=10.3.0.0/24
set api disabled=yes
set winbox address=10.3.0.0/24
set api-ssl disabled=yes
/lcd
set color-scheme=dark default-screen=informative-slideshow flip-screen=yes
/lcd pin
set pin-number=XXXXXXXX
/system clock
set time-zone-name=XXXXXXXX
/system identity
set name=XXXXXXXX
/tool graphing interface
add interface=sfp-sfpplus1
add interface=sfp-sfpplus2
add interface=ether1
I should note that the both the NAT rules work properly.
From user “mkx”:
So based on my exported config, do you see any glaring changes I should make ASAP?
I really only need this basic functionality:
- Masq NAT working so LAN can get to internet.
- A single DNAT entry to get to LAN server from internet.
- Block all other outside traffic from getting in (including block access to router management from internet).
- Get 10G to the internet in both directions (or at least 8+ gbps).
Any tuning optimizations, tips, or suggestions to improve my WAN → LAN performance are very much welcome and appreciated!
. Yes, you’re off base. It’s not the physical interfaces, but the number of data streams like TCP sessions that matters. Across the Internet, you don’t need to have 10G to run into this; that’s why things like BitTorrent and NNTP let you set up multiple sessions for a single download. And why iperf3 will do that if you tell it to.
