A proxmox host connected to the GB switch the vlan2 and an internal LAN vlan3 (both tagged)
A CHR VM (P10) on proxmox that has a VLAN aware bridge and creates vlan2 and vlan3 interfaces and routes between it
A Linux container
When I run the speed test from the Linux container (or any other machine on the internal network) my speed is limited to just 300-400Mbit, even though the connection should be ~1Gbit:
$ speedtest
Retrieving speedtest.net configuration...
Testing from Sonic.net, LLC (135.180.127.45)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Mach Dilemma (Fremont, CA) [33.33 km]: 9.458 ms
Testing download speed................................................................................
Download: 357.49 Mbit/s
Testing upload speed......................................................................................................
Upload: 370.92 Mbit/s
The speed should NOT be halved because the data only runs once through the switch. The traffic between CHR VM and Linux container is virtualized via Proxmox VM and does not limit the bandwidth.
I also confirmed:
When I connect a laptop to the FTTH model directly, I get 900Mbit
When I create VLAN enp1s0.2 on the proxmox host directly and run speedtest over it, I get 900Mbit as well. So the issue is not the switch, cables or the proxmox host
Unfortunately RouterOS does not allow me to run a speedtest and the proprietary “Bandwidth Test” is pretty annoying because I’d need another instance somewhere which for one I do not have and also does not give me same results as just the speedtest CLI test.
As it seems right so far, RouterOS is halving my bandwidth somewhere (despite P10 test license).
What could be the reason for this?
And how to most easily debug this further, without having to spin other CHR instances?
What is your configuration? I.e. firewall rules and so on.
Have you run perftests from CHR directly to some local host?
What are resources of the CHR VM? How many vCPU and RAM
What is your configuration? I.e. firewall rules and so on.
As for the config, it’s really big and I am not sure how relevant. Just basic firewall rules but no traffic shaping etc. I tried to strip away everything completely unnecessary and attach config at the bottom.
Have you run perftests from CHR directly to some local host?
Not yet, besides the bandwidth tests over the internet. Do you have a pointer how to best do this?
What are resources of the CHR VM? How many vCPU and RAM
2vCPUs, 512MB RAM (85MB used). Host is a 4core Celeron J1900 @ 2GHz and 8GB RAM (it’s this thing).
/interface bridge
add name=br-main vlan-filtering=yes
/interface ethernet
set [ find default-name=ether1 ] disable-running-check=no
/interface vlan
add comment=SONIC interface=br-main name=vlan2 vlan-id=2
add comment=LAN interface=br-main name=vlan3 vlan-id=3
/interface vrrp
add interface=vlan3 name=vrrp3 sync-connection-tracking=yes vrid=3
/interface list
add name=LAN
add name=WAN
add include=LAN name=trusted
/routing table
add disabled=no fib name=default_sonic
/interface bridge port
add bridge=br-main interface=ether1
/ip firewall connection tracking
set enabled=yes
/ip neighbor discovery-settings
set discover-interface-list=none lldp-med-net-policy-vlan=1
/interface bridge vlan
add bridge=br-main tagged=br-main,ether1 vlan-ids=2
add bridge=br-main tagged=br-main,ether1 vlan-ids=3
/interface detect-internet
set internet-interface-list=static lan-interface-list=static wan-interface-list=static
/interface list member
add interface=vlan3 list=LAN
add interface=vlan2 list=WAN
/ip address
add address=10.227.79.2/24 interface=vlan3 network=10.227.79.0
add address=10.227.79.254 interface=vrrp3 network=10.227.79.254
/ip firewall address-list
add address=10.227.0.0/16 list=own_hosts
add address=0.0.0.0/8 comment="defconf: RFC6890" list=no_forward_ipv4
add address=169.254.0.0/16 comment="defconf: RFC6890" list=no_forward_ipv4
add address=224.0.0.0/4 comment="defconf: multicast" list=no_forward_ipv4
add address=255.255.255.255 comment="defconf: RFC6890" list=no_forward_ipv4
add address=127.0.0.0/8 comment="defconf: RFC6890" list=bad_ipv4
add address=192.0.0.0/24 comment="defconf: RFC6890" list=bad_ipv4
add address=192.0.2.0/24 comment="defconf: RFC6890 documentation" list=bad_ipv4
add address=198.51.100.0/24 comment="defconf: RFC6890 documentation" list=bad_ipv4
add address=203.0.113.0/24 comment="defconf: RFC6890 documentation" list=bad_ipv4
add address=240.0.0.0/4 comment="defconf: RFC6890 reserved" list=bad_ipv4
add address=0.0.0.0/8 comment="defconf: RFC6890" list=not_global_ipv4
add address=10.0.0.0/8 comment="defconf: RFC6890" list=not_global_ipv4
add address=100.64.0.0/10 comment="defconf: RFC6890" list=not_global_ipv4
add address=169.254.0.0/16 comment="defconf: RFC6890" list=not_global_ipv4
add address=172.16.0.0/12 comment="defconf: RFC6890" list=not_global_ipv4
add address=192.0.0.0/29 comment="defconf: RFC6890" list=not_global_ipv4
add address=192.168.0.0/16 comment="defconf: RFC6890" list=not_global_ipv4
add address=198.18.0.0/15 comment="defconf: RFC6890 benchmark" list=not_global_ipv4
add address=255.255.255.255 comment="defconf: RFC6890" list=not_global_ipv4
add address=224.0.0.0/4 comment="defconf: multicast" list=bad_src_ipv4
add address=255.255.255.255 comment="defconf: RFC6890" list=bad_src_ipv4
add address=0.0.0.0/8 comment="defconf: RFC6890" list=bad_dst_ipv4
add address=224.0.0.0/4 comment="defconf: RFC6890" list=bad_dst_ipv4
add address=10.227.0.0/16 list=private_lans
add address=192.168.0.0/16 list=private_lans
/ip firewall filter
add action=accept chain=input comment="defconf: accept ICMP after RAW" protocol=icmp
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop all not coming from trusted" in-interface-list=!trusted
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=established,related hw-offload=yes
add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat connection-state=new in-interface-list=WAN
add action=drop chain=forward comment="defconf: drop bad forward IPs" src-address-list=no_forward_ipv4
add action=drop chain=forward comment="defconf: drop bad forward IPs" dst-address-list=no_forward_ipv4
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade WAN interface (Sonic)" out-interface=vlan2
/ip firewall raw
add action=accept chain=prerouting comment="defconf: enable for transparent firewall"
add action=accept chain=prerouting comment="defconf: accept DHCP discover" dst-address=\
255.255.255.255 dst-port=67 in-interface-list=LAN protocol=udp src-address=0.0.0.0 \
src-port=68
add action=drop chain=prerouting comment="defconf: drop bogon IP's" src-address-list=bad_ipv4
add action=drop chain=prerouting comment="defconf: drop bogon IP's" dst-address-list=bad_ipv4
add action=drop chain=prerouting comment="defconf: drop bogon IP's" src-address-list=bad_src_ipv4
add action=drop chain=prerouting comment="defconf: drop bogon IP's" dst-address-list=bad_dst_ipv4
add action=drop chain=prerouting comment="defconf: drop non global from WAN" in-interface-list=WAN src-address-list=not_global_ipv4
add action=drop chain=prerouting comment="defconf: drop forward to local lan from WAN" dst-address-list=private_lans in-interface-list=WAN
add action=drop chain=prerouting comment="defconf: drop local if not from default IP range" in-interface-list=LAN src-address=!10.227.79.0/24
add action=drop chain=prerouting comment="defconf: drop bad UDP" port=0 protocol=udp
add action=jump chain=prerouting comment="defconf: jump to ICMP chain" jump-target=icmp4 protocol=icmp
add action=jump chain=prerouting comment="defconf: jump to TCP chain" jump-target=bad_tcp protocol=tcp
add action=accept chain=prerouting comment="defconf: accept everything else from LAN" in-interface-list=trusted
add action=accept chain=prerouting comment="defconf: accept everything else from WAN" in-interface-list=WAN
add action=drop chain=prerouting comment="defconf: drop the rest"
/ip route
add comment=default_sonic disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
135.180.128.1 routing-table=default_sonic scope=30 suppress-hw-offload=no \
target-scope=10
/routing rule
add action=lookup comment="main table" disabled=no table=main
add action=lookup comment="default sonic" disabled=no table=default_sonic
It be curious if that also shows ~400Mb/s. FWIW the public test server runs Proxmox too…
Another question are you using VirtIO as the network interface in Proxmox config for CHR? Or some emulated network adapter? The later likely be much slower…
I’ll bet that CHR is using E1000 emulated network card, since AFAIK that’s the default for a new VM. I haven’t tested it, but CHR does support multiqueue with VirtIO - so another thing to try if already using VirtIO.
Try setting the VM CHR on your Proxmox system to use VirtIO network interfaces.
Also , make the VM CHR is a stand-alone VM ( not in a container ).
Verify your Mikrotik CHR license is the correct level.
Thank you all for helping to debug, I really appreciate it!
Some feedback on the suggestions:
Another forum member runs as public “Bandwidth Test” server, see:
viewtopic.php?t=104266
Great! I tried this out and the result is truly surprising! TX (i.e. upload) is around 700Mbps but RX (i.e., download) is just half of it. See: https://snipboard.io/ae5ERy.jpg
Can it have anything to do with RX of interfaces? Because then it would explain why both up/downlinks are slow with my container: Because CHR would “RX” both uplink and downlink (once from the LAN and once from the WAN interface)
You can try to spin iperf container inside CHR and test there (there are example of successful setups)
True. But guess all path will still show half.
Ok, I am trying this. One concern is, can I have two test licenses though?
(At this point I am so deep in I’ll buy it anyway but it would be great if I could still test/confirm the bandwidth issue before purchase)
I have spun up the VM already but need more time to set up and test … will do in a few hours.
Another question are you using VirtIO as the network interface in Proxmox config for CHR? Or some emulated network adapter? The later likely be much slower…
Try setting the VM CHR on your Proxmox system to use VirtIO network interfaces.
I’ll bet that CHR is using E1000 emulated network card, since AFAIK that’s the default for a new VM. I haven’t tested it, but CHR does support multiqueue with VirtIO - so another thing to try if already using VirtIO.
Unfortunately I am confirming that I am already using VirtIO (I used ttek’s script to create the VM).
That’s the config: https://snipboard.io/Tu9mqo.jpg
Also , make the VM CHR is a stand-alone VM ( not in a container ).
Sure, but I don’t think there is any conceivable way it could even be a container…
Verify your Mikrotik CHR license is the correct level.
Yes, verified, it shows P10 (it’s the test license)
Thank you @kleshki! See here:
Indeed, the CPU is under heavy load. It fluctuates but above is roughly the situation during the bandwidth test. Have of the CPU goes into “networking” (which is not very specific to me).
I think that the most important take-away from this thread so far is that throughput drops after applying (high capacity) CHR license … without changing any of configuration. As documented in post #10 above.
Which means that the code, which enforces licensed limits inside CHR somehow misses its target. And that’s something MT support should be looking into.
I mean … if proxmox in general would be directly to blame, then licensing level of ROS would not matter.
I know this seems to be related to a license at first, but maybe not. It’s interesting to see if the problem persists on another type of hypervisor, especially in Hyper-V since it doesn’t use virtio drivers for switches.
Also, what happens if you try to acquire different type of license for testing (P unlimited/P1)?
