Hi, I have a monster vlan on MT and when I send something in between the transfer is low, what could be the reason?
Hex S Lag 802.3ad router - ether4 sfp1
Switch CSS326-24G-2S +
At the other end, proxmox also with LAG 802.3ad 1 Gbps
Before split into vlans, it flicked a full 1 Gb / s
About 25 rules per firewall, all for pppoe-out1 only
Yes, one core is 100% loaded, I changed to ethernet and the transfer improved to 30MB / s, CPU0 load: 15% CPU1: 0% CPU2: 58% CPU3: 98% - load while copying, LAG on Ethernet port 3 and 4 without SFP
Without console config files( export ), its hard to see how you have configured devices correctly. Especially the hex S which is not designed for massive loads.
That said, if your are offloading to the SWITCH CHIP of each device and NOT the CPU as you have by the looks of your winbox screen shot, you should not be having a CPU issue at all as the data should be getting transferred at wire speed. But the screen shot you are showing looks like the hex and your trying to get the CPU to do the work !
Remember most MT Router gear works like Ether— ( <–>Switch chip )-<---->(CPU), you should program for hardware switch chip offload when you can and only bring vlans up to the cpu if you actually need to firewall/filter against…
As per : https://i.mt.lv/cdn/product_files/RB750Gr3-esw3_190642.png
Try and program the hex using the guides as per
https://wiki.mikrotik.com/wiki/Manual:Switch_Chip_Features
FYI ( CRS 3x series and beyond, are far easier to config for hardware(switch chip) offloading when doing up a RouterOS config !!!
yes, the picture shows everything
Well based on the winbox shot.. Yep CPU limiting issue !
What specific files would you need to evaluate my configuration?
I had to make a mistake with the configuration because I configured according to this link https://wiki.mikrotik.com/wiki/Manual:Switch_Chip_Features
# aug/19/2022 18:10:36 by RouterOS 6.49.6
# software id = NXZU-0IEV
#
# model = RB760iGS
# serial number =
/caps-man channel
add band=2ghz-g/n control-channel-width=20mhz extension-channel=disabled \
frequency=2412 name=channel1
add band=2ghz-g/n control-channel-width=20mhz extension-channel=disabled \
frequency=2437 name=channel6
add band=2ghz-g/n control-channel-width=20mhz extension-channel=disabled \
frequency=2462 name=channel11
add band=5ghz-n/ac control-channel-width=20mhz extension-channel=disabled \
frequency=5180 name=channel36
add band=5ghz-n/ac control-channel-width=20mhz extension-channel=disabled \
frequency=5200 name=channel40
/caps-man datapath
add client-to-client-forwarding=yes local-forwarding=yes name=TP-Link_1C50 \
vlan-id=2 vlan-mode=use-tag
/interface bridge
add name=bridge1 pvid=999 vlan-filtering=yes
/interface ethernet
set [ find default-name=ether1 ] mtu=1492
set [ find default-name=sfp1 ] auto-negotiation=no mtu=1596 rx-flow-control=\
on tx-flow-control=on
/interface vlan
add interface=bridge1 name=Dom vlan-id=2
add interface=bridge1 name=MGMT vlan-id=999
add interface=bridge1 name=Serwery vlan-id=10
add interface=bridge1 name=Storage vlan-id=21
add interface=ether5 name=WAN-VLAN vlan-id=200
add interface=bridge1 name=Wiktor vlan-id=4
add interface=bridge1 name=Wirtualizator-Prox vlan-id=20
/interface bonding
add lacp-rate=1sec mode=802.3ad name=bonding1 slaves=ether4,ether3,sfp1
/interface pppoe-client
add add-default-route=yes dial-on-demand=yes disabled=no interface=WAN-VLAN \
max-mru=1500 max-mtu=1500 name=pppoe-out1 password= user=\
/caps-man rates
add basic=6Mbps,9Mbps,12Mbps,18Mbps,24Mbps,36Mbps,48Mbps,54Mbps ht-basic-mcs=\
"" ht-supported-mcs="" name=rate1 supported=\
6Mbps,9Mbps,12Mbps,18Mbps,24Mbps,36Mbps,48Mbps,54Mbps vht-basic-mcs="" \
vht-supported-mcs=""
/caps-man security
add authentication-types=wpa2-psk encryption=aes-ccm name=security1 \
passphrase=
/caps-man configuration
add channel.tx-power=40 country=no_country_set datapath=TP-Link_1C50 \
installation=any mode=ap name=TP-Link_1C50 rx-chains=0,1,2,3 security=\
security1 ssid=TP-Link_1C50 tx-chains=0,1,2,3
add channel.tx-power=40 country="united states" datapath=TP-Link_1C50 \
installation=any mode=ap name=TP-Link_1C50_5G rx-chains=0,1,2,3 security=\
security1 ssid=TP-Link_1C50_5G tx-chains=0,1,2,3
/caps-man interface
add configuration=TP-Link_1C50 configuration.hw-protection-mode=rts-cts \
configuration.multicast-helper=full datapath=TP-Link_1C50 disabled=no \
l2mtu=1600 mac-address=DC:2C:6E:1B:33:07 master-interface=none name=2g \
radio-mac=DC:2C:6E:1B:33:07 radio-name=DC2C6E1B3307 rates.basic=\
6Mbps,9Mbps,12Mbps,18Mbps,24Mbps,36Mbps,48Mbps,54Mbps rates.ht-basic-mcs=\
"" rates.supported=6Mbps,9Mbps,12Mbps,18Mbps,24Mbps,36Mbps,48Mbps,54Mbps \
security=security1
add channel.tx-power=40 configuration=TP-Link_1C50_5G datapath=TP-Link_1C50 \
disabled=no l2mtu=1600 mac-address=DC:2C:6E:1B:33:08 master-interface=\
none name=5g radio-mac=DC:2C:6E:1B:33:08 radio-name=DC2C6E1B3308 \
security=security1
/interface list
add name=WAN
add name=LAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=dhcp_pool1 ranges=192.168.0.100-192.168.0.200
add name=Lab ranges=192.168.1.2-192.168.1.254
add name=VPN ranges=192.168.2.2-192.168.2.254
/ip dhcp-server
add address-pool=dhcp_pool1 disabled=no interface=Dom lease-time=2h name=\
dhcp1
add address-pool=Lab disabled=no name=Serwer_Lab
/lora servers
add address=eu.mikrotik.thethings.industries down-port=1700 name=TTN-EU \
up-port=1700
add address=us.mikrotik.thethings.industries down-port=1700 name=TTN-US \
up-port=1700
add address=eu1.cloud.thethings.industries down-port=1700 name=\
"TTS Cloud (eu1)" up-port=1700
add address=nam1.cloud.thethings.industries down-port=1700 name=\
"TTS Cloud (nam1)" up-port=1700
add address=au1.cloud.thethings.industries down-port=1700 name=\
"TTS Cloud (au1)" up-port=1700
add address=eu1.cloud.thethings.network down-port=1700 name="TTN V3 (eu1)" \
up-port=1700
add address=nam1.cloud.thethings.network down-port=1700 name="TTN V3 (nam1)" \
up-port=1700
add address=au1.cloud.thethings.network down-port=1700 name="TTN V3 (au1)" \
up-port=1700
/port
set 0 name=serial0
/ppp profile
add bridge=bridge1 dns-server=192.168.2.1 local-address=192.168.2.1 name=adam \
remote-address=VPN use-encryption=required
/queue simple
add dst=pppoe-out1 max-limit=7M/25M name=Szymon target=192.168.0.196/32 time=\
6h-22h,sun,mon,tue,wed,thu,fri,sat
add dst=pppoe-out1 max-limit=7M/25M name=Mariusz target=192.168.0.214/32 \
time=6h-22h,sun,mon,tue,wed,thu,fri,sat
/tool user-manager customer
set admin access=\
own-routers,own-users,own-profiles,own-limits,config-payment-gw
/user group
set full policy="local,telnet,ssh,ftp,reboot,read,write,policy,test,winbox,pas\
sword,web,sniff,sensitive,api,romon,dude,tikapp"
/caps-man manager
set ca-certificate=auto certificate=auto enabled=yes
/caps-man provisioning
add action=create-dynamic-enabled master-configuration=TP-Link_1C50 \
name-format=identity slave-configurations=TP-Link_1C50
add action=create-dynamic-enabled master-configuration=TP-Link_1C50_5G \
name-format=identity slave-configurations=TP-Link_1C50_5G
/dude
set enabled=yes
/interface bridge port
add bridge=bridge1 interface=Dom pvid=2
add bridge=bridge1 interface=MGMT pvid=999
add bridge=bridge1 interface=Serwery pvid=10
add bridge=bridge1 interface=Wirtualizator-Prox pvid=20
add bridge=bridge1 interface=bonding1 pvid=999
add bridge=bridge1 interface=Storage pvid=21
add bridge=bridge1 interface=Wiktor pvid=4
/ip neighbor discovery-settings
set discover-interface-list=!dynamic
/interface bridge vlan
add bridge=bridge1 tagged=bonding1,bridge1,MGMT vlan-ids=999
add bridge=bridge1 tagged=bridge1,bonding1 untagged=Dom vlan-ids=2
add bridge=bridge1 tagged=bridge1,bonding1 untagged=Serwery vlan-ids=10
add bridge=bridge1 tagged=bridge1,bonding1,Wirtualizator-Prox vlan-ids=20
add bridge=bridge1 tagged=bonding1,bridge1,Storage vlan-ids=21
add bridge=bridge1 tagged=bonding1,bridge1 vlan-ids=4
/interface l2tp-server server
set authentication=mschap2 default-profile=default ipsec-secret=Elektryk1@1 \
max-mru=1500 max-mtu=1500 use-ipsec=yes
/interface list member
add interface=ether5 list=WAN
add interface=ether4 list=LAN
/interface ovpn-server server
set auth=sha1 certificate=server cipher=aes128,aes192,aes256 default-profile=\
adam enabled=yes port=23119 require-client-certificate=yes
/ip address
add address=192.168.0.1/24 interface=Dom network=192.168.0.0
add address=10.70.1.1/29 interface=Wiktor network=10.70.1.0
add address=172.16.1.1/24 interface=MGMT network=172.16.1.0
add address=10.60.1.1/24 interface=Serwery network=10.60.1.0
add address=10.10.10.1/25 interface=Wirtualizator-Prox network=10.10.10.0
add address=10.10.10.129/25 interface=Storage network=10.10.10.128
/ip dhcp-server lease
add address=192.168.0.6 mac-address=00:D8:61:78:B2:34 use-src-mac=yes
add address=192.168.0.5 mac-address=00:D8:61:78:B2:35 use-src-mac=yes
add address=192.168.0.214 comment=Mariusz mac-address=B4:2E:99:67:A7:EB \
use-src-mac=yes
add address=192.168.0.196 comment=Szymon mac-address=4C:CC:6A:AF:11:51 \
use-src-mac=yes
add address=192.168.1.250 client-id=1:e0:db:55:c4:4c:b mac-address=\
E0:DB:55:C4:4C:0B server=Serwer_Lab use-src-mac=yes
add address=192.168.0.4 comment=PiHole mac-address=00:A0:98:5A:CC:A6 \
use-src-mac=yes
/ip dhcp-server network
add address=192.168.0.0/24 dns-server=192.168.0.4 gateway=192.168.0.1
add address=192.168.1.0/24 dns-server=192.168.0.4 gateway=192.168.1.1
/ip dns
set servers=1.1.1.1,1.0.0.1
/ip firewall address-list
add address=192.168.0.0/24 list=Adresy_calej_sieci
add address=192.168.0.196 list=Komp
add address=192.168.0.214 list=Komp
add list=ddos-attackers
add list=ddos-target
/ip firewall filter
add action=accept chain=input src-address=192.168.2.0/24
add action=accept chain=input src-address=
add action=accept chain=forward src-address=192.168.2.0/24
add action=accept chain=input dst-port=23119 protocol=tcp
add action=return chain=detect-ddos dst-limit=32,32,src-and-dst-addresses/10s
add action=add-dst-to-address-list address-list=ddos-target \
address-list-timeout=10m chain=detect-ddos
add action=add-src-to-address-list address-list=ddos-attackers \
address-list-timeout=10m chain=detect-ddos
add action=drop chain=input dst-port=8291 in-interface=pppoe-out1 protocol=\
tcp
add action=drop chain=input in-interface=pppoe-out1 protocol=icmp
add action=add-src-to-address-list address-list=Syn_Flooder \
address-list-timeout=30m chain=input comment=\
"Add Syn Flood IP to the list" connection-limit=30,32 protocol=tcp \
tcp-flags=syn
add action=drop chain=input comment="Drop to syn flood list" \
src-address-list=Syn_Flooder
add action=add-src-to-address-list address-list=Port_Scanner \
address-list-timeout=1w chain=input comment="Port Scanner Detect" \
protocol=tcp psd=21,3s,3,1
add action=drop chain=input comment="Drop to port scan list" \
src-address-list=Port_Scanner
add action=jump chain=input comment="Jump for icmp input flow" jump-target=\
ICMP protocol=icmp
add action=jump chain=forward comment="Jump for icmp forward flow" \
jump-target=ICMP protocol=icmp
add action=drop chain=forward comment="Drop to bogon list" dst-address-list=\
bogons
add action=drop chain=forward comment="Avoid spammers action" dst-port=25,587 \
protocol=tcp src-address-list=spammers
add action=accept chain=input comment="Accept DNS - UDP" port=53 protocol=udp
add action=accept chain=input comment="Accept DNS - TCP" port=53 protocol=tcp
add action=accept chain=input comment="Accept to established connections" \
connection-state=established
add action=accept chain=input comment="Accept to related connections" \
connection-state=related
add action=accept chain=input comment="Full access to SUPPORT address list" \
src-address-list=support
add action=accept chain=ICMP comment=\
"Echo request - Avoiding Ping Flood, adjust the limit as needed" \
icmp-options=8:0 limit=2,5:packet protocol=icmp
add action=accept chain=ICMP comment="Echo reply" icmp-options=0:0 protocol=\
icmp
add action=accept chain=ICMP comment="Time Exceeded" icmp-options=11:0 \
protocol=icmp
add action=accept chain=ICMP comment="Destination unreachable" icmp-options=\
3:0-1 protocol=icmp
add action=accept chain=ICMP comment=PMTUD icmp-options=3:4 protocol=icmp
add action=drop chain=ICMP comment="Drop to the other ICMPs" protocol=icmp
add action=jump chain=output comment="Jump for icmp output" jump-target=ICMP \
protocol=icmp
/ip firewall nat
add action=masquerade chain=srcnat out-interface=pppoe-out1
add action=dst-nat chain=dstnat dst-address= dst-address-type="" \
dst-port=80 protocol=tcp src-address-type="" to-addresses=10.60.1.8 \
to-ports=80
add action=dst-nat chain=dstnat dst-address= dst-port=443 \
protocol=tcp to-addresses=10.60.1.8 to-ports=443
add action=masquerade chain=srcnat disabled=yes dst-address=!10.60.1.8 \
src-address=192.168.0.0/24
add action=dst-nat chain=dstnat dst-address=10.60.1.8 dst-address-type=local \
dst-port=80 protocol=tcp to-addresses=10.60.1.8 to-ports=80
add action=dst-nat chain=dstnat dst-address=10.60.1.8 dst-address-type=local \
dst-port=443 protocol=tcp to-addresses=10.60.1.8 to-ports=443
add action=dst-nat chain=dstnat disabled=yes dst-address= \
dst-port=21 protocol=tcp to-addresses=192.168.0.10 to-ports=21
add action=dst-nat chain=dstnat disabled=yes dst-address= \
dst-port=20 protocol=tcp to-addresses=192.168.0.10 to-ports=20
add action=dst-nat chain=dstnat disabled=yes dst-address= \
dst-port=49000-49100 protocol=tcp to-addresses=192.168.0.10 to-ports=\
49000-49100
add action=dst-nat chain=dstnat dst-address=dst-port=\
65400-65410 protocol=tcp to-addresses=10.70.1.3 to-ports=65400-65410
/ip firewall raw
add action=drop chain=prerouting dst-address-list=ddos-target \
src-address-list=ddos-attackers
/ip firewall service-port
set ftp disabled=yes
set tftp disabled=yes
/ip route
add distance=1 gateway= pref-src=0.0.0.0 scope=10
/ip service
set telnet disabled=yes port=65408
set ftp disabled=yes port=960
set www disabled=yes port=65480
set ssh disabled=yes port=65400
set api disabled=yes
set api-ssl disabled=yes
/routing rip interface
add send=v1-2
/snmp
set enabled=yes location=192.168.0.20 trap-generators=interfaces \
trap-interfaces=all
/system clock
set time-zone-name=Europe/Warsaw
/system gps
set set-system-time=yes
/tool bandwidth-server
set authenticate=no max-sessions=1000
/tool user-manager database
set db-path=flash/user-manager
Hi.
Just FYI, when you do a export use : export hide-sensitive so that passwords etc are not shown..
Looking over your config further shows me that you are doing a LOT of CPU intensive work.
As for switch chip native & vlan traffic, it looks like all your processing needs are CPU based not switch based so there is no point trying to use the switch chip configs.
Notably you have a significant amount of firewall rules, which for each one there is a lot of work you should be able to realistically drop that to < 6 rules in the /firewall filter for efficiency, I know you have a lot there for ddos/ syn flood, but if you take a birds eye view of required rules to effectively achieve the same thing, as rules wont stop people auto scanning anyhow, because even if they are unsuccessful, your router needs to waste time still looking at the traffic, match the rule and still drop the traffic, so you better off just blocking everything except what you need to come in.
Look at https://help.mikrotik.com/docs/display/ROS/Building+Advanced+Firewall.
/ip firewall filter
add action=accept chain=input comment=“defconf: accept ICMP after RAW” protocol=icmp
add action=accept chain=input comment=“defconf: accept established,related,untracked” connection-state=established,related,untracked
add action=drop chain=input comment=“defconf: drop all not coming from LAN” in-interface-list=!LAN
That is probably all you need, even then you can remove the ICMP rule if you want so you cant ICMP to the device( or just add the “in-interface” to just the WAN for specific stuff)
Dont go overboard on filtering stuff, unless your an ISP and your passing on( forwarding ) public IP addresses to downstream - LAN devices, if your front end device is just the router and its NAT(Masquerading ), then those rules about right for ( input chain filtering rules) .
As for your slowness. I suspect your RB760iGS( Hex S), simply isn’t up for the Job to move 100’s of MB of traffic a second, that cpu cannot move that much data for for what you want @ gigabit. You need a bigger/better cpu device to do it.
E.G look at
https://mikrotik.com/product/hex_s#fndtn-testresults
compare that to
https://mikrotik.com/product/rb4011igs_rm#fndtn-testresults
or
https://mikrotik.com/product/rb5009ug_s_in#fndtn-testresults
I would suggest if you have a small i5/i7 computer lying around around install CHR version of RouterOS on that and forget your processing limitation troubles( a get a P1 license from mikrotik ). I run it in VMWare Esxi and can move significant amounts of VLAN traffic thats filtered on a x86 box and it doesnt break a sweat !!
I also thought so that my mikrotik does not do with such movement, if I would switch vlans to CRS305-1G-4S + IN, would I have full 1 Gb / s wetedy?
Edit: I just realized you were possibly asking about inter-vlan (i.e. between vlan) performance, and that is going to be limited by routing performance. The following will improve performance between devices in the same vlan however, since in that case the CPU will never even see the packets. But if you are talking about traffic between vlans, then that traffic must be routed, and the MT7621 isn’t a high performance router.
Edit2: And I didn’t read the part about bonding (lag) and one core at 100%. You have reached a CPU limitation. what does /tool profile show?
What specific files would you need to evaluate my configuration?
aug/19/2022 18:10:36 by RouterOS 6.49.6
model = RB760iGS
Hardware offloading of vlan-filtering to switch ASIC in MT7621 was not added until v7.1-rc5 and there have been more patches to bridge since then. See note 3 in Switch Chip Features Introduction
Backup your config, then upgrade the hEX S to 7.4 stable.
Then see if it works better.
What is the reason for the lag (bonding1) on the hEX S? My guess is that is won’t help performance, and will probably reduce it. That is all done by the CPU if it is anything like on the Ubituiti ER-X which is also based on the MT7621.
Bonding is primarily a High Availability solution. See High Availability Solutions - Bonding
Also see Manual:Layer2 misconfiguration - LAG interfaces and load balancing and
I updated and it didn’t do much. Why isn’t the whole CPU running 100% with just one core?
I am not sure exactly what that question means. The MT7621 has two cores, with hyperthreading to make it look like 4 virtual cores. Was the question “why isn’t one CPU 100% saturated”? I don’t know for sure, but I would guess is has to do with the load and the way the linux kernel does SMP scheduling.
You evidently didn’t see the edits I made. Since the hEX S is routing, offloading to the switch will have little effect, since all inter-vlan routing is done by the CPU.
It would be interesting to see what the performance was like without bonding.
I don’t know if the bonding CPU usage shows up under Networking or not. The profiler documentation for v6 only states “networking - core packet processing”, and the v7 documentation has nothing but the statement that the classifications “are self-explanatory and do not require detailed explanation.” This seems to be a case where the limited documentation in V6 was better than the V7 lack of any documentation, other than the “name”.
Can hex S not use a chip switch to transfer data at full speed? Maybe you need to change some settings of the chip switch?
I assume you understand the difference between routing and switching?
The hEX S running v7.4 can switch traffic with the ASIC. But that isn’t what is used when going between vlans; that requires routing. And Routing requires the CPU, and traversing the “internal trunk” link between the switch ASIC and the CPU. And that limits aggregate routing bandwidth.
Here’s a post from ycombinator by tbyehl - it had links to these images https://imgur.com/a/vtgUp02 which describe 1Gb aggregate limitation that applies to any “1-armed router” or “router on a stick” (e.g. a single full duplex 1Gbps link to a router).
this is how I understand the differences between routing and switching, will CRS305-1G-4S + IN under RouterOs control serve me traffic on the full range of the link, i.e. 1 GB / s or 10 GB / s?
Perhaps someone else can help you.
To me it seems you have at least one configuration error. If you are going to specify the pvid on the bridge, then you should not create a vlan interface with the same vlan id (the MGMT interface). Instead you should use bridge1 where you use MGMT. @anav will say you should never specify a pvid for a bridge. I am not that strict, but you shouldn’t do both at the same time. I am not even sure what it actually does, when you do. ROS is very happy to allow you to enter invalid configuration without any warnings; it assumes you know what you want to do, and interprets what you enter in some undefined way.
Normally the “link” between the CPU’s base bridge interface and the Switch ASIC will be untagged, but you have told it to use tagged (in the /interface bridge vlan stanza).
And if performance is your goal, I would at least try not using bond (LAG) between the hEX S and the CRS305. I don’t have the ability to test with the equipment I have, but my guess is that it will be slower than a single link (and force the involvement of the CPU) even within the same vlan.
I have highlighted what I think are incompatible parts of your config. Perhaps @sindy or @tdw will chime in, as they know much more about the ROS bridge than I do.
aug/19/2022 18:10:36 by RouterOS 6.49.6
software id = NXZU-0IEV
model = RB760iGS
/interface bridge
add name=bridge1 pvid=999 vlan-filtering=yes
/interface vlan
add interface=bridge1 name=MGMT vlan-id=999
/dude
set enabled=yes
/interface bridge port
add bridge=bridge1 interface=MGMT pvid=999
/interface bridge vlan
add bridge=bridge1 tagged=bonding1,bridge1,MGMT vlan-ids=999
/ip address
add address=172.16.1.1/24 interface=MGMT network=172.16.1.0