Community discussions

MikroTik App
 
thefriendlyguy
just joined
Topic Author
Posts: 15
Joined: Fri Jul 01, 2016 10:50 am

CRS317-1G-16+ on 7.1rc3: IPFix with wrong timestamp, terrible InterVLAN Routing performance

Sun Sep 12, 2021 2:00 pm

Hi there!
I am owner of a CRS317-1G-16+.
The device is pretty much sitting in a cupboard and collecting dust most of the time (1 year, give or take), since the features I am looking for are not working as one might would expect.
I pull it out and flash a new development release of router-os every couple of months.
The most important features for me are: wire-speed Inter-VLAN routing and IPFIX. That would be the first big step.
Current status is:
- Not even close to wirespeed(10G), i get throughput in the range of 300-400 mbit/s when measuring with ipferf between two vlans.
Yes, i enabled hw-offloading:
[admin@BackBone] > /interface/ethernet/switch/ print
Columns: NAME, TYPE, L3-HW-OFFLOADING
# NAME     TYPE              L3-HW-OFFLOADING
0 switch1  Marvell-98DX8216  yes    
I can "see" the packets hitting the cpu as its load increases dramatically as soon as i begin a test.
The bridge ports alll show as HW Offload, but its obviously not the case.
[admin@BackBone] /interface> /interface/bridge/port/ print
Flags: I - INACTIVE; H - HW-OFFLOAD
Columns: INTERFACE, BRIDGE, HW, PVID, PRIORITY, PATH-COST, INTERNAL-PATH-COST, HORIZON
 #    INTERFACE                   BRIDGE   HW   PVID  PRIORITY  PATH-COST  INTERNAL-PATH-COST  HORIZON
 0  H sfp-sfpplus1		   bridge1  yes     1  0x80             10                  10  none   
 1 IH sfp-sfpplus2                bridge1  yes     1  0x80             10                  10  none   
 2 IH sfp-sfpplus3		   bridge1  yes     1  0x80             10                  10  none   
 3 IH sfp-sfpplus4		   bridge1  yes     1  0x80             10                  10  none   
 4  H sfp-sfpplus5		   bridge1  yes     1  0x80             10                  10  none   
 5  H sfp-sfpplus6		   bridge1  yes     1  0x80             10                  10  none   
 6 IH sfp-sfpplus7		   bridge1  yes     1  0x80             10                  10  none   
 7 IH sfp-sfpplus8	  	   bridge1  yes     1  0x80             10                  10  none   
 8  H sfp-sfpplus9	  	   bridge1  yes     1  0x80             10                  10  none   
 9  H sfp-sfpplus10		   bridge1  yes   133  0x80             10                  10  none   
10 IH sfp-sfpplus11	 	   bridge1  yes     1  0x80             10                  10  none   
11 IH sfp-sfpplus12             bridge1  yes     1  0x80             10                  10  none   
12 IH sfp-sfpplus13             bridge1  yes     1  0x80             10                  10  none   
13  H sfp-sfpplus14		   bridge1  yes     1  0x80             10                  10  none   
14  H LACP-Trunk2CCR2004          bridge1  yes     1  0x80             10                  10  none  
And I also have a fasttrack-connecction fw rule:
/ip firewall filter
add action=fasttrack-connection chain=forward connection-state=established,related,untracked hw-offload=yes

- IPFix "works" to some degree: Its just that ALL timestamps are messed up: They are dated to 01.01.1970.
Yes, the system has a correct time and uses ntp:
[admin@BackBone] > /system/clock/ print
                  time: 12:40:05
                  date: sep/12/2021
  time-zone-autodetect: yes
        time-zone-name: Europe/Zurich
            gmt-offset: +02:00
            dst-active: yes
I tried all versions of netflow as well as ipfix, date issue is the same for all of them.
I hope my feedback helps in further development of routeros: feel free to ask questions.
If somebody has a advice for me: Please, tell me :)

Kind regards
 
User avatar
raimondsp
MikroTik Support
MikroTik Support
Posts: 267
Joined: Mon Apr 27, 2020 10:14 am

Re: CRS317-1G-16+ on 7.1rc3: IPFix with wrong timestamp, terrible InterVLAN Routing performance

Mon Sep 13, 2021 9:50 am

300-400 mbit/s is the maximum routing speed that CRS317's CPU is capable of. That means the routing is performed by the CPU, not the hardware.
Please provide the output of the following commands:
/interface export
/ip export
 
thefriendlyguy
just joined
Topic Author
Posts: 15
Joined: Fri Jul 01, 2016 10:50 am

Re: CRS317-1G-16+ on 7.1rc3: IPFix with wrong timestamp, terrible InterVLAN Routing performance

Tue Sep 14, 2021 12:40 pm

Hi there!
Thanks for your swift response. Sure, its using the cpu, thats obvious. Not sure why though.
Here is the output of /interface export:
/interface bridge
add name=bridge1 vlan-filtering=yes
/interface ethernet
set [ find default-name=ether1 ] l2mtu=1592 name=ether1
set [ find default-name=sfp-sfpplus1 ] l2mtu=1592 mtu=1592 name=sfp-sfpplus1
set [ find default-name=sfp-sfpplus2 ] disabled=yes l2mtu=1592 mtu=1592
set [ find default-name=sfp-sfpplus3 ] l2mtu=1592 mtu=1592 name=sfp-sfpplus3
set [ find default-name=sfp-sfpplus4 ] l2mtu=1592 mtu=1592 name=sfp-sfpplus4
set [ find default-name=sfp-sfpplus5 ] l2mtu=1592 mtu=1592 name=sfp-sfpplus5
set [ find default-name=sfp-sfpplus6 ] l2mtu=1592 mtu=1592 name=sfp-sfpplus6
set [ find default-name=sfp-sfpplus7 ] l2mtu=1592 mtu=1592 name=sfp-sfpplus7
set [ find default-name=sfp-sfpplus8 ] l2mtu=1592 mtu=1592 name=sfp-sfpplus8
set [ find default-name=sfp-sfpplus9 ] l2mtu=1592 mtu=1592 name=sfp-sfpplus9
set [ find default-name=sfp-sfpplus10 ] l2mtu=1592 mtu=1592 name=sfp-sfpplus10
set [ find default-name=sfp-sfpplus11 ] l2mtu=1592 mtu=1592 name=sfp-sfpplus11
set [ find default-name=sfp-sfpplus12 ] disabled=yes l2mtu=1592 mtu=1592
set [ find default-name=sfp-sfpplus13 ] disabled=yes l2mtu=1592 mtu=1592
set [ find default-name=sfp-sfpplus14 ] l2mtu=1592 mtu=1592 name=sfp-sfpplus14
set [ find default-name=sfp-sfpplus15 ] l2mtu=1592 mtu=1592 name=sfp-sfpplus15
set [ find default-name=sfp-sfpplus16 ] l2mtu=1592 mtu=1592 name=sfp-sfpplus16

/interface vlan
add interface=bridge1 name=vlan1 vlan-id=1
add interface=bridge1 name=vlan2 vlan-id=2
add interface=bridge1 name=vlan100 vlan-id=100
add interface=bridge1 name=vlan101 vlan-id=101
add interface=bridge1 name=vlan251 vlan-id=251
add interface=bridge1 name=vlan254 vlan-id=254
/interface bonding
add mode=802.3ad name=LACP slaves=sfp-sfpplus15,sfp-sfpplus16
/interface ethernet switch
set 0 l3-hw-offloading=yes
/interface lte apn
set [ find default=yes ] ip-type=ipv4
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/interface bridge port
add bridge=bridge1 interface=sfp-sfpplus1
add bridge=bridge1 interface=sfp-sfpplus2
add bridge=bridge1 interface=sfp-sfpplus3
add bridge=bridge1 interface=sfp-sfpplus4
add bridge=bridge1 interface=sfp-sfpplus5
add bridge=bridge1 interface=sfp-sfpplus6
add bridge=bridge1 interface=sfp-sfpplus7
add bridge=bridge1 interface=sfp-sfpplus8
add bridge=bridge1 interface=sfp-sfpplus9
add bridge=bridge1 interface=sfp-sfpplus10 pvid=133
add bridge=bridge1 interface=sfp-sfpplus11
add bridge=bridge1 interface=sfp-sfpplus12
add bridge=bridge1 interface=sfp-sfpplus13
add bridge=bridge1 interface=sfp-sfpplus14
add bridge=bridge1 interface=LACP

/interface bridge vlan
add bridge=bridge1 tagged=LACP,sfp-sfpplus14 vlan-ids=10
add bridge=bridge1 tagged=sfp-sfpplus1,sfp-sfpplus14,sfp-sfpplus11 vlan-ids=133
add bridge=bridge1 tagged=bridge1 untagged=sfp-sfpplus1,sfp-sfpplus14,vlan1 vlan-ids=1
add bridge=bridge1 tagged="LACP,sfp-sfpplus14,sfp-sfpplus1,sfp-sfpplus5,sfp-sfpplus6,sfp-sfpplus7,sfp-sfpplus8,s\
    fp-sfpplus3,sfp-sfpplus4,bridge1" vlan-ids=251
add bridge=bridge1 tagged=LACP,sfp-sfpplus5,sfp-sfpplus6,sfp-sfpplus7,sfp-sfpplus8,sfp-sfpplus3,sfp-sfpplus4 vlan-ids=66
add bridge=bridge1 tagged=sfp-sfpplus14,sfp-sfpplus5,sfp-sfpplus6,sfp-sfpplus7,sfp-sfpplus8,sfp-sfpplus3,sfp-sfpplus4 vlan-ids=\
    13
add bridge=bridge1 tagged=sfp-sfpplus5,sfp-sfpplus6,sfp-sfpplus7,sfp-sfpplus8,sfp-sfpplus9,sfp-sfpplus3,sfp-sfpplus4,bridge1 \
    vlan-ids=2
add bridge=bridge1 tagged=sfp-sfpplus5,sfp-sfpplus6,sfp-sfpplus7,sfp-sfpplus8,sfp-sfpplus14,sfp-sfpplus3,sfp-sfpplus4 vlan-ids=\
    117
add bridge=bridge1 tagged="sfp-sfpplus5,sfp-sfpplus6,sfp-sfpplus7,sfp-sfpplus8,sfp-sfpplus14,sfp-sfpplus9,sfp-sfpplus11,s\
    fp-sfpplus3,sfp-sfpplus4,bridge1" vlan-ids=100
add bridge=bridge1 tagged=sfp-sfpplus5,sfp-sfpplus6,sfp-sfpplus7,sfp-sfpplus8,sfp-sfpplus3,sfp-sfpplus4,bridge1 vlan-ids=101
add bridge=bridge1 tagged=sfp-sfpplus5,sfp-sfpplus6,sfp-sfpplus7,sfp-sfpplus8,sfp-sfpplus3,sfp-sfpplus4 vlan-ids=102
add bridge=bridge1 tagged=sfp-sfpplus5,sfp-sfpplus6,sfp-sfpplus7,sfp-sfpplus8,sfp-sfpplus3,sfp-sfpplus4,bridge1 vlan-ids=254
and the output of "/ip export":

/ip address
add address=10.0.2.254/24 interface=vlan2 network=10.0.2.0
add address=10.0.1.254/24 interface=vlan1 network=10.0.1.0
add address=10.0.100.254/24 interface=vlan100 network=10.0.100.0
add address=10.0.101.254/24 interface=vlan101 network=10.0.101.0
add address=10.0.254.254/24 interface=vlan254 network=10.0.254.0
add address=10.0.232.254/24 interface=vlan251 network=10.0.232.0
/ip dhcp-relay
add dhcp-server=10.0.2.30 disabled=no interface=vlan1 local-address=10.0.1.254 name=ip-helper-vlan1
add dhcp-server=10.0.2.30 disabled=no interface=vlan100 name=ip-helper-vlan100
add dhcp-server=10.0.2.30 disabled=no interface=vlan101 name=ip-helper-vlan101
/ip dns
set servers=10.0.2.30,10.0.2.40
/ip neighbor discovery-settings
set discover-interface-list=!dynamic
/ip settings
set rp-filter=strict tcp-syncookies=yes
/ip firewall filter
add action=reject chain=forward out-interface=vlan251 reject-with=icmp-admin-prohibited
add action=reject chain=forward in-interface=vlan251 reject-with=icmp-admin-prohibited
add action=fasttrack-connection chain=forward connection-state=established,related,untracked hw-offload=yes
add action=accept chain=forward connection-state=established,related,untracked
add action=reject chain=input disabled=yes reject-with=icmp-admin-prohibited src-address=!10.0.232.0/24
/ip route
add disabled=no dst-address=0.0.0.0/0 gateway=10.0.254.1
/ip service
set telnet address=10.0.232.0/24 disabled=yes
set ftp address=10.0.232.0/24
set www address=10.0.232.0/24
set ssh address=10.0.232.0/24
set www-ssl address=10.0.232.0/24
set api address=10.0.232.0/24
set winbox address=10.0.232.0/24
set api-ssl address=10.0.232.0/24
/ip traffic-flow
set enabled=yes
/ip traffic-flow target
add dst-address=10.0.232.16 port=4739 src-address=10.0.232.254 version=ipfix
Please note: There is another bug i discovered while testing.
Its with the "export" command: Some settings have a trailing and ending ". This prevents one to quickly import the config previously exported.
One has to remove those ". Example:
This one would not work:
/interface bridge vlan 
add bridge=bridge1 tagged="sfp-sfpplus5,sfp-sfpplus6,sfp-sfpplus7,sfp-sfpplus8,sfp-sfpplus14,sfp-sfpplus9,sfp-sfpplus11,s\
    fp-sfpplus3,sfp-sfpplus4,bridge1" vlan-ids=100
This one would work:
/interface bridge vlan 
add bridge=bridge1 tagged=sfp-sfpplus5,sfp-sfpplus6,sfp-sfpplus7,sfp-sfpplus8,sfp-sfpplus14,sfp-sfpplus9,sfp-sfpplus11,s\
    fp-sfpplus3,sfp-sfpplus4,bridge1 vlan-ids=100
Thanks for your help!
Kind regards
 
User avatar
raimondsp
MikroTik Support
MikroTik Support
Posts: 267
Joined: Mon Apr 27, 2020 10:14 am

Re: CRS317-1G-16+ on 7.1rc3: IPFix with wrong timestamp, terrible InterVLAN Routing performance

Fri Sep 17, 2021 1:45 pm

Hey again, and thanks for the feedback!

I reported the import/export issue and waiting for the fix.
Meanwhile, we are trying to reproduce your case. I'll keep you in touch.
 
User avatar
raimondsp
MikroTik Support
MikroTik Support
Posts: 267
Joined: Mon Apr 27, 2020 10:14 am

Re: CRS317-1G-16+ on 7.1rc3: IPFix with wrong timestamp, terrible InterVLAN Routing performance

Fri Sep 17, 2021 4:16 pm

I was unable to reproduce your issue. I did a similar setup with 802.3ad bonding and VLAN bridge with both tagged and untagged interfaces. And L3HW offloading clearly worked as intended.

Here is my setup:
/interface bridge
add name=bridge vlan-filtering=yes

/interface vlan
add interface=bridge name=vlan10 vlan-id=10
add interface=bridge name=vlan254 vlan-id=254

/interface bonding
add mode=802.3ad name=LACP slaves=sfp-sfpplus15,sfp-sfpplus16

/interface ethernet switch
set 0 l3-hw-offloading=yes

/interface bridge port
add bridge=bridge interface=ether1 pvid=254
add bridge=bridge interface=LACP pvid=10
add bridge=bridge interface=sfp-sfpplus1 pvid=254

/interface bridge vlan
add bridge=bridge tagged=bridge,LACP untagged=sfp-sfpplus1 vlan-ids=10
add bridge=bridge tagged=bridge untagged=ether1 vlan-ids=254

/ip address
add address=192.168.1.17/24 interface=vlan254 network=192.168.1.0
add address=192.168.16.17/24 interface=vlan10 network=192.168.16.0

/ip route
add dst-address=192.168.16.0/22 gateway=192.168.16.26
I did iperf3 from a client (192.168.17.5) connected to sfp-sfpplus1 (untagged VLAN 254) to a server (192.168.1.74) connected via another switch (CRS326), which, in turn, connected via LACP to CRS317 (tagged VLAN 10).

iperf stats with L3HW enabled (CRS317 CPU usage near 0%):
Accepted connection from 192.168.17.5, port 49320
[  5] local 192.168.1.74 port 5201 connected to 192.168.17.5 port 49322
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   111 MBytes   934 Mbits/sec                  
[  5]   1.00-2.00   sec   111 MBytes   934 Mbits/sec                  
[  5]   2.00-3.00   sec   111 MBytes   935 Mbits/sec                  
[  5]   3.00-4.00   sec   111 MBytes   934 Mbits/sec                  
[  5]   4.00-5.00   sec   111 MBytes   935 Mbits/sec                  
[  5]   5.00-6.00   sec   111 MBytes   934 Mbits/sec                  
[  5]   6.00-7.00   sec   111 MBytes   935 Mbits/sec                  
[  5]   7.00-8.00   sec   111 MBytes   935 Mbits/sec                  
[  5]   8.00-9.00   sec   111 MBytes   934 Mbits/sec                  
[  5]   9.00-10.00  sec   111 MBytes   935 Mbits/sec                  
[  5]  10.00-10.00  sec   197 KBytes   835 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  1.09 GBytes   934 Mbits/sec                  receiver
Note that both computers running iperf3 have a 1G interface, so that was a bottleneck. But you can clearly see a near-wire-speed performance. We can redo the tests in the lab with 10G interfaces, but I think it is redundant.


iperf stats with L3HW disabled (CRS317 CPU usage 100%):
Accepted connection from 192.168.17.5, port 49298
[  5] local 192.168.1.74 port 5201 connected to 192.168.17.5 port 49300
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  38.3 MBytes   321 Mbits/sec                  
[  5]   1.00-2.00   sec  39.0 MBytes   327 Mbits/sec                  
[  5]   2.00-3.00   sec  38.8 MBytes   325 Mbits/sec                  
[  5]   3.00-4.00   sec  38.8 MBytes   326 Mbits/sec                  
[  5]   4.00-5.00   sec  38.8 MBytes   326 Mbits/sec                  
[  5]   5.00-6.00   sec  38.8 MBytes   326 Mbits/sec                  
[  5]   6.00-7.00   sec  38.8 MBytes   326 Mbits/sec                  
[  5]   7.00-8.00   sec  38.9 MBytes   326 Mbits/sec                  
[  5]   8.00-9.00   sec  38.8 MBytes   326 Mbits/sec                  
[  5]   9.00-10.00  sec  38.8 MBytes   326 Mbits/sec                  
[  5]  10.00-10.04  sec  1.48 MBytes   321 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.04  sec   389 MBytes   325 Mbits/sec                  receiver
Keep in mind that enabling l3-hw-offloading=yes may not affect the existing connections, and the traffic still may continue via the CPU. Flush the ARP table to force offload of all active hosts:
/ip/arp/remove [find]
 
User avatar
raimondsp
MikroTik Support
MikroTik Support
Posts: 267
Joined: Mon Apr 27, 2020 10:14 am

Re: CRS317-1G-16+ on 7.1rc3: IPFix with wrong timestamp, terrible InterVLAN Routing performance

Fri Sep 17, 2021 4:22 pm

I have found an issue in your setup:
/interface bridge vlan
add bridge=bridge1 tagged=bridge1 untagged=sfp-sfpplus1,sfp-sfpplus14,vlan1 vlan-ids=1

/interface vlan
add interface=bridge1 name=vlan1 vlan-id=1
Circular reference: vlan1 marked as a tagged interface of bridge1, but bridge1 is the interface under vlan1.
Please remove vlan1 from tagged interfaces (and keep l3-hw-enabled=no while doing that) and redo the test.
Just to be sure, flush the ARP table before tests:
/ip/arp/remove [find]
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11381
Joined: Thu Mar 03, 2016 10:23 pm

Re: CRS317-1G-16+ on 7.1rc3: IPFix with wrong timestamp, terrible InterVLAN Routing performance

Fri Sep 17, 2021 5:34 pm

Circular reference: vlan1 marked as a tagged interface of bridge1, but bridge1 is the interface under vlan1.

Would it be possible for command interpreter to detect such circular references? They seem to be quite frequent for inexperienced users ...
 
thefriendlyguy
just joined
Topic Author
Posts: 15
Joined: Fri Jul 01, 2016 10:50 am

Re: CRS317-1G-16+ on 7.1rc3: IPFix with wrong timestamp, terrible InterVLAN Routing performance

Mon Sep 27, 2021 10:34 am

Good morning!
I had some time over the weekend to test the changes you suggested.
I can confirm: This moved routing away from the cpu. I achieved somewhat close to wirespeed. 9 Gbps. I am fine with that! Thanks a lot!
Also: I upgraded to RC4, which fixed the ipfix timestamps. Though i think the ipfix data is somewhat unreliable as i wasnt able to see my iperf testing, no matter how long i let it run. (At first i thought my test was to short.) I fear that only the connection gets tracked but as soon as the packets hit the switching fabric (is this a fpga or a asic?) no performance data is collected.

Also, I encountered some new problems. For a unknown reason the CRS decided to NOT route some of my vlans. On every new reboot of the switch another range of vlans was affected.
For example: I couldn't access the mikrotik from my management vlan. I tried to pinpoint the problem for over 2 hours, but i couldnt find the problem. So i rebooted the switch.
After a reboot I was able to access it but all my vms (from the vm vlan) couldnt reach the internet. CRS did use the default route and could contact internet hosts. Some VLANS could as well, but not my vm vlan.
After i rebooted again: They were working again. But another vlan / subnet wouldnt work...And so on.
Every time i rebooted a different vlan / subnet wasnt working.
I didint have enough time to determine the root cause. After about 12 hours straight i decided to power down the switch again and move back to my ES-16-XG.
 
User avatar
raimondsp
MikroTik Support
MikroTik Support
Posts: 267
Joined: Mon Apr 27, 2020 10:14 am

Re: CRS317-1G-16+ on 7.1rc3: IPFix with wrong timestamp, terrible InterVLAN Routing performance

Mon Sep 27, 2021 11:13 am

Good morning!

Nice to hear that hardware routing works and ipfix got fixed! Connection tracking does not work with hardware routing since the packets never enter the CPU, and the switch chip does not provide the routing performance data.

Your random VLAN inaccessibility sounds weird. Do VLAN members use dynamic or static IP addresses? If dynamic, where DHPC server is located? Please post your latest config:
/interface export
/ip export

Who is online

Users browsing this forum: No registered users and 21 guests