CCR2004-1G-12S+2XS: IPv4 routing performance less than IPv6?

RouterOS General
1

Hi all,

I’m trying to troubleshoot some speed issue which led me to this weird performance differences between IPv4 and IPv6 - hoping I can get some help narrow down the bottleneck.

I understand throughout for routing traffic going through CCR2004 is likely to be CPU bound, but I wasn’t expecting IPv4 to be slower than that of IPv6, thus this post.

Setup

All connections are with SFP+ auto negotiated at 10Gbps - I understand LAGG will sacrifice some performance, I’m only looking to understand the discrepancy of performance between IPv4 and IPv6 which I believe LAGG is unlikely to contribute to this issue.
All hosts get both IPv4 and ULA IPv6 addresses.

All interfaces on RouterOS has Tx and Rx flow control set to auto, though this did not seem to have affected the test result.

Benchmark

Sending from Host 2 to Host 1
IPv6 - 7.73 Gbits/sec

$ iperf3 -c host1.mydomain.example -t30 -i30 -P5 -6
Connecting to host host1.mydomain.example, port 5201
[  5] local fdfd::a00:a00:27ff:fe60:e32e port 46804 connected to fdfd:0:0:a00::abcd port 5201
[  7] local fdfd::a00:a00:27ff:fe60:e32e port 46820 connected to fdfd:0:0:a00::abcd port 5201
[  9] local fdfd::a00:a00:27ff:fe60:e32e port 46830 connected to fdfd:0:0:a00::abcd port 5201
[ 11] local fdfd::a00:a00:27ff:fe60:e32e port 46838 connected to fdfd:0:0:a00::abcd port 5201
[ 13] local fdfd::a00:a00:27ff:fe60:e32e port 46846 connected to fdfd:0:0:a00::abcd port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-30.00  sec  4.98 GBytes  1.43 Gbits/sec  1417    653 KBytes       
[  7]   0.00-30.00  sec  5.82 GBytes  1.67 Gbits/sec  1923    700 KBytes       
[  9]   0.00-30.00  sec  5.37 GBytes  1.54 Gbits/sec  1624    544 KBytes       
[ 11]   0.00-30.00  sec  4.99 GBytes  1.43 Gbits/sec  2301    718 KBytes       
[ 13]   0.00-30.00  sec  5.90 GBytes  1.69 Gbits/sec  2073    502 KBytes       
[SUM]   0.00-30.00  sec  27.1 GBytes  7.75 Gbits/sec  9338             
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-30.00  sec  4.98 GBytes  1.43 Gbits/sec  1417             sender
[  5]   0.00-30.03  sec  4.98 GBytes  1.42 Gbits/sec                  receiver
[  7]   0.00-30.00  sec  5.82 GBytes  1.67 Gbits/sec  1923             sender
[  7]   0.00-30.03  sec  5.81 GBytes  1.66 Gbits/sec                  receiver
[  9]   0.00-30.00  sec  5.37 GBytes  1.54 Gbits/sec  1624             sender
[  9]   0.00-30.03  sec  5.37 GBytes  1.54 Gbits/sec                  receiver
[ 11]   0.00-30.00  sec  4.99 GBytes  1.43 Gbits/sec  2301             sender
[ 11]   0.00-30.03  sec  4.98 GBytes  1.43 Gbits/sec                  receiver
[ 13]   0.00-30.00  sec  5.90 GBytes  1.69 Gbits/sec  2073             sender
[ 13]   0.00-30.03  sec  5.90 GBytes  1.69 Gbits/sec                  receiver
[SUM]   0.00-30.00  sec  27.1 GBytes  7.75 Gbits/sec  9338             sender
[SUM]   0.00-30.03  sec  27.0 GBytes  7.73 Gbits/sec                  receiver

iperf Done.

IPv4 - 2.57 Gbits/sec

$ iperf3 -c host1.mydomain.example -t30 -i30 -P5 -4
Connecting to host host1.mydomain.example, port 5201
[  5] local 10.10.11.143 port 33874 connected to 10.10.10.213 port 5201
[  7] local 10.10.11.143 port 33888 connected to 10.10.10.213 port 5201
[  9] local 10.10.11.143 port 33890 connected to 10.10.10.213 port 5201
[ 11] local 10.10.11.143 port 33892 connected to 10.10.10.213 port 5201
[ 13] local 10.10.11.143 port 33896 connected to 10.10.10.213 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-30.00  sec  2.24 GBytes   641 Mbits/sec  18934   49.5 KBytes       
[  7]   0.00-30.00  sec  1.61 GBytes   462 Mbits/sec  16267   73.5 KBytes       
[  9]   0.00-30.00  sec  1.87 GBytes   537 Mbits/sec  16611   24.0 KBytes       
[ 11]   0.00-30.00  sec  1.67 GBytes   477 Mbits/sec  16438   58.0 KBytes       
[ 13]   0.00-30.00  sec  1.59 GBytes   457 Mbits/sec  16502   38.2 KBytes       
[SUM]   0.00-30.00  sec  8.99 GBytes  2.57 Gbits/sec  84752             
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-30.00  sec  2.24 GBytes   641 Mbits/sec  18934             sender
[  5]   0.00-30.03  sec  2.24 GBytes   640 Mbits/sec                  receiver
[  7]   0.00-30.00  sec  1.61 GBytes   462 Mbits/sec  16267             sender
[  7]   0.00-30.03  sec  1.61 GBytes   461 Mbits/sec                  receiver
[  9]   0.00-30.00  sec  1.87 GBytes   537 Mbits/sec  16611             sender
[  9]   0.00-30.03  sec  1.87 GBytes   536 Mbits/sec                  receiver
[ 11]   0.00-30.00  sec  1.67 GBytes   477 Mbits/sec  16438             sender
[ 11]   0.00-30.03  sec  1.66 GBytes   476 Mbits/sec                  receiver
[ 13]   0.00-30.00  sec  1.59 GBytes   457 Mbits/sec  16502             sender
[ 13]   0.00-30.03  sec  1.59 GBytes   456 Mbits/sec                  receiver
[SUM]   0.00-30.00  sec  8.99 GBytes  2.57 Gbits/sec  84752             sender
[SUM]   0.00-30.03  sec  8.98 GBytes  2.57 Gbits/sec                  receiver

iperf Done.

Reverse mode
IPv6 - 7.33 Gbits/sec

$ iperf3 -c host1.mydomain.example -t30 -i30 -P5 -6 -R
Connecting to host host1.mydomain.example, port 5201
Reverse mode, remote host host1.mydomain.example is sending
[  5] local fdfd::a00:a00:27ff:fe60:e32e port 39440 connected to fdfd:0:0:a00::abcd port 5201
[  7] local fdfd::a00:a00:27ff:fe60:e32e port 39456 connected to fdfd:0:0:a00::abcd port 5201
[  9] local fdfd::a00:a00:27ff:fe60:e32e port 39458 connected to fdfd:0:0:a00::abcd port 5201
[ 11] local fdfd::a00:a00:27ff:fe60:e32e port 39474 connected to fdfd:0:0:a00::abcd port 5201
[ 13] local fdfd::a00:a00:27ff:fe60:e32e port 39488 connected to fdfd:0:0:a00::abcd port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-30.00  sec  5.19 GBytes  1.48 Gbits/sec                  
[  7]   0.00-30.00  sec  5.09 GBytes  1.46 Gbits/sec                  
[  9]   0.00-30.00  sec  5.08 GBytes  1.46 Gbits/sec                  
[ 11]   0.00-30.00  sec  5.16 GBytes  1.48 Gbits/sec                  
[ 13]   0.00-30.00  sec  5.09 GBytes  1.46 Gbits/sec                  
[SUM]   0.00-30.00  sec  25.6 GBytes  7.33 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-30.03  sec  5.20 GBytes  1.49 Gbits/sec  107             sender
[  5]   0.00-30.00  sec  5.19 GBytes  1.48 Gbits/sec                  receiver
[  7]   0.00-30.03  sec  5.10 GBytes  1.46 Gbits/sec  626             sender
[  7]   0.00-30.00  sec  5.09 GBytes  1.46 Gbits/sec                  receiver
[  9]   0.00-30.03  sec  5.10 GBytes  1.46 Gbits/sec  142             sender
[  9]   0.00-30.00  sec  5.08 GBytes  1.46 Gbits/sec                  receiver
[ 11]   0.00-30.03  sec  5.10 GBytes  1.46 Gbits/sec  314             sender
[ 11]   0.00-30.00  sec  5.16 GBytes  1.48 Gbits/sec                  receiver
[ 13]   0.00-30.03  sec  5.17 GBytes  1.48 Gbits/sec    7             sender
[ 13]   0.00-30.00  sec  5.09 GBytes  1.46 Gbits/sec                  receiver
[SUM]   0.00-30.03  sec  25.7 GBytes  7.34 Gbits/sec  1196             sender
[SUM]   0.00-30.00  sec  25.6 GBytes  7.33 Gbits/sec                  receiver

iperf Done.

IPv4 - 2.87 Gbits/sec

$ iperf3 -c host1.mydomain.example -t30 -i30 -P5 -4 -R
Connecting to host host1.mydomain.example, port 5201
Reverse mode, remote host host1.mydomain.example is sending
[  5] local 10.10.11.143 port 59334 connected to 10.10.10.213 port 5201
[  7] local 10.10.11.143 port 59340 connected to 10.10.10.213 port 5201
[  9] local 10.10.11.143 port 59342 connected to 10.10.10.213 port 5201
[ 11] local 10.10.11.143 port 59350 connected to 10.10.10.213 port 5201
[ 13] local 10.10.11.143 port 59356 connected to 10.10.10.213 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-30.00  sec  1.99 GBytes   570 Mbits/sec                  
[  7]   0.00-30.00  sec  2.02 GBytes   578 Mbits/sec                  
[  9]   0.00-30.00  sec  1.99 GBytes   570 Mbits/sec                  
[ 11]   0.00-30.00  sec  2.01 GBytes   576 Mbits/sec                  
[ 13]   0.00-30.00  sec  2.01 GBytes   576 Mbits/sec                  
[SUM]   0.00-30.00  sec  10.0 GBytes  2.87 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-30.03  sec  1.99 GBytes   570 Mbits/sec  9278             sender
[  5]   0.00-30.00  sec  1.99 GBytes   570 Mbits/sec                  receiver
[  7]   0.00-30.03  sec  2.02 GBytes   579 Mbits/sec  9186             sender
[  7]   0.00-30.00  sec  2.02 GBytes   578 Mbits/sec                  receiver
[  9]   0.00-30.03  sec  2.00 GBytes   571 Mbits/sec  9067             sender
[  9]   0.00-30.00  sec  1.99 GBytes   570 Mbits/sec                  receiver
[ 11]   0.00-30.03  sec  2.02 GBytes   577 Mbits/sec  9239             sender
[ 11]   0.00-30.00  sec  2.01 GBytes   576 Mbits/sec                  receiver
[ 13]   0.00-30.03  sec  2.01 GBytes   576 Mbits/sec  10164             sender
[ 13]   0.00-30.00  sec  2.01 GBytes   576 Mbits/sec                  receiver
[SUM]   0.00-30.03  sec  10.0 GBytes  2.87 Gbits/sec  46934             sender
[SUM]   0.00-30.00  sec  10.0 GBytes  2.87 Gbits/sec                  receiver

iperf Done.

The initial test result seem to suggest IPv4 routing is prone to a larger amount of retransmissions, in the region of 10~40x more with IPv4.
The throughput on IPv4 is very much maxed out at around just under 3 Gbps, there bandwidth on IPv6 goes up to just under 8Gbps, a difference of around 2.5x.

I have slightly more Firewall rules configured on IPv6 (25 rules) than IPv4 (21 rules) side.
NAT rules I have tried disabling most rules so both sides have 3 rules and that made no significant difference.

What could I be missing / worth looking into?

Thanks in advance!
untitled.png

2

Show us the config. From what is shown so far and what you explained it seems like IPv4 is being routed while IPv6 is being bridged … but only look at config can tell what you actually have.

3

Sure, attaching config below.

Since yesterday I have consolidated some IPv4 Firewall rules and NAT rules (though not sure NAT rules mattered), while there is a slight gain in IPv4 throughput the gap is still very wide.
Throughput on IPv4 averages ~3Gbps, while IPv6 averages at ~8Gbps.

I have also taken CRS328 out of the equation by connecting both Host 1 and Host 2 directly to the CCR2004, since the symptoms are basically identical.




Bridge / Interfaces configuration

/interface> export
# 2024-04-03 11:22:29 by RouterOS 7.14.1
# software id = JIJP-GLVA
#
# model = CCR2004-1G-12S+2XS
# serial number = REDACTED


/interface bridge
add name=bridge vlan-filtering=yes


/interface ethernet
set [ find default-name=sfp28-1 ] comment="ISP Uplink" mac-address=\
    50:42:89:7F:1B:9F rx-flow-control=auto tx-flow-control=auto
set [ find default-name=sfp28-2 ] rx-flow-control=auto tx-flow-control=auto
set [ find default-name=sfp-sfpplus1 ] name=sfpplus1 rx-flow-control=auto \
    tx-flow-control=auto
set [ find default-name=sfp-sfpplus2 ] name=sfpplus2 rx-flow-control=auto \
    tx-flow-control=auto
set [ find default-name=sfp-sfpplus3 ] name=sfpplus3 rx-flow-control=auto \
    tx-flow-control=auto
set [ find default-name=sfp-sfpplus4 ] name=sfpplus4 rx-flow-control=auto \
    tx-flow-control=auto
set [ find default-name=sfp-sfpplus5 ] name=sfpplus5 rx-flow-control=auto \
    tx-flow-control=auto
set [ find default-name=sfp-sfpplus6 ] name=sfpplus6 rx-flow-control=auto \
    tx-flow-control=auto
set [ find default-name=sfp-sfpplus7 ] name=sfpplus7 rx-flow-control=auto \
    tx-flow-control=auto
set [ find default-name=sfp-sfpplus8 ] name=sfpplus8 rx-flow-control=auto \
    tx-flow-control=auto
set [ find default-name=sfp-sfpplus9 ] name=sfpplus9 rx-flow-control=auto \
    tx-flow-control=auto
set [ find default-name=sfp-sfpplus10 ] name=sfpplus10 rx-flow-control=auto \
    tx-flow-control=auto
set [ find default-name=sfp-sfpplus11 ] name=sfpplus11 rx-flow-control=auto \
    tx-flow-control=auto
set [ find default-name=sfp-sfpplus12 ] name=sfpplus12 rx-flow-control=auto \
    tx-flow-control=auto



/interface vlan
add comment="Bridge VLAN VPN" interface=bridge name=bridge-vlan172 vlan-id=172
add comment="Bridge VLAN Guest" interface=bridge name=bridge-vlan253 vlan-id=\
    253
add comment="Bridge VLAN Guest" interface=bridge name=bridge-vlan254 vlan-id=\
    254


/interface bonding
add lacp-rate=1sec mode=802.3ad name=bond/9+10 slaves=sfpplus9,sfpplus10


/interface list
add comment="all local interfaces" name=spfpluses
add name=wan
add name=bridges-untrusted
add name=lan
add name=bridges-trusted
add name=mgmt
add include=spfpluses,mgmt name=local-interfaces-all
add include=bridges-trusted,bridges-untrusted name=bridges-all


/interface bridge port
add bridge=bridge interface=sfpplus1
add bridge=bridge interface=sfpplus2
add bridge=bridge interface=sfpplus3
add bridge=bridge interface=sfpplus4
add bridge=bridge interface=sfpplus5
add bridge=bridge interface=sfpplus6
add bridge=bridge interface=sfpplus7
add bridge=bridge interface=sfpplus8
add bridge=bridge disabled=yes interface=sfpplus9
add bridge=bridge disabled=yes interface=sfpplus10
add bridge=bridge interface=sfpplus11
add bridge=bridge interface=sfpplus12
add bridge=bridge interface=*36
add bridge=bridge interface=bond/9+10


/interface bridge vlan
add bridge=bridge tagged="sfpplus1,sfpplus2,sfpplus3,sfpplus4,sfpplus5,sfpplus6,\
    sfpplus7,sfpplus8,sfpplus11,sfpplus12,bridge,bond/9+10" vlan-ids=10
add bridge=bridge tagged="sfpplus1,sfpplus2,sfpplus3,sfpplus4,sfpplus5,sfpplus6,\
    sfpplus7,sfpplus8,sfpplus11,sfpplus12,bridge,bond/9+10" vlan-ids=172
add bridge=bridge tagged="sfpplus1,sfpplus2,sfpplus3,sfpplus4,sfpplus5,sfpplus6,\
    sfpplus7,sfpplus8,sfpplus11,sfpplus12,bridge,bond/9+10" vlan-ids=253
add bridge=bridge tagged="sfpplus1,sfpplus2,sfpplus3,sfpplus4,sfpplus5,sfpplus6,\
    sfpplus7,sfpplus8,sfpplus11,sfpplus12,bridge,bond/9+10" vlan-ids=254


/interface list member
add interface=sfpplus1 list=spfpluses
add interface=sfpplus2 list=spfpluses
add interface=sfpplus3 list=spfpluses
add interface=sfpplus4 list=spfpluses
add interface=sfpplus5 list=spfpluses
add interface=sfpplus6 list=spfpluses
add interface=sfpplus7 list=spfpluses
add interface=sfpplus8 list=spfpluses
add interface=sfpplus9 list=spfpluses
add interface=sfpplus10 list=spfpluses
add interface=sfpplus11 list=spfpluses
add interface=sfpplus12 list=spfpluses
add interface=sfp28-1 list=wan
add interface=sfp28-2 list=wan
add interface=bridge-vlan253 list=bridges-untrusted
add interface=bridge-vlan254 list=bridges-untrusted
add interface=bridge-vlan172 list=bridges-trusted
add interface=ether1 list=mgmt
add interface=bridge list=bridges-trusted

IPv4 Firewall

/ip/firewall/ export
# 2024-04-03 11:24:58 by RouterOS 7.14.1
# software id = JIJP-GLVA
#
# model = CCR2004-1G-12S+2XS
# serial number = REDACTED


/ip firewall filter
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" \
    connection-state=established,related hw-offload=yes
add action=accept chain=forward comment="defconf: accept established,related" \
    connection-state=established,related
add action=drop chain=forward comment=\
    "Deny traffic from untrusted bridges to local bridges" in-interface-list=\
    bridges-untrusted out-interface-list=bridges-all
add action=drop chain=forward comment="defconf: drop invalid" connection-state=\
    invalid
add action=drop chain=forward comment=\
    "defconf:  drop all from WAN not DSTNATed" connection-nat-state=!dstnat \
    connection-state=new in-interface=sfp28-1
add action=drop chain=input comment="Drop traffic from addr-list-drop-src list" \
    log=yes log-prefix=fw-drop-src src-address-list=addr-list-drop-src
add action=accept chain=input protocol=icmp
add action=accept chain=input connection-state=established
add action=accept chain=input connection-state=related
add action=accept chain=input comment=WireGuard dst-port=13231 protocol=udp
add action=accept chain=input comment=WireGuard dst-port=13232 protocol=udp
add action=drop chain=input in-interface=sfp28-1 log-prefix=rule20


/ip firewall nat
add action=masquerade chain=srcnat out-interface=sfp28-1
add action=masquerade chain=srcnat comment="hairpin NAT for bridge-10" \
    disabled=yes dst-address=10.10.10.0/23 src-address=10.10.10.0/23
add action=masquerade chain=srcnat comment="hairpin NAT for bridge-172" \
    disabled=yes dst-address=10.10.172.0/24 src-address=10.10.172.0/24
add action=masquerade chain=srcnat comment=\
    "hairpin NAT for bridge-254 - disabled by default" disabled=yes \
    dst-address=10.10.254.0/24 src-address=10.10.254.0/24
add action=dst-nat chain=dstnat comment=kvm-traefik-80 dst-address-list=wan-ip \
    dst-address-type="" dst-port=80 protocol=tcp to-addresses=10.10.10.213 \
    to-ports=80
add action=dst-nat chain=dstnat comment=kvm-traefik-443 dst-address-list=wan-ip \
    dst-address-type=local dst-port=443 protocol=tcp to-addresses=10.10.10.213 \
    to-ports=443
add action=dst-nat chain=dstnat comment=kvm-traefik-123/udp dst-address-list=\
    wan-ip dst-address-type=local dst-port=123 protocol=udp to-addresses=\
    10.10.10.251 to-ports=123
add action=dst-nat chain=dstnat comment=kvm-transmission1 dst-address-list=\
    wan-ip dst-address-type=local dst-port=65431 protocol=tcp to-addresses=\
    10.10.10.251 to-ports=65431
add action=dst-nat chain=dstnat comment=kvm-transmission1 dst-address-list=\
    wan-ip dst-address-type=local dst-port=65431 protocol=udp to-addresses=\
    10.10.10.251 to-ports=65431
add action=dst-nat chain=dstnat comment=kvm-transmission2 dst-address-list=\
    wan-ip dst-address-type=local dst-port=65432 protocol=tcp to-addresses=\
    10.10.10.251 to-ports=65432
add action=dst-nat chain=dstnat comment=kvm-transmission2 dst-address-list=\
    wan-ip dst-address-type=local dst-port=65432 protocol=udp to-addresses=\
    10.10.10.251 to-ports=65432
add action=dst-nat chain=dstnat comment=kvm-resilio-tcp dst-address-list=wan-ip \
    dst-address-type=local dst-port=55555 protocol=tcp to-addresses=\
    10.10.10.251 to-ports=55555
add action=dst-nat chain=dstnat comment=kvm-resilio-udp dst-address-list=wan-ip \
    dst-address-type=local dst-port=55555 protocol=udp to-addresses=\
    10.10.10.251 to-ports=55555
add action=dst-nat chain=dstnat comment=nas-transmission-20000 \
    dst-address-list=wan-ip dst-address-type=local dst-port=20000 protocol=tcp \
    to-addresses=10.10.10.254 to-ports=20000

IPv6 Firewall

/ipv6/firewall/ export
# 2024-04-03 11:27:26 by RouterOS 7.14.1
# software id = JIJP-GLVA
#
# model = CCR2004-1G-12S+2XS
# serial number = REDACTED


/ipv6 firewall filter
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=drop chain=forward comment=\
    "Deny from Untrusted Bridges to Local Bridges" in-interface-list=\
    bridges-untrusted out-interface-list=bridges-all
add action=accept chain=input comment="defconf: accept ICMPv6" protocol=icmpv6
add action=accept chain=input comment="defconf: accept UDP traceroute" port=\
    33434-33534 protocol=udp
add action=accept chain=input comment=\
    "defconf: accept DHCPv6-Client prefix delegation." dst-port=546 protocol=\
    udp src-address=fe80::/10
add action=accept chain=input comment="defconf: accept IKE" dst-port=500,4500 \
    protocol=udp
add action=accept chain=input comment="defconf: accept ipsec AH" protocol=\
    ipsec-ah
add action=accept chain=input comment="defconf: accept ipsec ESP" protocol=\
    ipsec-esp
add action=accept chain=input comment=\
    "defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=accept chain=input comment=WireGuard dst-port=13231,13232 \
    in-interface-list=wan protocol=udp
add action=drop chain=input comment=\
    "defconf: drop everything else not coming from LAN" in-interface-list=\
    !bridges-all
add action=accept chain=forward comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=\
    invalid
add action=drop chain=forward comment="defconf: drop packets with bad src ipv6" \
    src-address-list=bad_ipv6
add action=drop chain=forward comment="defconf: drop packets with bad dst ipv6" \
    dst-address-list=bad_ipv6
add action=drop chain=forward comment="defconf: rfc4890 drop hop-limit=1" \
    hop-limit=equal:1 protocol=icmpv6
add action=accept chain=forward comment="defconf: accept ICMPv6" protocol=\
    icmpv6
add action=accept chain=forward comment="defconf: accept HIP" protocol=139
add action=accept chain=forward comment="defconf: accept IKE" dst-port=500,4500 \
    protocol=udp
add action=accept chain=forward comment="defconf: accept ipsec AH" protocol=\
    ipsec-ah
add action=accept chain=forward comment="defconf: accept ipsec ESP" protocol=\
    ipsec-esp
add action=accept chain=forward comment=\
    "defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=accept chain=forward comment="ino ingress" dst-address-list=ino \
    dst-port=80,443 in-interface-list=wan protocol=tcp
add action=drop chain=forward comment=\
    "defconf: drop everything else not coming from LAN" in-interface-list=\
    !bridges-all


/ipv6 firewall nat
add action=masquerade chain=srcnat comment=\
    "NAT for ULA - unstable prefix delegation workaround" out-interface=sfp28-1
add action=dst-nat chain=dstnat comment="Allow GW to front IPv6 Ingress" \
    dst-address-list=wan-ip dst-port=80,443 protocol=tcp to-address=\
    fdfd::a00:dfd0:4339:3b07:d34/128

Please let me know if I have missed out any other relevant configurations!
untitled2.png

4

Both sfp-sfpplus8 and bond/9+10 are trunk (all tagged) ports. So how are hosts configured regarding VLANs?

And, BTW, you didn’t post full config. So I’ll assume you’re just trolling and not expecting to get any usable advice if you won’t post full config (sensitive data obfuscated, not left out)..

I’m still trying to find out whether IPv6 traffic is routed or switched … I’m having hard time to believe that this device can route at almost 8Gbps based on official test results (2-3Gbps would sound much more reasonable to me).

5

Looking at the IPv6 addresses from your first post, fdfd:0:0:a00::abcd and fdfd::a00:a00:​27ff:fe60:e32e, they are both in the same (assumed normal) /64 subnet, namely fdfd:0:0:a00::/64. When you run iperf3 between the two hosts using those addresses, the router will not need to perform any routing at all, only standard switching. The two hosts see that they are on the same subnet and will use ND to discover their MAC addresses and then talk directly to each other on Layer 2, no gateway (router) needs to be involved. The IPv6 firewall rules are not touched at all. That’s why you see switch-like performance figures from the benchmarks.

As for IPv4: It looks like you’ve configured your subnet as a /23 subnet, namely 10.10.10.0/23 which means both 10.10.11.143 and 10.10.10.213 should still theoretically be on the same subnet and if the clients are correctly configured (with /23 network) they should not use the router as gateway for routing traffic, only as a switch for switching packets. That you see lower performance values for IPv4 means that it was not the case, and the CCR2004 had to act as router. One possible cause could be that one of the clients has /24 as subnet in its configuration, or its route table misses a route entry for 10.10.10.0/23 to go through the link layer bypassing the gateway. Both would mean that it would see the other address as being outside of its subnet and will forward packets to the gateway (the router) instead of sending them directly on the link layer to the MAC address of the other host. If that’s the case it will be worsened by the fact that the router will even perform NAT on the packets due to the “hairpin NAT for bridge-10” rule that you have in the firewall configuration.

Side note: all the dst-nat rules for port forwarding to 10.10.10.251 in your firewall nat table could be consolidated into two rules, one for udp, one for tcp. dst-port can have a comma separated list of ports/port ranges, and you just need to omit “to-ports” (keep original port, no port translations needed). Same for 10.10.10.213 with dst-port=80,443