Performance Problem ?

RouterOS General
1

Hi all
i need to design solution with “multilink” for WAN connection and also i need a GRE encapsulation.
I buy two routers CCR1036-8G-2S+EM for that because all track i need to have 10Gb/s.

i connect like that
host ---- router ---- router —host
this connection is for lab but on production there wiil be two ISP between the routers. so .. on my lab i set two adresses on WAN connection.

on start (on clear configuration) i run iperf from host to host (only one direction) i have 9,6 Gb/s
then i add GRE tunnel(sinle) iperf result 9,5Gb/s

but when i add second adresses on WAN connection, add second gre tunnel, and define ECMP (with PCC on mangle firewall) i have a lot of packet retransmission and 1,3Gb/s. is this an performance problem ??? (as they said on their webpage this is the fastest router in portfolio so where is the problem ? )

[ 4] local 172.31.19.50 port 44458 connected to 172.31.0.50 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 154 MBytes 1.29 Gbits/sec 462 266 KBytes
[ 4] 1.00-2.00 sec 167 MBytes 1.40 Gbits/sec 657 273 KBytes
[ 4] 2.00-3.00 sec 175 MBytes 1.46 Gbits/sec 858 287 KBytes
[ 4] 3.00-4.00 sec 167 MBytes 1.40 Gbits/sec 1103 276 KBytes
[ 4] 4.00-5.00 sec 166 MBytes 1.39 Gbits/sec 1017 262 KBytes
[ 4] 5.00-6.00 sec 170 MBytes 1.42 Gbits/sec 904 270 KBytes
[ 4] 6.00-7.00 sec 176 MBytes 1.47 Gbits/sec 1560 216 KBytes
[ 4] 7.00-8.01 sec 175 MBytes 1.46 Gbits/sec 695 284 KBytes
[ 4] 8.01-9.00 sec 177 MBytes 1.49 Gbits/sec 1000 284 KBytes
[ 4] 9.00-10.00 sec 170 MBytes 1.43 Gbits/sec 902 189 KBytes
[ 4] 10.00-11.00 sec 164 MBytes 1.38 Gbits/sec 926 197 KBytes
[ 4] 11.00-12.00 sec 167 MBytes 1.40 Gbits/sec 540 279 KBytes
^C[ 4] 12.00-12.84 sec 150 MBytes 1.50 Gbits/sec 662 279 KBytes

2

Is your 10G interface listed under WAN? (/interface list member print)
Without seeing your config settings nobody really can help. One needs to see the settings of the involved interfaces, incl. all the involved IPs, as well the route table etc.

3

You are probably dropping out of the “fastpath” ? Due to you PCC/Mangle rules, so performance will take a huge hit.
And something with fragmentation also I think?
This device, when routing small 64byte packets with some queues & ip-filter rules etc only reaches 1.5Gbits/sec anymore…(see official test-results)

Normally, with larger packets (1518 / 512 bytes) it should still > 10Gbps with ip-filters/queues applied according to official test results.

4

Combining ECMP and PCC is pointless - ECMP is based on spreading the traffic across multiple gateways (typically one per WAN uplink) based on destination address alone (thanks to the routing cache, which remembers the gateway chosen for the first packet for a given destination address and re-uses that gateway for packets to the same destination address “forever”) and without need to use a routing-mark, whereas the PCC approach chooses the routing table (routing-mark) based on selectable criteria. Packet retransmissions only make sense to me in the context of assigning routing-mark using /ip firewall mangle rules and having an action=fasttrack-connection rule in chain=forward of /ip firewall filter, but do you use the stateful firewall at all?

5

Maybye my config will help you to fix the my problem

router1

/interface gre
add local-address=1.1.1.1 name=gre-tunnel1 remote-address=1.1.1.2
add local-address=1.1.1.5 name=gre-tunnel2 remote-address=1.1.1.6
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip address
add address=192.168.88.1/24 comment=defconf interface=ether8 network=192.168.88.0
add address=1.1.1.1/30 interface=sfp-sfpplus1 network=1.1.1.0
add address=172.31.19.250/24 interface=sfp-sfpplus2 network=172.31.19.0
add address=192.168.69.1/30 interface=gre-tunnel1 network=192.168.69.0
add address=1.1.1.5/30 interface=sfp-sfpplus1 network=1.1.1.4
add address=192.168.69.5/30 interface=gre-tunnel2 network=192.168.69.4
/ip firewall address-list
add address=172.31.19.0/24 list=LAN
/ip firewall mangle
add action=mark-connection chain=prerouting comment="PCC rules" connection-mark=no-mark dst-address-list=!LAN dst-address-type=!local new-connection-mark=LAN-to-WAN1 passthrough=yes per-connection-classifier=both-addresses-and-ports:2/0 \
    src-address-list=LAN
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-list=!LAN dst-address-type=!local new-connection-mark=LAN-to-WAN2 passthrough=yes per-connection-classifier=both-addresses-and-ports:2/1 src-address-list=LAN
add action=mark-routing chain=prerouting comment="Mark routing for upload packets from marked connections" connection-mark=LAN-to-WAN1 dst-address-list=!LAN new-routing-mark=WAN1 passthrough=no src-address-list=LAN
add action=mark-routing chain=prerouting connection-mark=LAN-to-WAN2 dst-address-list=!LAN new-routing-mark=WAN2 passthrough=no src-address-list=LAN
/ip route
add distance=1 dst-address=172.31.0.0/24 gateway=gre-tunnel1 routing-mark=WAN1
add distance=1 dst-address=172.31.0.0/24 gateway=gre-tunnel2 routing-mark=WAN2
add check-gateway=ping distance=1 dst-address=172.31.0.0/24 gateway=gre-tunnel1
add check-gateway=ping distance=1 dst-address=172.31.0.0/24 gateway=gre-tunnel2

router2

/interface gre
add local-address=1.1.1.2 name=gre-tunnel1 remote-address=1.1.1.1
add local-address=1.1.1.6 name=gre-tunnel2 remote-address=1.1.1.5
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip address
add address=192.168.88.1/24 comment=defconf interface=ether7 network=192.168.88.0
add address=1.1.1.2/30 interface=sfp-sfpplus1 network=1.1.1.0
add address=172.31.0.250/24 interface=sfp-sfpplus2 network=172.31.0.0
add address=192.168.69.2/30 interface=gre-tunnel1 network=192.168.69.0
add address=1.1.1.6/30 interface=sfp-sfpplus1 network=1.1.1.4
add address=192.168.69.6/30 interface=gre-tunnel2 network=192.168.69.4
/ip firewall address-list
add address=172.31.0.0/24 list=LAN
/ip firewall mangle
add action=mark-connection chain=prerouting comment="PCC rules" connection-mark=no-mark dst-address-list=!LAN dst-address-type=!local new-connection-mark=LAN-to-WAN1 passthrough=yes per-connection-classifier=both-addresses-and-ports:2/0 \
    src-address-list=LAN
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-list=!LAN dst-address-type=!local new-connection-mark=LAN-to-WAN2 passthrough=yes per-connection-classifier=both-addresses-and-ports:2/1 src-address-list=LAN
add action=mark-routing chain=prerouting comment="Mark routing for upload packets from marked connections" connection-mark=LAN-to-WAN1 dst-address-list=!LAN new-routing-mark=WAN1 passthrough=no src-address-list=LAN
add action=mark-routing chain=prerouting connection-mark=LAN-to-WAN2 dst-address-list=!LAN new-routing-mark=WAN2 passthrough=no src-address-list=LAN
/ip route
add distance=1 dst-address=172.31.19.0/24 gateway=gre-tunnel1 routing-mark=WAN1
add distance=1 dst-address=172.31.19.0/24 gateway=gre-tunnel2 routing-mark=WAN2
add check-gateway=ping distance=1 dst-address=172.31.19.0/24 gateway=gre-tunnel1
add check-gateway=ping distance=1 dst-address=172.31.19.0/24 gateway=gre-tunnel2