Upgrading RB450G from ROS 5.26 to 6.28 resulted in massive packet loss on WAN traffic (NATed). As a result my single TCP throughput has been cut in half (5 mbit/s instead of 9 mbit/s) and the overall upload bandwidth degraded accordingly.
According to tcpdump analysis, with ROS 6.28 outgoing NAT traffic is affected by 10% packet loss which explains the degraded TCP throughput. Incoming traffic on the other hand does not exhibit significant packet loss.
After i downgraded to 5.26, everything went back to normal (no packet loss, full throughput).
Here are the iperf/tcpdump results before and after the upgrade.
AFTER UPGRADE - ROS 6.28 / Firmware 3.07
Observation: 10% packet loss, Single TCP throughput: 5mbit/s
iperf 3.0.9
Time: Sat, 09 May 2015 09:40:31 GMT
Starting Test: protocol: UDP, 1 streams, 1390 byte blocks, omitting 0 seconds, 10 second test
------------------------------------------------
Test Complete. Summary Results:
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 4] 0.00-10.01 sec 10.91 MBytes 9.95 Mbits/sec 1.592 ms 79/756 (10%)
iperf 3.0.9
Time: Sat, 09 May 2015 09:41:41 GMT
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds,
10 second test
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 1.0 sec 384 KBytes 3.15 Mbits/sec
[ 3] 1.0- 2.0 sec 384 KBytes 3.15 Mbits/sec
[ 3] 2.0- 3.0 sec 512 KBytes 4.19 Mbits/sec
[ 3] 3.0- 4.0 sec 640 KBytes 5.24 Mbits/sec
[ 3] 4.0- 5.0 sec 768 KBytes 6.29 Mbits/sec
[ 3] 5.0- 6.0 sec 768 KBytes 6.29 Mbits/sec
[ 3] 6.0- 7.0 sec 1.00 MBytes 8.39 Mbits/sec
[ 3] 7.0- 8.0 sec 768 KBytes 6.29 Mbits/sec
[ 3] 8.0- 9.0 sec 128 KBytes 1.05 Mbits/sec
[ 3] 9.0-10.0 sec 256 KBytes 2.10 Mbits/sec
------------------------------------------------
Test Complete. Summary Results:
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.01 sec 5.63 MBytes 4.60 Mbits/sec sender
[ 4] 0.00-10.01 sec 5.63 MBytes 4.60 Mbits/sec receiver
===================================================================================
| Tcpdump IO Statistics |
| |
| Duration: 11.149112 secs |
| Interval: 1 secs |
| |
| Col 1: COUNT(tcp.analysis.retransmission) tcp.analysis.retransmission |
| 2: COUNT(tcp.analysis.duplicate_ack)tcp.analysis.duplicate_ack |
| 3: COUNT(tcp.analysis.lost_segment) tcp.analysis.lost_segment |
| 4: COUNT(tcp.analysis.fast_retransmission) tcp.analysis.fast_retransmission |
|---------------------------------------------------------------------------------|
| |1 |2 |3 |4 | |
| Interval | COUNT | COUNT | COUNT | COUNT | |
|------------------------------------------| |
| 0 <> 1 | 0 | 0 | 0 | 0 | |
| 1 <> 2 | 1 | 18 | 0 | 1 | |
| 2 <> 3 | 0 | 0 | 0 | 0 | |
| 3 <> 4 | 0 | 0 | 0 | 0 | |
| 4 <> 5 | 0 | 0 | 0 | 0 | |
| 5 <> 6 | 0 | 0 | 0 | 0 | |
| 6 <> 7 | 0 | 0 | 0 | 0 | |
| 7 <> 8 | 0 | 0 | 0 | 0 | |
| 8 <> 9 | 5 | 137 | 0 | 2 | |
| 9 <> 10 | 53 | 102 | 0 | 2 | |
| 10 <> Dur| 15 | 0 | 0 | 0 | |
===================================================================================
BEFORE UPGRADE - ROS 5.26 / Firmware 2.39
Observation: 0% packet loss, Single TCP throughput: 9mbit/s
iperf 3.0.9
Time: Sat, 09 May 2015 09:34:57 GMT
Starting Test: protocol: UDP, 1 streams, 1390 byte blocks, omitting 0 seconds, 10 second test
------------------------------------------------
Test Complete. Summary Results:
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 4] 0.00-10.01 sec 10.91 MBytes 9.95 Mbits/sec 1.661 ms 0/756 (0%)
iperf 3.0.9
Time: Sat, 09 May 2015 09:32:24 GMT
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds,
10 second test
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.01 sec 693 KBytes 6.61 Mbits/sec
[ 4] 1.01-2.01 sec 189 KBytes 2.55 Mbits/sec
[ 4] 2.01-3.01 sec 693 KBytes 6.68 Mbits/sec
[ 4] 3.01-4.01 sec 1008 KBytes 9.26 Mbits/sec
[ 4] 4.01-5.01 sec 1.05 MBytes 9.77 Mbits/sec
[ 4] 5.01-6.01 sec 1008 KBytes 9.26 Mbits/sec
[ 4] 6.01-7.01 sec 1.05 MBytes 9.78 Mbits/sec
[ 4] 7.01-8.01 sec 1008 KBytes 9.25 Mbits/sec
[ 4] 8.01-9.01 sec 1008 KBytes 9.26 Mbits/sec
[ 4] 9.01-10.01 sec 1008 KBytes 9.26 Mbits/sec
------------------------------------------------
Test Complete. Summary Results:
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.01 sec 8.55 MBytes 8.17 Mbits/sec sender
[ 4] 0.00-10.01 sec 8.50 MBytes 8.12 Mbits/sec receiver
===================================================================================
| Tcpdump IO Statistics |
| |
| Duration: 10.263404 secs |
| Interval: 1 secs |
| |
| Col 1: COUNT(tcp.analysis.retransmission) tcp.analysis.retransmission |
| 2: COUNT(tcp.analysis.duplicate_ack)tcp.analysis.duplicate_ack |
| 3: COUNT(tcp.analysis.lost_segment) tcp.analysis.lost_segment |
| 4: COUNT(tcp.analysis.fast_retransmission) tcp.analysis.fast_retransmission |
|---------------------------------------------------------------------------------|
| |1 |2 |3 |4 | |
| Interval | COUNT | COUNT | COUNT | COUNT | |
|------------------------------------------| |
| 0 <> 1 | 0 | 0 | 0 | 0 | |
| 1 <> 2 | 0 | 0 | 0 | 0 | |
| 2 <> 3 | 0 | 0 | 0 | 0 | |
| 3 <> 4 | 0 | 0 | 0 | 0 | |
| 4 <> 5 | 0 | 0 | 0 | 0 | |
| 5 <> 6 | 0 | 0 | 0 | 0 | |
| 6 <> 7 | 0 | 0 | 0 | 0 | |
| 7 <> 8 | 0 | 0 | 0 | 0 | |
| 8 <> 9 | 0 | 0 | 0 | 0 | |
| 9 <> 10 | 0 | 0 | 0 | 0 | |
| 10 <> Dur| 2 | 3 | 0 | 1 | |
===================================================================================
CONFIGURATION
# may/09/2015 12:03:30 by RouterOS 5.26
# software id = *******
#
/interface bridge
add l2mtu=1520 name=local
/interface ethernet
set 0 name=WAN-1
set 1 name=WAN-2
set 2 name=LAN
set 3 name=ether4-local
set 4 name=ether5-local
/interface pppoe-client
add disabled=no interface=WAN-1 max-mtu=1492 name=pppoe-wan1 password=\
XU***** user=X91004******
/interface wireless security-profiles
set [ find default=yes ] group-ciphers="" unicast-ciphers=""
/ip hotspot user profile
set [ find default=yes ] idle-timeout=none keepalive-timeout=2m
/ip pool
add name=default-dhcp ranges=192.168.0.50-192.168.0.249
/ip dhcp-server
add address-pool=default-dhcp authoritative=yes disabled=no interface=local \
name=default
/ppp profile
set 0 use-compression=no use-encryption=no use-vj-compression=no
/interface pppoe-client
add allow=chap disabled=no interface=WAN-2 max-mtu=1492 name=pppoe-wan2 \
password=cv***** user=X9100*******
/routing bgp instance
set default disabled=yes
/interface bridge port
add bridge=local interface=LAN
/ip address
add address=192.168.0.1/24 interface=local
/ip dhcp-client
add default-route-distance=0 interface=WAN-1
add default-route-distance=0 interface=WAN-2
/ip dhcp-server network
add address=192.168.0.0/24 comment="default configuration" dns-server=\
192.168.0.1 gateway=192.168.0.1
/ip dns
set allow-remote-requests=yes cache-size=4048KiB max-udp-packet-size=512 \
servers=8.8.8.8,8.8.4.4
/ip dns static
add address=192.168.0.1 name=router
/ip firewall filter
add chain=input comment="default configuration" protocol=icmp
add chain=input comment="default configuration" connection-state=established \
in-interface=pppoe-wan1
add chain=input comment="default configuration" connection-state=related \
in-interface=pppoe-wan1
add action=log chain=input comment="default configuration" disabled=yes \
in-interface=pppoe-wan1 log-prefix=firewall-wan1
add action=drop chain=input comment="default configuration" in-interface=\
pppoe-wan1
add chain=input comment="default configuration" connection-state=established \
in-interface=pppoe-wan2
add chain=input comment="default configuration" connection-state=related \
in-interface=pppoe-wan2
add action=log chain=input comment="default configuration" disabled=yes \
in-interface="(unknown)" log-prefix=firewall-wan2
add action=drop chain=input comment="default configuration" in-interface=\
pppoe-wan2
/ip firewall mangle
add action=mark-connection chain=input connection-mark=no-mark in-interface=\
pppoe-wan1 new-connection-mark=WAN-1_conn
add action=mark-connection chain=input connection-mark=no-mark in-interface=\
pppoe-wan2 new-connection-mark=WAN-2_conn
add action=mark-routing chain=output connection-mark=WAN-1_conn \
new-routing-mark=to_WAN-1
add action=mark-routing chain=output connection-mark=WAN-2_conn \
new-routing-mark=to_WAN-2
add chain=prerouting dst-address=192.168.0.0/24
add action=mark-connection chain=prerouting comment=\
"PCC load balancing outgoing connections WAN-1" connection-mark=no-mark \
dst-address-type=!local in-interface=local new-connection-mark=WAN-1_conn \
per-connection-classifier=both-addresses:2/0
add action=mark-connection chain=prerouting comment=\
"Mark Incoming connections WAN-1" connection-mark=no-mark \
dst-address-type=!local in-interface=pppoe-wan1 new-connection-mark=\
WAN-1_conn per-connection-classifier=both-addresses:2/0
add action=mark-connection chain=prerouting comment=\
"PCC load-balancing outgoing connections WAN-2" connection-mark=no-mark \
dst-address-type=!local in-interface=local new-connection-mark=WAN-2_conn \
per-connection-classifier=both-addresses:2/1
add action=mark-connection chain=prerouting comment=\
"Mark Incoming connections WAN-2" connection-mark=no-mark \
dst-address-type=!local in-interface=pppoe-wan2 new-connection-mark=\
WAN-2_conn per-connection-classifier=both-addresses:2/1
add action=mark-routing chain=prerouting connection-mark=WAN-1_conn \
in-interface=local new-routing-mark=to_WAN-1
add action=mark-routing chain=prerouting connection-mark=WAN-2_conn \
in-interface=local new-routing-mark=to_WAN-2
/ip firewall nat
add action=masquerade chain=srcnat out-interface=pppoe-wan1
add action=masquerade chain=srcnat out-interface=pppoe-wan2 to-addresses=\
0.0.0.0
/ip neighbor discovery
set ether4-local disabled=yes
set ether5-local disabled=yes
set pppoe-wan1 disabled=no
/ip route
add check-gateway=ping distance=1 gateway=82.135.16.28%pppoe-wan1 \
routing-mark=to_WAN-1
add check-gateway=ping distance=1 gateway=82.135.16.28%pppoe-wan2 \
routing-mark=to_WAN-2
add distance=1 gateway=82.135.16.28%pppoe-wan1
add distance=2 gateway=82.135.16.28%pppoe-wan2
/ip upnp
set enabled=yes
/ip upnp interfaces
add interface=WAN-1 type=external
add interface=WAN-2 type=external
add interface=LAN type=internal
/queue interface
set WAN-1 queue=ethernet-default
set WAN-2 queue=ethernet-default
set LAN queue=ethernet-default
set ether4-local queue=ethernet-default
set ether5-local queue=ethernet-default