Page 1 of 1

Massive packet loss on WAN after upgrading ROS 5.26 -> 6.28

Posted: Sat May 09, 2015 2:52 pm
by solibra
Upgrading RB450G from ROS 5.26 to 6.28 resulted in massive packet loss on WAN traffic (NATed). As a result my single TCP throughput has been cut in half (5 mbit/s instead of 9 mbit/s) and the overall upload bandwidth degraded accordingly.
According to tcpdump analysis, with ROS 6.28 outgoing NAT traffic is affected by 10% packet loss which explains the degraded TCP throughput. Incoming traffic on the other hand does not exhibit significant packet loss.
After i downgraded to 5.26, everything went back to normal (no packet loss, full throughput).

Here are the iperf/tcpdump results before and after the upgrade.

AFTER UPGRADE - ROS 6.28 / Firmware 3.07
Observation: 10% packet loss, Single TCP throughput: 5mbit/s
iperf 3.0.9
Time: Sat, 09 May 2015 09:40:31 GMT
Starting Test: protocol: UDP, 1 streams, 1390 byte blocks, omitting 0 seconds, 10 second test
------------------------------------------------
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  4]   0.00-10.01  sec  10.91 MBytes  9.95 Mbits/sec  1.592 ms  79/756 (10%)
iperf 3.0.9
Time: Sat, 09 May 2015 09:41:41 GMT
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds,
 10 second test
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 1.0 sec   384 KBytes  3.15 Mbits/sec
[  3]  1.0- 2.0 sec   384 KBytes  3.15 Mbits/sec
[  3]  2.0- 3.0 sec   512 KBytes  4.19 Mbits/sec
[  3]  3.0- 4.0 sec   640 KBytes  5.24 Mbits/sec
[  3]  4.0- 5.0 sec   768 KBytes  6.29 Mbits/sec
[  3]  5.0- 6.0 sec   768 KBytes  6.29 Mbits/sec
[  3]  6.0- 7.0 sec  1.00 MBytes  8.39 Mbits/sec
[  3]  7.0- 8.0 sec   768 KBytes  6.29 Mbits/sec
[  3]  8.0- 9.0 sec   128 KBytes  1.05 Mbits/sec
[  3]  9.0-10.0 sec   256 KBytes  2.10 Mbits/sec
------------------------------------------------
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.01  sec  5.63 MBytes  4.60 Mbits/sec                 sender
[  4]   0.00-10.01  sec  5.63 MBytes  4.60 Mbits/sec                 receiver
===================================================================================
| Tcpdump IO Statistics                                                           |
|                                                                                 |
| Duration: 11.149112 secs                                                        |
| Interval:  1 secs                                                               |
|                                                                                 |
| Col 1: COUNT(tcp.analysis.retransmission) tcp.analysis.retransmission           |
|     2: COUNT(tcp.analysis.duplicate_ack)tcp.analysis.duplicate_ack              |
|     3: COUNT(tcp.analysis.lost_segment) tcp.analysis.lost_segment               |
|     4: COUNT(tcp.analysis.fast_retransmission) tcp.analysis.fast_retransmission |
|---------------------------------------------------------------------------------|
|          |1      |2      |3      |4      |                                      |
| Interval | COUNT | COUNT | COUNT | COUNT |                                      |
|------------------------------------------|                                      |
|  0 <>  1 |     0 |     0 |     0 |     0 |                                      |
|  1 <>  2 |     1 |    18 |     0 |     1 |                                      |
|  2 <>  3 |     0 |     0 |     0 |     0 |                                      |
|  3 <>  4 |     0 |     0 |     0 |     0 |                                      |
|  4 <>  5 |     0 |     0 |     0 |     0 |                                      |
|  5 <>  6 |     0 |     0 |     0 |     0 |                                      |
|  6 <>  7 |     0 |     0 |     0 |     0 |                                      |
|  7 <>  8 |     0 |     0 |     0 |     0 |                                      |
|  8 <>  9 |     5 |   137 |     0 |     2 |                                      |
|  9 <> 10 |    53 |   102 |     0 |     2 |                                      |
| 10 <> Dur|    15 |     0 |     0 |     0 |                                      |
===================================================================================

BEFORE UPGRADE - ROS 5.26 / Firmware 2.39
Observation: 0% packet loss, Single TCP throughput: 9mbit/s
iperf 3.0.9
Time: Sat, 09 May 2015 09:34:57 GMT
Starting Test: protocol: UDP, 1 streams, 1390 byte blocks, omitting 0 seconds, 10 second test
------------------------------------------------
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  4]   0.00-10.01  sec  10.91 MBytes  9.95 Mbits/sec  1.661 ms  0/756 (0%)
iperf 3.0.9
Time: Sat, 09 May 2015 09:32:24 GMT
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds,
 10 second test
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.01   sec   693 KBytes  6.61 Mbits/sec
[  4]   1.01-2.01   sec   189 KBytes  2.55 Mbits/sec
[  4]   2.01-3.01   sec   693 KBytes  6.68 Mbits/sec
[  4]   3.01-4.01   sec  1008 KBytes  9.26 Mbits/sec
[  4]   4.01-5.01   sec  1.05 MBytes  9.77 Mbits/sec
[  4]   5.01-6.01   sec  1008 KBytes  9.26 Mbits/sec
[  4]   6.01-7.01   sec  1.05 MBytes  9.78 Mbits/sec
[  4]   7.01-8.01   sec  1008 KBytes  9.25 Mbits/sec
[  4]   8.01-9.01   sec  1008 KBytes  9.26 Mbits/sec
[  4]   9.01-10.01  sec  1008 KBytes  9.26 Mbits/sec
------------------------------------------------
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.01  sec  8.55 MBytes  8.17 Mbits/sec                  sender
[  4]   0.00-10.01  sec  8.50 MBytes  8.12 Mbits/sec                  receiver
===================================================================================
| Tcpdump IO Statistics                                                           |
|                                                                                 |
| Duration: 10.263404 secs                                                        |
| Interval:  1 secs                                                               |
|                                                                                 |
| Col 1: COUNT(tcp.analysis.retransmission) tcp.analysis.retransmission           |
|     2: COUNT(tcp.analysis.duplicate_ack)tcp.analysis.duplicate_ack              |
|     3: COUNT(tcp.analysis.lost_segment) tcp.analysis.lost_segment               |
|     4: COUNT(tcp.analysis.fast_retransmission) tcp.analysis.fast_retransmission |
|---------------------------------------------------------------------------------|
|          |1      |2      |3      |4      |                                      |
| Interval | COUNT | COUNT | COUNT | COUNT |                                      |
|------------------------------------------|                                      |
|  0 <>  1 |     0 |     0 |     0 |     0 |                                      |
|  1 <>  2 |     0 |     0 |     0 |     0 |                                      |
|  2 <>  3 |     0 |     0 |     0 |     0 |                                      |
|  3 <>  4 |     0 |     0 |     0 |     0 |                                      |
|  4 <>  5 |     0 |     0 |     0 |     0 |                                      |
|  5 <>  6 |     0 |     0 |     0 |     0 |                                      |
|  6 <>  7 |     0 |     0 |     0 |     0 |                                      |
|  7 <>  8 |     0 |     0 |     0 |     0 |                                      |
|  8 <>  9 |     0 |     0 |     0 |     0 |                                      |
|  9 <> 10 |     0 |     0 |     0 |     0 |                                      |
| 10 <> Dur|     2 |     3 |     0 |     1 |                                      |
===================================================================================
CONFIGURATION
# may/09/2015 12:03:30 by RouterOS 5.26
# software id = *******
#
/interface bridge
add l2mtu=1520 name=local
/interface ethernet
set 0 name=WAN-1
set 1 name=WAN-2
set 2 name=LAN
set 3 name=ether4-local
set 4 name=ether5-local
/interface pppoe-client
add disabled=no interface=WAN-1 max-mtu=1492 name=pppoe-wan1 password=\
    XU***** user=X91004******
/interface wireless security-profiles
set [ find default=yes ] group-ciphers="" unicast-ciphers=""
/ip hotspot user profile
set [ find default=yes ] idle-timeout=none keepalive-timeout=2m
/ip pool
add name=default-dhcp ranges=192.168.0.50-192.168.0.249
/ip dhcp-server
add address-pool=default-dhcp authoritative=yes disabled=no interface=local \
    name=default
/ppp profile
set 0 use-compression=no use-encryption=no use-vj-compression=no
/interface pppoe-client
add allow=chap disabled=no interface=WAN-2 max-mtu=1492 name=pppoe-wan2 \
    password=cv***** user=X9100*******
/routing bgp instance
set default disabled=yes
/interface bridge port
add bridge=local interface=LAN
/ip address
add address=192.168.0.1/24 interface=local
/ip dhcp-client
add default-route-distance=0 interface=WAN-1
add default-route-distance=0 interface=WAN-2
/ip dhcp-server network
add address=192.168.0.0/24 comment="default configuration" dns-server=\
    192.168.0.1 gateway=192.168.0.1
/ip dns
set allow-remote-requests=yes cache-size=4048KiB max-udp-packet-size=512 \
    servers=8.8.8.8,8.8.4.4
/ip dns static
add address=192.168.0.1 name=router
/ip firewall filter
add chain=input comment="default configuration" protocol=icmp
add chain=input comment="default configuration" connection-state=established \
    in-interface=pppoe-wan1
add chain=input comment="default configuration" connection-state=related \
    in-interface=pppoe-wan1
add action=log chain=input comment="default configuration" disabled=yes \
    in-interface=pppoe-wan1 log-prefix=firewall-wan1
add action=drop chain=input comment="default configuration" in-interface=\
    pppoe-wan1
add chain=input comment="default configuration" connection-state=established \
    in-interface=pppoe-wan2
add chain=input comment="default configuration" connection-state=related \
    in-interface=pppoe-wan2
add action=log chain=input comment="default configuration" disabled=yes \
    in-interface="(unknown)" log-prefix=firewall-wan2
add action=drop chain=input comment="default configuration" in-interface=\
    pppoe-wan2
/ip firewall mangle
add action=mark-connection chain=input connection-mark=no-mark in-interface=\
    pppoe-wan1 new-connection-mark=WAN-1_conn
add action=mark-connection chain=input connection-mark=no-mark in-interface=\
    pppoe-wan2 new-connection-mark=WAN-2_conn
add action=mark-routing chain=output connection-mark=WAN-1_conn \
    new-routing-mark=to_WAN-1
add action=mark-routing chain=output connection-mark=WAN-2_conn \
    new-routing-mark=to_WAN-2
add chain=prerouting dst-address=192.168.0.0/24
add action=mark-connection chain=prerouting comment=\
    "PCC load balancing outgoing connections WAN-1" connection-mark=no-mark \
    dst-address-type=!local in-interface=local new-connection-mark=WAN-1_conn \
    per-connection-classifier=both-addresses:2/0
add action=mark-connection chain=prerouting comment=\
    "Mark Incoming connections WAN-1" connection-mark=no-mark \
    dst-address-type=!local in-interface=pppoe-wan1 new-connection-mark=\
    WAN-1_conn per-connection-classifier=both-addresses:2/0
add action=mark-connection chain=prerouting comment=\
    "PCC load-balancing outgoing connections WAN-2" connection-mark=no-mark \
    dst-address-type=!local in-interface=local new-connection-mark=WAN-2_conn \
    per-connection-classifier=both-addresses:2/1
add action=mark-connection chain=prerouting comment=\
    "Mark Incoming connections WAN-2" connection-mark=no-mark \
    dst-address-type=!local in-interface=pppoe-wan2 new-connection-mark=\
    WAN-2_conn per-connection-classifier=both-addresses:2/1
add action=mark-routing chain=prerouting connection-mark=WAN-1_conn \
    in-interface=local new-routing-mark=to_WAN-1
add action=mark-routing chain=prerouting connection-mark=WAN-2_conn \
    in-interface=local new-routing-mark=to_WAN-2
/ip firewall nat
add action=masquerade chain=srcnat out-interface=pppoe-wan1
add action=masquerade chain=srcnat out-interface=pppoe-wan2 to-addresses=\
    0.0.0.0
/ip neighbor discovery
set ether4-local disabled=yes
set ether5-local disabled=yes
set pppoe-wan1 disabled=no
/ip route
add check-gateway=ping distance=1 gateway=82.135.16.28%pppoe-wan1 \
    routing-mark=to_WAN-1
add check-gateway=ping distance=1 gateway=82.135.16.28%pppoe-wan2 \
    routing-mark=to_WAN-2
add distance=1 gateway=82.135.16.28%pppoe-wan1
add distance=2 gateway=82.135.16.28%pppoe-wan2
/ip upnp
set enabled=yes
/ip upnp interfaces
add interface=WAN-1 type=external
add interface=WAN-2 type=external
add interface=LAN type=internal
/queue interface
set WAN-1 queue=ethernet-default
set WAN-2 queue=ethernet-default
set LAN queue=ethernet-default
set ether4-local queue=ethernet-default
set ether5-local queue=ethernet-default

Re: Massive packet loss on WAN after upgrading ROS 5.26 -> 6.28

Posted: Sun May 10, 2015 12:42 pm
by Thalid
Routerboot 3.07 is abit old acourding to http://routerboard.com/RB450G
3.18 should be the latest.

To upgrade:
System->routerboard-> see if you are offered 3.18 hit upgrade and do a reboot

Re: Massive packet loss on WAN after upgrading ROS 5.26 -> 6.28

Posted: Tue May 12, 2015 6:10 am
by solibra
Routerboot 3.07 is abit old acourding to http://routerboard.com/RB450G
3.18 should be the latest.
3.07 is actually the latest available Routerboot version for the RB450G.