weird network performance issue (Possible MTU fault)

mikea · March 20, 2015, 12:43pm

First of all excuse my writing since i m not very good in english,

Second because image is more than words here is a diagram of my setup:
original.jpg
i have the above betwork and the strange issue is that on Local Server with PPPoe connection A (1462 MTU) on MT and Rate-Limit unlimited i get SpeedTest.net approx 45mb/9,5mb. Same server PPPoe connection B same good results.
On client side When i connect to PPPoE server (1492 MTU) connection B route i get 37mb/9,5mb BUT when i connect PPPoE server (1492 MTU) connection A route i get 22mb/9,5mb only. I tested the wireless path from client to last MT GW that handle the PPPoE Client (1462 MTU) from my ISP with btest.exe and i get 39mb / 9,5mb.
Also when 2 or more clients connect with PPPoe connection A they share the above mentioned bandwith of 22mb/9,5mb and they are not get any advantage of the total 45mb pipe
I have tried multiple MTU setups but with no success. All AP and CPE are in WDS.
Any Help would be much appreciated.
Thank you in advance

samsung172 · March 20, 2015, 11:34pm

i think the NAS might have a 100% CPU ? 25mb/s over pppoe use a bit CPU power. I dont think this is a mtu issue.

Is your net capable of more than 1500bytes? - Try running 1492 to pppoe server.

mikea · March 21, 2015, 12:01am

dublicate post

mikea · March 22, 2015, 5:41pm

Thank you for your responce,
its has nothing to do with CPU.
NAS - cpu.jpg
I m sure it is a misconf cause i have two ISP Providers one link from (ISP #A) and two from the other (ISP #B)
when my clients have ip range from the ISP A then speed is normal as expected but when they have ip range from ISP B they have top limit about 20mbit download. If i disable for test ISP A uplink then they get full speed.
I suspect that the issue is that clients with iprange from ISP B when ISP A active they get path MTU from ISP A and not ISP B as they supposed to and there is difference on those two MTUs.
Am i right? is this maybe the issue? and how i can solve this.
Thank you for your time.

mikea · September 29, 2015, 6:33pm

I ve upgraded the Mtik that handles the PPPoE server to CCR1009-8G-1S but still having this issue!
Any help would be appreciated.

bajodel · October 1, 2015, 6:47am

Do you have any MSS adjustment in your mangle section at the moment? Which?

mikea · October 1, 2015, 8:42pm

/ip firewall mangle
add action=change-mss chain=forward new-mss=1440 out-interface=all-ppp
protocol=tcp tcp-flags=syn tcp-mss=1441-65535
add action=change-mss chain=forward in-interface=all-ppp new-mss=1372
protocol=tcp tcp-flags=syn tcp-mss=1373-65535

bajodel · October 2, 2015, 5:44pm

You can simply test lowering both mtu, if the problem vanish you have the diagnosis.
Remove { in/out-interface=all-ppp } matchers from your current MSS adjustment (why do you set them asymmetric?) and test the game

bajodel · October 4, 2015, 4:24am

any updates ? ..

mikea · October 4, 2015, 10:27am

Tryed both of your recomendations but in vein.
Btest from one end to the other gives full bandwidth.
The issue is when traffic comes from one isp through the wireless.
The second ISP has no issues passing full bandwidth over the same wireless links.
When i access the same ISP that has issues over wireless from the lan of the PPPoE concentrator i get full bandwidth.

This is what i call brain damaging issue!!!

bajodel · October 4, 2015, 6:14pm

Maybe provide the MTs export ..so can go deeper

Another thing.. are you using bridges?

mikea · October 4, 2015, 6:38pm

No no bridges at all i ll post export in a few minutes.

# oct/04/2015 21:37:57 by RouterOS 6.32.2
# software id = SR1P-HGKP
#
/interface ethernet
set [ find default-name=ether5 ] comment="ISP A" name=\
    ether1-wan1
set [ find default-name=ether6 ] comment="ISP B1" name=\
    ether2-wan2
set [ find default-name=ether7 ] comment=ISP B2 name=ether3-wan3
set [ find default-name=ether8 ] name=ether4
set [ find default-name=ether1 ] name=ether5
set [ find default-name=ether2 ] name=ether6
set [ find default-name=ether3 ] name=ether7
set [ find default-name=ether4 ] name=ether8
/interface pppoe-client
add comment=ISP A disabled=no interface=ether1-wan1 max-mru=1480 max-mtu=1480 \
    mrru=1600 name=pppoe-wan1 password=xxxxxx user=xxxxxx
add comment=ISP B1 disabled=no interface=ether2-wan2 max-mru=1480 max-mtu=1480 \
    mrru=1600 name=pppoe-wan2 password=xxxxxx user=xxxxxx
add comment=ISP B2 disabled=no interface=ether3-wan3 max-mru=1480 max-mtu=1480 \
    mrru=1600 name=pppoe-wan3 password=xxxxxx user=xxxxxx
/ip neighbor discovery
set ether1-wan1 comment="ISP A"
set ether2-wan2 comment="ISP B1"
set ether3-wan3 comment=ISP B2
set pppoe-wan1 comment=ISP A
set pppoe-wan2 comment=ISP B1
set pppoe-wan3 comment=ISP B2
/interface vlan
add interface=ether8 l2mtu=1576 name=vlan.customers vlan-id=300
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=pool.managment ranges=10.12.12.150-10.12.13.254
add name=pool.pppoe ranges=10.15.15.100-10.15.15.254
add name=pool.local ranges=192.168.18.200-192.168.18.240
add name=pool.expired ranges=10.19.19.100-10.19.19.254
/ip dhcp-server
add add-arp=yes address-pool=pool.managment disabled=no interface=ether8 \
    lease-time=1w name=dhcp.managment
add add-arp=yes address-pool=pool.local disabled=no interface=ether5 \
    lease-time=1w name=dhcp.local
/ppp profile
add change-tcp-mss=yes dns-server=10.15.15.1 local-address=10.15.15.1 name=\
    pppoe-profile queue-type=default remote-address=pool.pppoe
/interface pppoe-server server
add authentication=pap,chap default-profile=pppoe-profile disabled=no \
    interface=vlan.customers max-mru=1480 max-mtu=1480 mrru=1600 \
    one-session-per-host=yes service-name=pppoe-server
add authentication=pap,chap default-profile=pppoe-profile disabled=no \
    interface=ether8 max-mru=1480 max-mtu=1480 mrru=1600 \
    one-session-per-host=yes service-name=service1
/ip address
add address=10.12.12.1/23 interface=ether8 network=10.12.12.0
add address=192.168.18.10/24 interface=ether5 network=192.168.18.0
add address=xxxxxxxxxx.70/28 interface=ether5 network=xxxxxxxxxx.64
add address=xxxxxxxxxx.230/28 interface=ether5 network=xxxxxxxxxx.224
add address=169.254.1.2 interface=ether8 network=255.255.0.0
/ip dhcp-server lease
add address=10.12.13.254 client-id=1:4:18:d6:23:e7:cd mac-address=\
    04:18:D6:23:E7:CD server=dhcp.managment
add address=10.12.13.253 client-id=1:4:18:d6:ce:7e:6 mac-address=\
    04:18:D6:CE:7E:06 server=dhcp.managment
add address=192.168.18.238 client-id=1:4:18:d6:23:b5:fa mac-address=\
    04:18:D6:23:B5:FA server=dhcp.local
add address=192.168.18.237 client-id=1:60:a4:4c:51:fe:20 mac-address=\
    60:A4:4C:51:FE:20 server=dhcp.local
add address=10.12.13.251 client-id=1:dc:9f:db:f2:42:94 mac-address=\
    DC:9F:DB:F2:42:94 server=dhcp.managment
add address=10.12.13.250 client-id=1:dc:9f:db:68:c3:e4 mac-address=\
    DC:9F:DB:68:C3:E4 server=dhcp.managment
add address=192.168.18.236 client-id=1:24:a4:3c:b3:cb:49 mac-address=\
    24:A4:3C:B3:CB:49 server=dhcp.local
add address=10.12.13.249 client-id=1:dc:9f:db:f0:4c:88 mac-address=\
    DC:9F:DB:F0:4C:88 server=dhcp.managment
add address=10.12.13.248 client-id=1:44:d9:e7:26:6:0 mac-address=\
    44:D9:E7:26:06:00 server=dhcp.managment
add address=10.12.13.247 client-id=1:4:18:d6:c8:48:ad mac-address=\
    04:18:D6:C8:48:AD server=dhcp.managment
add address=10.12.13.246 client-id=1:4:18:d6:26:b7:fc mac-address=\
    04:18:D6:26:B7:FC server=dhcp.managment
add address=10.12.13.243 client-id=1:dc:9f:db:92:53:aa mac-address=\
    DC:9F:DB:92:53:AA server=dhcp.managment
/ip dhcp-server network
add address=10.12.12.0/23 dns-server=10.12.12.1 gateway=10.12.12.1
add address=192.168.18.0/24 dns-server=192.168.18.10 gateway=192.168.18.10
/ip dns
set allow-remote-requests=yes servers=8.8.8.8,8.8.4.4
/ip firewall address-list
add address=10.15.15.2-10.15.15.254 list=masq
add address=192.168.18.0/24 list=masq
add address=10.12.12.0/23 list=masq
/ip firewall filter
add chain=output connection-state=established
add chain=output connection-state=related
add action=drop chain=output connection-state=invalid
add chain=forward connection-state=established disabled=yes
add chain=forward connection-state=related disabled=yes
add action=drop chain=forward connection-state=invalid disabled=yes
add action=drop chain=input dst-port=53 in-interface=pppoe-wan1 protocol=udp
add action=drop chain=input dst-port=53 in-interface=pppoe-wan2 protocol=udp
add action=drop chain=input dst-port=53 in-interface=pppoe-wan3 protocol=udp
add action=drop chain=forward dst-port=!80,8080 protocol=tcp src-address=\
    10.19.19.0/24
/ip firewall mangle
add action=change-mss chain=forward in-interface=all-ppp new-mss=1460 \
    protocol=tcp tcp-flags=syn tcp-mss=1461-65535
add chain=prerouting dst-address=xxxxxxxxxx.64/28
add chain=prerouting dst-address=xxxxxxxxxx.224/28
add chain=prerouting dst-address=10.15.15.0/24 src-address=xxxxxxxxxx.224/28
add chain=prerouting dst-address=10.15.15.0/24 src-address=xxxxxxxxxx.64/28
add chain=prerouting src-address=192.168.18.115
add chain=prerouting dst-address=192.168.18.115
add action=mark-connection chain=input in-interface=pppoe-wan1 \
    new-connection-mark=inet1_connection
add action=mark-connection chain=input in-interface=pppoe-wan2 \
    new-connection-mark=inet2_connection
add action=mark-connection chain=input in-interface=pppoe-wan3 \
    new-connection-mark=inet3_connection
add action=mark-connection chain=prerouting connection-mark=no-mark \
    in-interface=pppoe-wan1 new-connection-mark=inet1_connection
add action=mark-connection chain=prerouting connection-mark=no-mark \
    in-interface=pppoe-wan2 new-connection-mark=inet2_connection
add action=mark-connection chain=prerouting connection-mark=no-mark \
    in-interface=pppoe-wan3 new-connection-mark=inet3_connection
add action=mark-routing chain=output connection-mark=inet1_connection \
    new-routing-mark=inet1
add action=mark-routing chain=output connection-mark=inet2_connection \
    new-routing-mark=inet2
add action=mark-routing chain=output connection-mark=inet3_connection \
    new-routing-mark=inet3
add action=mark-connection chain=prerouting connection-mark=no-mark \
    dst-address-type=!local in-interface=ether8 new-connection-mark=\
    inet1_connection per-connection-classifier=both-addresses:3/0 \
    src-address=10.15.15.2-10.15.15.254
add action=mark-connection chain=prerouting connection-mark=no-mark \
    dst-address-type=!local in-interface=ether8 new-connection-mark=\
    inet2_connection per-connection-classifier=both-addresses:3/1 \
    src-address=10.15.15.2-10.15.15.254
add action=mark-connection chain=prerouting connection-mark=no-mark \
    dst-address-type=!local in-interface=ether8 new-connection-mark=\
    inet3_connection per-connection-classifier=both-addresses:3/2 \
    src-address=10.15.15.2-10.15.15.254
add action=mark-routing chain=prerouting connection-mark=inet1_connection \
    in-interface=ether8 new-routing-mark=inet1 src-address=\
    10.15.15.2-10.15.15.254
add action=mark-routing chain=prerouting connection-mark=inet2_connection \
    in-interface=ether8 new-routing-mark=inet2 src-address=\
    10.15.15.2-10.15.15.254
add action=mark-routing chain=prerouting connection-mark=inet3_connection \
    in-interface=ether8 new-routing-mark=inet3 src-address=\
    10.15.15.2-10.15.15.254
add action=mark-connection chain=prerouting connection-mark=no-mark \
    dst-address-type=!local new-connection-mark=inet1_connection src-address=\
    xxxxxxxxxx.64/28
add action=mark-routing chain=prerouting connection-mark=inet1_connection \
    new-routing-mark=inet1 src-address=xxxxxxxxxx.64/28
add action=mark-connection chain=prerouting connection-mark=no-mark \
    dst-address-type=!local new-connection-mark=inet2_connection src-address=\
    xxxxxxxxxx.224/28
add action=mark-routing chain=prerouting connection-mark=inet2_connection \
    new-routing-mark=inet2 src-address=xxxxxxxxxx.224/28
/ip firewall nat
add action=masquerade chain=srcnat dst-address=xxxxxxxxxx.75 out-interface=\
    ether5
add action=masquerade chain=srcnat out-interface=pppoe-wan1 src-address-list=\
    masq
add action=masquerade chain=srcnat out-interface=pppoe-wan2 src-address-list=\
    masq
add action=masquerade chain=srcnat out-interface=pppoe-wan3 src-address-list=\
    masq
add action=redirect chain=dstnat dst-port=80 in-interface=vlan.customers \
    protocol=tcp src-address=10.19.19.0/24 to-ports=8080
/ip firewall service-port
set sip disabled=yes
/ip proxy
set enabled=yes
/ip proxy access
add action=deny redirect-to=192.168.18.10 src-address=10.19.19.0/24
/ip route
add check-gateway=ping distance=1 gateway=pppoe-wan1 routing-mark=inet1
add check-gateway=ping distance=1 gateway=pppoe-wan2 routing-mark=inet2
add check-gateway=ping distance=1 gateway=pppoe-wan3 routing-mark=inet3
add check-gateway=ping distance=1 gateway=pppoe-wan3
add check-gateway=ping distance=2 gateway=pppoe-wan2
add check-gateway=ping distance=3 gateway=pppoe-wan1
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www disabled=yes
set ssh disabled=yes
/ppp aaa
set interim-update=5m use-radius=yes
/ppp profile
add change-tcp-mss=yes dns-server=10.19.19.1 local-address=10.19.19.1 name=\
    pppoe-expired queue-type=default-small remote-address=pool.local
/radius
add address=192.168.18.115 secret=xxxxxx service=ppp
/radius incoming
set accept=yes
/system clock
set time-zone-name=Europe/Athens
/system identity
set name=xxxxxxxxxx
/system routerboard settings
set cpu-frequency=1200MHz memory-frequency=1066DDR protected-routerboot=\
    disabled

bajodel · October 6, 2015, 7:05pm

I suspect you have more than a single mss adjustment; this is not necessarily a problem but surely it doesn’t help to clean the picture. In ppp/pppoe profiles you now have mss clamp auto adjust checked (in/out conn), AND you have also the manually inserted ones.
Take a look in live mangle table and look for ‘D’ marked rules (dynamic).
In your shoes I would disable all auto adjustments and I would set 2 global manual mss clamps; furthermore.. I would try to set a low value to start (1420, both directions and for all interfaces so both links).

bajodel · October 11, 2015, 8:46am

any updates?

mikea · September 21, 2016, 9:14pm

After a year i can say that this issue is still present!

P.S. After Getting MTCNA & MTCRE i m still struggling with this issue (wonder why… )

Any help will be more than welcome.