PCC upload speed issue

eslamxboss · Sun May 27, 2018 3:06 pm

Hello All,
I'm using PCC method with 4 WAN connection
Every WAN has Download Speed:10M and Upload Speed: 1M
after PCC configuration Download speed: 40M but upload speed:1M
I need to get the upload speed of 4 WANs

/interface ethernet
set [ find default-name=ether1 ] name=WAN1
set [ find default-name=ether2 ] name=WAN2
set [ find default-name=ether3 ] name=WAN3
set [ find default-name=ether4 ] name=WAN4
set [ find default-name=ether10 ] name=LAN

/ip address
add address=192.168.0.2/24 interface=WAN1 network=192.168.0.0
add address=192.168.2.2/24 interface=WAN2 network=192.168.2.0
add address=192.168.3.2/24 interface=WAN3 network=192.168.3.0
add address=192.168.4.2/24 interface=WAN4 network=192.168.4.0
add address=10.0.0.1/16 interface=LAN network=10.0.0.0

/ip firewall mangle
add action=accept chain=prerouting disabled=yes dst-address=\
    192.168.0.0-192.168.4.255 in-interface=LAN
add action=mark-connection chain=prerouting connection-mark=no-mark \
    dst-address-type=!local in-interface=LAN new-connection-mark=ISP1_conn \
    passthrough=yes per-connection-classifier=both-addresses-and-ports:4/0
add action=mark-connection chain=prerouting connection-mark=no-mark \
    dst-address-type=!local in-interface=LAN new-connection-mark=ISP2_conn \
    passthrough=yes per-connection-classifier=both-addresses-and-ports:4/1
add action=mark-connection chain=prerouting connection-mark=no-mark \
    dst-address-type=!local in-interface=LAN new-connection-mark=ISP3_conn \
    passthrough=yes per-connection-classifier=both-addresses-and-ports:4/2
add action=mark-connection chain=prerouting connection-mark=no-mark \
    dst-address-type=!local in-interface=LAN new-connection-mark=ISP4_conn \
    passthrough=yes per-connection-classifier=both-addresses-and-ports:4/3
add action=mark-routing chain=prerouting connection-mark=ISP1_conn \
    in-interface=LAN new-routing-mark=to_ISP1
add action=mark-routing chain=prerouting connection-mark=ISP2_conn \
    in-interface=LAN new-routing-mark=to_ISP2
add action=mark-routing chain=prerouting connection-mark=ISP3_conn \
    in-interface=LAN new-routing-mark=to_ISP3
add action=mark-routing chain=prerouting connection-mark=ISP4_conn \
    in-interface=LAN new-routing-mark=to_ISP4
add action=mark-connection chain=prerouting connection-state=new \
    in-interface=WAN1 new-connection-mark=ISP1_conn passthrough=no
add action=mark-connection chain=prerouting connection-state=new \
    in-interface=WAN2 new-connection-mark=ISP2_conn passthrough=no
add action=mark-connection chain=prerouting connection-state=new \
    in-interface=WAN3 new-connection-mark=ISP3_conn passthrough=no
add action=mark-connection chain=prerouting connection-state=new \
    in-interface=WAN4 new-connection-mark=ISP4_conn passthrough=no
add action=mark-routing chain=output connection-mark=ISP1_conn \
    new-routing-mark=to_ISP1 passthrough=no
add action=mark-routing chain=output connection-mark=ISP2_conn \
    new-routing-mark=to_ISP2 passthrough=no
add action=mark-routing chain=output connection-mark=ISP3_conn \
    new-routing-mark=to_ISP3 passthrough=no
add action=mark-routing chain=output connection-mark=ISP4_conn \
    new-routing-mark=to_ISP4 passthrough=no

/ip firewall nat
add action=masquerade chain=srcnat out-interface=WAN1
add action=masquerade chain=srcnat out-interface=WAN2
add action=masquerade chain=srcnat out-interface=WAN3
add action=masquerade chain=srcnat out-interface=WAN4

/ip route
add check-gateway=ping distance=1 gateway=192.168.0.1 routing-mark=to_ISP1
add check-gateway=ping distance=1 gateway=192.168.2.1 routing-mark=to_ISP2
add check-gateway=ping distance=1 gateway=192.168.3.1 routing-mark=to_ISP3
add check-gateway=ping distance=1 gateway=192.168.4.1 routing-mark=to_ISP4
add check-gateway=ping distance=1 gateway=192.168.0.1
add check-gateway=ping distance=2 gateway=192.168.2.1
add check-gateway=ping distance=3 gateway=192.168.3.1
add check-gateway=ping distance=4 gateway=192.168.4.1

sindy · Sun May 27, 2018 3:57 pm

The first question is how exactly do you measure that?

Any single TCP session or UDP exchange uses only one WAN at a time, so the result should be 10 Mbit/s download and 1 Mbit/s upload.

eslamxboss · Sun May 27, 2018 6:29 pm

The first question is how exactly do you measure that?

Any single TCP session or UDP exchange uses only one WAN at a time, so the result should be 10 Mbit/s download and 1 Mbit/s upload.

I mean when i downloading MicroTik using 4 WANs in same time

sindy · Sun May 27, 2018 7:21 pm

Okay, so the fair answer is "I don't know how I measure it because a blackbox measures it for me".

And my short fair response is "I have no idea why you get such result and whether it is realistic or not".

The long response is that I've checked how speedtest.net works - it first opens four sessions in parallel to measure download speed, and then it opens another four TCP sessions in parallel to measure upload speed. This is the case on my connection with a single uplink; it would be interesting to know whether the server detects that the connections from the same client arrive from different public IPs and informs the client so that it could modify its behaviour (open more sessions), but I don't have a second connection handy to test that.

The per-connection-classifier setting in your rules says both-addresses-and-ports:4/n. The hashing function translates two 32-bit values and two 16-bit values into a single 32-bit one, and the client side port increments by one as the TCP sessions are initiated. Although common sense would say that the same change (+1 on source port) should have the same effect on the two least significant bits of the hash for both cases, in reality it may not be so straightforward.

So I'm afraid that the only way to find out what is really going on would be to add count-only rules (action=passthrough) for the speedtest traffic on the wan interfaces, which requires to know the IP address of the chosen speedtest server in advance, and then run the speedtest and see whether all four interfaces got the same amount of traffic.

msatter · Sun May 27, 2018 8:37 pm

I use PCC and don't have any problem with speedtest. I am looking at NAT part because there was no selection on what traffic goes to what line.
To me it seems that one connection is sent out on all lines.

Using source port for PCC is the best when just accessing the internet.
Connection tracking will catch the returning traffic by looking at the source port if I am correct.

sindy · Sun May 27, 2018 9:42 pm

I am looking at NAT part because there was no selection on what traffic goes to what line.

??? The selection of WAN interface is done by the pcc (in case of connections established from inside, which is our case) which assigns the connection-mark translated into a routing-mark in the next step. srcnat happens next, after the WAN has already been chosen.

To me it seems that one connection is sent out on all lines.

??? RouterOS is not forking packets unless you specially ask it to do so. What did you actually have in mind?

Connection tracking will catch the returning traffic by looking at the source port if I am correct.

Connection tracking matches the incoming packets to existing connections by fully matching source and destination socket addresses (for tcp and udp, for icmp something else is used instead of port), not interfaces, but that's not the point here. If the return traffic would not be recognized as such and would thus be dropped, I would expect this to affect the download test at first place, which obviously doesn't happen.

sindy · Sun May 27, 2018 9:47 pm

One idea for the OP, your setup seems quite exotic to me with private subnets on the WANs, so maybe you actually use some 3G modems with SIMs from the same operator? If so, it could be that the summary upload capacity of the cell is limited to 1 Mbit/s no matter how many subscribers use it.

msatter · Sun May 27, 2018 11:34 pm

I am looking at NAT part because there was no selection on what traffic goes to what line.

To me it seems that one connection is sent out on all lines.
??? RouterOS is not forking packets unless you specially ask it to do so. What did you actually have in mind?
!!! I never understood why I should have that many NAT lines for only one translation. I used one and later I went multiple very specific which route matched what line.
Default I would have one NAT line for the traffic that goes through secure line (!no-mark) and the rest is going through the 'unsecure' line.

sindy · Mon May 28, 2018 12:05 am

I never understood why I should have that many NAT lines for only one translation.

Oh, that one

action=masquerade automatically chooses the address assigned to the out-interface chosen by routing as the new source address, but if you would not restrict the scope of that rule, it would src-nat all initial packets, including those to local destinations. That's not a big deal if you don't use dstnat but it is better to have it under explicit control. And you can use src-address or src-address-list for that, or out-interface-list, all that in a single rule, or out-interface, but in this latter case, you need one rule per interface (well, yes, you can also use a single rule with out-interface=!LAN, but that's only true if you only have a single LAN interface).

And if you don't use masquerade, which should only be used at interfaces which get addresses dynamically as it has some "side effects" which are necessary for dynamic addresses but very unpleasant otherwise, and use src-nat with manually specified to-addresses instead, you must choose the right rule (with the right to-addresses) for each interface.

I admit that this particular configuration (with individual masquerade rule per WAN) may even come from me and may look like that because a) I mostly think the src-nat way and b) I may have wanted to make it simple to understand rather than introducing the interface list.

eslamxboss · Mon May 28, 2018 1:52 pm

I use PCC and don't have any problem with speedtest. I am looking at NAT part because there was no selection on what traffic goes to what line.
To me it seems that one connection is sent out on all lines.

Using source port for PCC is the best when just accessing the internet.
Connection tracking will catch the returning traffic by looking at the source port if I am correct.

I tried to use source port too but upload speed goes from 1 wan only

sindy · Mon May 28, 2018 9:47 pm

The reason why it is hard to suggest a solution is that it is not clear what the problem is.

The fact that English is neither your nor my native language doesn't help it either, and my knowledge of 10 words in Arabic makes little difference.

What I've tried to explain in the previous posts was that the allocation of TCP sessions to your 4 WANs works and the measurent method you use works with several sessions in parallel; both these conclusions can be taken from the fact that the download rate depends on whether the PCC is active or not.

In the next step we come to the fact that although the strategy used to choose WANs for sessions measuring upload is exactly the same (because although the sessions are used to measure opposite the direction of data transfer, their establishment is also initiated from the client side, so it is again your PCC rules what chooses the WAN for each of the four sessions), the upload speed measured remained the same when you've activated the PCC.

To find out what is actually wrong, I've asked you to perform additional testing:

So I'm afraid that the only way to find out what is really going on would be to add count-only rules (action=passthrough) for the speedtest traffic on the wan interfaces, which requires to know the IP address of the chosen speedtest server in advance, and then run the speedtest and see whether all four interfaces got the same amount of traffic.

I'll tell you how I did the test: I've started capturing using Wireshark on the machine running the speedtest before starting the actual test, ran the speedtest once, and looked for most data-intensive TCP sessions which, in accord with the assumptions, were all established to the same remote IP address.

For your purpose, you have to do the same first (start capturing using Wireshark, then open the speedtest page, run the test once, keep the speedtest page open, stop capturing), but instead of looking for biggest sessions in the capture, it should be enough for you to apply the following display filter expression:
(http.request.uri.path == "/hello") && (http.referer == "http://www.speedtest.net/")
This should reduce the packet list shown to just a few http GET packets with the same destination IP address. Let's assume it is 1.2.3.4 for the purpose of the code below, you have to set the actual one before applying that code.

After doing that, you have to paste the following code to your CLI (substitute 1.2.3.4 by the actual address):

/ip firewall address-list add list=speedtest address=1.2.3.4
/ip firewall mangle print where chain=postrouting dynamic=no
/ip firewall mangle add chain=postrouting action=passthrough protocol=tcp dst-address-list=speedtest out-interface=WAN4 place-before=0
/ip firewall mangle print where chain=postrouting dynamic=no
/ip firewall mangle add chain=postrouting action=passthrough protocol=tcp dst-address-list=speedtest out-interface=WAN3 place-before=0
/ip firewall mangle print where chain=postrouting dynamic=no
/ip firewall mangle add chain=postrouting action=passthrough protocol=tcp dst-address-list=speedtest out-interface=WAN2 place-before=0
/ip firewall mangle print where chain=postrouting dynamic=no
/ip firewall mangle add chain=postrouting action=passthrough protocol=tcp dst-address-list=speedtest out-interface=WAN1 place-before=0

This will create counting rules for upload traffic to the speedtest server.

After doing that, you have to run speedtest again, using the the same server like before, and send the following CLI commands:

/ip firewall mangle print where dst-address-list="speedtest"
/ip firewall mangle print stats where dst-address-list="speedtest"

The whole exercise should show whether the upload traffic is distributed more or less evenly among the WANs or not.

frank333 · Fri Jul 20, 2018 3:36 pm

also 'I, I have the same problem, thanks to sindy the pcc worked, then after a reboot measuring with speedtest.net the bandwidth in both download and upload no longer add up.
I'm trying to do the test with wireshark.
Someone has a suggestion to give me ?

sindy · Fri Jul 20, 2018 3:46 pm

The first question is whether you still can see traffic on both WANs. Use two action=passthrough out-interface=xxx chain=postrouting rules in mangle to collect outgoing traffic statistics on both WANs, it will tell you whether the PCC still works or not. Further steps depend on the result. And check that src-nat rule which miraculously changed after reboot or upgrade back then, maybe it has changed again?

frank333 · Fri Jul 20, 2018 3:53 pm

hi sindy ; thanks for your patience,
yes the traffic works on both WAN
I'm trying to do the test you suggested above but it's something new to me and I have to study it a bit...
The rule that was changed during the upgrade is intact and the configuration is the same.

frank333 · Fri Jul 20, 2018 5:06 pm

I entered the in address-list there ip computer from which launch speedtest.net
I inserted the rule:

/ip firewall mangle print where chain=postrouting dynamic=no comment=test_wireshark

but the 'Tik does not accept the others instead are accepted

/ip firewall address-list add list=speedtest address=10.0.0.130

/ip firewall mangle print where chain=postrouting dynamic=no comment=test_wireshark
/ip firewall mangle add chain=postrouting action=passthrough protocol=tcp dst-address-list=speedtest out-interface=pppoe-out1 place-before=0

/ip firewall mangle print where chain=postrouting dynamic=no comment=test_wireshark
/ip firewall mangle add chain=postrouting action=passthrough protocol=tcp dst-address-list=speedtest dst-address=192.168.8.1 place-before=0

Schermata del 2018-07-20 15.52.42.png

sindy · Fri Jul 20, 2018 7:08 pm

Forget about particular addresses. The idea behind the passthrough rules was to count summary traffic being sent via the two WANs. Only if they show approximately equal amounts of traffic, it makes sense to analyse speedtest behaviour and how the PCC handles it. Otherwise it means that the PCC broke and has to be fixed.

frank333 · Fri Jul 20, 2018 7:32 pm

I managed to understand that in the address-list I have to insert the ip of the server speedtest that I use for the test 88.149.202.248 in my case, moreover since I have a gateway connected directly in lan I modified the rule of mangle so tell me if it seems correct to you :

add chain=postrouting action=passthrough protocol=tcp dst-address=192.168.8.1 dst-address-list=speedtest 
      out-interface=bridge log=no log-prefix=""

.

what is the point of this line: /ip firewall mangle print where chain=postrouting dynamic=no comment=test_wireshark ?

The wireshark filter (http.request.uri.path == "/hello") && (http.referer == "http://www.speedtest.net/") appears not to work due to a syntax error.
http.referer should I enter the http address of the speedtest server I'm testing on ?

Schermata del 2018-07-20 19.03.36.png

now something is passing on wan1

sindy · Fri Jul 20, 2018 9:28 pm

Sorry, I'm away from the PC till tomorrow noon at least so I can't read the previous posts easily. However, although the Wireshark filter seems correct to me (and to Wireshark too as its background is green so I don't understand what syntax error you talk about), its sole purpose was to identify the test server's IP address, so as you've managed to find it some other way, you don't need to care much about the filter.

To the rules - a packet which is routed to the address of the server via 192.168.8.1 does not have dst-address 192.168.8.1, the dst-address remains the one of the server. So this rule won't count.

Maybe it would be best to modify the counting rules to check dst-address=server.ip and routing-mark=to_wan1 (or what routing-mark values you use).

frank333 · Fri Jul 20, 2018 11:44 pm

I did the count for the test on speedtest.net, I hope I wrote the right rules for mangle, it seems that much of the traffic passes on WAN2 (????) and yet is also committed WAN1
as for the download seems to me that there is some problem, for the upload instead I see that the band is summed up correctly.

> /ip firewall mangle print where dst-address-list="speedtest"      
Flags: X - disabled, I - invalid, D - dynamic 
 0    ;;; test_speedtest.net (server 88.149.202.248)
      chain=postrouting action=passthrough routing-mark=to_WAN1 protocol=tcp dst-address-list=speedtest log=no 
      log-prefix="" 

 1    chain=postrouting action=passthrough routing-mark=to_WAN2 protocol=tcp dst-address-list=speedtest log=no 
      log-prefix="" 

> /ip firewall mangle print stats where dst-address-list="speedtest"
Flags: X - disabled, I - invalid, D - dynamic 
 #    CHAIN                                                      ACTION                            BYTES         PACKETS
 0    ;;; test_speedtest.net (server 88.149.202.248)
      postrouting                                                passthrough                       6 364             145
 1    postrouting                                                passthrough                   7 524 615           6 404

sindy · Sat Jul 21, 2018 12:19 am

The counter rules seem fine to me, the traffic distribution does not. So the next step would be to remove the dst-address-list from these rules, reset the counters and try to run normal traffic for some time. If the counter rules show again that most traffic goes via wan2, there is something wrong with the PCC, otherwise PCC works fine but the source TCP port allocation at the PC running the speedtest yields the same remainder after hash division and thus hits the same wan.

frank333 · Sat Jul 21, 2018 1:32 am

> /ip firewall mangle print                                  
Flags: X - disabled, I - invalid, D - dynamic 
 0    ;;; test_speedtest.net (server 88.149.202.248)
      chain=postrouting action=passthrough routing-mark=to_WAN1 protocol=tcp log=no log-prefix="" 

 1    chain=postrouting action=passthrough routing-mark=to_WAN2 protocol=tcp log=no log-prefix=""

Schermata del 2018-07-21 00.22.24.png

I checked the interface graph so much that I downloaded an ISO linux via torrent and the traffic comes out mainly from WAN1 (pppoe) and not from WAN2 (LTE gateway)!?

sindy · Sat Jul 21, 2018 2:21 pm

The second rule (index 1) has counted much more and it refers to to_wan2, but I'm not sure whether a torrent download always opens many connections or it depends on some circumstances. So hard to say. By "normal traffic" I had in mind rather many different pages on different servers, even though small, than a single large file.

frank333 · Sat Jul 21, 2018 2:33 pm

you have also browsed the internet normally ; it seems that all traffic passes via WAN2 instead it happens from WAN1 I'll attach the firewall prints

also I have this strange situation: when I do a speedtest with the ipad I have correct upload values (combined bandwidth) instead if I do the test with the computer linux I write I have values of a single wan (WAN1)

firewal mangle print.txt

ip firewall filterprint.txt

ip firewall nat print .txt

frank333 · Sat Jul 21, 2018 2:52 pm

test.jpeg

maybe I understood the problem! I tried to turn off all the rules of dscp and the load balancing works finally. now however how can I do to keep the dscp?

sindy · Sat Jul 21, 2018 2:54 pm

I hazily remember you wanted an exceptional handling for one LAN IP address which should always use only wan2. This is rule 16 in file firewal mangle print.txt above and it is disabled there; however, there is also the rule originally intended to handle traffic incoming via the wireless gateway (192.168.8.1), which is rule 19 in that file.

Could it be that the PC on which you were running the speedtest before has (or had while you were testing) an IP address from 192.168.8.0 subnet rather than from 10.0.0.0/24?

sindy · Sat Jul 21, 2018 2:57 pm

All the dscp-marking rules are in the postrouting chain, which means that they cannot affect the already chosen route for a packet any more, so I assume it is just a coincidence that it started working after you've disabled them and that you have changes something else in parallel.

frank333 · Sat Jul 21, 2018 3:00 pm

All the dscp-marking rules are in the postrouting chain, which means that they cannot affect the already chosen route for a packet any more, so I assume it is just a coincidence that it started working after you've disabled them and that you have changes something else in parallel.

as soon as you reactivate the dscp rules, the load balancing doesn't work. It's really a complex thing!

frank333 · Sat Jul 21, 2018 5:16 pm

/queue tree> print
Flags: X - disabled, I - invalid 
 0   ;;; Uplink QoS
     name="QoS_pppoe-out1" parent=pppoe-out1 packet-mark="" limit-at=0 queue=wireless-default priority=8 max-limit=2900k burst-limit=0 burst-threshold=0 
     burst-time=0s bucket-size=0.1 
     
     .....
      9 XI ;;; Downlink QoS
     name="QoS_bridge" parent=bridge packet-mark="" limit-at=0 queue=wireless-default priority=8 max-limit=30M burst-limit=0 burst-threshold=0 burst-time=0s 
     bucket-size=0.1

I realized that the qos on the bridge was also limiting the gatewayLTE traffic (which is in the bridge on eth10) I thought I would solve by separating the eth10 on a second bridge (or some other solution) so that I could manage the dscp excluding the gateway lte, what do you think sindy?

Ps: I can't connect to mikrotik forum if I use load balancing.

sindy · Sat Jul 21, 2018 11:39 pm

I thought I would solve by separating the eth10 on a second bridge (or some other solution) so that I could manage the dscp excluding the gateway lte, what do you think sindy?

I think that this has moved so far away from the theme of this topics that we should return to your original one where it fits much better. So copy the idea there in a way that it would make sense if someone would read that topic only, without jumping to this current one and back, and we shall continue there.

frank333 · Sat Jul 21, 2018 11:58 pm

frank333 · Sun Jul 22, 2018 12:01 am

PCC upload speed issue

PCC upload speed issue

Re: PCC upload speed issue

Re: PCC upload speed issue

Re: PCC upload speed issue

Re: PCC upload speed issue

Re: PCC upload speed issue

Re: PCC upload speed issue

Re: PCC upload speed issue

Re: PCC upload speed issue

Re: PCC upload speed issue

Re: PCC upload speed issue

Re: PCC upload speed issue

Re: PCC upload speed issue

Re: PCC upload speed issue

Re: PCC upload speed issue

Re: PCC upload speed issue

Re: PCC upload speed issue

Re: PCC upload speed issue

Re: PCC upload speed issue

Re: PCC upload speed issue

Re: PCC upload speed issue

Re: PCC upload speed issue

Re: PCC upload speed issue

Re: PCC upload speed issue

Re: PCC upload speed issue

Re: PCC upload speed issue

Re: PCC upload speed issue

Re: PCC upload speed issue

Re: PCC upload speed issue

Re: PCC upload speed issue

Re: PCC upload speed issue

Who is online