Community discussions

 
User avatar
dgnevans
Member
Member
Topic Author
Posts: 463
Joined: Fri Mar 08, 2013 11:24 am
Location: Zimbabwe
Contact:

PCC Load balancing and https connection issues

Mon Jun 16, 2014 1:26 pm

I have an interesting one. Recently I changed my service provider for two satellite links. I have 3 satellite links all connected together using PCC load balancing. Last week I updated my software to 6.15 on my routers which is around the same time that the new link were installed. The load balancing seems to work well the only problem we have is when a user opens a site that is https. The browser seems to hang on establishing a secure link and often the page times out. If you refresh generally on the second or third time it loads. If you disable the links at test each individually there does not seem to be an issue. This tends to happen after some time. does anyone have any ideas on what could be going wrong. Below is my mangle rules and nat

IP Mangle

0 chain=input action=mark-connection new-connection-mark=WAN1_CONN
passthrough=yes in-interface=WAN1

1 chain=input action=mark-connection new-connection-mark=WAN2_CONN
passthrough=yes in-interface=WAN2

2 chain=input action=mark-connection new-connection-mark=WAN3_CONN
passthrough=yes in-interface=WAN3

3 chain=output action=mark-routing new-routing-mark=TO_WAN1 passthrough=yes
connection-mark=WAN1_CONN

4 chain=output action=mark-routing new-routing-mark=TO_WAN2 passthrough=yes
connection-mark=WAN2_CONN

5 chain=output action=mark-routing new-routing-mark=TO_WAN3 passthrough=yes
connection-mark=WAN3_CONN

6 chain=prerouting action=accept dst-address=192.168.2.0/24
in-interface=BACKBONE

7 chain=prerouting action=accept dst-address=192.168.3.0/24
in-interface=BACKBONE

8 chain=prerouting action=accept dst-address=192.168.4.0/24
in-interface=BACKBONE

9 chain=prerouting action=mark-connection new-connection-mark=WAN1_CONN
passthrough=yes dst-address-type=!local in-interface=BACKBONE
per-connection-classifier=src-address:3/0

10 chain=prerouting action=mark-connection new-connection-mark=WAN2_CONN
passthrough=yes dst-address-type=!local in-interface=BACKBONE
per-connection-classifier=src-address:3/1

11 chain=prerouting action=mark-connection new-connection-mark=WAN3_CONN
passthrough=yes dst-address-type=!local in-interface=BACKBONE
per-connection-classifier=src-address:3/2

12 chain=prerouting action=mark-routing new-routing-mark=TO_WAN1
passthrough=yes in-interface=BACKBONE connection-mark=WAN1_CONN

13 chain=prerouting action=mark-routing new-routing-mark=TO_WAN3
passthrough=yes in-interface=BACKBONE connection-mark=WAN3_CONN

14 chain=prerouting action=mark-routing new-routing-mark=TO_WAN2
passthrough=yes in-interface=BACKBONE connection-mark=WAN2_CONN

IP Nat

1 chain=srcnat action=masquerade out-interface=WAN1

2 chain=srcnat action=masquerade out-interface=WAN2

3 chain=srcnat action=masquerade out-interface=WAN3
 
User avatar
dgnevans
Member
Member
Topic Author
Posts: 463
Joined: Fri Mar 08, 2013 11:24 am
Location: Zimbabwe
Contact:

Re: PCC Load balancing and https connection issues

Mon Jun 16, 2014 5:15 pm

I downgraded the software from 6.15 to 6.7 and it worked for about 2 hours and then https sites stopped loading. Hopefully someone has an idea for me.
 
TikUser
newbie
Posts: 48
Joined: Thu Jul 04, 2013 2:40 pm
Location: EU

Re: PCC Load balancing and https connection issues

Tue Jun 17, 2014 3:56 am

- Use "passthrough=no" in the rules 3,4,5,12,13,14.
- Put "connection-state=new" in the rules 0,1,2,9,10,11.
- Put rules 6,7,8 at the top.
- Use public DNS servers (GoogleDNS, OpenDNS, NortonDNS,...) for all your customers.
- Decrease (or increase, if your size is 512) the Max UDP packet size to 1410. Try also with 4096.
- Increase the connection tracking timeouts.

- Reboot the router after the changes.
- If you activated the hotspot feature, then you need to adapt these PCC rules.
 
User avatar
dgnevans
Member
Member
Topic Author
Posts: 463
Joined: Fri Mar 08, 2013 11:24 am
Location: Zimbabwe
Contact:

Re: PCC Load balancing and https connection issues

Tue Jun 17, 2014 10:15 am

Thank you TikUser.
I have made the changes and am testing. Will let you know how tests go later today.
 
User avatar
dgnevans
Member
Member
Topic Author
Posts: 463
Joined: Fri Mar 08, 2013 11:24 am
Location: Zimbabwe
Contact:

Re: PCC Load balancing and https connection issues

Tue Jun 17, 2014 2:05 pm

Changed seems to work well for +- 5 hours and then suddenly time-outs or ssl error started to occur on traffic flowing through Wan2. Cannot see the reason for it. Any Suggestions
Below is the updated config

0 chain=prerouting action=accept dst-address=192.168.2.0/24
in-interface=BACKBONE

1 chain=prerouting action=accept dst-address=192.168.3.0/24
in-interface=BACKBONE

2 chain=prerouting action=accept dst-address=192.168.4.0/24
in-interface=BACKBONE

3 chain=input action=mark-connection new-connection-mark=WAN1_CONN
passthrough=yes connection-state=new in-interface=WAN1

4 chain=input action=mark-connection new-connection-mark=WAN2_CONN
passthrough=yes connection-state=new in-interface=WAN2

5 chain=input action=mark-connection new-connection-mark=WAN3_CONN
passthrough=yes connection-state=new in-interface=WAN3

6 chain=output action=mark-routing new-routing-mark=TO_WAN1 passthrough=no
connection-mark=WAN1_CONN

7 chain=output action=mark-routing new-routing-mark=TO_WAN2 passthrough=no
connection-mark=WAN2_CONN

8 chain=output action=mark-routing new-routing-mark=TO_WAN3 passthrough=no
connection-mark=WAN3_CONN

9 chain=prerouting action=mark-connection new-connection-mark=WAN1_CONN
passthrough=yes connection-state=new dst-address-type=!local
in-interface=BACKBONE
per-connection-classifier=both-addresses-and-ports:3/0

10 chain=prerouting action=mark-connection new-connection-mark=WAN2_CONN
passthrough=yes connection-state=new dst-address-type=!local
in-interface=BACKBONE
per-connection-classifier=both-addresses-and-ports:3/1

11 chain=prerouting action=mark-connection new-connection-mark=WAN3_CONN
passthrough=yes connection-state=new dst-address-type=!local
in-interface=BACKBONE
per-connection-classifier=both-addresses-and-ports:3/2

12 chain=prerouting action=mark-routing new-routing-mark=TO_WAN1 passthrough=n>
in-interface=BACKBONE connection-mark=WAN1_CONN

13 chain=prerouting action=mark-routing new-routing-mark=TO_WAN2 passthrough=n>
in-interface=BACKBONE connection-mark=WAN2_CONN

14 chain=prerouting action=mark-routing new-routing-mark=TO_WAN3 passthrough=n>
in-interface=BACKBONE connection-mark=WAN3_CONN
 
TikUser
newbie
Posts: 48
Joined: Thu Jul 04, 2013 2:40 pm
Location: EU

Re: PCC Load balancing and https connection issues

Tue Jun 17, 2014 2:59 pm

- Do NOT use “both-addresses-and-ports“ as the PCC classifier. This option breaks https connections. Use „src-address“. You can also use „both-addresses“, but then you need to EXCLUDE https connections from PCC rules.
- Check the settings in the satellite "modems".
- Have you properly configured the routes (IP-> Routes)?
- Do you need rules 3,4,5,6,7,8? If you don't need remote access from the internet to your router (NOT from your LAN), then you can disable these rules.
- These problems can also occur if there are problems with the:
a) customer’s pc,
b) browser,
c) ssl certifikate.

See this:
http://www.digicert.com/ssl-problems.htm
http://www.ehow.com/facts_7297602_do-pa ... -out_.html
http://laptapos.com/internet/how-to-fix ... le-chrome/

- How have you configured the connection tracking timeouts?
Try these settings:
/ip firewall connection tracking
set enabled=yes generic-timeout=10m icmp-timeout=10s tcp-close-timeout=10s tcp-close-wait-timeout=1m tcp-established-timeout=1h tcp-fin-wait-timeout=1m \
    tcp-last-ack-timeout=30s tcp-syn-received-timeout=1m tcp-syn-sent-timeout=10s tcp-time-wait-timeout=1m udp-stream-timeout=3m udp-timeout=30s
Last edited by TikUser on Tue Jun 17, 2014 3:13 pm, edited 1 time in total.
 
User avatar
dgnevans
Member
Member
Topic Author
Posts: 463
Joined: Fri Mar 08, 2013 11:24 am
Location: Zimbabwe
Contact:

Re: PCC Load balancing and https connection issues

Tue Jun 17, 2014 3:09 pm

Where we are noticing it is https://malil.google.com this is the first to freak out. We do not require access from outside.
On the satellite modems there is nothing to configure. I have a router on each modem which connects us to the outside world and handles nat to individual servers. As I did not want to mess around with PCC and natting to servers etc. It does seem to be PCC that is breaking the ssl connections.
 
User avatar
dgnevans
Member
Member
Topic Author
Posts: 463
Joined: Fri Mar 08, 2013 11:24 am
Location: Zimbabwe
Contact:

Re: PCC Load balancing and https connection issues

Tue Jun 17, 2014 3:21 pm

I have changed it from both addresses and ports to src and also disabled statement 3 through 8 and shifted others up. testing again.
 
TikUser
newbie
Posts: 48
Joined: Thu Jul 04, 2013 2:40 pm
Location: EU

Re: PCC Load balancing and https connection issues

Tue Jun 17, 2014 3:23 pm

PCC breaks https connections when you are NOT using "src-address". Read my previous post. I updated it. I also posted a connection tracking configuration. This is not a problem with the PCC. HTTPS doesn't like when connections are coming from more than 1 ip address.
Every time you modify the PCC rules, you need to reboot the router.
 
User avatar
dgnevans
Member
Member
Topic Author
Posts: 463
Joined: Fri Mar 08, 2013 11:24 am
Location: Zimbabwe
Contact:

Re: PCC Load balancing and https connection issues

Tue Jun 17, 2014 3:37 pm

Great have made changes and am testing. Have been rebooting after every change made. So far so good lets see how it goes. Thanks for the help. Will keep posted.
 
User avatar
dgnevans
Member
Member
Topic Author
Posts: 463
Joined: Fri Mar 08, 2013 11:24 am
Location: Zimbabwe
Contact:

Re: PCC Load balancing and https connection issues

Tue Jun 17, 2014 5:21 pm

Thanks for your help Tikuser. I am starting to suspect i am having an issue with my Wan2. Everytime I get a failure or time out , the https traffic passes through this connection. If it passes through the other connections I do no have a problem. It works well for a number of hours at time and then start to see it happening. I have removed potential failure points. Have tried it with src only and then with src and dst no difference. so at this point it points to either the connection or the router randomly pushing traffic to the wrong connection.
 
TikUser
newbie
Posts: 48
Joined: Thu Jul 04, 2013 2:40 pm
Location: EU

Re: PCC Load balancing and https connection issues

Tue Jun 17, 2014 6:25 pm

It could be, or you are missing some small detail. I'm using PCC without problems so far.
Mikrotik users reported routing mark problems in the most recent versions of ROS. Make a text backup (not full backup). Reset your router configuration (with "No default configuration" enabled). Import the text backup. Maybe it helps. Netinstall only as last resort.
If possible, replace the router with another one to test if the problem repeats.

Good luck!
Last edited by TikUser on Tue Jun 17, 2014 7:23 pm, edited 1 time in total.
 
User avatar
dgnevans
Member
Member
Topic Author
Posts: 463
Joined: Fri Mar 08, 2013 11:24 am
Location: Zimbabwe
Contact:

Re: PCC Load balancing and https connection issues

Tue Jun 17, 2014 7:22 pm

I have ruled out the links. If I point traffic through one of the links directly I dont get any time outs or https errors. I had PCC working previously without issues. It even worked with both and ports. The only thing I have changed since then is the software version on the routers and the vsat connections. Otherwise the configs have not changed. I have contacted mikrotik just in case it is the router mark issue that other users have encountered. I have seen errors on my routers. showind traffic destined for one connection appearing on another conneciton. Each of the logs for the connections shows similar errors.
On Wan 3 192.168.4.0/24
Jun 17 13:48:43.704: %SEC-6-IPACCESSLOGP: list 101 denied tcp 192.168.3.2(64288) -> 66.223.170.168(80), 1 packet
Jun 17 13:48:43.704: %SEC-6-IPACCESSLOGP: list 101 denied tcp 192.168.3.2(35471) -> 108.168.176.241(443), 1 packet
Jun 17 13:48:43.704: %SEC-6-IPACCESSLOGP: list 101 denied tcp 192.168.3.2(55472) -> 31.13.91.97(443), 2 packets
Jun 17 13:48:43.704: %SEC-6-IPACCESSLOGP: list 101 denied tcp 192.168.3.2(63004) -> 23.78.232.93(80), 3 packets
Jun 17 13:48:43.704: %SEC-6-IPACCESSLOGP: list 101 denied tcp 192.168.3.2(49571) -> 17.172.192.25(443), 3 packets
Jun 17 13:48:43.704: %SEC-6-IPACCESSLOGP: list 101 denied tcp 192.168.3.2(49831) -> 173.194.41.200(80), 1 packet
Jun 17 13:48:43.704: %SEC-6-IPACCESSLOGP: list 101 denied tcp 192.168.3.2(55477) -> 31.13.91.97(443), 1 packet
Jun 17 13:48:43.704: %SEC-6-IPACCESSLOGP: list 101 denied tcp 192.168.3.2(63094) -> 23.99.20.254(443), 1 packet
Jun 17 13:48:43.704: %SEC-6-IPACCESSLOGP: list 101 denied tcp 192.168.3.2(65182) -> 31.13.91.97(443), 1 packet
Jun 17 13:48:43.704: %SEC-6-IPACCESSLOGP: list 101 denied tcp 192.168.3.2(65183) -> 31.13.91.97(443), 1 packet
Jun 17 13:48:43.704: %SEC-6-IPACCESSLOGP: list 101 denied tcp 192.168.3.2(55476) -> 31.13.91.97(443), 1 packet
Jun 17 13:48:43.704: %SEC-6-IPACCESSLOGP: list 101 denied tcp 192.168.3.2(55470) -> 31.13.69.80(443), 1 packet
Jun 17 13:48:43.704: %SEC-6-IPACCESSLOGP: list 101 denied tcp 192.168.3.2(33754) -> 184.173.179.37(443), 1 packet
Jun 17 14:37:43.356: %SEC-6-IPACCESSLOGP: list 101 denied tcp 192.168.2.2(4798) -> 59.104.186.71(443), 1 packet

I have another router I can put in and test with. Will do that tomorrow and see how that goes. Will wait to see what mikrotik come back with as well.

Thanks again
 
TikUser
newbie
Posts: 48
Joined: Thu Jul 04, 2013 2:40 pm
Location: EU

Re: PCC Load balancing and https connection issues

Tue Jun 17, 2014 7:32 pm

I'm also interested to hear where the problem is. Keep us informed. :D
 
User avatar
dgnevans
Member
Member
Topic Author
Posts: 463
Joined: Fri Mar 08, 2013 11:24 am
Location: Zimbabwe
Contact:

Re: PCC Load balancing and https connection issues

Tue Jun 17, 2014 7:36 pm

will update when I have further info
 
User avatar
dgnevans
Member
Member
Topic Author
Posts: 463
Joined: Fri Mar 08, 2013 11:24 am
Location: Zimbabwe
Contact:

Re: PCC Load balancing and https connection issues

Wed Jun 18, 2014 5:32 pm

So our provider change the route of one connection which seems to have solved most of the broken https. There are still some packets ending up at the wrong routers so router marking must still be a problem waiting for Mikrotik support to assist with this. Will update when finalised issues.
 
User avatar
dgnevans
Member
Member
Topic Author
Posts: 463
Joined: Fri Mar 08, 2013 11:24 am
Location: Zimbabwe
Contact:

Re: PCC Load balancing and https connection issues

Sat Jul 12, 2014 10:49 pm

Ok so some weeks of working on this. I have reconfigured my load balancing router from scratch. And have been working with my service provider. Found a number of issues on their side. Their DNS servers were not 100% they fixed that. Got them to disable the dns cache on the modem as that would hijack dns queries. THey disabled their bandwidth management on our connections as that was slowing our browsing down and causing issues. Now the load balancing seems to be working well. after they made these changes.
 
TikUser
newbie
Posts: 48
Joined: Thu Jul 04, 2013 2:40 pm
Location: EU

Re: PCC Load balancing and https connection issues

Mon Jul 14, 2014 1:36 am

I'm glad to hear that now is all working well!
 
iglobal
just joined
Posts: 11
Joined: Thu Mar 07, 2013 6:34 am

Re: PCC Load balancing and https connection issues

Fri Jan 23, 2015 2:38 pm

I'm glad to hear that now is all working well!

Hello TikUser,

Can you public the best script for load balancing please?
 
TikUser
newbie
Posts: 48
Joined: Thu Jul 04, 2013 2:40 pm
Location: EU

Re: PCC Load balancing and https connection issues

Sat Jan 24, 2015 4:53 pm

There is no best script. Every script needs to be adapted to your needs.
I would suggest you to search this forum and the Mikrotik wiki for Load Balancing (PCC) and QOS discussions/examples. There are also great TikTube videos which explain QoS and PCC.

You can start building your script with these general examples:
http://wiki.mikrotik.com/wiki/Manual:PCC
How PCC works (beginner)
Mikrotik DUAL WAN Load Balancing using PCC method. by zaiB
Mikrotik 4 WAN Load Balancing using PCC method. by zaiB
 
User avatar
dgnevans
Member
Member
Topic Author
Posts: 463
Joined: Fri Mar 08, 2013 11:24 am
Location: Zimbabwe
Contact:

Re: PCC Load balancing and https connection issues

Mon Feb 23, 2015 1:05 pm

Hi Iglobal
As tikuser says the pcc load balancing really changes depending on what you trying to do. It is not difficult if you follow the tutorials. My problems were all ISP related. I moved ISP's late last year and have not had an issue since. If you have problems post on this forum and guys will help you.
 
ik3umt
Member Candidate
Member Candidate
Posts: 248
Joined: Tue Jul 08, 2014 3:58 pm

Re: PCC Load balancing and https connection issues

Tue Aug 04, 2015 6:26 pm

- Use "passthrough=no" in the rules 3,4,5,12,13,14.
- Put "connection-state=new" in the rules 0,1,2,9,10,11.
- Put rules 6,7,8 at the top.
- Use public DNS servers (GoogleDNS, OpenDNS, NortonDNS,...) for all your customers.
- Decrease (or increase, if your size is 512) the Max UDP packet size to 1410. Try also with 4096.
- Increase the connection tracking timeouts.

- Reboot the router after the changes.
- If you activated the hotspot feature, then you need to adapt these PCC rules.
Quite old post but.....few questions:

What does "passthrough=" parameter really do ?
What's the importance of mangle rule order ?
Does the "distance" parameter in marked routes make any difference? Should it be set to same "1" distance or 1,2,3 per route ?

Thank you
 
TikUser
newbie
Posts: 48
Joined: Thu Jul 04, 2013 2:40 pm
Location: EU

Re: PCC Load balancing and https connection issues

Fri Sep 25, 2015 3:17 pm

a)
http://forum.mikrotik.com/viewtopic.php?t=51346#p261176
http://forum.mikrotik.com/viewtopic.php?t=96273

b)
"When processing a chain, rules are taken from the chain in the order they are listed there from top to bottom. If a packet matches the criteria of the rule, then the specified action is performed on it, and no more rules are processed in that chain (the exception is the passthrough action)."
Link.

c)
Routes with lower distance parameter are more important. 1 is higher than 10.
This is useful, if you have some sort of failover option. If one link goes down, the backup route will be activated and traffic will go through the other link.
Also important if you are using dynamic routing protocols like OSPF and BGP.

Manual:Route Selection Algorithm in RouterOS
What Is Administrative Distance?
 
NerdMobile
just joined
Posts: 2
Joined: Fri Oct 02, 2015 5:31 pm

Re: PCC Load balancing and https connection issues

Fri Oct 02, 2015 5:38 pm

don't know if will work out for you too, but i had the same problem, and all i did is set the rules where the pcc are in to dst port !443 and its working so far

Who is online

Users browsing this forum: Google [Bot] and 61 guests