Community discussions

MUM Europe 2020
 
godovic
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 52
Joined: Sun Mar 08, 2009 11:52 pm

ECMP vs NTH vs PCC

Sun May 10, 2009 12:09 am

Can someone write something like pros and cons for this three load balancing solutions?

Does multi wan configuration with NTH can be used on only one client PC? Lets say will the download accelerators or torrents works with all wans?
Is there solutions for ECMP flushing?

PCC is more like NTH or ECMP?

tnx
 
User avatar
Chupaka
Forum Guru
Forum Guru
Posts: 8345
Joined: Mon Jun 19, 2006 11:15 pm
Location: Minsk, Belarus
Contact:

Re: ECMP vs NTH vs PCC

Sun May 10, 2009 12:26 am

PCC is like extended ECMP without flushing =)

the main ideological difference between Nth and PCC is the client: in Nth each client uses one and only WAN interface; in PCC it uses many WAN interfaces
Russian-speaking forum: https://forum.mikrotik.by/. Welcome!

For every complex problem, there is a solution that is simple, neat, and wrong.

MikroTik. Your life. Your routing.
 
godovic
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 52
Joined: Sun Mar 08, 2009 11:52 pm

Re: ECMP vs NTH vs PCC

Sun May 10, 2009 10:45 pm

PCC is like extended ECMP without flushing =)

the main ideological difference between Nth and PCC is the client: in Nth each client uses one and only WAN interface; in PCC it uses many WAN interfaces
Tnx a lot - waiting for 3.24 :D
 
User avatar
macgaiver
Forum Guru
Forum Guru
Posts: 1730
Joined: Wed May 18, 2005 5:57 pm
Location: Sol III, Sol system, Sector 001, Alpha Quadrant

Re: ECMP vs NTH vs PCC

Mon May 11, 2009 8:27 am


the main ideological difference between Nth and PCC is the client: in Nth each client uses one and only WAN interface; in PCC it uses many WAN interfaces
eeee..... if you write it other way around it can also be correct :) - need to be more precise

PCC is the same nth, but with additional feature - it can remember where does the previous packet from the "flow" went, and send the next packet the same way. You can identify flow by specifying set of options from src,dst addresses and ports
With great knowledge comes great responsibility, because of ability to recognize id... incompetent people much faster.
 
godovic
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 52
Joined: Sun Mar 08, 2009 11:52 pm

Re: ECMP vs NTH vs PCC

Mon May 11, 2009 11:32 am


the main ideological difference between Nth and PCC is the client: in Nth each client uses one and only WAN interface; in PCC it uses many WAN interfaces
eeee..... if you write it other way around it can also be correct :) - need to be more precise

PCC is the same nth, but with additional feature - it can remember where does the previous packet from the "flow" went, and send the next packet the same way. You can identify flow by specifying set of options from src,dst addresses and ports

So, that is great improvement for messingers, p2p, https .... tnx a lot. I got 3.24 beta from the support so I will test on my "guinea pigs" @ work :D
 
studyete
just joined
Posts: 20
Joined: Wed Jun 20, 2012 2:37 pm

Re: ECMP vs NTH vs PCC

Tue Feb 05, 2013 8:25 am

I've tried both pcc and nth. I'm having trouble understanding it. When i configure my mikrotik with PCC load balancing, bandwidth from both wan seems to marge. like i have 1MB in wan1 and 512 in wan2. in pcc i got around 1.4mb. but when i use nth. i seems to using only one wan1. the other wan BW remain useless. So my quarry is, Can i get BW using nth same as when i get bw using pcc?
 
aymoon22
just joined
Posts: 11
Joined: Sun Jun 17, 2012 1:26 am

Re: ECMP vs NTH vs PCC

Mon Feb 10, 2014 12:36 pm

so if i have 2 adsl line both is 2 Mb i have to use pcc to get 4Mb in internet surfing (download and browsing)?????
 
LaZyLion
just joined
Posts: 23
Joined: Fri May 09, 2014 10:27 am

Re: ECMP vs NTH vs PCC

Wed Sep 09, 2015 3:26 pm

For anyone like me who has been doing endless googling and finding lots of examples that lack decent explanations, here is how I understand load balancing; nth vs pcc.

This assumes you read examples like https://aacable.wordpress.com/2011/08/2 ... te-script/ and are going cross-eyed.

Let's assume we have two equal wan links connecting us to the internet, one dsl and one cable modem.

Using firewall mangle rules 'nth' option we send every first packet out the dsl connection and every second packet out the cable connection:
/ip firewall mangle 

	add chain=prerouting 	connection-mark=no-mark 	dst-address-type=!local \
	 	in-interface=bridge-local nth=2,1 connection-state=new \
		action=mark-connection 	new-connection-mark=cn_ether1 
    
	add chain=prerouting 	connection-mark=no-mark 	dst-address-type=!local 	\
		in-interface=bridge-local nth=2/2 connection-state=new \
    		action=mark-connection 	new-connection-mark=cn_ether2 
    	


Doing this will give you a completely balanced distribution across your two wan links, which is great if all your traffic is http unsecured stuff like in the good old days.

With this method, a user (host machine) can theoretically download two pictures at once giving the illusion of having doubled their speed. Why is this an illusion? Two trucks doing the speed limit each carrying a load of bricks will get the job done sooner than one truck doing the speed limit carrying a load of bricks. Nothing is moving any faster, you are just getting more done in the same time period.

But this method will break secure sessions such as ssl (https), ssh, tls, etc...

Why? Picture this overly simplified example of an ssh session:
you-dsl: hellow server
server-to-dsl: acknowledge, username please
you-cbl: my username is barney
server-to-cbl: wait what? who are you?
you-dsl: my password is hunter2
server-to-dsl: what? no, i didn't ask for your password, i asked for your username
See the difference? Our dsl link and our cable link have different public facing ip addresses.
Whenever you want to establish a secure session, it's connections have to come from the same
ip address every time. If the address suddenly changes, then the server can't verify that
it is coming the same user/host machine. It suspects a man in the middle attack, or that some third party machine is trying to hi-jack the data stream. Better safe than sorry, so it drops the connection and possibly breaks the session.

With nth, you can't run a secure session because your connections are always going out over different links.

"Ah" you ask, "but what if I mark-connections instead of marking packets?"

You should be doing this anyway as it is less processor intensive. Packets are already marked as being part of connections so marking them again is redundant, and a lot of extra work for the router.

Butt still no difference as far as sessions are concerned.

Connections are not the same as Sessions. A session can have multiple connections to a server. Facebook is a great example of this. One their main page, each picture, games, ads, status's are all transported over their own individual 'connection'. Once that object loads, that connection terminates and another can begin. (There is a limit on how many simultaneous connections you can run.)
All these connections are part of your session with Facebook.

(Yes, you can also have multiple sessions to a service, but lets stay on track.)

This is where PCC comes into play.
/ip firewall mangle 

	add chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface=bridge-local \
		per-connection-classifier=src-address:2/0 action=mark-connection new-connection-mark=cn_ether1 
    
	add chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface=bridge-local \
		per-connection-classifier=src-address:2/1 action=mark-connection new-connection-mark=cn_ether2 
		
In this example, instead of just dealing out packets like a deck of cards ( one for you, one for you, one for you), we do a hash match on the details of a potential connection. You can choose from 9 combinations of details to hash to decide which modem link the connection should traverse, but each option has it's draw backs.

Long story short, "src-address" is the safest. Why? Back to Facebook: all those status updates, pictures, games, and ads all come from different servers all of which may have different ip addresses, but through the magic of money and shared session id's, they all remain part of the same session on their end. Good programmers and network engineers are worth the money.

Buttt this only works as long as all the requests from your computers session come from same ip address. If not, then the connection is broken for security reasons and that picture won't load, even though the rest of the page does. If you have been playing with load balancing, you have probably seen this happen.

Buttttt if you use only src-address in your pcc rules, then your computer will always use the same wan link every time because the source address is the ip address of your computer which probably doesn't change very often, and likely won't change in the middle of a session.

Unfortunately this means you have no load balancing at all and thus no illusion of increased speed.
Your sisters machine in the next room has a different ip address than yours and so she is probably using the cable modem, which at least means you have the dsl link to yourself. This is all well and good until your mother gets online to play farmville. Now one of you has to share a link.

So you use "both-addresses" in your pcc rule. This means that the hash will be different some times because even though your src-address is the same, if your destination-address is different, the link used may be different. This is the desired result of using PCC to load balance connections.

Butttttt now you run the risk of breaking sessions because the connections, though part of the same session, may arrive at the server farm from different ip addresses. Only one of those addresses is associated with the session. The other is ignored at best, or treated as a potential security problem at worst. A lot of this depends on how the service you are using distributes their content.

There are 9 options in the PCC drop down list. "src-address" is the only one guaranteed to maintain sessions such as ssh, tsl, ssl (https) etc... Many people use "both-addresses" and say they have no problems. Experiment at your own risk.

Your non-session http traffic will seem faster if you use one of the other 8 options, but almost everything is https these days (thanks to the paranoia about a certain TLA) and well it should be. Do you really want your buddies at work wiresharking your unencrypted voip conversation with your 'wife' and playing it back at the Christmas party? (yes this is possible, even if you don't have Christmas parties).

So what is the point of load balancing if everyday use won't be any faster?

In a single user environment, there's very little point, especially if you have a good ISP link and a high data cap. Or zero data-cap. You'd probably be better off bonding multiple connections (where possible but that's way outside our scope here).

If you have a low data cap and are worried about overages, you can use PCC rules to swap between
your links. Of course, you could also do this just by swapping cables twice a month, but what fun is that.

I have a script that runs once a day via /system scheduler that rotates my PCC rules for me to better utilize my multiple connections. Yes, this can still break sessions, but I'm not usually doing anything at 4 am when it happens (except posting rants on forums).

The real benefit of load balancing these days is in a multiple user environment. The more hosts on your network, the more even your wan utilization will be. You might not be any faster, but you won't slow down as much.

The other benefit is automatic failover. When one wan link goes down, a properly configured router
will automatically send the traffic out the next default route with the longest bit match. This will still
break sessions. Things like a vpn connection will drop and need to reconnect, but you will still be "online". Voip calls and other udp (connectionless) connections may even survive the switch if you have a script triggered by the fail, and clear any voip connections in /ip firewall connections. I have yet to test this.


One final note: The PCC rules in all these examples you find will not catch traffic originating from the router itself. DNS lookups, pinging from the console, /tool fetch; all of these will go out whatever link matches first, which will probably be the same link every time. This is uneven distribution.

Here's what I do to load balance dns queries from the router:
	add chain=postrouting connection-mark=no-mark connection-state=new dst-address-list=dns nth=2,2 \
		action=mark-connection new-connection-mark=cn_ether2 
	add  chain=postrouting connection-mark=no-mark connection-state=new dst-address-list=dns nth=2,1 \
		action=mark-connection new-connection-mark=cn_ether1 
DNS lookups are not sessions, so there is no need to maintain connections for them.

The dst-address list 'dsn' contains several known dns servers open to the public:

Google: 8.8.8.8, 8.8.4.4
Level 3: 4.2.2.x
Open DNS: 208.67.222.222, 208.67.220.220
Norton Connect Safe: 199.85.126.20, 199.85.127.20 ....

You should add your ISP's dns servers here as well if you use them.



I know this isn't a technically accurate explanation, but it isn't meant to be.
I hope this saves someone like me a lot of fruitless Googling and frustration.

If there is anything I have glaringly wrong, please feel free to contribute to the conversation.

Good luck.

(edited to correct 4am sleep deprived typos)
Last edited by LaZyLion on Sat Sep 12, 2015 8:24 am, edited 5 times in total.
 
User avatar
macgaiver
Forum Guru
Forum Guru
Posts: 1730
Joined: Wed May 18, 2005 5:57 pm
Location: Sol III, Sol system, Sector 001, Alpha Quadrant

Re: ECMP vs NTH vs PCC

Wed Sep 09, 2015 3:50 pm

Living with this setup for long time:
http://wiki.mikrotik.com/wiki/Manual:PC ... _Balancing


i haven't encounter what you stated here:
So you use "both-addresses" in your pcc rule. This means that the hash result will be different every time because
even though your src-address is always the same, your destination-address will be different.

Butttttt again, this will break your sessions because the connections, though part of the same session, will arrive at the
server farm from different ip addresses. Only one of those addresses is associated with the session.
The other is ignored at best, or treated as a potential security problem at worst.
i'm pretty sure that all services that runs many parallel connections ether connect to same dst-address, or doesn't care about src-address of each connection.
With great knowledge comes great responsibility, because of ability to recognize id... incompetent people much faster.
 
Insahnity
just joined
Posts: 1
Joined: Wed Sep 30, 2015 10:59 pm

Re: ECMP vs NTH vs PCC

Wed Sep 30, 2015 11:54 pm

LaZyLion, your post has in fact saved me much Googling and frustration, thank you.

I live in a building in a major city, but due to lack of density in my area, there is no drive for ISPs to upgrade infrastructure beyond the most basic DSL, compounded by a huge distance attenuating the DSL and no alternatives (cable, etc.). There are several high bandwidth businesses in a building all limited to about 200kbps upload and 3-4 Mbps download (at best). I have been salivating at the prospect of implementing some sort of link aggregation scheme (multiple users, multiple lines), but I have limited to no experience on the subject.

Could bonding/ECMP/PCC/<insert any other link aggregation scheme>/Magical Pink Unicorn to grant all my wishes/etc. be used to transparently choose when to aggregate links and not to aggregate links? If the user is doing a secured activity like internet banking, they could probably do it with a 14.4 modem so load balancing is not required. But if they are doing a bandwidth intensive application, like a UDP media stream from a website, I'd like it to aggregate its brains out. I think things get a bit trickier where you need a side order of security with the bandwith, such as an FTP upload. The link that macgaiver posted in his reply talks of routing interfaces but doesn't appear to pay attention to packet types.

This is also further complicated by bandwidth caps. Say one link can download 3 Mbps, and another could download at 4 Mbps, but both have the same badwidth cap. As-is, the ECMP protocol just looks at the fastest route, not how often the path is used. You could potentially overflow one cap while being under the cap in the other link.

The system doesn't have to be perfect. If a secure session breaks due to failover, that's an acceptable loss, as it isn't likely going to happen frequently (one hopes).
 
edisonch
just joined
Posts: 1
Joined: Thu Oct 01, 2015 9:38 am

Re: ECMP vs NTH vs PCC

Thu Oct 01, 2015 9:54 am

Hi Guys,

Using PCC, slow down my LAN connections a lot whenever I have to go WAN through mikrotik. I use 2 of unequal WAN by aacble.wordpress.com - and the connections are just fine but the download speed decrease tremendously.

1st WAN = 3 Mbps
2nd WAN = 40 Mbps

Using 2nd WAN, I usually download 100 Mb file in approximately a minutes or less. Now my LAN is behind mikrotik and it took about +30 minutes download. I lost quite a lot by having PCC on for load balancing in Routerboard 750.

Does it make it any different if I use more powerful routerboard? like rb 450g ?
Or is it something wrong with my script?

hardware problem or software problem?
The link has no problem to download in hurry.

thanks
 
renedr
just joined
Posts: 2
Joined: Fri Apr 10, 2015 5:14 pm

Re: ECMP vs NTH vs PCC

Tue Mar 08, 2016 2:01 pm

LaZyLion, I also recognized that only a PCC based on the source address will preserve the session. However, my experience shows that most of the times using the destination address works fine even with secure connections like online banking. I don't know about Facebook because I don't use facebook.

IN the few times I have experienced problems that may be caused by PCC, I disable one of the WAN connections using the interface menu. Although this is quite clumsy, being able to bind together the bandwidth of my two 2Mbps connections for the extra bandwidth is worth the extra effort.

BTW, to simplify router processing, I only use destination addresses because most of the time I am the only one using my connections, so including source address doesn't really add any additional hash.

Great post! Cheers.
 
User avatar
ZeroByte
Forum Guru
Forum Guru
Posts: 4051
Joined: Wed May 11, 2011 6:08 pm

Re: ECMP vs NTH vs PCC

Tue Mar 08, 2016 4:53 pm

One thing to consider when using load balancing/sharing in this scenario, any connection which gets mapped to the 3M ISP won't be able to go faster than 3M. Unfortunately, there's nothing that can be done about this - it's just how things are. Perhaps your test was being mapped to the 3M connection. Depending on your PCC mapping, you might be always getting mapped to the 3M connection based on your IP address. Open two or three different "what is my IP address" sites like myipaddress.com, ipchicken.com, and ipquail.com - see which provider's IP each comes from, and if they're all coming from one or the other. Of course, no matter what that shows, it's not necessarily going to be the same for your download test.

You could start a download and if it's going slow, look in torch and connection tracking to see which reply-address is being used - that will tell you which ISP the download is coming across - if it's the fast ISP, then there is a performance issue. Also check CPU load whenever a download is going slow.
When given a spoon,
you should not cling to your fork.
The soup will get cold.

Who is online

Users browsing this forum: ahmedit, HWTest, rado3105 and 99 guests