Some forums or sites are not exactly ‘load balancing friendly’ - if we balance each new connection ( for example PCC both-addr-ports /2:0 or NTH 2,1 ), those sites see user coming from ‘other’ IP, and ask for login again (and again, and again …). Luckily, most popular forum/boards like phpBB by default allow user to login simultaneously from multiple IP addresses - only hassle is that you will have to enter login twice instead of once.
I would expect that this Mikrotik forum would be ‘Load Balancing Friendly’, especially since it uses phpBB … but unfortunately, that is not the case. It frequently asks me to login again, which is especially visible if I go to “User Control Panel” - almost every click on some tab on left side requires new login. But even outside of User Control Panel, just browsing forum posts, I get frequently auto-logged out ( presumably when forum notices I changed IP address due to Load Balancing ).
I wonder if it would be possible for this forum to become ‘Load Balancing Friendly’, seeing as Mikrotik is company that sells devices which enable Load Balancing. Also, I wonder if reason it is not ‘friendly’ right now is due to phpBB or some custom Mikrotik setup ? And if it is phpBB, fixing it to be friendly and upstreaming that to phpBB original source would help majority of forums around world to become Load Balancing Friendly … which would be worthy goal for Mikrotik I presume.
Some see it as security feature. If user is logged in from address X, then another connection from address Y could be evil hacker who stole poor user’s session, so it’s safer to require new login. It doesn’t work well with certain kinds of load balancing, but question is how many people use that. I guess that some try and give up for this exact reason - you never know what website won’t like it. So you can try to convince them to change it, set exceptions if they don’t, or use something more static like per-connection-classifier=both-addresses. I’m just saying, it’s nothing against your goal.
This forum does indeed not like that you are logged in from different IP addresses. Thst is why it is on the fixed-VPN, self created, address list in my router so that only IP address is used during a session. Connection or router marking is then your friend.
My list is not that long, so most sites don’t filter on different IP source addresses.
My reasoning is that a website could be hosted on multiple servers (e.g. via multiple DNS entries) and it could be that they are surprised that the same client comes from different source IP on different TCP connections.
So on my network I only balance on src-address. Of course this means that a single user always uses the same outgoing line no matter what they do. But as we have hundreds of users the whole workload is still pretty well balanced over the lines. That may be different when you have only a small number of (active) users, of course.
It is quite probable that someone ‘see’ this as security feature, but is it not one. It will not stop evil hacker at all, if he got your credentials, since he will login again if needed. And if hacker somehow really steal your ‘session’ and not your credentials, then he must be either able to compromise your channel (in which case forcing HTTPS on site is much more helpful) or has compromised your computer, but in both cases he can steal that session again.
Well, it is not exactly better , since it defeat primary Load Balancing goal - to increase available bandwidth when you have multiple ISP providers.
Specifically, only source-address would make single PC always use single ISP, so in home environment it will minimize benefits of load balancing. In large organization, ie place where simultaneously many PCs are accessing internet, it is quite good approach. But for home where usually only single user at a time is doing some heavy downloading (and even that happens only few times per day), this approach will still limit any heavy download to one ISP.
Compare that to balancing on src-port , where even single home user will get his heavy downloads at double speed when load balancing is used.
That is excellent point - if that list is not that long, such filtering on IP source addresses is probably not default on phpBB.
I also did not experience such obvious problems (like frequent auto-logouts) on many other sites, but I started using per-connection load balancing only recently.
If you want to spread traffic easy for a loadbalancer mark it on basis of dst-src address. So you get the spreading but traffic from one source is taking always the same exit for a single destination.
Different destinations could take then other exit points and so you don’t need a list to keep.
I am using different exit points to scatter my traffic…even in connections to the same target so I need to keep a list. Also a list for sites that don’t want VPN traffic.
Interesting. If I understood your comment, you are saying that phpBB/site is checking when same browser/HTTPS session is coming from different IP, and then it auto-logout that session ?
That is questionable from security benefits point, as I mentioned in second part of previous post. Looking at usual ways to compromise session:
compromised PC, and session is stolen from process/memory - in this case, hacker can initiate his new connection from same PC ( or even inject/use same TCP connection). Checking session help: NONE
compromised network, and HTTP session is stolen from TCP stream - in this case, if hacker initiate new connection using session data, above check can help. But if hacker inject packets in existing session, it will not help. Therefore, checking session help: PARTIAL
compromised network, and HTTPS session is used - there is no known practical way to break SSL, and even if there was, hackers stealing forum credentials would be last of our worries. Therefore, checking session help: NONE
So, from my point of view, that “security measure” could be disabled without noticeable effect on security, especially if HTTPS is used. And since most security conscious sites should enforce HTTPS anyway (it appears this forum does), effects of that check on security is minimal. Then it turns to following questions:
I’m tempted to ask who even uses load balancing at home, but that would not be helpful. I agree that requiring the same IP address could be one of those things that may seem more secure at first, but don’t really add much. Instead it can be unnecessarily annoying. Not only with your setup, also when device moves between different networks (work, home, …), but it won’t be as annoying there. Anyway, good luck, you will need it, who knows how many not LB friendly websites are there.
Granted, load balancing at home is still somewhat unusual but I guess it will become more common as coverage of homes with multiple ISPs improve. But all my points about home are also valid for small business networks (SMBs), where up to dozen people are connected to internet via two decent ISP connections. In those cases, just like at home, even single ISP link is enough to cover all usual internet usage patterns ( browsing, emails, chats, voip, video streams …) , since none of them gets over several Mbs. Practically only scenario where increasing internet speed via load balancing would be visible would be occasional (rare) situations when someone is downloading something really big. And even at SMB that would happen mostly few times per day, by single user at a time ( for example loading new drivers etc), and only rarely it will be several PCs at same time. So when we use conservative per-PC balancing instead of more aggressive per-connection balancing, workers in that SMB will ( in most cases) miss opportunity to have their big downloads done faster - even if they have load balancing.
In my particular case, I have at home 300 Mbs from ISP1 and 200 Mbs download speed from ISP2. Either of them is more than enough for 99% of usual home usage ( browsing, video streams, online games, voip/chats …), but occasionally I do have those “big downloads” - mostly installing Steam games, where 30-50 GB is usual size of game these days, and frequently it needs to be installed fast if it was “what should we play now?” decision for online coop with another person. Since downloading 30-50GB over 300Mbs link can take 20+min … shortening that to just 10min is quite visible benefit of per-connection load balancing. Other “big download” cases are also rare ( chrome downloads, torrent downloads ), but they too benefit from per-connection load balancing since they all can use multiple connections for download.
Unfortunately, per-connection LB has those issues we discussed here, so this is sort of “chicken vs egg” problem : people rarely decide to use per-connection balancing (even if it would give more benefits) due to those problems on server side… and people setting those servers rarely decide to care for per-connection LB since low number of people use it. Hopefully, if enough people start asking those sites if it is possible to allow client per-connection LB on their site, eventually we can get more LB-friendly environment. One server at a time - and Mikrotik’s own forum seemed as good place to start, seeing as Mikrotik makes devices that enable load balancing in first place.
In the meantime, only sensible approach that I see here is to do more conservative per-destination balancing, and see it it is possible for only some destinations to use per-connection balancing … like for Steam, but that would work only if Steam download from their own servers and not from game developer servers ( I think Steam uses its own download servers), but also it would require same set of servers ( skipping CDNs and Steam changing from which server it downloads …). Anyway, that is something I will have to investigate.
TL/DR: When home or small business has two decent ISP links, only type of load balancing that would bring visible benefit ( and even that rarely) is per-connection balancing to speed up occasional big downloads. Anything else can run just fine over single ISP link, so per-computer balancing would rarely have visible benefits over just failover setup. But that per-connection LB has issues on some servers that do not allow same session/user from multiple IPs … and only thing to do currently is to keep asking such servers to allow it, hoping that in due time things will get better.
When you really want to use the multiple internet connections to get faster download speeds (vs the use that small businesses have, which is more focused on redundancy and uses the balancing more to spread the download amount over the different subscriptions), I would advise to setup some system of bundling.
There are many names for such technology (of course determined by the marketing folks, buzzwords like “SDN”) but the idea is that you have some externally hosted service that provides your external IP address, and to there you have two different connections (tunnels or whatever you want to call them) to transport your traffic to the local site. With such a service you can actually add the download performance of your connections for a single connection, e.g. a download.
Interesting option, but I guess it would have two downsides:
I would have to find such service close to both of my ISPs, otherwise I would add unwanted latency ( and neither of my ISPs offer such bundling as of now )
it would be additional cost
Regarding my specific issue, I just found workable solution. I investigated what I suggested in previous post, about only doing per-connection balancing for Steam downloads (since that is almost only case where I need double speed), and I found that:
steam indeed download from its own download servers, and not from game developer sites
while steam has 70+ registered network ranges, they could all be grouped in three loose groups: 146.66.x.x , 185.25.x.x , other
my own downloads always used IP from 146.66.x.x group, few connections at start, then steam adds tens of connections as time pass
for really big downloads, Steam starts downloading from another IP ( not necessarily from 146.66.x.x range), also up to tens of connections.
it does not go beyond those two IPs for real big download, although 2-3 other IPs appear for some small traffic
Therefore, if “dst address” PCC balancing is used, for really big downloads there is 50% chance that Steam will use both my ISPs anyway ( if second server in #4 has different even/odd hash from first server in #3 ). In fact, that makes Steam very “LB friendly”, since their client can use multiple server IPs, and thus even per ‘dst address’ balancing can sometimes utilize both ISPs.
But since I was not happy with just 50% chance on really big Steam downloads ( and even less on other ) to use 2nd ISP, I set my load balancing rules in this order:
*** IP/FIREWALL/MANGLE ***
mark connection for_ISP2 on prerouting/bridge using PCC “dst address” /2:0 on new connections if destination is not 146.66.x.x or local // so per-destination balancing for non-Steam traffic
mark connection for_ISP2 on prerouting/bridge using NTH 2,1 on new connections if destination is 146.66.x.x // so per-connection balancing for default Steam traffic
mark connection for_ISP2 on input/ISP2link // I only allow incoming VPN connections
mark routing to TO_ISP2 on prerouting/bridge if connection mark==for_ISP2
mark routing to TO_ISP2 on output if connection mark==for_ISP2
*** IP/ROUTES ***
default route distance 10 to ISP2 if it has route mark TO_ISP2 ( check:ping)
default route distance 20 to ISP1 ( check:ping)
default route distance 30 to ISP2
Above resulted in conservative and safe load balancing ( per destination address) to internet, with aggressive and faster load balancing for Steam downloads. I actually used address list instead of fixed 146.66.x.x , so that I can later easily add some other site from which I need fast download. For now, this “opt in” approach ( where I add sites to list if I need fast download from them) is good enough for me - sites like Mikrotik forum now work as usual, without problems, and Steam download at near 500Mbs speeds. Although I still hope internet will in general become more per-connection LB friendly
I think that home with multiple ISPs, if you do it to get higher speeds, is dead end. It doesn’t make sense in long term. If you want higher speed, get faster connection from one ISP. It will be trouble free, no problems like you have now. And even cheaper, because generally there’s starting price for some basic offer and each next step is to pay X% more for Y% speed increase, where Y is significantly higher than X. The obvious problem is when any single ISP doesn’t offer high enough speed. But that’s today, they will offer it tomorrow (ok, not literally). It’s just because there’s not enough demand yet.
Really, few hundered megabits here and there, who cares, anything over 100Mbit is perfectly fine for vast majority of users. Most people don’t even have that and it would be dream come true for them. Sure, it doesn’t allow each family member to simultaneously stream own 8K video, downloading a game can take an hour, etc. My heart bleeds for everyone who’s suffering that much. But in fact, I’m glad for people with seemingly excessive demands, because they help to push things forward.
More interesting aspect of this is multiple ISPs for failover purposes. Functioning internet is increasingly important to people, so it would make sense to have backup. One way is to have something much slower only for emergencies. But if you’re not going to use it otherwise, then everything is probably too expensive just for that. And if you do want to use it, then it won’t help you much if it’s slow. So maybe this could be the way how you end up with two roughly equal connections. Although maybe not at home, because if your single ISP is down for few hours per year, it’s just not worth it. But for small business it’s more likely.
So yes, I wish you luck with your quest, it can be useful if you succeed. But truth is, I’d much rather see some standard and easily available solution for real full redundancy, i.e. with same addresses over multiple ISPs. I mean in future with IPv6, there’s no hope for IPv4. It’s possible, there are provider-independent IP addresses, but my dream is something that small business or even home enthusiast can afford. And no, I don’t think that doing it using some cloud service is the right solution.
Why not? It is the only technically viable solution, and people already are using it (“VPN” in the everyman’s meaning of the word).
It just requires a bundling capability to be added to the service, and of course a hoster that has a lot of bandwidth available.
I don’t see why it would not be “the right solution” and how it could work without a point outside your premises where the traffic is split.
It’s an opinion, possibly wrong one. I realize that it may be the best (or only) realistic solution, but it doesn’t feel right to route everything through another party like this.
I agree that requiring the same IP address could be one of those things that may seem more secure at first, but don’t really add much. Instead it can be unnecessarily annoying. Not only with your setup, also when device moves between different networks (work, home, …), but it won’t be as annoying there. Anyway, good luck, you will need it, who knows how many not LB friendly websites are there.
But in fact, I’m glad for people with seemingly excessive demands, because they help to push things forward.