Some forums or sites are not exactly 'load balancing friendly' - if we balance each new connection ( for example PCC both-addr-ports /2:0 or NTH 2,1 ), those sites see user coming from 'other' IP, and ask for login again (and again, and again ....). Luckily, most popular forum/boards like phpBB by default allow user to login simultaneously from multiple IP addresses - only hassle is that you will have to enter login twice instead of once.

I would expect that this Mikrotik forum would be 'Load Balancing Friendly', especially since it uses phpBB ... but unfortunately, that is not the case. It frequently asks me to login again, which is especially visible if I go to "User Control Panel" - almost every click on some tab on left side requires new login. But even outside of User Control Panel, just browsing forum posts, I get frequently auto-logged out ( presumably when forum notices I changed IP address due to Load Balancing ).

I wonder if it would be possible for this forum to become 'Load Balancing Friendly', seeing as Mikrotik is company that sells devices which enable Load Balancing. Also, I wonder if reason it is not 'friendly' right now is due to phpBB or some custom Mikrotik setup ? And if it is phpBB, fixing it to be friendly and upstreaming that to phpBB original source would help majority of forums around world to become Load Balancing Friendly ... which would be worthy goal for Mikrotik I presume.

Some see it as security feature. If user is logged in from address X, then another connection from address Y could be evil hacker who stole poor user's session, so it's safer to require new login. It doesn't work well with certain kinds of load balancing, but question is how many people use that. I guess that some try and give up for this exact reason - you never know what website won't like it. So you can try to convince them to change it, set exceptions if they don't, or use something more static like per-connection-classifier=both-addresses. I'm just saying, it's nothing against your goal.

It's just a regular PHPBB setup, nothing specific has been changed. And most sites are the same, they check sessions as a security measure

This forum does indeed not like that you are logged in from different IP addresses. Thst is why it is on the fixed-VPN, self created, address list in my router so that only IP address is used during a session. Connection or router marking is then your friend.

My list is not that long, so most sites don't filter on different IP source addresses.

Some forums or sites are not exactly 'load balancing friendly' - if we balance each new connection ( for example PCC both-addr-ports /2:0 or NTH 2,1 )

That is why it is better to balance only on both-addr or even only on src-address, without including ports.

Should that not be, "both-addr or even only on dst-address" because you are going through the loadbalancer outwards?!

My reasoning is that a website could be hosted on multiple servers (e.g. via multiple DNS entries) and it could be that they are surprised that the same client comes from different source IP on different TCP connections.
So on my network I only balance on src-address. Of course this means that a single user always uses the same outgoing line no matter what they do. But as we have hundreds of users the whole workload is still pretty well balanced over the lines. That may be different when you have only a small number of (active) users, of course.

Some see it as security feature. If user is logged in from address X, then another connection from address Y could be evil hacker who stole poor user's session, so it's safer to require new login. It doesn't work well with certain kinds of load balancing, but question is how many people use that. I guess that some try and give up for this exact reason - you never know what website won't like it. So you can try to convince them to change it, set exceptions if they don't, or use something more static like per-connection-classifier=both-addresses. I'm just saying, it's nothing against your goal.

It is quite probable that someone 'see' this as security feature, but is it not one. It will not stop evil hacker at all, if he got your credentials, since he will login again if needed. And if hacker somehow really steal your 'session' and not your credentials, then he must be either able to compromise your channel (in which case forcing HTTPS on site is much more helpful) or has compromised your computer, but in both cases he can steal that session again.

That is why it is better to balance only on both-addr or even only on src-address, without including ports.

Well, it is not exactly better , since it defeat primary Load Balancing goal - to increase available bandwidth when you have multiple ISP providers.

Specifically, only source-address would make single PC always use single ISP, so in home environment it will minimize benefits of load balancing. In large organization, ie place where simultaneously many PCs are accessing internet, it is quite good approach. But for home where usually only single user at a time is doing some heavy downloading (and even that happens only few times per day), this approach will still limit any heavy download to one ISP.

Compare that to balancing on src-port , where even single home user will get his heavy downloads at double speed when load balancing is used.

This forum does indeed not like that you are logged in from different IP addresses. Thst is why it is on the fixed-VPN, self created, address list in my router so that only IP address is used during a session. Connection or router marking is then your friend.

My list is not that long, so most sites don't filter on different IP source addresses.

That is excellent point - if that list is not that long, such filtering on IP source addresses is probably not default on phpBB.
I also did not experience such obvious problems (like frequent auto-logouts) on many other sites, but I started using per-connection load balancing only recently.

This forum does indeed not like that you are logged in from different IP addresses. Thst is why it is on the fixed-VPN, self created, address list in my router so that only IP address is used during a session. Connection or router marking is then your friend.

My list is not that long, so most sites don't filter on different IP source addresses.

That is excellent point - if that list is not that long, such filtering on IP source addresses is probably not default on phpBB.
I also did not experience such obvious problems (like frequent auto-logouts) on many other sites, but I started using per-connection load balancing only recently.

If you want to spread traffic easy for a loadbalancer mark it on basis of dst-src address. So you get the spreading but traffic from one source is taking always the same exit for a single destination.
Different destinations could take then other exit points and so you don't need a list to keep.

I am using different exit points to scatter my traffic....even in connections to the same target so I need to keep a list. Also a list for sites that don't want VPN traffic.

It's just a regular PHPBB setup, nothing specific has been changed. And most sites are the same, they check sessions as a security measure

Interesting. If I understood your comment, you are saying that phpBB/site is checking when same browser/HTTPS session is coming from different IP, and then it auto-logout that session ?

That is questionable from security benefits point, as I mentioned in second part of previous post. Looking at usual ways to compromise session:
- compromised PC, and session is stolen from process/memory - in this case, hacker can initiate his new connection from same PC ( or even inject/use same TCP connection). Checking session help: NONE
- compromised network, and HTTP session is stolen from TCP stream - in this case, if hacker initiate new connection using session data, above check can help. But if hacker inject packets in existing session, it will not help. Therefore, checking session help: PARTIAL
- compromised network, and HTTPS session is used - there is no known practical way to break SSL, and even if there was, hackers stealing forum credentials would be last of our worries. Therefore, checking session help: NONE

So, from my point of view, that "security measure" could be disabled without noticeable effect on security, especially if HTTPS is used. And since most security conscious sites should enforce HTTPS anyway (it appears this forum does), effects of that check on security is minimal. Then it turns to following questions:

1) is it possible to disable that check in phpBB ?
2) would this Mikrotik forum be willing to do it?

Almost 6000 postings. Are you going to celebrate that?

Specifically, only source-address would make single PC always use single ISP, so in home environment it will minimize benefits of load balancing.

I'm tempted to ask who even uses load balancing at home, but that would not be helpful. I agree that requiring the same IP address could be one of those things that may seem more secure at first, but don't really add much. Instead it can be unnecessarily annoying. Not only with your setup, also when device moves between different networks (work, home, ...), but it won't be as annoying there. Anyway, good luck, you will need it, who knows how many not LB friendly websites are there.

I'm tempted to ask who even uses load balancing at home, but that would not be helpful. I agree that requiring the same IP address could be one of those things that may seem more secure at first, but don't really add much. Instead it can be unnecessarily annoying. Not only with your setup, also when device moves between different networks (work, home, ...), but it won't be as annoying there. Anyway, good luck, you will need it, who knows how many not LB friendly websites are there.

Granted, load balancing at home is still somewhat unusual but I guess it will become more common as coverage of homes with multiple ISPs improve. But all my points about home are also valid for small business networks (SMBs), where up to dozen people are connected to internet via two decent ISP connections. In those cases, just like at home, even single ISP link is enough to cover all usual internet usage patterns ( browsing, emails, chats, voip, video streams ...) , since none of them gets over several Mbs. Practically only scenario where increasing internet speed via load balancing would be visible would be occasional (rare) situations when someone is downloading something really big. And even at SMB that would happen mostly few times per day, by single user at a time ( for example loading new drivers etc), and only rarely it will be several PCs at same time. So when we use conservative per-PC balancing instead of more aggressive per-connection balancing, workers in that SMB will ( in most cases) miss opportunity to have their big downloads done faster - even if they have load balancing.

In my particular case, I have at home 300 Mbs from ISP1 and 200 Mbs download speed from ISP2. Either of them is more than enough for 99% of usual home usage ( browsing, video streams, online games, voip/chats ...), but occasionally I do have those "big downloads" - mostly installing Steam games, where 30-50 GB is usual size of game these days, and frequently it needs to be installed fast if it was "what should we play now?" decision for online coop with another person. Since downloading 30-50GB over 300Mbs link can take 20+min ... shortening that to just 10min is quite visible benefit of per-connection load balancing. Other "big download" cases are also rare ( chrome downloads, torrent downloads ), but they too benefit from per-connection load balancing since they all can use multiple connections for download.

Unfortunately, per-connection LB has those issues we discussed here, so this is sort of "chicken vs egg" problem : people rarely decide to use per-connection balancing (even if it would give more benefits) due to those problems on server side... and people setting those servers rarely decide to care for per-connection LB since low number of people use it. Hopefully, if enough people start asking those sites if it is possible to allow client per-connection LB on their site, eventually we can get more LB-friendly environment. One server at a time - and Mikrotik's own forum seemed as good place to start, seeing as Mikrotik makes devices that enable load balancing in first place.

In the meantime, only sensible approach that I see here is to do more conservative per-destination balancing, and see it it is possible for only some destinations to use per-connection balancing ... like for Steam, but that would work only if Steam download from their own servers and not from game developer servers ( I think Steam uses its own download servers), but also it would require same set of servers ( skipping CDNs and Steam changing from which server it downloads ...). Anyway, that is something I will have to investigate.

TL/DR: When home or small business has two decent ISP links, only type of load balancing that would bring visible benefit ( and even that rarely) is per-connection balancing to speed up occasional big downloads. Anything else can run just fine over single ISP link, so per-computer balancing would rarely have visible benefits over just failover setup. But that per-connection LB has issues on some servers that do not allow same session/user from multiple IPs ... and only thing to do currently is to keep asking such servers to allow it, hoping that in due time things will get better.

When you really want to use the multiple internet connections to get faster download speeds (vs the use that small businesses have, which is more focused on redundancy and uses the balancing more to spread the download amount over the different subscriptions), I would advise to setup some system of bundling.
There are many names for such technology (of course determined by the marketing folks, buzzwords like "SDN") but the idea is that you have some externally hosted service that provides your external IP address, and to there you have two different connections (tunnels or whatever you want to call them) to transport your traffic to the local site. With such a service you can actually add the download performance of your connections for a single connection, e.g. a download.

When you really want to use the multiple internet connections to get faster download speeds (vs the use that small businesses have, which is more focused on redundancy and uses the balancing more to spread the download amount over the different subscriptions), I would advise to setup some system of bundling.
There are many names for such technology (of course determined by the marketing folks, buzzwords like "SDN") but the idea is that you have some externally hosted service that provides your external IP address, and to there you have two different connections (tunnels or whatever you want to call them) to transport your traffic to the local site. With such a service you can actually add the download performance of your connections for a single connection, e.g. a download.

Interesting option, but I guess it would have two downsides:
- I would have to find such service close to both of my ISPs, otherwise I would add unwanted latency ( and neither of my ISPs offer such bundling as of now )
- it would be additional cost

Regarding my specific issue, I just found workable solution. I investigated what I suggested in previous post, about only doing per-connection balancing for Steam downloads (since that is almost only case where I need double speed), and I found that:
1) steam indeed download from its own download servers, and not from game developer sites
2) while steam has 70+ registered network ranges, they could all be grouped in three loose groups: 146.66.x.x , 185.25.x.x , other
3) my own downloads always used IP from 146.66.x.x group, few connections at start, then steam adds tens of connections as time pass
4) for really big downloads, Steam starts downloading from another IP ( not necessarily from 146.66.x.x range), also up to tens of connections.
5) it does not go beyond those two IPs for real big download, although 2-3 other IPs appear for some small traffic

Therefore, if "dst address" PCC balancing is used, for really big downloads there is 50% chance that Steam will use both my ISPs anyway ( if second server in #4 has different even/odd hash from first server in #3 ). In fact, that makes Steam very "LB friendly", since their client can use multiple server IPs, and thus even per 'dst address' balancing can sometimes utilize both ISPs.

But since I was not happy with just 50% chance on really big Steam downloads ( and even less on other ) to use 2nd ISP, I set my load balancing rules in this order:
*** IP/FIREWALL/MANGLE ***
- mark connection for_ISP2 on prerouting/bridge using PCC "dst address" /2:0 on new connections if destination is not 146.66.x.x or local // so per-destination balancing for non-Steam traffic
- mark connection for_ISP2 on prerouting/bridge using NTH 2,1 on new connections if destination is 146.66.x.x // so per-connection balancing for default Steam traffic
- mark connection for_ISP2 on input/ISP2link // I only allow incoming VPN connections
- mark routing to TO_ISP2 on prerouting/bridge if connection mark==for_ISP2
- mark routing to TO_ISP2 on output if connection mark==for_ISP2
*** IP/ROUTES ***
- default route distance 10 to ISP2 if it has route mark TO_ISP2 ( check:ping)
- default route distance 20 to ISP1 ( check:ping)
- default route distance 30 to ISP2

Above resulted in conservative and safe load balancing ( per destination address) to internet, with aggressive and faster load balancing for Steam downloads. I actually used address list instead of fixed 146.66.x.x , so that I can later easily add some other site from which I need fast download. For now, this "opt in" approach ( where I add sites to list if I need fast download from them) is good enough for me - sites like Mikrotik forum now work as usual, without problems, and Steam download at near 500Mbs speeds. Although I still hope internet will in general become more per-connection LB friendly

I think that home with multiple ISPs, if you do it to get higher speeds, is dead end. It doesn't make sense in long term. If you want higher speed, get faster connection from one ISP. It will be trouble free, no problems like you have now. And even cheaper, because generally there's starting price for some basic offer and each next step is to pay X% more for Y% speed increase, where Y is significantly higher than X. The obvious problem is when any single ISP doesn't offer high enough speed. But that's today, they will offer it tomorrow (ok, not literally). It's just because there's not enough demand yet.

Really, few hundered megabits here and there, who cares, anything over 100Mbit is perfectly fine for vast majority of users. Most people don't even have that and it would be dream come true for them. Sure, it doesn't allow each family member to simultaneously stream own 8K video, downloading a game can take an hour, etc. My heart bleeds for everyone who's suffering that much. But in fact, I'm glad for people with seemingly excessive demands, because they help to push things forward.

More interesting aspect of this is multiple ISPs for failover purposes. Functioning internet is increasingly important to people, so it would make sense to have backup. One way is to have something much slower only for emergencies. But if you're not going to use it otherwise, then everything is probably too expensive just for that. And if you do want to use it, then it won't help you much if it's slow. So maybe this could be the way how you end up with two roughly equal connections. Although maybe not at home, because if your single ISP is down for few hours per year, it's just not worth it. But for small business it's more likely.

So yes, I wish you luck with your quest, it can be useful if you succeed. But truth is, I'd much rather see some standard and easily available solution for real full redundancy, i.e. with same addresses over multiple ISPs. I mean in future with IPv6, there's no hope for IPv4. It's possible, there are provider-independent IP addresses, but my dream is something that small business or even home enthusiast can afford. And no, I don't think that doing it using some cloud service is the right solution.

So yes, I wish you luck with your quest, it can be useful if you succeed. But truth is, I'd much rather see some standard and easily available solution for real full redundancy, i.e. with same addresses over multiple ISPs. I mean in future with IPv6, there's no hope for IPv4. It's possible, there are provider-independent IP addresses, but my dream is something that small business or even home enthusiast can afford. And no, I don't think that doing it using some cloud service is the right solution.

Why not? It is the only technically viable solution, and people already are using it ("VPN" in the everyman's meaning of the word).
It just requires a bundling capability to be added to the service, and of course a hoster that has a lot of bandwidth available.
I don't see why it would not be "the right solution" and how it could work without a point outside your premises where the traffic is split.

It's an opinion, possibly wrong one. I realize that it may be the best (or only) realistic solution, but it doesn't feel right to route everything through another party like this.

It's an opinion, possibly wrong one. I realize that it may be the best (or only) realistic solution, but it doesn't feel right to route everything through another party like this.

EVERYTHING outside your own network is routed through another party! That is the Internet.
When you mean: I don't want it to be managed by others, it does not need to be that way. You can host your own CHR in the cloud, you can host your own server or router somewhere in a datacenter.
But most people want something they can "pull off the shelf" so a standardized cloud service or package would be what they need.
That is quite similar to routing all your traffic through some VPN provider for whatever reason. Not something I would (want to) do, but quite convenient for lots of people.

I meant that when I have full-featured connection (public IP addresses and everything) from ISP who brings it to my house, with this cloud thing I'd use it only as dumb tunnel to the other virtual ISP. It's not necessarily a problem. It will add few milliseconds of extra latency, but not much if it's close. If the goal is to combine speeds or have addresses completely independent on local ISPs, it's the way to go.

But if I actually like my local ISP and only want good backup (with right addresses, speed doesn't matter that much), it's excessive, because I'll probably only need it for few hours per year. Maybe what I really want (but I admit that I haven't thought it through completely, just thinking out loud) is my main ISP to be also VPN provider. Because if there's going to be an outage, it's more likely to be on "last mile" (wind knocks down an antenna, etc..). So I could connect using e.g. mobile ISP to my regular ISP and have my normal addresses temporarily rerouted to tunnel. But I don't see it happening, demand would be low, because it's mostly interesting if you run some servers and need same addresses to keep them accessible. For most people it's probably enough to have any backup that allows them to access internet and addresses don't matter. And lack of complete protection (even ISP's uplink can go down) doesn't help either.

It depends on what kind of ISP you currently have. At my ISP it is also possible to host a system and you could do that and put the necessary software on it.
Of course that is quite expensive. Other local companies provide virtual servers (I don't think my ISP offers that; only virtual webhosting and physical servers).
I have a virtual server at another company that runs my website and mail and it would be possible to add such a service there, but I likely would have to buy extra data budget.

I agree that it would not be very useful, but ever increasing datarates are apparently something that can be sold. I have a 100/30 VDSL connection but others have faster versions and pay for it.
And the local cable company offers 1000/50 connections. It must be attractive to some customers.

Outbound per-connection load balancing that groups source/dest connections onto the same WAN [so when client A visits site A, the first connection goes out of the least-loaded WAN - say WAN A - then all connections between client A and site A use WAN A until client A no longer has any connections to site A] is adequate for 99% of use cases. The more client devices you have, the more evenly balanced the utilisation will be.
Yes, it's nice to have a block of IPs available on all WANs so you can balance inbound.
Yes, per-packet load balancing would much more efficiently fill each pipe but you will run into problems as soon as the latencies of the various links differ too much.