Community discussions

MikroTik App
 
gotsprings
Forum Guru
Forum Guru
Topic Author
Posts: 2102
Joined: Mon May 14, 2012 9:30 pm

Same IP FAIL OVER/BONDING from multiple ISPS

Fri Dec 09, 2022 2:42 pm

One of our clients has Big Leaf as a service.

What Big Leaf does is provide a box that sits Infront of our network. Any working ISP connection is jacked in to The Big Leaf box... And one feed comes out to my router.

The Big Leaf box provides One Static Public IP to my router. This IP does not change no matter what or how many ISPs are connected.

So I am guessing the box is using something like MPTCP to send traffic to a VPS somewhere, then onto the internet.

Rip 1 or 2 of the 3 ISPs out of the box, and a zoom call still doesn't drop.

My only problem with the service is it's way too expensive for me to use more often. I have dozens of home users who would Kill for this service. Right up until I show them the price sheet.

So I set out to build my "poor man's version".

Using a small forum computer I tried to build OpenMPTCP. Gave up on that due to various issues I couldn't figure out and the lack of support.

I loaded speedify onto the computer and after messing with bypasses and a finding the "right server"...

I lost my public IP as I am behind carrier grade NAT with their service. But I have pulled the plug on my primary ISP with a wifi call going... And it didn't drop.

They offer a "private server option"... Where you get your own Public IP address from a server they set up for you. Since this server is "yours"... You don't have to share bandwidth, can do port forwarding, and shouldn't have to worry about getting your IP blocked as services see it as a VPN.

That server has a bandwidth limit of 3TB per month. Which is suspiciously close to the light sail plan... At 10 bucks a month. But I digress.

Unfortunately... The server IP they gave me... Mapped to Canada, and broke all sorts of IP Geolocated services. Making me shut off the dedicated server and go back to the NAT'd IP. Which actually saved me 90 bucks a month.

The closest server to me maps as Canada... Breaking things again. The next public server has been blacklisted as a VPN. By several services. Then I somehow found another server, that is close enough and not black listed, YET.

So if you made it this far... What are other using for this sort of application?

I really wish this was something I could handle at the Tik level. Having a computer running Ubuntu then a 3rd party software, INFRONT OF MY ROUTER, seems like a service problem waiting to happen.
 
sup5
Member
Member
Posts: 359
Joined: Sat Jul 10, 2010 12:37 am

Re: Same IP FAIL OVER/BONDING from multiple ISPS

Fri Dec 09, 2022 4:33 pm

You can handle this with Mikrotik alone, but it's quite convoluted. You'll run into plenty of issues. Most notable are MTU and MSS-Clamping issues.

I did WAN-bonding before. You need a server or at least another internet connection to connect to.

I was able to do Layer-4 hashing to distribute the load as even as possible.
Per-Packet-Round-Robin (XOR) will make VoIP-calls impossible
 
gotsprings
Forum Guru
Forum Guru
Topic Author
Posts: 2102
Joined: Mon May 14, 2012 9:30 pm

Re: Same IP FAIL OVER/BONDING from multiple ISPS

Fri Dec 09, 2022 10:16 pm

I was reading an article that makes me think Zerotier might handle this soon.
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Same IP FAIL OVER/BONDING from multiple ISPS

Sun Dec 11, 2022 8:01 pm

In my experience, all you need is a virtual server in some hosting where you can choose the country, which is a matter of $6 per month, running the Mikrotik CHR. And then create one L2TP/IPsec tunnel to that machine per each uplink ISP connected to the on-site Mikrotik, and use OSPF with BFD to take care of the failover. So the SPOF becomes the datacenter's hardware rather than the on-site uplinks. On the customer site, you can use a pair of Tiks with VRRP.

But I have seen VMs in datacenters to get disconnected from the internet for tens of minutes, too.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 19105
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Same IP FAIL OVER/BONDING from multiple ISPS

Sun Dec 11, 2022 8:35 pm

Sindy can you provide a diagram of what that would look like, I am having visualizing what you are proposing........
Doesnt have to win the engineering diagram of the year award..........


This has potential........... lets say between two families we combine our internet connections..............
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 3253
Joined: Sun May 01, 2016 7:12 pm
Location: California

Re: Same IP FAIL OVER/BONDING from multiple ISPS

Mon Dec 12, 2022 9:58 am

Ah, I too love the idea of "magic bonding" to make a fatter pipe. But the BIG thing to be aware of bonding comes at the cost of BOTH added latency and additional overhead in all cases. Certainly some approaches less than others.

So my advice is avoid bonding if you can...unless you really do have a single, long duration TCP connection that needs ever ounce of bandwidth for SINGLE TCP connection (the one case I can think of is a RTMP-based livestream). It does create single point of failure as @sindy notes; also, since all traffic goes over the internet (via some form of tunnel) to one place before the internet your may further be taking a sub-optimal path to dst-host than without bonding. In more typical traffic patterns, e.g. if you have multiple users using a web browser, you really may be better offer load balancing across the WANs (e.g. something like PCC or ECMP) locally & no need for a server anywhere to "merge the tunnels" that make up the bond. While no client user connection would ever exceed the max speed of any SINGLE WAN connection using load balancing, you do avoid the latency/overhead penalty of bonding thus likely more effectively use ALL of the bandwidth (when you consider there are actually multiple connections to load balances. All you have to do it ignore a potentially higher, but singular, speed test result that bonding MAY give you.

We've actually used Speedify VM with CHR in a couple places in past. Speedify uses UDP-based DLTS tunnels (which I believe what CAPsMAN's tunnels use internally) over each WAN to their servers (that then put the bonded traffic out on the internet). But without their dedicated server, speed isn't actually that good since I think their "shared servers" (which just VPN concentrators) get over-loaded. But it's recurring cost for their dedicated server, and more than a DIY hosted solution. Now imagine it work as a container (but never tried)...setup require putting your all your WANs on a VLAN trunk, along with another VLAN for Speedify generated bonded LAN out. That, and you need to buy a subscription to Speedify.

I do think @sindy's approach is likely best for Mikrotik – without needing any containers. Now I haven't tried it, since our LTE devices need V7 for MBIM and as far I know BFD isn't supported in V7 yet. Could be wrong here. But BFD is what gives you a quicker recovery time from a failed tunnel in the bond...so if that's missing L2TP alone is somewhat limited.

And, ZeroTier would operate pretty much same as L2TP – but you still need a node on the ZT network that acts as the "hub" where the bond go out to the internet someplace. ZeroTier multipath support makes it slightly easier since it's figuring out how to make the tunnels (e.g. paths) for you – but you don't escape the same requirement to have a single point that you use as the route out of ZT to internet. While you'd avoid some MTU issues with ZeroTier since the tunnels are >1500 MTU, you kinda relaying on ZT to distribute the load with little visibility/control on it's actually balance between paths. While I do see multiple paths being used (e.g. "/zerotier/peer print proplist=path"), hard to know how much traffic is going out each. And Mikroitk hasn't even made clear what ZT bonding policy it's using (guess is "balance-aware", but even if right, how well it works is unknown).

To me, Multipath TCP is more for clients, than a router. It's actually what apple uses for Siri (since your iPhone often has both Wi-Fi and cell, iOS takes advantage via Multipath TCP to multiplex the same request to ensure fast response). If any of your WANs are wireless, the risk is it's still TCP, so still dealing handshakes that can get lost and add delay to session. estestablishment. Not discounting it could work well, but OpenMPTCP been around a bit now and hasn't seen much traction.
 
gotsprings
Forum Guru
Forum Guru
Topic Author
Posts: 2102
Joined: Mon May 14, 2012 9:30 pm

Re: Same IP FAIL OVER/BONDING from multiple ISPS

Mon Dec 12, 2022 5:08 pm

My primary concern is redundancy.

Don't really pay attention or care about the whole "speeding things up".

What I have seen from SERVICES... they often are looking at the IP they are sending traffic too. If that IP changes... I am subject to all the issues that can follow.
1. The service usually drops.
2. Sometimes they require signing back in from the new IP.
3. Sometimes one ISP is geolocated to one place and the other... somewhere else. This really can go so far as having one ISP banned from the service.

I found an old project called Enguard... WHAT A WASTE OF BANDWIDTH! But... it looks like it would really solve some issues.

Duplicate packets go over as many links as are available. Packet that gets there first, wins. The other gets dropped.
You run at the speed of the fastest connection. And if it stops... the packets from the second are used.

But the big point... you are always the same IP to the service. Removing the whole step of making a new connection when IPs change.

When you have stood there and watched someone talking on the phone and seen the call drop as the primary went out.
They called back when WiFI calling made a IPSec connection from the new IP.
Talked for a a few words, then the primary came back.
Call dropped again.
Wait for link...
Call back.

I have not watched it first hand... but I figure zoom would do the same thing...
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 3253
Joined: Sun May 01, 2016 7:12 pm
Location: California

Re: Same IP FAIL OVER/BONDING from multiple ISPS

Mon Dec 12, 2022 7:11 pm

I'm not sure I understand here, connection tracking is sticky... In original example... a call is going on when primary goes down, call drops, & secondary/backup takes over, after reconnection - but at that point as long as the secondary also doesn't go down, the should call remain on the secondary WAN & and NOT flop back, assuming you are using PCC/ECMP, or even just simple route-distance failover.

Obviously avoiding the first call drop be ideal, esp if you have multiple WANs, so get that. So some bonding MAYBE what you want, if that use case is more important than bandwidth efficiency and latency. If you links are of the same size, same speed, same latency, then there is nothing stopping from using a L2 bond (e.g. LAG). But if all your WAN have variable speed/latency, that any bonding solution has to make choices to divide packets among the LANs, detect "dead WANs" & importantly, using protocol-dependant things like ping/flowrate/drops/latency/AI/ML/etc/etc to ideally guide the bonding protocol. BFD is the standard approach to detecting path failures and pretty quick, thus @sindy's approach.

MT's ZeroTier support doesn't let you set the bonding preferences, but as a datapoint, ZT does have a "duplicate packets across all paths" option in the non-Mikrotik builds:
active-backup: Use only one primary link at a time and failover to another designated link.
broadcast: Duplicate traffic across all available links at all times.
balance-rr: Stripe packets across multiple links (not for use with TCP.)
balance-xor: Hash flows to specific links.
balance-aware: Auto-balance flows across links.
But Mikrotik doesn't let you set this. And you'd still need to have a host with stable IP somewhere within in the ZT network (and then all the Mikrotik just use that as the upstream default route, with ZT applying the the multipath policy to get packet there). But if you goal is reliability, not sure ZT is ready for primetime here – especially if MT hasn't even said that they support multipath in ZT....
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Same IP FAIL OVER/BONDING from multiple ISPS

Tue Dec 13, 2022 8:33 am

My primary concern is redundancy.

Don't really pay attention or care about the whole "speeding things up".
Regarding "speeding things up" - you do need to speed things up as compared to what OSPF alone can give you, in order to make the failover faster than the application timeout (and human tolerance threshold) - not every user will patiently keep waiting for a call to recover if it has already been silent/frozen for 10 seconds. And that's where BFD is essential, because it is the only way to move the failover time into the sub-second area.

As for redundancy - the multiple tunnels I suggest provide redundancy of the physical connections (ISP uplinks), but not of the physical server you run the CHR at, nor of the connectivity of the datacenter. If you wanted these to be redundant, you would need your own AS and BGP to host the same public IP at two distinct data centers, and some other stuff (like connection tracking synchronisation between the two routers). On top of all that, I doubt you could get sub-second failover times in this case.

Another user here on the forum has taken the "bandwidth waste" approach - instead of using OSPFv to provide redundancy at routing level, he uses the broadcast mode of bonding, but that brings the issue of spamming the communication endpoints with duplicates of packets - bonding adds no individual IDs to the frames that would allow the receiving bond to drop the duplicates. His solution was to use Wireguard for this purpose, as it is apparently the only tunneling protocol available in RouterOS that drops transport packets with old sequence numbers. I haven't tested this myself, though.

So the whole setup would be L2TP with MLPPP (and possibly IPsec-encrypted) as a carrier layer for EoIP as a carrier layer for bonding as a carrier layer for Wireguard. MLPPP is needed to prevent the L2TP transport packets from getting fragmented, because non-first fragments of packets get often lost in the internet and you need a really high MTU on the L2TP payload side to accommodate all the overhead of the inner tunnels and still keep a decent MTU for the Wireguard payload. You could probably use BCP instead of EoIP, and bond together bridges rather than EoIP interfaces, but I've never tried that (the virtual interface of a BCP tunnel can only be made a port of a bridge, no way to make it a member of a bond directly).
 
gotsprings
Forum Guru
Forum Guru
Topic Author
Posts: 2102
Joined: Mon May 14, 2012 9:30 pm

Re: Same IP FAIL OVER/BONDING from multiple ISPS

Wed Dec 14, 2022 4:32 pm

So... found a server where my services were not getting blocked...

But do I trust this little box running Ubuntu to be my only point of failure... THAT WOULD BE A NO.

Need to build:
2 Feeds into the Tik.
2 Feeds out to the Bonder
1 feed back to the Tik Or maybe leave it in the LAN and make it a gateway...

Recursive routing to make a "bypass for the bonder". So go out A or B.

Set up a mangle and route (and I guess routing table now in RouterOS7) to let the bonder get Data from both ISPs, And not have Roll OVER.

Make 2 VLANs (Minimum) to select if you "use the hot interface" or use the bonded one.

Like I typed... this allows for one IP with no DROP as one or the other connection goes down or gets flakey.
It allows for a "direct connection" to a non bonded IP address incase a service is BLOCKED at the VPS.

(A DHCP Option that would let me pick a gateway without changing the LAN IP scope would be even better.)
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 3253
Joined: Sun May 01, 2016 7:12 pm
Location: California

Re: Same IP FAIL OVER/BONDING from multiple ISPS

Wed Dec 14, 2022 5:09 pm

Recursive routing to make a "bypass for the bonder". So go out A or B.
Recursive routes are not quick approach to failure detection.

On the "sketchy bonding host"... Beyond ports or public IP getting block, you're relying on their internet too, this might have packet loss or latency too, that negatively effect things. See packet loss see at L3 would cause TCP to slow down, thus not even be able to achieve the bonded capacity. And you haven't mention what you're doing about queue'ing, which should also be a consideration.

I'd recommend you re-read @sindy's posts, since if you want to do this, his BFD approach is best you got on a Mikrotik today. Unless your original call example is now not important... So at this point, I'm lost at what you're trying to do. What problem are you really trying to solve?
 
gotsprings
Forum Guru
Forum Guru
Topic Author
Posts: 2102
Joined: Mon May 14, 2012 9:30 pm

Re: Same IP FAIL OVER/BONDING from multiple ISPS

Wed Dec 14, 2022 8:20 pm

Ammo

My chain of thought and rambling can be hard to follow.

Looking to solve multiple issues. Trying to build a solution that "catches" most of the issues.

The Recursive route is the "1 is down... everyone go to 2".

The bonded connection maintained the call and other services that were running at the same time. When i pulled the plug out of my modem and left the bonder with my cell phone as a tethered connection.

So far... the connection beyond server has been at worst 14 ms and at best 4 ms. The public server seems to be able to handle about 200/200.

Strangely... the software saw my connection to another Tik over zerotier as a "stream" and actually upped its priority compared to my other traffic. Odd.
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 3253
Joined: Sun May 01, 2016 7:12 pm
Location: California

Re: Same IP FAIL OVER/BONDING from multiple ISPS

Wed Dec 14, 2022 9:29 pm

Strangely... the software saw my connection to another Tik over zerotier as a "stream" and actually upped its priority compared to my other traffic. Odd.
Don't argue if it works :).

Depending on your ZT instance's interfaces setting (e.g. if it's "all", it would try the WANs and bonded LAN returned from your "bonding hub"). AFAIK it doesn't mess with QoS headers, (but it's possible, during path discovery/monitoring, you'd see a small number of priority marked packets that probe the network). ZT largely uses variances in latency to determining the ZT path packets take at Layer2; and at Layer 3, the ZT interfaces should follow their assigned Mikrotik route distance & scope shown in /ip/route.
 
gotsprings
Forum Guru
Forum Guru
Topic Author
Posts: 2102
Joined: Mon May 14, 2012 9:30 pm

Re: Same IP FAIL OVER/BONDING from multiple ISPS

Fri Dec 16, 2022 6:16 pm

Tried a Lazy idea and took the feed from my network to the bonder. Then out of the bonder to a port on my switch. UNTagged that port 404 as joke. Added a VLAN tag of 404 to the port that feeds my WAP. Added an SSID with a 404 as the VLAN tag... the bonder handed out DHCP addresses on the SSID.

Who is online

Users browsing this forum: No registered users and 19 guests