Community discussions

MikroTik App
 
VladOst
just joined
Topic Author
Posts: 9
Joined: Fri Dec 08, 2017 2:22 pm

Bonding LTE Interface

Sun Sep 06, 2020 10:31 am

Hi! I want to aggregate several LTE interfaces to increase the bandwidth. I expected to use the standard "Interface -> Bonding" functionality, but I can only add physical Eth, Sfp or WiFi interfaces there. Why I can't add LTE interfaces there?
 
jarda
Forum Guru
Forum Guru
Posts: 7763
Joined: Mon Oct 22, 2012 4:46 pm

Re: Bonding LTE Interface

Sun Sep 06, 2020 10:34 am

Is it for ptp link or for connection to the mobile operator?
 
VladOst
just joined
Topic Author
Posts: 9
Joined: Fri Dec 08, 2017 2:22 pm

Re: Bonding LTE Interface

Sun Sep 06, 2020 11:16 am

For traffic transmission via LTE Modems of different mobile operators.
 
User avatar
doneware
Trainer
Trainer
Posts: 596
Joined: Mon Oct 08, 2012 8:39 pm
Location: Hungary

Re: Bonding LTE Interface

Sun Sep 06, 2020 11:19 am

Why I can't add LTE interfaces there?
bonding is for ethernet interfaces, and happens at OSI Layer 2. as you might have seen, it's about sending frames in various fashion across multiple interfaces, like round robin or using some sort of hashing algorithm.
with LTE - which might look as a 'regular' ethernet interface - your device build a PDP context to the P-GW in the operator's network (which is subsequently tunnelled via GTP-U from the e-nodeB on). It is - at least as you experience - a L3 connection, so bonding/etherchannel/LACP can not work. you don't have access to the raw PDP context, but even if you would, the far end (P-GW) will not do the same multiplexing for you.

there are implementations in various LTE modems that really act like an ethernet interface, i.e you can run DHCP on it to acquire an IPv4 address, but those are in reality point-to-point virtual ethernet connections between your routerboard and the (invisible) embedded router in your modem, which will then perform NAT44. from the modem it's the same PDP context as described above.

the best thing you can do is to use ECMP across multiple PDP contexts (APNs established on lte interface). in routeros6 this requires manual ip route entries. on the other hand, with IPv4 you will be quite limited as your operator will either do NAT44 or assign 2 separate public IPv4 addresses to your sessions, so unless
- your application is MP-TCP aware
- your application uses user space networking (like QUIC)
- you build separate overlay tunnels (i.e. 2 L2TP tunnels) and set up ECMP routes on both ends
a single flow will not be able to utilise the increased bandwidth available. moreover you will definitely experience variations in latency across multiple PDP contexts, which can lead in OOS packet deliver in round robin scenarios. [don't try this at home :-]

routerOS7 handles ECMP differently, so there can be two active forwarding entries with equal metric in the routing table towards the same destination (default route in your case), and it will multiplex outgoing packets across the multiple path, based a hash of src/dst addresses and src/dst ports if available. if i remember correctly, it can be configured as well, what algorithms should be used. i have to emphasise, even if round robin operation is available here (which i seriously doubt), it will result the same OOS issues as with overlays. so totally not a good idea.
#TR0359
 
jarda
Forum Guru
Forum Guru
Posts: 7763
Joined: Mon Oct 22, 2012 4:46 pm

Re: Bonding LTE Interface

Sun Sep 06, 2020 11:19 am

Then bonding cannot work at all. Search for load balancing.
 
VladOst
just joined
Topic Author
Posts: 9
Joined: Fri Dec 08, 2017 2:22 pm

Re: Bonding LTE Interface

Sun Sep 06, 2020 1:02 pm

WOW! Really, thanks for such a detailed answer! Then I will try to explain in detail what I wanted to achieve :)
Goal: organize sufficient bandwidth and redundant (if possible) channel for conducting live broadcast to the social network (Instagram). We will leave the problems of logging into someone else's IGTV account outside the scope of the case. The stream will be standard IG media from the phone.
I planned a scheme like this:
1.png
We will skip fine settings. I understand how this can be done through the balancing and packet marking in PREROUTING. I just wanted to shift this task to the existing standard "Interface - Bonding" function.
I do not understand why Mikrotik does not consider the LTE interface to be identical to Eth or Wlan, if it is also assigned a MAC, and you can interact with it at the L2 OSI level. Obviously, on the side of telecom operators, other levels arise, perhaps this will create collisions, but on the side of Mikrotik, LTE is also an interface.
By the way, in the functional Bonding I can even include created by ovpn-out interfaces! But LTE can't ...
You do not have the required permissions to view the files attached to this post.
 
User avatar
doneware
Trainer
Trainer
Posts: 596
Joined: Mon Oct 08, 2012 8:39 pm
Location: Hungary

Re: Bonding LTE Interface  [SOLVED]

Sun Sep 06, 2020 10:25 pm

ok, let's go step by step.
1. the MAC address on the LTE interface is a bit misleading. never the less, you can't do bonding there.
2. ECMP (equal cost multipath) can work with multiple independent operators, you just need to persuade routerOS 6.x to take the multiple available paths into consideration. in routeros6 you have to have multiple gateways associated with a single route entry to achieve ECMP. something like this:
/ip route add dst-address=0.0.0.0/0 gateway=lte1,lte2,lte3
make sure this is the best path. this can be achieved by bumping up the
default-route-distance
value under
/interface lte apn
to 2 or more. after this your ECMP static route will be the best path and become active.

3. this however will not grant you superpowers. your smartphone will most probably build a single TCP or UDP session to instagram that will carry the livestream. now - as routeros will do per destination load sharing (based on the src ip/dst ip a hash will be generated) each outgoing flow will be routed through a certain gateway - in your case through one of the lte interfaces. that is a single flow (tcp or udp session) cannot achieve higher throughput than one of the connections in the ECMP route. in case of multiple outgoing/incoming flows they would be distributed (somewhat) across the 3 interfaces, but not evenly. that means you can get 2 low speed connections on one link, and 2 high speed connections on the other - the router doesn't know the speed of the connections, it just does the hash calculation and sends out the packet based on the hash value on the chosen interface.

4. yes, you can build openvpn tunnels - and i totally discourage you to do that, as openvpn's encryption is not hw accelerated and in routeros6 it runs over TCP (read tcp-over-tcp meltdown) so it's a pretty poor choice. you could just build L2TP tunnels, and achieve superior performance compared to openvpn as you don't need encryption anyways. with PPP you could essentially do ML-PPP to combine their capacity, but routerOS doesn't support that with L2TP. you will always end up with 3 different L3 interfaces and the same load balancing limitations apply.
technically you could run BCP over L2TP (see: https://wiki.mikrotik.com/wiki/Manual:BCP_bridging_(PPP_tunnel_bridging but disregard PPTP, use L2TP instead), build 3 different tunnels that end up in 3 different bridge interfaces, add a VLAN interface to each bridge and you can put all the VLAN interfaces into a bond with balance-rr algorithm.
but (BUT! BUT!!)
- you need to do it on the far end as well (another mikrotik (chr))
- Out of sequence packet delivery will eventually happen which will ruin the throughput (TCP might drop the transmit window to minimum) as the packets in your flow will take different paths and their order can change and you don't want that.
- quite a complicated setup, might work but consistent performance is not guaranteed.

if routerOS had HTTPS proxy with MP-TCP support you could benefit from the multiple links even with a single high speed flow.
#TR0359
 
VladOst
just joined
Topic Author
Posts: 9
Joined: Fri Dec 08, 2017 2:22 pm

Re: Bonding LTE Interface

Mon Sep 07, 2020 9:46 am

Great thank you! Everything is detailed and understandable!
I will try different options, if something works out, I'll write it. Perhaps it will be useful to someone too.
 
Wmillo
newbie
Posts: 27
Joined: Fri Apr 18, 2008 1:00 pm
Location: India

Re: Bonding LTE Interface

Thu Oct 15, 2020 12:50 pm

Thanks for the detailed information.
I'm not sure how MP-TCP works and I'd like to ask you one thing: when you say "if routerOS had an HTTPS proxy with MP-TCP support you could benefit from multiple connections even with a single high-speed stream" this would only be true for https traffic and with servers that support MP-TCP?
Thank you






ok, let's go step by step.
1. the MAC address on the LTE interface is a bit misleading. never the less, you can't do bonding there.
2. ECMP (equal cost multipath) can work with multiple independent operators, you just need to persuade routerOS 6.x to take the multiple available paths into consideration. in routeros6 you have to have multiple gateways associated with a single route entry to achieve ECMP. something like this:
/ip route add dst-address=0.0.0.0/0 gateway=lte1,lte2,lte3
make sure this is the best path. this can be achieved by bumping up the
default-route-distance
value under
/interface lte apn
to 2 or more. after this your ECMP static route will be the best path and become active.

3. this however will not grant you superpowers. your smartphone will most probably build a single TCP or UDP session to instagram that will carry the livestream. now - as routeros will do per destination load sharing (based on the src ip/dst ip a hash will be generated) each outgoing flow will be routed through a certain gateway - in your case through one of the lte interfaces. that is a single flow (tcp or udp session) cannot achieve higher throughput than one of the connections in the ECMP route. in case of multiple outgoing/incoming flows they would be distributed (somewhat) across the 3 interfaces, but not evenly. that means you can get 2 low speed connections on one link, and 2 high speed connections on the other - the router doesn't know the speed of the connections, it just does the hash calculation and sends out the packet based on the hash value on the chosen interface.

4. yes, you can build openvpn tunnels - and i totally discourage you to do that, as openvpn's encryption is not hw accelerated and in routeros6 it runs over TCP (read tcp-over-tcp meltdown) so it's a pretty poor choice. you could just build L2TP tunnels, and achieve superior performance compared to openvpn as you don't need encryption anyways. with PPP you could essentially do ML-PPP to combine their capacity, but routerOS doesn't support that with L2TP. you will always end up with 3 different L3 interfaces and the same load balancing limitations apply.
technically you could run BCP over L2TP (see: https://wiki.mikrotik.com/wiki/Manual:BCP_bridging_(PPP_tunnel_bridging but disregard PPTP, use L2TP instead), build 3 different tunnels that end up in 3 different bridge interfaces, add a VLAN interface to each bridge and you can put all the VLAN interfaces into a bond with balance-rr algorithm.
but (BUT! BUT!!)
- you need to do it on the far end as well (another mikrotik (chr))
- Out of sequence packet delivery will eventually happen which will ruin the throughput (TCP might drop the transmit window to minimum) as the packets in your flow will take different paths and their order can change and you don't want that.
- quite a complicated setup, might work but consistent performance is not guaranteed.

if routerOS had HTTPS proxy with MP-TCP support you could benefit from the multiple links even with a single high speed flow.
[/quote]
 
WeWiNet
Member
Member
Posts: 432
Joined: Thu Sep 27, 2018 4:11 pm

Re: Bonding LTE Interface

Thu Oct 15, 2020 5:39 pm

Wait here, taking a step back:

You use a mobile phone (= LTE device , soon maybe even 5G) and go over Wifi to an AP and to LTE modems? Sounds like not the easiest one
Why not go directly from phone into internet? (you can add VPN onto the phone)

Current Mobile phones (mid/high end smart phones) have CAT10 or more inside (up to CAt18), these are beasts in terms of achieving high throughput, low latency ...
You can even have dual sim, dual standby etc..
I don't know what LTE modems you connect to your router, but many are CAT4 (lousy), CAT 6 (so-so), and only few are higher CAT than 6.
Achieving the performance of a modern mobile phone which has 3 or more carrier aggregation is quit difficult with standard LTE modems for routers...

I would give it a try directly with the phone rather than try to do what you planned...
WeWiNet

**
MTCNA
hapac2, map, hap-lite, ltap-mini, RB4011 (good!), Audience (better), Audience + Wap R ac LTE Hybrid DSL + LTE FWA (best :-) ) !!!

Who is online

Users browsing this forum: robsgax and 45 guests