Split traffic then merge

Hi, sindy!

I decided to try your above suggestion to avoid throttling and fragmentation as much as possible (and keeping the MTU as 1500):

I understand there is no such thing in MikroTik as MLPPP, and I have to do it manually. I have a lot of questions, because I think there is no material online which would cover my use-case.

  • So how shall my L2TP tunnels look like on initiator and responder side (connect to, allow fast path, use ipsec, user)?
  • I guess original responder shall be the L2TP server, and initiator shall be the L2TP client. Is this assumption correct?
  • Shall I still use “manual” IPsec tunnels for this artificial MLPPP with multiple L2TPs?
  • How can I “connect” IPsec tunnels (previously they were connected via IP address to the EoIPs) to L2TP connections? By doing the same with IPIPs?
  • What kind of mangles shall I create to distribute traffic between L2TPs? Like, defining routes, and use some IP header field (DSCP)?
  • Both IPIP and L2TP client has an option to specify IPsec secret. As I have manually created IPsec tunnels, I’m assuming I don’t need to set these, is this correct?
  • So IPsecs are built up, they will encapsulate IPIP, and with IPIP addresses I have to create L2TP connections?

Thank you very much, I’m very interested if this can help me getting a bit further :slight_smile:

This sounds like an understandable misunderstanding :slight_smile: MLPPP indeed stands for “Multi-Link PPP”, but nevertheless it can be used even on a single link. The benefit of using MLPPP on a single link is that it allows to split large payload packets into multiple transport ones that do not exceed a certain size. So instead of fragmentation of the transport packets at IP level, which can cause packet loss because non-first fragments are treated badly in some parts of the internet, the payload packets get split. So the packet rate still doubles like in case of IP level fragmentation, but other negative effects of fragmentation are suppressed.

In routerOS, a real multi-link operation is only supported for PPPoE client, which isn’t helpful for your use case.

So the first thing I would try would be to use a single L2TP/IPsec tunnel with MLPPP enabled; to enable MLPPP, you have to set, at both the client and the server, max-mtu and max-mru allowed for the transport packets to something like 1300 (it can be fine-tuned later but for most traffic a small max-mtu doesn’t make much difference as it doesn’t matter whether you split a 1500-byte payload packet into a 1000-byte one and a 500-byte one or into a 1499-byte one and a 1-byte one) and mrru to 1504.

If that is not sufficient (the per-flow bandwidth limitation is still in place), you need to distribute the transport packets of the same L2TP flow among multiple unerlying IPIP tunnels, each of them encrypted using its own IPsec tunnel, same like we did with EoIP. You can keep the multi-EoIP setup in place, add the multi-IPIP one to it (using the same transport addresses like the EoIP tunnels do), and add routing tables and mangle rules to evenly distribute the traffic between the L2TP client and the L2TP server among those IPIP tunnels. You cannot use ECMP (multiple gateways for the same route) as route caching would cause all packets to use the same IPIP tunnel.



  • [quote=“, post:101, topic:139236”]
    So how shall my L2TP tunnels look like on initiator and responder side (connect to, allow fast path, use ipsec, user)?
    [/quote]

There will be just a single L2TP tunnel even if distribution of the traffic over multiple IPIP tunnels will be used. You will have to link the L2TP client to some local address c.c.c.c at router C - client (by setting src-address=c.c.c.c), and let it connect to some local address s.s.s.s at router S - server; at router C, the routing to s.s.s.s will be controlled by the routing tables and mangle rules, and so will be the routing to c.c.c.c at router S. There is no need to assign an address to each of the IPIP tunnel interfaces, I even prefer to assign s.s.s.s and c.c.c.c to port-less bridge interfaces than to assign them to any of the IPIPs. The user and password of the /interface l2tp-client row on C must match the name and password of a /ppp secret row on S.

  • [quote=“, post:101, topic:139236”]
    I guess original responder shall be the L2TP server, and initiator shall be the L2TP client. Is this assumption correct?
    [/quote]

Technically it is not mandatory in this setup with the IPIP tunnels sandwiched between L2TP and IPsec, but I would also colocate L2TP server with IPsec responder.

  • [quote=“, post:101, topic:139236”]
    Shall I still use “manual” IPsec tunnels for this artificial MLPPP with multiple L2TPs?
    [/quote]

Yes, as said above, keep the “manual” IPsec tunnels you have, and just add IPIP tunnels in parallel to the EoIP ones (you can remove the EoIP ones in future if this approach proves better).

  • [quote=“, post:101, topic:139236”]
    How can I “connect” IPsec tunnels (previously they were connected via IP address to the EoIPs) to L2TP connections? By doing the same with IPIPs?
    [/quote]

There will be no direct relationship between the IPsec and the L2TP. The IPsec encrypts the transport packets of IPIP that flow between their transport addresses; the payload of IPIP will be the transport of L2TP, with an unrelated set of addresses (this is mandatory - c.c.c.c and s.s.s.s must not match the IPsec policies)

  • [quote=“, post:101, topic:139236”]
    What kind of mangles shall I create to distribute traffic between L2TPs? Like, defining routes, and use some IP header field (DSCP)?
    [/quote]

Just as many routing tables as there are IPsec-encrypted IPIP tunnels at each device, with a single default route each, each using another IPIP tunnel interface as gateway. And then, mangle rules in chain output will assign the corresponding routing marks to packets from c.c.c.c to s.s.s.s (at C) and from s.s.s.s to c.c.c.c (at S) using the nth match, to distribute the LT2P transport traffic evenly:
/ip route
add routing-mark=via-ipip1 gateway=ipip1
add routing-mark=via-ipip2 gateway=ipip2
add routing-mark=via-ipip3 gateway=ipip3
/ip firewall mangle
add chain=output src-address=!c.c.c.c action=accept
add chain=output nth=3,1 action=mark-routing new-routing-mark=via-ipip1
add chain=output nth=2,1 action=mark-routing new-routing-mark=via-ipip2
add chain=output action=mark-routing new-routing-mark=via-ipip3

  • [quote=“, post:101, topic:139236”]
    Both IPIP and L2TP client has an option to specify IPsec secret. As I have manually created IPsec tunnels, I’m assuming I don’t need to set these, is this correct?
    [/quote]

You even must not set these, as it would create another set of peers and policies that could interfere with the manually configured ones.

  • [quote=“, post:101, topic:139236”]
    So IPsecs are built up, they will encapsulate IPIP, and with IPIP addresses I have to create L2TP connections?
    [/quote]

As stated above, do not link the c.c.c.c/32 and s.s.s.s/32 addresses to the IPIP interfaces, use some empty bridges for that - again, it’s just for clarity reasons, technically you can attach them to any of the IPIP tunnel interfaces.

Wow, thank you, this is thorough (again)! I did even have to let my coffee work to process :slight_smile:


Yes, limitation is still in place.


I’m a bit lost on these:

  • no need to assign an address to each of the IPIP tunnel interfaces?
  • technically you can attach them to any of the IPIP tunnel interfaces?
  • IPIP tunnels shall use the very same addresses as EoIPs do, right? So EoIP tunnels will remain intact (usable), but due to routing, packets will follow a different path via IPIPs?

Thank you!

The payload packet enters a tunnel interface; the tunneling process encapsulates the payload packet into a transport one and sends it to the remote-address configured for that tunnel interface; the source address is chosen by routing if no local-address is specified, or the local-address is used no matter what the routing suggests.

But the addresses I am talking about, c.c.c.c and s.s.s.s ones, are addresses of the IPIP payload, which is the L2TP transport. An L3 tunnel (point-to-point) interface does not necessarily need an IP address attached to it - if you tell a route that its gateway is the tunnel interface, routing will send packets that match that route’s destination via that interface. On the other hand, these addresses must be assigned to some interface at their respective routers, to allow the routers to use them as their own ones. So to avoid confusion half a year later, I suggest to create dedicated bridge interfaces.

The other way round would be to attach an individual /32 address and network to each of the IPIP interfaces, and set the _gateway_s of routes to the network addresses.

And yes, EoIP tunnels will remain usable, but routing will make sure that packets between c.c.c.c and s.s.s.s will run via IPIP tunnels (unless you’d choose c.c.c.c and s.s.s.s from the same subnet and attach them to the bond interfaces using the EoIP ones).

Hi!

Thanks. I’m just started to make this happen, but already stucked:

Shall I create one “L2TP Client” Interface in both initiator and responder routers? So “L2TP Server Binding” is not necessary in either router? On my Tik, "L2TP Server Binding doesn’t have an option to set an address:

Even, for “L2TP Client” I don’t seem to have src-address, but I have Connect-to:

IPIP tunnels anyway are already setup, and running. Strangely, at C (client, initiator), IPIPs are getting 1442 as MTU, however on S (server, responder), they are having 1434. I can’t seem to be able to change it though.

Ah, I haven’t realized that “L2TP Server” and “L2TP Server Binding” are different things.

Now what you wrote, makes sense to me, I’ll try it tomorrow.

Okay, something I miss:

“You will have to link the L2TP client to some local address c.c.c.c at router C - client (by setting src-address=c.c.c.c)”

What do you mean here by “link”? I tried to set an address to L2TP Client (/ip address), but it’s an invalid interface, I can’t set its address.

Moreover, in PPP, I have both “Secrets” and “L2TP Secrets”. Shouldn’t I use “L2TP Secrets” to let this L2TP connect?

Currently I added a few test items in both C and S:

  • Virtual bridge in C: address: 100.100.103.2
  • Virtual bridge in S: address: 100.100.103.1
  • Route In C: 100.100.103.1 gw: ipip1
  • Route In S: 100.100.103.2 gw: ipip1
  • No mangles as of now
  • L2TP Client in C: connect to: 100.100.103.1

Ipip is running, but L2TP can’t connect (it stays in “Connecting…” status).

What do I miss? :confused:

At the server side, forget about L2TP server bindings and L2TP secrets, use just regular PPP secrets, the server binding will be created dynamically. You can create a static binding between the (user)name of the secret and the server binding interface name later if needed.

At the client side, if the src-address field is missing in the L2TP Client settings, you either have an outdated RouterOS or an outdated Winbox:
l2tpclient.png
In RouterOS, it has been added somewhere between 6.45 and 6.47; in Winbox, I’ve got no idea as I rarely use it.

Hi!

Yes, indeed, I was a bit out-dated (6.46.xy), now updated to latest and I have the src-address option at the client, and L2TP is up and running now :slight_smile:

Thank you!

I’ll continue with mangles, but those will be easy now.

Hm.

Why is “Actual MTU” not respecting “Max MTU”?

Because max-mtu has a different meaning if MLPPP is enabled.

Unfortunately I can’t make this work. Not even the L2TP:

If I route packets to this “running” l2tp gateway, those packets are not arriving on the other end of the L2TP.

So packet is going from C-> 100.100.103.2 → IPIP1(C) → IPIP1(S) → 100.100.103.1(S) → some resource at S, but there are no packets in S on the L2TP interface.

If I set the local and remote addresses of the secret, it is “kindof” working, however the l2tp is hanging up at every 20-30s.

So what I don’t get:

  • Can I still mark my packets I wish to push through this L2TP tunnel in “prerouting” chain with routing-mark “route-through-l2tp”?
  • Output chain happens later than preroute chain, so can I apply the output mangles to packets which already has this “route-through-l2tp” routing-mark?
  • If I set local and remote addresses of the PPP secret in S, it creates dynamic routes for 100.100.103.2 via l2tp, which is wrong, as I won’t be able to round-rubin the route through IPIPs.
  • If I don’t set these addresses, there is no connection between the 2 Tiks, they can’t even ping the other half (C can’t ping S with 100.100.103.1, and S can’t ping C with 100.100.103.2).

So somewhat I’m missing, but can’t seem to solve it.

As far as I understand:

In client router: My payload → mangle (mark routing) → marked payload’s gw is l2tp → l2tp’s dst-addr’s gw is ipip1 → IPsec → INTERNET, Is this correct?

In server router: INTERNET → IPsec → ipip1 → How this will know that the payload is l2tp?

You’ve said the /interface l2tp-client was running, haven’t you? If so, /interface l2tp-server print at the server side should show an interface as well.

You have to set the remote-address and local-address on the /ppp secret row, but you have to use yet another pair of addresses than the c.c.c.c and s.s.s.s used to establish the L2TP tunnel. And there seems to still remain some misunderstanding - it is correct that there is only a single L2TP tunnel. The round-robin distribution will be applied to the L2TP transport packets, not to the L2TP payload ones.

A packet that passes through an output chain is a packet sent by the router itself; such packets never pass through prerouting. And vice versa, packets that came from the outside do pass through prerouting but never pass through output.

But once a payload packet gets encapsulated into a transport one, the transport one is a separate packet from the point of view of the firewall.

So the actual payload packet arrives via an Ethernet interface, gets handled by prerouting, filter and postrouting, and “leaves” via L2TP. L2TP encapsulates it into a transport packet, which is sent by the router itself, so it passes through output and postrouting and “leaves” via IPIP. IPIP encapsulates it into its own transport packet, which is also sent by the router itself, passes through output and postrouting, and gets caught by the IPsec policy and encapsulated again.

Of course you can (and have to) assign routing-mark values to packets you want to send via the L2TP tunnel (unless you would want almost everything to go through there, or just a few destination subnets no matter the source).

Yes, it was up, but with some invalid (or empty) IP addresses, as I haven’t set those remote-address and local-address in the secret - my bad, I have never used L2TP with MikroTik (yet).


Thank you very much for your patience, I now understand fully this concept.

Now everything is up and running, however there is not too much of performance difference compared to Bonding/EoIPs. Transfer speed (measured with speedtest.net) almost seems decreasing as I increase the IPIPs, ie. with a single IPIP it reached ~21MBps (Note: ISP’s limitation is only activated after a few minutes of constant transfer, so this test was not yet affected), then I added 2 more IPIPs, and transfer decreased to somewhere below 10MBps (but this can be due to daylight timings: I tested in the morning with one IPIP, by that time it was 5am at S, and I tested with more IPIPs when it was already 8 or 9am at S - who knows).

However it works as it should, so every packet reaches now its destination, I don’t have any NATting or routing problem now. Thank you!

I have only two questions left: is it possible that creating more transports (IPIPs) for L2TP cause lower passthrough between 2 routers than if L2TP has only one transport IPIP? Maybe it’s due to mangles? CPU load doesn’t reach even 25% though.

It seems the best result can be achieved by using as less transport as possible (2), however they seem can’t provide the latency needed for streaming video needs.

Normally the stream I wish to watch is having a huge first impact (keyframe), then transport is relaxed for a few seconds, then another huge impact (next keyframe) on transport. However if transport is not fast enough (both in terms of bandwidth and latency), video will stall, as it’s not getting the next frame in proper time.

I have no affect on the stream itself (i.e. I can’t change it, but another codec would be much more suitable for this tunnel :frowning: ), but I have another Tik at another location, which is not behind NAT so I have an idea (question2): if due to this stream’s behavior, ISP limitation might be not an issue here; what would be the fastest, most reactive connection between two Tiks if they are:

Server (Video stream comes through) -- MT1 -- direct connection to internet -- ISP2 -- MT2 -- Client (wish to watch the stream)

No need to be secured, I wish to use this purely for video passthrough, as fast and responsive as possible.

Thank you!

I wasn’t expecting the throughput to be higher - my sole expectation from this approach was that a single stream connection would be distributed among multiple UDP sessions at the lowest transport level (which is the same for bonding of EoIP and for L2TP over IPIP), and that the payload would have a full 1500-byte MTU without IP fragments at the lowest transport level. L2TP is not famous for its speed, especially on your relatively weak hardware.


It is possible but strange. The amount of packet handling doesn’t depend on the number of IPIP channels used. And mangling is a relatively lightewight operation as compared to encapsulation of payload packets into L2TP transport ones.


Without encryption and without NAT traversal, IPIP is the simplest, and therefore fastest, encapsulation method. If you need NAT traversal, you have to use L2TP or IPsec (SSTP can do that too but for some reason it is way slower than L2TP). For IPsec, you can set enc-algorithms and auth-algorithms to null in the phase 2 proposal if you don’t care about security, so it should be as fast as a hardware-accelerated encryption would be on the same hardware.

Thank you, sindy!

It’s always a pleasure learning new things :slight_smile:

My video stream has been “kindof” solved by transcoding it on-the-fly to a better codec (h265), which reduced the bandwidth requirement, and can utilize the earlier created EoIP tunnels perfectly.

Sometimes we have to think “out-of-the-box”, to have the right solution :slight_smile:

Hi!

I’m facing two issues with this configuration currently, and I can’t debug what’s the heck. Everything was worked perfectly, but I had to reboot the “initiator” router (due to other reasons).

After initiator TIK rebooted, tunnel is not building up correctly, to be more exact:

  • IPSec tunnels in both TIKs are “established”: so I think all ports, NATs are configured correctly (TBH why they wouldn’t, I just rebooted the router)
  • IPSec tunnels in initiator has “Rx Packets” but no “Tx Packets” (i.e. all “Tx Packets” are 0)
  • IPSec tunnels in responder has “Tx Packets” but no “Rx Packets” (i.e. all “Rx Packets” are 0) → this seems that responder is OK, but initiator is not (makes sense, as I rebooted the initiator)
  • EoIP tunnels are in “RS” state in initiator, but they are in “S” state in responder (i.e. no EoIP tunnels are usable → hence tunnel is down)

IPSec in initiator also has a problem: I tried disabling all IPSec policies except one, and then I killed all the active connections.
What I’d expect after this: 1 IPSec to become “established”. However, all (even disabled) policies are coming back as “established”, with only one having “PH2 Total = 1” and “Rx bytes”.

Do you have any recommendation on what could I try? It really seems my IPSec configuration is not going to survive a reboot for some weird reason.

I sorted it out: TIK ran out of free space (3% based on “Files” in winbox). After removal of a few stuff managed to get 9% free, and rebooted, and now all tunnels are properly creating.

I’m afraid it is actually unrelated. Being out of RAM could be related (although it usually causes a reboot quite soon), being out of file space is not really likely.

What often happens is that after reboot of the initiator or some NATing router on the path between the initiator and the responder, the packets from the initiator start arriving to the responder from a different socket than before. The responder handles them because it ignores the source socket and only checks the SPI, but it sends the response packets to the previously learned socket (which also keeps the eventual tracked connection alive in its own firewall), so the response packets never reach the initiator as the NAT on its end of the path does not let them through.

But your case seems different as you’ve seen only Rx packets and no Tx ones. I’d have to see that live.

Yes I guess you are right, free space is increased by removing massive amount of configuration done recently, which might caused high RAM usage.

What was interesting though, after tunnel was up and running, router could manage these additional configuration without problems, but it was not able to properly reboot with them.

I believe RAM consumption during boot is a little higher than normally and it caused problems with building up the IPSec tunnels properly.

Although your point with different routes makes sense too, but in this particular case that was not the problem.

Anyway, I’m happy that it’s sorted out, I could even reach higher throughput (this time from initiator to responder) than before.

Thank you!