Community discussions

MikroTik App
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Split traffic then merge

Tue May 05, 2020 8:39 pm

I'd like to solve a strange ISP limitation.

Let's consider this topology:
Server -- MT1 -- ISP1 -- inland internet -- ISP2 -- MT2 -- Client
In this topology, if I run iperf3, which is a network bandwidth tester, bandwidth from server->client can easily reach the uplink bandwidth limit which is currently 25MBps.

However if the client is abroad:
Server -- MT1 -- ISP1 -- abroad internet -- ISPAbroad -- MT2 -- Client
I can only measure 5MBps from the server->client. But only if I measure with a single thread.

If I measure with parallel option of iperf3 (this way it creates multiple connections), the bandwidth can reach the uplink limits (25MBps) even through the abroad ISP.

So technically I have the bandwidth, but not on a single connection.

Is there a way to use those MT1 and MT2 to defeat this limitation?

I want to use a single TCP traffic from the server, then MT1 shall split the traffic into multiple concurrent connections, and then MT2 shall reassemble them and for the client it would seem as if traffic was never splitted.

What do you think? Is this something achievable with MikroTik? (As the bandwidth is quite low compared to processing power, I don't see any reasons why it should not be possible, just I don't know which direction should I go).

Thank you!
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Split traffic then merge

Tue May 05, 2020 11:02 pm

Try setting up a GRE tunnel between the two if both have public IP addresses. If it works, it could be that the ISP won't limit GRE per connection. If it doesn't work, the only way to split a single TCP flow among several paths is to use bonding in balance-rr mode, but bonding can only bond together L2 interfaces, so it needs EoIP tunnels. But EoIP is seen to the outside world as plain GRE, so even packets belonging to different EoIP tunnels will most likely be seen by the ISP's limiter as belonging to the same flow. However, there is the GRE ID field which Mikrotik misuses, so you can give it a try.

If it doesn't help, or if GRE doesn't pass through the ISPs at all (beware, there are issues with GRE handling in Mikrotik's firewall in current ROS versions), or because one of the Mikrotiks doesn't have a public IP on itself, you would have to establish multiple SSTP tunnels between the Tiks, and run an EoIP tunnel over each of them. This would make sure that each EoIP would be seen as a different TCP flow, but the overhead of the nested encapsulations would be terrible and TCP over TCP is also a headache source once packets get lost. Multiple L2TP tunnels won't help as the client side uses port 1701 as well so a single flow again. On the other hand, if the limitation only affects TCP connections, a single L2TP tunnel might hide the TCP from the ISP.
 
User avatar
floaty
Member
Member
Posts: 314
Joined: Sat Oct 20, 2018 1:24 am
Location: 52°08'32.34"N 14°39'05.0"E

Re: Split traffic then merge

Wed May 06, 2020 1:31 am

can only measure 5MBps from the server->client. But only if I measure with a single thread.

If I measure with parallel option of iperf3 (this way it creates multiple connections), the bandwidth can reach the uplink limits (25MBps) even through the abroad ISP.
.
little bit unclear how long you did the test ? ... maybe only the policy-engine of your fancy provider and/or your fancy country*, would have needed a litlle more time to compute you are trying to cheat ?!
.
so while iperf3 is threading ( in network terms better multiplexing ) over different ports ... I would prefer a bonding interface over multiple open-vpn tunnels :!:
.
guess iperf is using udp per default ( not 100% pos.) ... MTik open-vpn is udp-too ... guess you would need different ppp-profiles to have multiple tunnel-ends on the same ip's ??!
.
... figure it out before a warlord overthrowes your current gouvernment and you need a complete new approach 8)
.
[*recently found my own ass a banana-republican]
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Wed May 06, 2020 9:24 am

can only measure 5MBps from the server->client. But only if I measure with a single thread.

If I measure with parallel option of iperf3 (this way it creates multiple connections), the bandwidth can reach the uplink limits (25MBps) even through the abroad ISP.
.
little bit unclear how long you did the test ? ... maybe only the policy-engine of your fancy provider and/or your fancy country*, would have needed a litlle more time to compute you are trying to cheat ?!
.
so while iperf3 is threading ( in network terms better multiplexing ) over different ports ... I would prefer a bonding interface over multiple open-vpn tunnels :!:
.
guess iperf is using udp per default ( not 100% pos.) ... MTik open-vpn is udp-too ... guess you would need different ppp-profiles to have multiple tunnel-ends on the same ip's ??!
.
... figure it out before a warlord overthrowes your current gouvernment and you need a complete new approach 8)
.
[*recently found my own ass a banana-republican]
Iperf does tcp by default.
And it does not do on different ports because I manually needed to create a port forward and give this port as parameter to iperf.
So it goes on the same port but multiple "routes".

For how long? I tried 10s, 30s, 300s, and 40mins.
All measurements were the same. Ie. 1 thread 5mbps, 8thread 25mbps.
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Wed May 06, 2020 9:29 am

Try setting up a GRE tunnel between the two if both have public IP addresses. If it works, it could be that the ISP won't limit GRE per connection. If it doesn't work, the only way to split a single TCP flow among several paths is to use bonding in balance-rr mode, but bonding can only bond together L2 interfaces, so it needs EoIP tunnels. But EoIP is seen to the outside world as plain GRE, so even packets belonging to different EoIP tunnels will most likely be seen by the ISP's limiter as belonging to the same flow. However, there is the GRE ID field which Mikrotik misuses, so you can give it a try.

If it doesn't help, or if GRE doesn't pass through the ISPs at all (beware, there are issues with GRE handling in Mikrotik's firewall in current ROS versions), or because one of the Mikrotiks doesn't have a public IP on itself, you would have to establish multiple SSTP tunnels between the Tiks, and run an EoIP tunnel over each of them. This would make sure that each EoIP would be seen as a different TCP flow, but the overhead of the nested encapsulations would be terrible and TCP over TCP is also a headache source once packets get lost. Multiple L2TP tunnels won't help as the client side uses port 1701 as well so a single flow again. On the other hand, if the limitation only affects TCP connections, a single L2TP tunnel might hide the TCP from the ISP.
Neither of them have public address at the moment.
Both sits behind a nat, but I can bring both to public with dmz. However I have no idea what these 3rd party isp routers' dmzs allow.

Extra info: I already tried PPTP, but same limitation happened.
As I needed to manually forward a port for iperf measurement, I think ISP limits only single tcp connections. Multiple connections on the same tcp port can go higher.

Would this change anything?
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Split traffic then merge

Wed May 06, 2020 11:07 am

Would this change anything?
As you have tried PPTP and it was throttled, it means that GRE is throttled too (PPTP uses GRE as transport).

What I haven't realized yesterday is that with plain IPsec with exchange-mode=ike2, you can use different ports at both ends for each connection relatively easily, and it will use UDP also for transport packets rather than naked ESP, as due to the port forwarding necessary to differentiate the ports there will be NAT (well, OK, NPT) in the path. The rest of the concept remains unchanged - /interface bonding in balance-rr mode, bonding together five or six eoip tunnels, each transported using its own IPsec peer.

So I'd start with a single plain IPsec IKEv2 connection, to see whether UDP is getting throttled as well. If it is, you'd have to create multiple IPsec initiators at one end, and a single responder on the other, but at the responder side, you'd port-forward several outside UDP ports to 4500; on the initiator side, you'd set the these ports as port parameters of /ip ipsec peer rows. send-initial-contact must be set to no at all peers involved, otherwise it won't work. To make it even more stealth, you can use src-nat rules at the initiator side: chain=output dst-address=ip.of.responder.peer protocol=udp dst-port=one-of-the-remote-ports action=src-nat to-ports=some-random-port.

To make each of the EoIP tunnels use its dedicated IPsec tunnel, you have to create several local IP addresses at one device (I'd recommend the initiator to keep things a bit simpler, as the policies at initiator side will be configured manually and the responder will mirror them using a template) and link the EoIP tunnels at this end to those auxiliary addresses. The IPsec policies associated to the peers would have to tunnel between one of these addresses each at one side and the single address at the other one.
 
User avatar
floaty
Member
Member
Posts: 314
Joined: Sat Oct 20, 2018 1:24 am
Location: 52°08'32.34"N 14°39'05.0"E

Re: Split traffic then merge

Wed May 06, 2020 9:50 pm

.
sorry to interfere only for beeing so nitpicky ... I have to give the tcp-default ...
but ... mutiple "routes" ? ... nöh
.
39494
39500
39498
39496
.
root@badger:~# iperf -c  192.168.67.140 -P 4
------------------------------------------------------------
Client connecting to 192.168.67.140, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.100.78 port 39494 connected with 192.168.67.140 port 5001
[  6] local 192.168.100.78 port 39500 connected with 192.168.67.140 port 5001
[  5] local 192.168.100.78 port 39498 connected with 192.168.67.140 port 5001
[  4] local 192.168.100.78 port 39496 connected with 192.168.67.140 port 5001
[ ID] Interval       Transfer     Bandwidth
[  6]  0.0-10.0 sec   286 MBytes   240 Mbits/sec
[  5]  0.0-10.0 sec   218 MBytes   183 Mbits/sec
[  3]  0.0-10.0 sec   219 MBytes   183 Mbits/sec
[  4]  0.0-10.0 sec   395 MBytes   331 Mbits/sec
[SUM]  0.0-10.0 sec  1.09 GBytes   936 Mbits/sec
root@badger:~# 
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Wed May 06, 2020 10:07 pm

I'm not sure that I understand correctly what you are trying to mean here, sorry.
.
sorry to interfere only for beeing so nitpicky ... I have to give the tcp-default ...
but ... mutiple "routes" ? ... nöh
.
39494
39500
39498
39496
.
root@badger:~# iperf -c  192.168.67.140 -P 4
------------------------------------------------------------
Client connecting to 192.168.67.140, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.100.78 port 39494 connected with 192.168.67.140 port 5001
[  6] local 192.168.100.78 port 39500 connected with 192.168.67.140 port 5001
[  5] local 192.168.100.78 port 39498 connected with 192.168.67.140 port 5001
[  4] local 192.168.100.78 port 39496 connected with 192.168.67.140 port 5001
[ ID] Interval       Transfer     Bandwidth
[  6]  0.0-10.0 sec   286 MBytes   240 Mbits/sec
[  5]  0.0-10.0 sec   218 MBytes   183 Mbits/sec
[  3]  0.0-10.0 sec   219 MBytes   183 Mbits/sec
[  4]  0.0-10.0 sec   395 MBytes   331 Mbits/sec
[SUM]  0.0-10.0 sec  1.09 GBytes   936 Mbits/sec
root@badger:~# 
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Wed May 06, 2020 10:25 pm

Thank you! This seems a bit complicated for me from the first sight but I like MTs very much and always up for learning new things.

So, you're suggesting me to first creat an IPsec tunnel and retry the iperf tests, am I right?
Creating IPsec tunnel would be good based on this example: https://wiki.mikrotik.com/wiki/Manual:I ... sec_tunnel?

Would this change anything?
As you have tried PPTP and it was throttled, it means that GRE is throttled too (PPTP uses GRE as transport).

What I haven't realized yesterday is that with plain IPsec with exchange-mode=ike2, you can use different ports at both ends for each connection relatively easily, and it will use UDP also for transport packets rather than naked ESP, as due to the port forwarding necessary to differentiate the ports there will be NAT (well, OK, NPT) in the path. The rest of the concept remains unchanged - /interface bonding in balance-rr mode, bonding together five or six eoip tunnels, each transported using its own IPsec peer.

So I'd start with a single plain IPsec IKEv2 connection, to see whether UDP is getting throttled as well. If it is, you'd have to create multiple IPsec initiators at one end, and a single responder on the other, but at the responder side, you'd port-forward several outside UDP ports to 4500; on the initiator side, you'd set the these ports as port parameters of /ip ipsec peer rows. send-initial-contact must be set to no at all peers involved, otherwise it won't work. To make it even more stealth, you can use src-nat rules at the initiator side: chain=output dst-address=ip.of.responder.peer protocol=udp dst-port=one-of-the-remote-ports action=src-nat to-ports=some-random-port.

To make each of the EoIP tunnels use its dedicated IPsec tunnel, you have to create several local IP addresses at one device (I'd recommend the initiator to keep things a bit simpler, as the policies at initiator side will be configured manually and the responder will mirror them using a template) and link the EoIP tunnels at this end to those auxiliary addresses. The IPsec policies associated to the peers would have to tunnel between one of these addresses each at one side and the single address at the other one.
 
User avatar
floaty
Member
Member
Posts: 314
Joined: Sat Oct 20, 2018 1:24 am
Location: 52°08'32.34"N 14°39'05.0"E

Re: Split traffic then merge

Wed May 06, 2020 10:33 pm

I'm not sure that I understand correctly
.
just a side-degression in iperf ... please proceed
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Split traffic then merge

Wed May 06, 2020 10:40 pm

Creating IPsec tunnel would be good based on this example: https://wiki.mikrotik.com/wiki/Manual:I ... sec_tunnel?
Unfortunately, none of the examples matches the needs (NAT at both ends, one with port forwarding) exactly. The example you have chosen is OK as a base, but
  • use exchange-mode=ikev2 (at peers at both ends)
  • at one machine (the one which you've configured as the PPTP client, as it is probably easier for you to configure port forwarding for the other one), set the peer's address to the public IP behind which the other machine is NATed, and in the identity, set my-id=fqdn:some.name
  • at the other machine, set the peer's address to 0.0.0.0/0, and in the identity, set remote-id=some.name, and on the router between it and the internet, set port-forwarding of UDP port 4500 (unless you already use 1:1 NAT there)
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Thu May 07, 2020 1:04 pm

Thanks!

I have started to create the IPsec tunnel, and I'd appreciate if you could validate this before I run into a problems that's caused by myself:

Server side MikroTik's name: server
Server side NAT router of ISP: isp_server
Client side MikroTik's name: client
Client side NAT router of ISP: isp_client

Topology:
server - isp_server(a.b.c.d) - WAN - isp_client - client

Now I created the profiles, proposals, peers and indentities on both parties:
[server] > ip ipsec profile add dh-group=modp2048 enc-algorithm=aes-128 name ike1-site2
[server] > ip ipsec proposal add enc-algorithms=aes-128-cbc name=ike1-site2 pfs-group=modp2048
[server] > ip ipsec peer add address=0.0.0.0/0 name=ike1-site2 profile=ike1-site2 exchange-mode=ike2 send-initial-contact=no
[server] > ip ipsec identity add peer=ike1-site2 secret=thisisnotasecurepsk remote-id=fqdn:some.name

[client] > ip ipsec profile add dh-group=modp2048 enc-algorithm=aes-128 name ike1-site2
[client] > ip ipsec proposal add enc-algorithms=aes-128-cbc name=ike1-site2 pfs-group=modp2048
[client] > ip ipsec peer add address=a.b.c.d/32 name=ike1-site2 profile=ike1-site2 exchange-mode=ike2 send-initial-contact=no
[client] > ip ipsec identity add peer=ike1-site2 secret=thisisnotasecurepsk my-id=fqdn:some.name
At this point there is no tunnel created (yet). I think I also need to set the policy, am I right?
Are the above configuration correct until this point?

Note: on isp_server, UDP4500 is forwarded to server (MTik).

Thank you!
Creating IPsec tunnel would be good based on this example: https://wiki.mikrotik.com/wiki/Manual:I ... sec_tunnel?
Unfortunately, none of the examples matches the needs (NAT at both ends, one with port forwarding) exactly. The example you have chosen is OK as a base, but
  • use exchange-mode=ikev2 (at peers at both ends)
  • at one machine (the one which you've configured as the PPTP client, as it is probably easier for you to configure port forwarding for the other one), set the peer's address to the public IP behind which the other machine is NATed, and in the identity, set my-id=fqdn:some.name
  • at the other machine, set the peer's address to 0.0.0.0/0, and in the identity, set remote-id=some.name, and on the router between it and the internet, set port-forwarding of UDP port 4500 (unless you already use 1:1 NAT there)
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Split traffic then merge

Thu May 07, 2020 2:48 pm

At this point there is no tunnel created (yet). I think I also need to set the policy, am I right?
I don't remember whether peers connect if no policy refers to them, but I think they should. I forgot to tell you that you have to set passive=yes at the peer which has address=0.0.0.0/0. This may cause some error (you cannot initiate connection to 0.0.0.0/0) so the peer may be inactive.

Are the above configuration correct until this point?
Yes, but there is more to the policies.

As you've decided to set up your own proposals instead of using the default ones, you'll have to add something more at the responder (server) side:
/ip ipsec policy group add name=ike1-site2
/ip ipsec policy add group=ike1-site2 template=yes proposal=ike1-site2


On the initiator (client) side, create some policy not conflicting with any address you use for starters, something like src-address=1.2.3.4 dst-address=5.6.7.8 peer=ike1-site2 tunnel=yes proposal=. On the responder (server) side, add generate-policy=port-strict policy-template-group=ike1-site2 to the identity, so that the responder would generate a mirror policy to what the initiator suggests its own one from the template with the proper proposal.

If these policies get up at both ends, the IPsec setup as such works, and you can start setting up the auxiliary addresses for the EoIP tunnels, and policies for interconnecting these addresses. The policy above is intended only to test the IPsec without breaking anything.
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Thu May 07, 2020 6:43 pm

Okay, thank you, I'm here now:

On the server, passive=yes was set (I think that was automatical).

"On the responder (server) side, add generate-policy=port-strict policy-template-group=ike1-site2 to the identity" - OK, done this.

Moreover did these on the server:
[server] > ip ipsec policy group add name=ike1-site2
[server] > ip ipsec policy add group=ike1-site2 template=yes proposal=ike1-site2

On the client I've done this:
[client] > ip ipsec policy group add name=ike1-site2
[client] > ip ipsec policy add group=ike1-site2 src-address=1.2.3.4 dst-address=5.6.7.8 peer=ike1-site2 tunnel=yes proposal=ike1-site2
Also I did set policy-template-group=ike1-site2 on the client's identity (it was set to defaults before).

I think the tunnel is still not running (there are no active peers on either MTiks).

Those src and dst addresses needs to be in MTik's LAN range? Or they can be random (they are random now).
Server and client MTik has different subnets for their LAN. Don't really know if this is important now though.
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Thu May 07, 2020 6:46 pm

Don't I need to add accept rule for UDP4500 on input chain (server side)?
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Split traffic then merge

Thu May 07, 2020 7:16 pm

On the server, passive=yes was set (I think that was automatical).
Maybe. Mikrotik keeps improving things, I may have missed that.

On the client I've done this:
...
[client] > ip ipsec policy group add name=ike1-site2
[client] > ip ipsec policy add group=ike1-site2 src-address=1.2.3.4 dst-address=5.6.7.8 peer=ike1-site2 tunnel=yes proposal=ike1-site2
Also I did set policy-template-group=ike1-site2 on the client's identity (it was set to defaults before).
The policy groups are relevant only where policies are generated from templates, so on client side adding them does no harm but is useless.

I think the tunnel is still not running (there are no active peers on either MTiks).
Stupid question - are incoming connections to UDP port 4500 permitted in chain=input of /ip firewall filter at the server Tik?

Those src and dst addresses needs to be in MTik's LAN range? Or they can be random (they are random now).
Server and client MTik has different subnets for their LAN. Don't really know if this is important now though.
The idea was to first try with a policy which matches on addresses which don't exist in the Mikrotiks, so that it wouldn't steal any traffic by mistake (IPsec policies override even routing to connected subnets).
Since there are different LAN subnets (which is in general good), you can use a policy interconnecting these subnets if you don't need to set up any firewall rules between them; if you do, let's skip it for the moment, we'll eventually add the firewall rules first and policies next.
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Thu May 07, 2020 11:06 pm

Okay, that was the problem, accepting 4500UDP @server Tik eventually let the IPsec to stand up.

One quick question: now that I opened UDP4500, anyone knows my IP (and preshared key + sets the correct ID for identity) can create an IPsec tunnel to my MT (As long as server's identity is set for matching by remote id)?

Also, there is a new policy on the server (responder): <5.6.7.8:0->1.2.3.4:0>, it seems readonly, I guess this is good and valid and comes from the initiator.

So now if I'd create some nat rules I could access responder's lan from initiator, right?

Thanks very much. How shall I go forward? Firewall rules or EoIP tunnel(s)?
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Split traffic then merge  [SOLVED]

Thu May 07, 2020 11:28 pm

One quick question: now that I opened UDP4500, anyone knows my IP (and preshared key + sets the correct ID for identity) can create an IPsec tunnel to my MT (As long as server's identity is set for matching by remote id)?
Correct.

Also, there is a new policy on the server (responder): <5.6.7.8:0->1.2.3.4:0>, it seems readonly, I guess this is good and valid and comes from the initiator.
Correct. It proves that the complete setup works, and you can now replace this test policy at initiator by some more useful one(s).

So now if I'd create some nat rules I could access responder's lan from initiator, right?
Yes, depending on the existing setup, some exceptions from src-nat/masquerade may be necessary to make the added policy client.lan.subnet->server.lan.subnet see the packets. But you don't need this policy if you take the EoIP over IPsec way.

Thanks very much. How shall I go forward? Firewall rules or EoIP tunnel(s)?
I think the initial idea was to first check whether UDP (IPsec transport packets) is also throttled?

But it's up to you of course.
To transport the EoIP using the IPsec, I'd recommend to create, at the initiator, an /interface bridge without any member ports, attach to it a private /32 IP address outside the LAN subnets of both the initiator and the responder, and create a policy with this address as src-address and the responder's WAN IP as dst-address. A mirror policy will be added at the responder. It should not be necessary to disable and re-enable the peers to that the additional policy would come up.

Once you can see the policy up, add an action=accept protocol=gre ipsec-policy=in,ipsec rule to chain=input of /ip firewall filter at both machines, and it must be before the "drop invalid" one (there's currently a bug regarding GRE handling in firewall; EoIP uses GRE protocol). Once this is done, you can add the EoIP interfaces, with properly set local and remote addresses, at both ends (the tunnel-id must be the same at both ends). You can then attach IP addresses from the same subnet to their ends, and you should be able to use the IP address of the far end as a gateway for a route to the far end LAN subnet (symmetrically at both machines of course) so that you could test the TCP client/server connection this way, and check the speed.

The EoIP interfaces are interfaces like any other, so depending on how the firewall looks like, you may have to add permissive rules to chain=forward of /ip firewall filter at both ends to make the client-server connection possible.

If the speed is OK, that's it; but since it wasn't OK using PPTP, I assume UDP will be throttled too, so the next step would be to add a second IPsec tunnel using a different UDP port on at least one end.

Out of curiosity, can you reveal what countries are we talking about?
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Fri May 08, 2020 11:46 am

To transport the EoIP using the IPsec, I'd recommend to create, at the initiator, an /interface bridge without any member ports, attach to it a private /32 IP address outside the LAN subnets of both the initiator and the responder, and create a policy with this address as src-address and the responder's WAN IP as dst-address. A mirror policy will be added at the responder. It should not be necessary to disable and re-enable the peers to that the additional policy would come up.
What is this bridge used for? (I gave it the 10.0.0.1/32 address)

... Once this is done, you can add the EoIP interfaces, with properly set local and remote addresses, at both ends (the tunnel-id must be the same at both ends). You can then attach IP addresses from the same subnet to their ends, and you should be able to use the IP address of the far end as a gateway for a route to the far end LAN subnet (symmetrically at both machines of course) so that you could test the TCP client/server connection this way, and check the speed.
What shall be the local and remote address of EoIPs at initiator and responder?

Now my policies look like this:
initator: 10.0.0.1 -> WAN IP of responder (192.168.100.100)
responder (mirrored): WAN IP of responder (192.168.100.100) -> 10.0.0.1

I tried adding EoIPs like this:
initiator: local IP=10.0.0.1, remote IP=192.168.100.100
responder: local IP=192.168.100.100, remote IP=10.0.0.1

Then attached one IP of local subnets to both of them respectively, then tried to ping from one Tik the other, but it did not go through.
Some firewall rules might still be needed, could you please confirm those upper questions for me?

Thank you!
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Split traffic then merge

Fri May 08, 2020 1:00 pm

What is this bridge used for? (I gave it the 10.0.0.1/32 address)
Just for clarity of the configuration. It is actually not important to which particular interface a "virtual" IP address is attached, but if you attach it to a dedicated interface which is always up because it inherits nothing from any physical one, it becomes always up no matter what, plus it is clearly logically separated from the other interfaces. And an empty bridge is the only L2 virtual interface you can create and it stays up.

If you ever get to using several EoIP tunnels in parallel, you'll need one such address at initiator side for each tunnel, but all of them may be configured on the same interface.

What shall be the local and remote address of EoIPs at initiator and responder?

Now my policies look like this:
initator: 10.0.0.1 -> WAN IP of responder (192.168.100.100)
responder (mirrored): WAN IP of responder (192.168.100.100) -> 10.0.0.1
I tried adding EoIPs like this:
initiator: local IP=10.0.0.1, remote IP=192.168.100.100
responder: local IP=192.168.100.100, remote IP=10.0.0.1
This is correct.

Then attached one IP of local subnets to both of them respectively
This is wrong. Imagine the two EoIP interfaces as two Ethernet interfaces interconnected using a cable, so you have to assign to each of them an address from the same subnet so that they could ping each other without any route. A /30 is enough, but it must not collide with any other subnet used at either site (not just machine).
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Fri May 08, 2020 2:22 pm

Can I chose an address from the virtual bridge's network? (Actually it's not from its network, but like 10.0.0.10 and 10.0.0.20 for the two sides?)
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Split traffic then merge

Fri May 08, 2020 2:38 pm

The addresses attached to the bridge should be /32 ones so no network as such. And the A, B, C classes have been obsoleted quite some time ago, so use 10.0.1.1/30 at one end and 10.0.1.2/30 on the other end, again just to reduce confusion. You could use 10.0.0.10/24 and 10.0.0.20/24 and still 10.0.0.1/32 would not be considered to be within that /24 subnet (because longer prefix always beats a shorter one), but it's an unnecessary additional mental load :)
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Fri May 08, 2020 2:57 pm

The addresses attached to the bridge should be /32 ones so no network as such. And the A, B, C classes have been obsoleted quite some time ago, so use 10.0.1.1/30 at one end and 10.0.1.2/30 on the other end, again just to reduce confusion. You could use 10.0.0.10/24 and 10.0.0.20/24 and still 10.0.0.1/32 would not be considered to be within that /24 subnet (because longer prefix always beats a shorter one), but it's an unnecessary additional mental load :)
Just have did this while you wrote :)
Now I have two full independent addresses (10.0.1.0/30, so EoIPs (responder, initiator) now have 10.0.1.1 and 10.0.1.2 respectively).

Both Tiks can ping each other with 220ms roundtrip time. :) Crossing my fingers.
However client behind initiator is not able to ping responder's IP (10.0.1.1).

How can I setup the firewall to be able to test this iperf thing?
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Split traffic then merge

Fri May 08, 2020 4:04 pm

client behind initiator is not able to ping responder's IP (10.0.1.1).
I don't know how the path between the iperf client and the iperf server was done before (i.e. using port forwarding or some other type of VPN), but if both the client and the server are in the LAN subnets of the respective sites, you have to add routes at both Tiks: /ip route add dst-address=remote.lan.sub.net/mask gateway=IP.of.remote.EoIP - without that, the ping response to the client address is sent down the default gateway rather than the EoIP tunnel.

Depending on how the firewalls are set up, they may allow any forwarded connections except those coming in via WAN, or they may be more restrictive and you have to add a rule accepting whatever emerges from the EoIP tunnel. Not knowing your complete configuration I cannot suggest anything more precise.
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Fri May 08, 2020 4:58 pm

Thank you!

It was simple port fwd, but now with these new routes there is no need to forward anything, I can ping each other, and direct TCP works perfect too.

So now I had the chance to redo the iperf tests too, here are the results via EoIP/IPsec:
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-60.00  sec  19.6 MBytes  2.74 Mbits/sec  189             sender
[  4]   0.00-60.00  sec  19.1 MBytes  2.67 Mbits/sec                  receiver
During this test, the server (or responder or sender)'s CPU is ~15%. Is it normal (MIPS 600MHz CPU). Other side is 0% (ARMv7 716MHz CPU).
Do you think I will need to finetune the security options - or just leave it as this? After all, 15% might not be a big deal (as this session is not planned to run 24/7).

I also did the parallel test via EoIP/IPsec:
[SUM]   0.00-60.00  sec  53.7 MBytes  7.51 Mbits/sec  671             sender
[SUM]   0.00-60.00  sec  52.2 MBytes  7.30 Mbits/sec                  receiver
During this test, the server (or responder or sender)'s CPU is ~30%. Other side is 1%.

I believe now I would move on to the parallelization.
For that I need to create new EoIP tunnels, based on the first EoIP right?
My assumptions are:
  • I would need to create a new address for the dummybridge, like 10.0.0.2/32 (first was 10.0.0.1/32)
  • Then I would create new EoIP pair with same parameters for local and remote address as for the first one but with different Tunnel ID
  • And then I need to give them addresses, ie: 10.0.2.0/30
  • Then I'd add a new route, similar like for the first one.

Then how shall I continue with the bonding interface? Or I don't need the routes?
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Split traffic then merge

Fri May 08, 2020 6:20 pm

So now I had the chance to redo the iperf tests too, here are the results via EoIP/IPsec:
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-60.00  sec  19.6 MBytes  2.74 Mbits/sec  189             sender
[  4]   0.00-60.00  sec  19.1 MBytes  2.67 Mbits/sec                  receiver
During this test, the server (or responder or sender)'s CPU is ~15%. Is it normal (MIPS 600MHz CPU). Other side is 0% (ARMv7 716MHz CPU).
Do you think I will need to finetune the security options - or just leave it as this? After all, 15% might not be a big deal (as this session is not planned to run 24/7).
If the throughput is reduced due to all the overhead (EoIP eats a couple of bytes from the packet MTU, and so does IPsec), reducing the MTU on the EoIP interfaces will help (TCP will accommodate to smaller MTU available so each packet won't be transported using one large and one small one, reducing the packet-per-second rate to one half which helps the CPU a lot).

If the throughput is reduced due to the IPsec encryption, you can save some CPU by using null encryption if the actual client-server connection is TLS-encrypted or if you don't care.


I also did the parallel test via EoIP/IPsec:
[SUM]   0.00-60.00  sec  53.7 MBytes  7.51 Mbits/sec  671             sender
[SUM]   0.00-60.00  sec  52.2 MBytes  7.30 Mbits/sec                  receiver
During this test, the server (or responder or sender)'s CPU is ~30%. Other side is 1%.
Now wait... what did you actually want to write? You've described both tests as "running via EoIP/IPsec", are you telling me that the result depends on the number of IPsec streams you use (the first test runs a single stream, the second one several ones)? Because if so, the international link is not the root cause of the bandwidth limitation issue - via EoIP/IPsec, the ISPs cannot distinguish the multiple sessions used by iperf from one another. So after all, the speed reduction on the international link may be caused simply by the round trip delay (200 ms is quite a lot), which means that splitting the single TCP session between several virtual paths will not help.

For that I need to create new EoIP tunnels, based on the first EoIP right?
My assumptions are:
  • I would need to create a new address for the dummybridge, like 10.0.0.2/32 (first was 10.0.0.1/32)
  • Then I would create new EoIP pair with same parameters for local and remote address as for the first one but with different Tunnel ID
  • And then I need to give them addresses, ie: 10.0.2.0/30
  • Then I'd add a new route, similar like for the first one.

Then how shall I continue with the bonding interface? Or I don't need the routes?
Starting from the end, you'll insert the bonding interface between the EoIPs and the IP configuration at each end. So the IP configuration at each end will migrate to the bond interface which will have all the EoIP interfaces as slaves, with no IP configurations on any of the EoIP ones. The routes will stay as they are.

But if the ISP throttling theory is correct (which I doubt now as I've seen the test results), several EoIP tunnels are not enough, you need a dedicated IPsec tunnel for each of them. So try with just one more EoIP tunnel first.

To do that, at the initiator side, before adding the EoIP tunnel, you have to add another auxiliary IP, a peer, and a policy for that other auxiliary IP which will use that peer:
/ip ipsec peer
add copy-from=ike1-site2 port=14500 my-id=fqdn:some-other-name
/ip ipsec policy
add copy-from=[find peer=ike1-site2] src-address=the.other.auxiliary.ip
/interface eoip
add copy-from=0 local-address=the.new.auxiliary.ip


At responder side, you have to do add an identity and a dst-nat rule:
/ip ipsec identity
add copy-from=[find peer=ike1-site2] remote-id=fqdn:some-other-name
/ip firewall nat
add chain=dstnat in-interface=your-wan-interface-name protocol=udp ipsec-policy=in,none dst-port=14500 action=dst-nat to-ports=4500
/interface eoip
add copy-from=0 remote-address=the.new.auxiliary.ip
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Fri May 08, 2020 7:42 pm

If the throughput is reduced due to the IPsec encryption, you can save some CPU by using null encryption if the actual client-server connection is TLS-encrypted or if you don't care.
Yes, I did this way, as the connection is already TLS-encrypted.

Now wait... what did you actually want to write? You've described both tests as "running via EoIP/IPsec", are you telling me that the result depends on the number of IPsec streams you use (the first test runs a single stream, the second one several ones)? Because if so, the international link is not the root cause of the bandwidth limitation issue - via EoIP/IPsec, the ISPs cannot distinguish the multiple sessions used by iperf from one another. So after all, the speed reduction on the international link may be caused simply by the round trip delay (200 ms is quite a lot), which means that splitting the single TCP session between several virtual paths will not help.
Indeed it was surprised me too, but this seems to be the case. But doing multiple test results that iperf single performs around 2-2.5MBps, whereas parallel increases this to somewhere 4-4.5MBps. I know (hope) that noone can distinguish between these iperf connections in the EoIP/IPsec tunnel, but there are too many variables in the equation now, so I consider this only some measurement error and keep going forward! :)

To do that, at the initiator side, before adding the EoIP tunnel, you have to add another auxiliary IP, a peer, and a policy for that other auxiliary IP which will use that peer:
/ip ipsec peer
add copy-from=ike1-site2 port=14500 my-id=fqdn:some-other-name
This is not working for me unfortunately:
> ip ipsec peer add copy-from=ike1-site2 port=14500 name=ike1-site2-2
failure: Multiple initiator peers for the same address/dns
It seems I can't have more than one IPsec peer with the same address. Might this be some kind of license limitation?

Edit:
I checked the manual here: https://wiki.mikrotik.com/wiki/Manual:IP/IPsec#Peers:
Exchange mode is the only unique identifier between the peers, meaning that there can be multiple peer configurations with the same remote-address as long as different exchange-mode is used.
This means I won't be able to create more than one IPsec with IKEv2 to Server Tik. Or I misunderstand something here.
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Split traffic then merge

Fri May 08, 2020 9:08 pm

This is not working for me unfortunately:
> ip ipsec peer add copy-from=ike1-site2 port=14500 name=ike1-site2-2
failure: Multiple initiator peers for the same address/dns
It seems I can't have more than one IPsec peer with the same address. Might this be some kind of license limitation?

Edit:
I checked the manual here: https://wiki.mikrotik.com/wiki/Manual:IP/IPsec#Peers:
Exchange mode is the only unique identifier between the peers, meaning that there can be multiple peer configurations with the same remote-address as long as different exchange-mode is used.
This means I won't be able to create more than one IPsec with IKEv2 to Server Tik. Or I misunderstand something here.
Well, this is just one of the few "we know better than you what you want to do" features of RouterOS. Luckily it can be overcome using a simple trick:
/ip dns static
add name=peer1.hu address=actual.responder.public.ip
add name=peer2.hu address=actual.responder.public.ip


And then you just set the address of each /ip ipsec peer row to peer1.hu and peer2.hu, respectively.
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Fri May 08, 2020 10:46 pm

Well, this is just one of the few "we know better than you what you want to do" features of RouterOS.
What a nice hack! Real think-out-of-the-box solution :)

I think I've setup everything in order, but the performance is getting worse.
Things I've done:
  • Static DNS
  • New peer, policy, identity (only at initiator)
  • New EoIP interface (both side)
  • New bonding interface (both side), and put 2 EoIPs in as slaves + giving it the IP address that previously was linked to EoIPs
(14500 is fwd to 4500, but as the policies are up, this is something I've done right :) )

Status:
IPsec tunnels are up, at least I have 2 active peers on both Tiks.
(Interesting: one peer's PH2 total is 2, other one's is 1. Also one peer's TX seems counting up, other one's not.)
At initiator both EoIPs are in RS state and seems alternating the TX (as supposed, I guess).
At responder, EoIP1 is in S state, the other one is in RS. (If I disable/enable the IPsec policy, both EoIPs come up to RS, but then after a few seconds, one of them eventually stops and falls back to S state).

Performance:
Pinging foreign subnet's server results sometimes replies, sometimes timeout.
Iperf starts very slow (I mean I needed to wait for 10+ seconds to be able to connect), but then it results 1.80MBps (but I realized that only one EoIP is running, so this is not something unexpected).

Now I'm trying to figure out why only one EoIP can run.

Thank you very much!
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Sat May 09, 2020 8:44 am

In long run, during the night, did not change, one EoIP of responder is not running.

If I disable the other IPsec tunnel (which is under the EoIP that runs), the not running EoIP starts running.

Then after reenabling the IPsec tunnel does not ruins this, i.e. both EoIPs are running, but only for a short (10-20seconds), then one of them stops.

What can cause this if I have 2 established IPsec peers both at responder and initiator's side?

I didn't create new peer at the responder side, only add a new identity.
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Split traffic then merge

Sat May 09, 2020 10:03 am

I have replicated your setup on one of my international links, and everything works as expected, both EoIP tunnels are constantly up, balance-rr causes odd pings to take one EoIP and even ones take the other... and I can see on the /ip ipsec installed-sa that each EoIP tunnel is carried by its own IPsec SA. I have also configured a single responder peer with two identities, it would actually be complex to do it any other way.

The only thing to come to my mind is that you should set level=unique on the /ip ipsec policy rows (the default value is require which may not be enough, I forgot to tell you).

If that doesn't help, I'll need your config exports to check.
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Sat May 09, 2020 10:20 am

Thank you, yes, changing level to unique makes the connection rock solid, now ping works, and it seems roundrubin also.
Bandwidth is not increased too much yet with only 2 EoIPs, now I'm gonna add 1-2 more then retest, and post my updates here.
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Sat May 09, 2020 11:42 am

Now, my results:
  • There are 4 IPsec tunnels, all spawn 1 EoIP tunnel
  • EoIPs are bonded together with a bonding interface in balance-rr mode
  • Everything is up and running! 8)

Bandwith however is not so great:
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-60.00  sec  16.3 MBytes  2.27 Mbits/sec   69             sender
[  4]   0.00-60.00  sec  16.1 MBytes  2.25 Mbits/sec                  receiver

There is a good chance now that iperf was wrong. I mean I used it wrong by supplying -P 8: with 8 parallel socket connections iperf can measure the maximized throughput, but those 8 connections seem to be independent of each other. When the connections are depending on each other this throughput maximization technique is not working so great.

Root cause can be the high ping which is something we can't make any lower I guess.

Ping statistics:
  • When pinging from a remote computer the home router via public ip: average 216ms.
  • When pinging from the remote computer the home Tik via bonding/eoip-slaves/ipsec: average 220ms

Although ping is high, but I don't need low latency (don't play online games through home Tik, or live multimedia, etc).
So, technically, even when the ping is high, but the bandwidth is large, I can transfer files fast. If I have 5Mpbs link with 200ms delay vs 20Mbps link with 200ms delay I will feel the 20Mbps transfers my files faster. Am I wrong?

Putting it together, high ping (=long geographical distance) shall not decrease the bandwidth by its own. Actually, iperf -P 8 was great for this, it proved that it can create 8 independent IP connections which altogether can be transfered from HomeTik to RemoteTik with high speed.
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Split traffic then merge

Sat May 09, 2020 12:12 pm

Although ping is high, but I don't need low latency (don't play online games through home Tik, or live multimedia, etc).
So, technically, even when the ping is high, but the bandwidth is large, I can transfer files fast. If I have 5Mpbs link with 200ms delay vs 20Mbps link with 200ms delay I will feel the 20Mbps transfers my files faster. Am I wrong?
You are right, but it depends on the TCP stack behavior. The idea of TCP is that it provides guaranteed delivery of packets to the application which uses it, which requires that the recipient acknowledges the delivery of the packets to the sender. And it is implementation dependent how many packets (or rather bytes) the sender is ready to have "on the way", i. e. unacknowledged, before it stops sending and waits for acknowledge.

So see how your actual application will perform.
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Sat May 09, 2020 1:27 pm

Just have tried with the problematic software, but it does not really work OK.
It's a NAS streaming my media => latency can be high, but bandwidth is needed, now the stream plays, then stops for buffering and then play again.
At least now it's going through the tunnel :)

If this is because the application does not let more packets to be unack'd before sending out new ones, is it possible to introduce some buffering mechanism? Or that's not gonna help?
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Split traffic then merge

Sat May 09, 2020 1:40 pm

is it possible to introduce some buffering mechanism? Or that's not gonna help?
The buffering mechanism is expected to be embedded into the TCP receiving side - it should indicate a large enough buffer to the sender. Try another player, if it doesn't help, you'll have to copy the media to a local file.
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Sat May 09, 2020 1:45 pm

is it possible to introduce some buffering mechanism? Or that's not gonna help?
The buffering mechanism is expected to be embedded into the TCP receiving side - it should indicate a large enough buffer to the sender. Try another player, if it doesn't help, you'll have to copy the media to a local file.
That's indeed my last resort, I'm trying to avoid copying :)

Thank you very much!
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Sat May 09, 2020 1:52 pm

One more question:

What shall be the bottleneck of this setup? Without the tunnel, I can measure 5-6Mbps, but within the tunnel it's only 2-2.5Mbps. I'm pretty sure I can't reach the native speed because obviously there is overhead, but how shall I check which point is the weakest one? Encryption is already set to null in the proposals on both sides.
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Split traffic then merge

Sat May 09, 2020 2:12 pm

What shall be the bottleneck of this setup? Without the tunnel, I can measure 5-6Mbps, but within the tunnel it's only 2-2.5Mbps. I'm pretty sure I can't reach the native speed because obviously there is overhead, but how shall I check which point is the weakest one? Encryption is already set to null in the proposals on both sides.
People around here recently started complaining about EoIP bandwidth issues (with recent RouterOS versions) when you let the EoIP interfaces advertise larger MTU than they can actually provide without fragmenting the packets. So maybe set MTU on the EoIP tunnels to 1400. But since the whole exercise has not brought what you've expected, the question is whether it makes sense to debug this further.
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Sat May 09, 2020 2:51 pm

Do I need to have IPsec tunnels to use EoIP and bonding?

I'm thinking on utilizing the bonding somehow without complex tunneling, with just basic NAT tools as they seem faster here.
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Split traffic then merge

Sat May 09, 2020 4:18 pm

It depends on how the devices between the Mikrotiks and the internet behave. GRE has no notion of ports, so only one GRE flow per remote address can be NATed. Mikrotik doesn't use GRE ID in what they call GRE, and use it in a non-standard way in what they call EoIP. So you can have multiple EoIP tunnels between two Mikrotiks if necessary, but the behaviour of the NATs and firewalls on the path between them has to be tested.

In any case, you have to run some kind of traffic check from both ends to open the pinholes on those NAT devices, as I doubt they allow to manually configure per-IP-protocol forwarding on them. So at each end, you would configure the remote-address of the tunnel as the public one behind which the remote Mikrotik is NATed, and you have to activate the keepalive testing on them.
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Sat May 09, 2020 4:58 pm

Yes, I understand this, might try out later once get back to the router.

In the meantime I did some iperf tests and realized a new aspect:

During the day I could measure 10MBps avg bandwidth with native (without tunnels) single tcp flow.

But later, and night it drops back to even 2-3.

Did also a traceroute test, my packets go around half of the world :)

So this is easy. Usually when I have time to see my videos, the routes are busy and I dont get higher bandwidth.

I dont think there is an easy way to fix this, at least at the endpoints. It might (will) eventually "fix itself" once they put in a few more routers, or upgrade some of them.

Thank you for all your efforts! :)
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Sun May 10, 2020 10:54 am

There is still something interesting in this topic:

Now the current native speed of iperf (with public IPs and basic NAT): 10.1Mbits/sec at receiver's side. Ping is 160ms, constantly.
If I create only 1 IPsec policy and peer, and one EoIP tunnel on top of it, with correctly set routes, I got for the same measure: 3.39Mbits/sec.

The 10.1 seems an international per socket limit (it's probably 10).

However the tunnel bandwidth compared to it seems small.

IPsec profile enc: aes-128 with modp2048.
Proposal auth: null, encr alg: null.
EoIP MTU: 1400 automatically-adjusted

There is no bonding, only one pure EoIP tunnel.

Both Tiks CPU is not raising more then 15-18%, but anyway hardwares:
Initiator Tik: hAP ac^2
Responder Tik: RB2011UiAS-2HnD

I know there is overhead of IPsec of 50-57 bytes, and for EoIP it's said to be the overhead of 42bytes, that's altogether less than 100bytes.
Compared it to MTU, which is now 1400, that's around 7%, but let's consider it generous 10%.

Having a native 10Mbps link, with 10% overhead without considering any HW limitation that would be 9Mbps. Or am I wrong here in some aspects?
Or maybe my HW adds an extra 40% tradeoff to overhead?

I've read comparisons and IPsec shall be considered one of the fastest VPN solutions. Of course now I don't have HW support, but even, if both Tiks serving the tunnel by SW, should not their CPU usage go high during the TX operation?

Thank you
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Split traffic then merge

Sun May 10, 2020 11:11 am

I understand your reasoning. As said, I've seen complaints on EoIP performance recently, so what MTU have you configured on the EoIP tunnels to advertise? The point is that if you advertise a higher one than the real WAN MTU minus IPsec overhead minus EoIP's own overhead, the EoIP fragments larger packets and reassembles them without the IP stack using it to know about that, so a) the overhead becomes double and b) if those complaints are reasoned, it's probably related to some bug in the fragmentation/reassembly process.
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Sun May 10, 2020 11:26 am

Now EoIPs are set to use 1400 as MTU, all other interfaces use 1500 by default (I haven't changed).
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Split traffic then merge

Sun May 10, 2020 11:44 am

Try to lower the EoIP to 1300 and test again.
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Sun May 10, 2020 1:04 pm

Thanks, MTU is 1400 on both sides of EoIP. Now the avg bandwidth is 3.93, but sometimes it goes up to even above 8:
With 1300 it was worse than with 1400.

The current bandwidth can be changing, but in general testing almost at identical time, with 1300 and 1400, 1400 gives better results. This image is for 1400:

Image
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Split traffic then merge

Sun May 10, 2020 1:10 pm

Try adding a policy with the server and client LAN addresses on the respective ends, it will override the regular routing so you don't need to modify anything else. You'll see whether the EoIP or the IPsec is the bottleneck.
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Sun May 10, 2020 2:54 pm

Wow! There is some secret here.

I created 4 EoIPs earlier with 4 different policies.
After we realized that it won't work I disabled everything.

Today I started my measures with EoIP tunnel1 (firstly created). Measurements were disappointing.

But then I disabled EoIP tunnel1, and started to play with EoIP tunnel2 (on the second IPsec tunnel which is not on UDP4500 but on UDP14500).
And guess what:

Bitrate hyperjumped to 9Mbps, and sometimes it even goes to 10.0 which is almost the max limitation of the native connection. :shock: :shock: :shock:

Now I'm gonna retest with UDP4500.
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Sun May 10, 2020 3:52 pm

This is my latest test with 2 EoIPs in one bonding (MTUs on EoIPs are 1400 everywhere, thus IPsec shall not use the default 4500port at all):

Image

I can't describe how happy I'm now! Thank you so much, 5star solution and support!

Videos now can be played directly on TV. Amazing quality without glitches.
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Sun May 10, 2020 4:55 pm

There is a small mistake though.

After some time an idle IPsec tunnel gets destroyed.
Only one thing changes: EoIPs stop running. (Responder's IP is unchanged)

Looking at IPsec, it shows active peers as before. However the only way (or one way) to restore is disabling and reenabling the IPsec peers one by one.

Is it by design? Or I need some idle packet sender?

I can accept if IPsecs are shutting down if there is no traffic on them, but then there should be a way to signal them to reinitialize the connection if needed.

Thank you!
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Split traffic then merge

Sun May 10, 2020 5:15 pm

That's really strange.

When IPsec uses the NAT traversal mechanism, i.e. when it encapsulates the transport ESP packets into UDP ones, it automatically sends keepalive packets every 30 seconds regardless whether any "real" traffic is being sent or not, and this cannot be switched off. On top of that, regardless whether bare ESP or one encapsulated into UDP is used, the peers exchange Dead Peer Detection messages every 2 minutes by default. So at IPsec layer it should be OK.

EoIP keepalives are not sent by default and you have to configure them explicitly, so something in your firewall could prevent the EoIP from being established in client->server direction, but if that was the case, I cannot see how disabling and re-enabling of IPsec peer should affect that.
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Sun May 10, 2020 11:01 pm

Ok, I finetune this observation: I think some ISP in the middle blocks the large traffic.
I've tried to stream a 20mins video, it plays smooth until 10mins, then immediately stopped. For some time EoIPs seem to be up, but then the stop as well, however IPsec tunnels are shown as active.
If I disable and reenable both IPsec tunnels, they restart, EoIPs restart too, and video continues.
Tried with a longer video, once in about every 10 mins, the traffic seem to get banned, but then with this trick I can reset it.

Does this make any sense? If a router bans a port which creates high usage, why does it allow after 5seconds again?

What happens with the TCP flow in these parallel tunnels if one gets banned or stuck? Shall it continue with lower speed, or stop completely?
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Split traffic then merge

Sun May 10, 2020 11:16 pm

It doesn't make much sense to me - when you disable and re-enable the IPsec, the UDP ports at either end don't change, so for the ISP, it is still the same UDP stream. So unless the ISP would know it is an IPsec traffic, there is no explanation why they should re-enable the traffic once you restart it. And if they did know it was IPsec, I'd expect them to throttle it the same way like they do on port 4500, so it doesn't fit together.
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Mon May 11, 2020 9:52 am

Indeed strange for me too, I will post some more details if I find out the reason.

My uplink ISP can ban the connection without explicit closing, then when I close it manually it release and let the new one (with same port) connect.

Normally, is it wise to keep these tunnels open alltime 7/24?
Or it is preferred to open them only on request?
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Split traffic then merge

Mon May 11, 2020 10:46 am

"Opening" and "closing" normally has no meaning with UDP as such, the closest notion to "closing" is "no packet in either direction for longer than some timeout", so the ISP would have to understand the payload of the UDP, i.e. the IKEv2, to recognize an initiation of a new session.

So unless you disable the peer for more than three minutes, I can't see how the ISP could deem it a new connection without understanding that it is IPsec.

I have IPsec connection established for months, but obviously the ISPs in the countries involved are not so creative.

I would recommend you to start sniffing to file after the connection breaks, at both WAN interfaces, limited to the public IP of the remote site, to see whether the ISP blocks the connections or whether it is something inside the Mikrotiks. Sniff before restarting the peers of course. Maybe SA rekeying fails? The SA normally expires on time, but maybe there is some maximum number of bytes which triggers a rekey (it normally is with other vendors, I just haven't noticed anything related in RouterOS configuration)? Plus, as you use null encryption, I have no idea what may be broken there - there is no point in rekeying with null encryption, but maybe they trigger it anyway? Too much speculation with too few data, so sniff first.
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Split traffic then merge

Mon May 11, 2020 11:08 am

To add, my imitation of your setup (a bond of two EoIP tunnels over two SAs controlled by two IKEv2 sessions) is still transparent two days later, but I don't use null encryption and it's idling all the time, so the rekeying only takes place on timeout. And no, there is nothing configurable about max transferred bytes per SA key even in the most current stable RouterOS, 6.46.6, and I doubt there is any hardcoded value.
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Mon May 11, 2020 10:28 pm

I haven't done any sniffing yet, but here is a diagram of the behavior:
Image

The video was streamed, and where I've drew the black line there it stopped (roughly). So the player has some buffer.
At the black line when I realized it stop, I disable/enable both IPsec tunnels, and then after some time it built up again and video continued.

I also tried during a working transmission, to quickly disable/enable the IPsec tunnels, but they come up slower then the buffer time of the player (which I can not control) = video stops.

It happens too often, therefore I'm thinking on finetuning this nice solution:

How does this sound:
I create another two IPsecs with EoIPs. Basically a copy of the current two, but I don't make them slaves for the current bonding, I'd rather create a new bonding.
This way I'd have two bonding interfaces, both of them bond two-two EoIPs.

Then I'd like to use bonding1 for 2minutes, then bonding2 for 2 mins, kind of roundrubin, but time-based. Is this possible? balanced-rr can do this somehow?

I don't know if this would solve the problem, but definitely seems something worth trying out. (It can even go further, like after changing to bonding2, reset bonding1)
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Split traffic then merge

Mon May 11, 2020 10:35 pm

I prefer solutions to workarounds.

There is no need for another bond, you can make all the four EoIPs members of the same bond and enable/disable the EoIPs (synchronously at both ends) cyclically, but that's a workaround, and I'd only use it if you'd find out that the problem is outside, which I still hesitate to believe. Even if it wasn't a total blocking of the UDP stream between the two ports but just severe throttling, the probability that payload packets would be dropped and all keepalives would get through is quite low, so the IPsec should go down as well, and it doesn't.

So I would sniff on WAN in the state where EoIP stops running, as I've suggested.
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Mon May 11, 2020 10:47 pm

Understood, alright.
I'll sniff the tunnel traffic, and post it tomorrow.

Thank you!
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Tue May 12, 2020 9:23 am

I have created two sniffs (2Mb each), on both ends' WAN traffic towards other ends' public IP.

I can only start the sniffers once the connection drops (ie. Videoplayer stops), I hope that would be enough.

What shall I check in the sniff files?

Thank you!
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Split traffic then merge

Tue May 12, 2020 9:44 am

I have created two sniffs (2Mb each), on both ends' WAN traffic towards other ends' public IP.
"created" means you have actually sniffed them or just prepared a size limit of 2 MB?

I can only start the sniffers once the connection drops (ie. Videoplayer stops), I hope that would be enough.
Yes, sure, otherwise nothing interesting would fit into the files.

Just double-check you've set filter-ip-address at each end to the public IP of the remote site, and better also filter-ip-protocol to udp, because otherwise your management connection will be sniffed too and the 2 Mbytes of file site may be full of management connection packets.

What shall I check in the sniff files?
The basic comparison - whether there is something at all (as a minimum, there should be the NAT-T keepalives every 20 or 30 seconds in both directions), and whether everything that is sent from one side is seen also in the other side's capture. If there are any ESP transport packets, you can use their sequence numbers for comparison; if there are no transport packets sent at all but you can see the keepalives, it means that the issue is inside the Mikrotik; if you can see transport packets to be sent at one side but never reaching the remote one, it's the ISP.
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Tue May 12, 2020 11:54 am

"created" means you have actually sniffed them or just prepared a size limit of 2 MB?
I have actually sniffed until around 2Mb (limit was set to 4096, but it increased so slow that I stopped around 2megs).

Just double-check you've set filter-ip-address at each end to the public IP of the remote site, and better also filter-ip-protocol to udp, because otherwise your management connection will be sniffed too and the 2 Mbytes of file site may be full of management connection packets.
I've set the IPs only, direction stayed on 'any', filter operation is 'or'. But once this situation happened, there were almost no packets on these IP addresses.

Side note: after some time, EoIPs also stop, and then after some more time, all IPsec tunnels reinitialize themselves (then EoIPs come back to running state).

Based on my sniffs, there are ESP, UDPENCAP and ISAKMP packets (nothing else):
  • ESP packets are present with the same seq id on both ends => ISP is good
  • UDPENCAP packets are the NAT-keepalives, also present on both ends, in around every 10-15 seconds
  • ISAKMP is for the SA, also present on both sides, usually two packets next to each other: either "Initiator Request + Responder Response", or "Responder request + Initiator Response".
I don't really have an idea on what can be the problem. Based on our conversation here, everything seems correct. When the video stopped, I immediately went to WinBox and tried pinging the other side it was timeout from both Tiks to other. So there is indeed some problem, but now it seems not on IPsec level?

I haven't restarted IPsecs today, but in "Active peers", their uptime now is around 10 minutes. Is you IPsec tunnels' uptime going higher? (This might be no problem at all, just interesting for my eyes)
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Tue May 12, 2020 1:26 pm

New fact:

In IPsec proposal at responder PFS group was configured to modp2048 while at initiator it was set to none.
Now I set to none also at responder (sorry!).

It seems working now (at least for 35mins).

Proposal's lifetime is not being respected now, active peers' uptime can raise above 30mins (in spite of I changed the default 30min to 20min in the proposals everywhere).

Does this makes sense?
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Split traffic then merge

Tue May 12, 2020 2:57 pm

In IPsec proposal at responder PFS group was configured to modp2048 while at initiator it was set to none.
Now I set to none also at responder (sorry!).
Setting it to modp2048 at both ends would serve the same purpose, but since you use null encryption anyway, pfs-group=none at both ends causes no harm.

Proposal's lifetime is not being respected now, active peers' uptime can raise above 30mins (in spite of I changed the default 30min to 20min in the proposals everywhere).
Proposal lifetime says how long the SA can use a single key; after that, a rekey happens (which is actually a creation of a new SA and switching the traffic seamlessly from the old one to the new one), but it has no influence to IKE lifetime.

Active peer uptime can be days - the default lifetime in /ip ipsec peer profile is 1d, but it's also a matter of rekeying the IKE session, so it does not necessarily terminate and get re-initiated from scratch.
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Tue May 12, 2020 5:15 pm

Thank you.

I'd like to ask one more thing here:

To access my devices on the remote LAN, I used simple port forwards with dyndns earlier.
Now as the tunnel is up, probably faster (and some time later can also be more secure), I'd like to use it for accessing remote LAN devices.

In the initiator Tik I have added a new static DNS entry with my (responder's) dyndns name and the WAN address is the responder Tik,
thus I added a new static route with the WAN ip of the responder Tik/32, and IP of responder Tik's EoIPs bonding.

While this is working, I'm sure this is not the best approach: if the tunnel is down, I won't have access to my remote devices via portfw.
(I can create a script to disable the static DNS entry, but if clients cache the DNS it would take at least minutes for them to realize the change)

So I'm looking more for a solution which is pure firewall or route based, and can be activated/deactivated on the status change of the bonding (or IPsec).

How is this done in MikroTik's world?
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Split traffic then merge

Tue May 12, 2020 5:41 pm

It's not so much a matter of Mikrotik world, it's a general networking matter :)

Do you need the DDNS to work so that your IPsec tunnels would come up, or is the IP of the responder site static?
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Tue May 12, 2020 6:03 pm

:D

All sides are dynamic, so I need DDNS for the IPsecs too, yes.
For now I used the responder's public (not so often changing) dynamic IP.
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Split traffic then merge

Tue May 12, 2020 6:52 pm

All sides are dynamic, so I need DDNS for the IPsecs too, yes.
For now I used the responder's public (not so often changing) dynamic IP.
OK. So first, use the DDNS name of the responder as the peers' address items at the initiator if you haven't yet - that's still within the scope of the previous discussion.

Second, what would be the purpose of maintaining the "VPN-less" backup path, to manage the remote device via port-forwarding of SSH or Winbox? Leaving aside that opening access to management interfaces for the whole internet is a Bad Idea, you still need the DDNS record to resolve the name to an IP in order to get there. So if the update of the DDNS record of the initiator fails, you can still get to the initiator via the VPN; if the DDNS record update fails for the responder, you'll be unable to connect to the responder both directly and via the VPN anyway. So if something, I'd rather create a second pair of peers, where the current initiator would listen as a responder and the current responder would initiate connections, as a protection against DDNS failure at any single end. If both DDNS updates fail, you're doomed anyway.

As for your original idea, you can use an /ip firewall address list item with the DDNS name as the address field to track the IP (it generates a dynamic item with the IP which is updated at each expiry of the DNS record), and use a rule in chain=prerouting of /ip firewall mangle to assign a routing-mark to all forwarded packets towards this dst-address-list. A default route marked with this routing-mark will route everything accordingly marked through the bond; if the bond fails because both EoIP member interfaces will go down as their keepalive won't be responded, that route will become inactive, and the default route without any routing-mark will be used instead. Packets originated by Mikrotik itself will not be affected so they will take the "main default" route in any case.
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Tue May 12, 2020 10:21 pm

OK. So first, use the DDNS name of the responder as the peers' address items at the initiator if you haven't yet - that's still within the scope of the previous discussion.
Yes, I'm using this, but with caution. Whenever you write DNS names in these fields, it's get resolved only once, and then when your IP under your DNS changes your configuration will fail.
So I've created a static DNS instead (anyway I needed more peers with the same IP address) which get update regularly by a ROS script. This might have changed in some recent ROS update, I just updated ROS when we started the work with IPsec.

Second, what would be the purpose of maintaining the "VPN-less" backup path, to manage the remote device via port-forwarding of SSH or Winbox? Leaving aside that opening access to management interfaces for the whole internet is a Bad Idea, you still need the DDNS record to resolve the name to an IP in order to get there. So if the update of the DDNS record of the initiator fails, you can still get to the initiator via the VPN; if the DDNS record update fails for the responder, you'll be unable to connect to the responder both directly and via the VPN anyway. So if something, I'd rather create a second pair of peers, where the current initiator would listen as a responder and the current responder would initiate connections, as a protection against DDNS failure at any single end. If both DDNS updates fail, you're doomed anyway.
That's a correct reasoning, I accept that. Maintaining the old port forward rules are staying for accessing my devices even from pure 4G connection via phone if needed. Of course this is not covering a DDNS failure, but I trust in that solution very much. I have backup anyway if DDNS fails which never happened during 10 years. :)

As for your original idea, you can use an /ip firewall address list item with the DDNS name as the address field to track the IP (it generates a dynamic item with the IP which is updated at each expiry of the DNS record), and use a rule in chain=prerouting of /ip firewall mangle to assign a routing-mark to all forwarded packets towards this dst-address-list. A default route marked with this routing-mark will route everything accordingly marked through the bond; if the bond fails because both EoIP member interfaces will go down as their keepalive won't be responded, that route will become inactive, and the default route without any routing-mark will be used instead. Packets originated by Mikrotik itself will not be affected so they will take the "main default" route in any case.
ROS updating firewall's address list dns items is a new thing, in previous versions I needed a script doing that. Cool this is now implemented! :)
I did know the mangle thing too, but I just felt it more a workaround than a solution. So mangle is really a solution for handling multi-routes?

Thank you!
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Split traffic then merge

Wed May 13, 2020 12:18 am

Whenever you write DNS names in these fields, it's get resolved only once, and then when your IP under your DNS changes your configuration will fail.
That's has not been true since long ago (if it ever was). Each DNS answer contains a lifetime of the information, and once it expires, a new query is sent and the IP number is updated accordingly if the answer contains a different address than the previous one did (which spawns a reinitialization of an IPsec connection).

ROS updating firewall's address list dns items is a new thing, in previous versions I needed a script doing that. Cool this is now implemented! :)
I've been using this for at least two years, no idea when it has been actually introduced.

I did know the mangle thing too, but I just felt it more a workaround than a solution. So mangle is really a solution for handling multi-routes?
You can choose a routing table (assign a routing-mark to a packet) using /ip route rule, mangle rules, or vrf. Each method has its pros and cons. Mangle rules are the most flexible and most CPU intensive way. You only need dedicated routing tables for cases when some particular traffic to a given destination IP address needs to be sent via some other route than the other traffic to the same address. Here, the only reason to use mangle is that the destination IP address is dynamic; the other way would be to update the route's dst-address by a script watching for the DDNS changes, but that requires flash writes as you change the configuration. Translating a match to dynamically updated dst-address-list into a routing-mark by means of a mangle rule doesn't cause any configuration change so no flash writes.
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Wed May 13, 2020 10:33 pm

Perfect, it works very nice, thank you!

I've realized one thing: players are different. There are players which buffers, i.e it uses the maximal uplink at the beginning, and there are players which doesn't buffer.

Now I'm focusing the latter, which does not buffer. Now my connection seems more solid than few days ago, I think it also works pretty much without the tunnel.
But as we built this tunnel I'd like to use it.

So here is the thing:
I've created a mangle on preroute, then a route with the mark points to the tunnel.
If I watch video without the tunnel (ie.: I disable the mangle marking), it pretty much works. Sometimes stops, but overall good.
However if I route the traffic through the tunnel, the video hardly starts, then stop and it takes a lot of time until it can continue:

Image

Now I know this is mostly application based problem, but I'm very much interested, on why it can play over internet and not over the tunnel.
And I'd like to know which things would you check and test in my situation? I'd like to use the tunnel, but if it can not serve the unbuffered video, I won't be able to use this for videos.

Thank you!
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Split traffic then merge

Wed May 13, 2020 10:39 pm

Do you have an action=fasttrack-connection rule in chain=forward of your /ip firewall filter? If yes, disable it and try again. Mangling is skipped by vast majority of packets belonging to fasttracked connections.
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Wed May 13, 2020 11:00 pm

OMG! This was it. :?
After disabling the fasttrack, it works perfectly.

I should leave fasttrack disabled when I use the tunnel? Or filter more for dst addresses that are not my home ip?
What is the best solution for keeping fasttrack and having mangle too?

Thank you so much!
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Split traffic then merge

Wed May 13, 2020 11:17 pm

What is the best solution for keeping fasttrack and having mangle too?
viewtopic.php?t=134048#p659676
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Thu May 14, 2020 9:17 am

Thank you sindy!

This is another great post!

Just one quick question left here:
I disabled fasttrack on initator and suddenly everything started working.

But you mentioned fasttrack is for both directions. So in theory I have to disable it for the responder side also, no?
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Thu May 14, 2020 12:18 pm

I just realized by myself: fasttrack on responder is OK, as it has no policy-based routing with mangle rules.

All stuff seems working very smooth and fast with low CPU on Tiks now, thank you very much. This topic can be marked as solved now, I guess.

Thanks!
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Thu May 14, 2020 12:23 pm

One small addition to your linked topic:

In your example, at the last line there is a typo:
add chain=prerouting connection-mark=handling-A action=mark-routing new-routing-mark=handling-B
Here I think connection-mark shall be also "handling-B".

Otherwise, I needed another rule into mangle which matches for packets originating from then tunnel: these (along with incoming from wan) shall also not be marked.
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Split traffic then merge

Thu May 14, 2020 8:31 pm

This topic can be marked as solved now, I guess.
You, as the Original Poster (creator of the topic), are the only one who can do that. Check the post which you deem to contain the most of the overall solution (a tough decision, I know) and use the checkmark icon next to its number (I have no idea how it looks in different skins, the screenshot is from the default one):
icons.png

In your example, at the last line there is a typo:
Correct, thank you , fixed.
You do not have the required permissions to view the files attached to this post.
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Mon Jul 27, 2020 2:50 pm

Hello, just wanted to give a quick review of this solution. It is working flawless since then with 100% reliability. Rock solid, thanks a lot for this!

And I'm thinking on opening the IPSEC VPN to smaller clients (like Windows notebook) to this router.

I know it's possible, but I'd like to know if I should configure anything different due to this EoIP configuration?

Thanks, cheers
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Split traffic then merge

Mon Jul 27, 2020 4:31 pm

I'm thinking on opening the IPSEC VPN to smaller clients (like Windows notebook) to this router.
Which router is "this", the initiator in the multi-EoIP arrangement or the responder?
  • if the initiator in the multi-EoIP setup is the responder for the individual clients, there will be no conflict.
  • if the responder in the multi-EoIP setup is also the responder for the individual clients:
    • if you use IKEv2 to connect those clients, there will be no conflict either because exchange-mode is one of the criteria to chose a peer (as the exchange mode is idnicated in the very first packet to come from the initiator).
    • if you want to use L2TP/IPsec (which I don't recommend but you may be scared by the need to create all those certificates necessary to make IKEv2 work with the embedded VPN clients of Windows and iOS, and by the need to install Strongswan to Android devices), you have to either specify the public address of the initiator at the responder side (as the exchange-mode of L2TP/IPsec is main, use of other IKE(v1) exchange modes is really not a good idea, and there is no way to use a dedicated proposal etc. for the clients matching on the same peer as the initiator's identity is its IP address so there is no identifier statically linked to the client), or switch the existing tunnels to exchange-mode=ike2, as with IKEv2 you can use the certificate itself as a selector of the identity row.
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Tue Jul 28, 2020 12:30 am

Which router is "this", the initiator in the multi-EoIP arrangement or the responder?
The responder. I'd like to connect securely from a Windows machine to the current responder, when I'm out of the range of the current initiator, that's the main goal.

  • if you use IKEv2 to connect those clients...
  • if you want to use L2TP/IPsec...
Which one is more secure, if we take into account that my computer might not even have fix and/or public IP address?
Managing certificates is fine, but I prefer builtin solutions (unless they are said to be insecure, like PPTP).
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Split traffic then merge

Wed Jul 29, 2020 1:58 pm

I'd like to connect securely from a Windows machine to the current responder, when I'm out of the range of the current initiator, that's the main goal.
Yes, that's why you need a different proposal at the responder side.



Which one is more secure, if we take into account that my computer might not even have fix and/or public IP address?
Managing certificates is fine, but I prefer builtin solutions (unless they are said to be insecure, like PPTP).
I'm no cryptoanalyst, so I have to believe that IKEv2 is really more secure than IKE(v1) even if the latter uses main mode (other modes of IKE(v1) are definitely weaker).
Certificate-based authentication, if used properly, is more secure than one based on a pre-shared key.

The embedded VPN client of Windows supports both L2TP/IPsec (with IKE(v1) in main mode and with pre-shared key authentication) and IKEv2 with certificate authentication). In addition to higher security and lack of some NAT-related issues of L2TP/IPsec, there's another advantage of IKEv2 - gents in Riga have implemented pushing routes from the server to the client via DHCPINFORM, something that they never bothered to implement for PPP-based VPNs although Windows itself do support that also for these protocols (L2TP, PPTP).

On Android, IKEv2 is not available natively, and with L2TP, the default gateway of the device becomes the other end of the tunnel as soon as the VPN goes up and there is no way to change that.

On iOS/MacOS, IKEv2 works, with some tighter requirements for certificate validity time (it must not be too long) and subject-alt-name contents.
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Fri Jul 31, 2020 8:33 am

Hi,

I'm trying to figure out this IKEv2 connection, but something seems missing. I've basically copied everything from the previous tunnel configs:
[admin@MT_Responder] > ip ipsec proposal print 
Flags: X - disabled, * - default 
 3    ;;; Proposal for Windows clients
	  name="default-Remote" auth-algorithms=null enc-algorithms=null lifetime=20m 
      pfs-group=none 
	                                               
[admin@MT_Responder] > ip ipsec policy print detail 
Flags: T - template, X - disabled, D - dynamic, I - invalid, A - active, 
* - default 
 3 T   ;;; Remote - policy for Windows client to be able to connect from anywhere
       group=default-Remote src-address=::/0 dst-address=::/0 protocol=all 
       proposal=default-Remote template=yes 

[admin@MT_Responder] > ip ipsec peer print detail 
Flags: X - disabled, D - dynamic, R - responder 
 0   R ;;; Main IKE acceptor peer
       name="Main" passive=yes profile=default-Main exchange-mode=ike2 send-initial-contact=no 

[admin@MT_Responder] > ip ipsec identity print detail 
Flags: D - dynamic, X - disabled 
 2    ;;; Remote tunnel for Windows clients
      peer=Main auth-method=pre-shared-key remote-id=fqdn:remote secret="MYSECRET" generate-policy=port-strict 
      policy-template-group=default-Remote 

[admin@MT_Responder] > ip ipsec profile print detail  
Flags: * - default 
 1   name="default-Main" hash-algorithm=sha1 enc-algorithm=aes-128 dh-group=modp2048 lifetime=1d proposal-check=obey 
     nat-traversal=yes dpd-interval=2m dpd-maximum-failures=5 
And on Windows I chose IKEv2 VPN, and filled in "remote" for username and "MYSECRET" for password, but it fails connecting:

Image

I think this is some port issue, but I could not find anything on Windows to set the ports for the VPN to use. What do you think?
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Split traffic then merge

Fri Jul 31, 2020 12:39 pm

If ports 500 and 4500 are open for incoming requests in the input chain of the responder, it's not a port issue.

However, between the Windows embedded client and Mikrotik as a responder, IKEv2 only works with authentication by machine certificate. Username/password authentication is not possible, at least at the moment.
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Tue Jul 06, 2021 6:33 pm

@sindy:

I hope you're all good.
I'm taking into account you don't have a crystal ball, but without too much details, can you give me some lights in this please:

I've setup the IPSec tunnels, and EoIP tunnels, and bonding, which was working soo great for long months or even a year.
Now I realized it stopped working (haven't changed anything on either router).

What I'm seeing is on the initiator side, there are no active peers. (No Phase2 on either IPSec policy).

UDP4500 is still open on both Tiks, what would be your steps to debug this kind of problem? "IPSec tunnel is not establishing"?

As both TIKs are intact, I assume there is some misconfiguration due to power-outage or similar (which is bad, but imaginable).
I checked UDP@4500 from a 3rd party server: both TIKs are showing IKE over that port.

Thanks a ton!
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Wed Oct 13, 2021 9:22 am

Hello!

Today I've came across to a strange situation:

@Sindy, I have two EoIPs which is then bonded together, as you might not remember this.

I'm routing the desired traffic to that bonding, (instead going out on default gw).

However this bonding interface can't really go higher than 8-10MBps, while when I disable the route (letting the traffic flow through default gateway), traffic reaches up to 24MBps (on default gw, without bonding).
And I measured this with various tools, did router reboots, nothing seems help. Also, router's CPU usage is below 20% on both side.

While this might be another ISP limitation, can you maybe think on something worth checking? Or I just simply disable this for now, as the original issue might already gone?
(ISP limited international bandwidth)

Let me know your thoughts, please.

Thank you!
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Split traffic then merge

Sat Oct 16, 2021 10:12 pm

I do remember the setup. However, what I didn't get from what you wrote now, nor could I find it in the history of this topic, is whether the throughput via the bonded interface was ever higher that this. MTU is generally an issue with tunnels - if set too high on the EoIP interfaces, the TCP sends large packets which are then fragmented into two, doubling the packet rate and slightly increasing the bitrate. So maybe have a look at that first.

In any case, since the suspected throttling of single connections seems to be gone, I can see no reason to stick with this workaround.
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Sat Oct 16, 2021 10:27 pm

Sindy: after we succeed with the creation of the bonding, it could easily reached 20-24MBps. So it worked at my full available TX speed.

MTU is a nice idea, I checked it, on both routers (initiator, responder):

Image

It's kindof odd that the bonding itself has 1500 while the eoips are having 1400 only. But I don't really remember if we've changed that for some reason, or it's just the default.

---UPDATE
It was set by design, we talked a lot about this earlier, quick search showed why it's 1400. However what came into my mind reading back some posts (earlier default UDP4500 was speed-limited), now it seems my other chosen custom UDP ports are learnt by the ISP, and they are limiting them as well. I'm thinking on improving this solution to kindof jumping the ports:

Having two simulteanous eoip, and 1 "jumper": jumper can constantly recreate one eoip with an ipsec, while 2 are working in a round-rubin fashion:
Bondig: T0-T1
Jumper: T2
...
Bonding: T2-T0
Jumper: T1
...
Bonding: T1-T2
Jumper: T0
Motivation: I've tried disabling our solution, and then high-speed connection was worked via the default GW, but only for a short time, after half an hour it got speed-limited again.
This never happened with this nice eoip technique, before they "find out" my used ports. But that's not too bad, it took them a year, so now a quick workaround might be to just change those ports to something else.

What do you think about my MTUs? Taking into account that it was worked earlier?
Last edited by danergo on Sat Oct 16, 2021 10:38 pm, edited 1 time in total.
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Split traffic then merge

Sat Oct 16, 2021 10:35 pm

Try lowering the MTU to 1300 on the bonding interfaces at both ends and see whether it changes anything, But if you say it worked at full speed and now it doesn't, it sounds like another trick of the ISP. What if you disable one of the EoIP tunnels at both ends, does the speed fall to 1/2?
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Sat Oct 16, 2021 10:39 pm

Surely will do tomorrow, it's getting too late now, and I don't want to make any troubles :)

Thank you!
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Sun Oct 17, 2021 8:53 am

If I disable one EoIP at both ends, overall bonding's speed is not concerned. I.e. with 2 EoIPs it can reach now 3.5Mbps, and with disabled one channel can go up to ~3.3.
Same test method can reach 24Mbps now on default gateway.

Lowering the MTU to 1300 makes it a bit worse, not reaching above 2Mbps at all.

I'll try chancing the ports, and report back, I have high hopes on that.
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Sun Oct 17, 2021 9:08 am

Sindy, sorry for robbing your precious time. I have overlooked it.
Changing the UDP ports immediately solved the speed-limitation.

Now I'm considering this post as a "startpoint" when the tunnels are got back to work. I'll know then how much time could they survive.
Also I've realised that bonding can take care my "interchanging" of ports, as I don't have to manipulate the EoIPs, it's enough to change the port(s) of the underlying IPSec(s).
Those IPSec(s) will re-establish themselves, and the EoIP(s) will come up afterwards, and the bonding is intact during the whole operation (its speed might got reduced but that's minor).

Thank you!
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Sun Oct 17, 2021 10:41 am

Uhh. It works for 10 minutes only. It's a hard limit by isp. Moving them to another port resets the speed, but I guess they keep remembering on my earlier banned ports for a while, so port-hopping technique seems not feasible. I might have to use a lot of combinations to fool them.

Instead, I'm thinking on adding way more eoips. But as I remember that was not too efficient because of the overhead. I'll have to try anyway, because 10min is too frequent.

I don't have any questions now, but if you have any additional thoughts, please share them :)

Cheers, thank you!
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Sun Oct 17, 2021 1:10 pm

One more update (I guess last for now) :)

Using 5 tunnels, problem seems to be solved: all tunnels are low-speed (4-5Mbps) which doesn't bother the provider. I could use that for more than 30mins.
This is well suited for my needs, and from time to time MikroTik always amazes me how flexible and configurable it is.

Although for these tunnels responder side consumes 35-40% CPU, it's not at its peak level yet, so I'm satisfied.

Cheers,
Thank you!
D.
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Sun Oct 17, 2021 2:21 pm

One more thing, as I have added more IPSec tunnels, 5 is working, but then the 6th is not:

It's copied from the others with care (as the previous 3 on top of the existing 2).
Policies are communicated into responder, and both routers are seeing the Active peers with 6 IPSec tunnels.

On Initiator side, there is only RX bytes and packets, while on the responder side there is only TX.

Do I have a limitation of max 5 concurrent IPSec in MikroTik? Or I just have something else?

Six tunnels in initiator side:

Image

On initiator side: 6th EoIP interface is in RS status, while on responder side it's in S (as it's attached to bonding), but it doesn't get R(unning).

Initiator's EoIP's address is pingable from responder.

This is shot from "Active Peers", so actually it gets active, but then one way doesn't seem to work.
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Sun Oct 17, 2021 3:03 pm

Changing "Tunnel ID" to a different value, then saving EoIP interface solved this. Now I have 6 channels. :)
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Sun Nov 28, 2021 8:01 pm

Hi!

I have came across to an interesting detail. Let's reverse now the direction (sending files from initiator to responder).

Bandwith in this direction is ~7MBps (according to Ookla):

Image

This might be inaccurate, as it measured ~10MBps for the download direction, and I can easily reach ~20MBps with this tunnel implementation.

Now to the interesting part: I'm sending files (via scp) on the WAN interface and through the tunnel. (same file, same destination, different route):

Image

Even on WAN interface, sending happens at 3.2MBps, which is half of the measured rate of Ookla (but let's skip this for now).

See the right side of the diagram, that's tunnel's graph. It can send at lower speed than the default gateway (2.2MBps), and the traffic shape is really different from the WAN.
(Please note: I've done more tests, WAN diagram is recorded separately!)

Interesting facts:
  • RX packet rate (in Tunnel send) is mostly equal to RX packet rate (in WAN send)
  • TX packet rate (in Tunnel send) is ~33% less than in WAN send
  • Most problematic part: TX rate fluctuates in tunnel send, but constant in WAN send (note: in WAN send, I stopped and restarted once, that you can see on the chart)
It might be an MTU misconfig? L2 MTUs are 65535 on all EoIP tunnels and Bonding interface, and they seem to be not changable (grayed out).

Do you folks have some idea by any chance?

Should I decrease the L2 MTU on bonding and EoIPs?

Why can the scp on standard route send more packets, than on this, and why the bandwith fluctuates like this on the tunnel?

Thank you!
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Split traffic then merge

Sun Nov 28, 2021 8:57 pm

As long as the Rx packet rate is below the Tx packet rate, it's still reasonable, as TCP doesn't acknowledge every single packet. Also the speed fluctuation seems a TCP thing to me, because delay of ACK, as well as laziness of the destination to fetch the received data from the input buffer, causes throttling of sending.

What surprises me are the Tx drops on the tunnel interfaces - that's a local thing, the statistics doesn't look into TCP ACK to identify packets lost on their way to destination.

How does that look without the bonding, with just a single EoIP tunnel? Second, can you watch both the WAN traffic and the tunnel traffic as you do the test, even if it would require running two Winbox instances simultaneously?

As for MTU, EoIP simply hides the MTU of the transport side from the payload side, which leads to fragmentation, and fragments are basically handled awfully in the internet. To prevent that and at the same time keep the payload MTU at the headache-free 1500 bytes, the only current choice is to use L2TP which uses payload splitting and reassembly using MLPPP means, i.e. the transport packets are never "fragmented" in the IP sense but the payload they carry is. But unfortunately, Mikrotik doesn't support true MLPPP in terms of an ability to distribute the transport packets across multiple links directly, except when acting as PPPoE client.

So if you still need to use two streams to overcome the ISP's bandwidth throttling, and you want a payload MTU of 1500, you could replace the bonding (which doesn't add any own headers to the packets passing through it but may generate own traffic depending on the setting) by L2TP with MLPPP and distribution of its transport packets between two tunnels using mangle rules, but these two tunnels would still have to be "something over IPsec" ones as you cannot use NAT to distribute the L2TP transport traffic between two policies. But as without bonding, you don't need the tunnels to be L2, you can use IPIP (ipencap) instead of EoIP, saving a few bytes of the packet size as ipencap has smaller headers than EoIP. So it may compensate for the extra headers of (ML)PPP.
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Mon Nov 29, 2021 8:21 am

As long as the Rx packet rate is below the Tx packet rate, it's still reasonable, as TCP doesn't acknowledge every single packet. Also the speed fluctuation seems a TCP thing to me, because delay of ACK, as well as laziness of the destination to fetch the received data from the input buffer, causes throttling of sending.

What surprises me are the Tx drops on the tunnel interfaces - that's a local thing, the statistics doesn't look into TCP ACK to identify packets lost on their way to destination.

How does that look without the bonding, with just a single EoIP tunnel? Second, can you watch both the WAN traffic and the tunnel traffic as you do the test, even if it would require running two Winbox instances simultaneously?

Here is how sender's WAN looks like during a tunnel TX (I'd say it correlates, two charts are very similar):
Image

Here is how it changes if I enable only one EoIP:
Image

As for MTU, EoIP simply hides the MTU of the transport side from the payload side, which leads to fragmentation, and fragments are basically handled awfully in the internet. To prevent that and at the same time keep the payload MTU at the headache-free 1500 bytes, the only current choice is to use L2TP which uses payload splitting and reassembly using MLPPP means, i.e. the transport packets are never "fragmented" in the IP sense but the payload they carry is. But unfortunately, Mikrotik doesn't support true MLPPP in terms of an ability to distribute the transport packets across multiple links directly, except when acting as PPPoE client.

So if you still need to use two streams to overcome the ISP's bandwidth throttling, and you want a payload MTU of 1500, you could replace the bonding (which doesn't add any own headers to the packets passing through it but may generate own traffic depending on the setting) by L2TP with MLPPP and distribution of its transport packets between two tunnels using mangle rules, but these two tunnels would still have to be "something over IPsec" ones as you cannot use NAT to distribute the L2TP transport traffic between two policies. But as without bonding, you don't need the tunnels to be L2, you can use IPIP (ipencap) instead of EoIP, saving a few bytes of the packet size as ipencap has smaller headers than EoIP. So it may compensate for the extra headers of (ML)PPP.

This seems a bit overcomplicated for me currently, and as everything works great, I'd stick with the current implementation. As we can see, a single EoIP (I haven't even disabled the bonding, just I disabled all EoIPs except one) the traffic shape is much healthier.
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Wed Nov 30, 2022 3:52 pm

Hi, sindy!

I decided to try your above suggestion to avoid throttling and fragmentation as much as possible (and keeping the MTU as 1500):
So if you still need to use two streams to overcome the ISP's bandwidth throttling, and you want a payload MTU of 1500, you could replace the bonding (which doesn't add any own headers to the packets passing through it but may generate own traffic depending on the setting) by L2TP with MLPPP and distribution of its transport packets between two tunnels using mangle rules, but these two tunnels would still have to be "something over IPsec" ones as you cannot use NAT to distribute the L2TP transport traffic between two policies. But as without bonding, you don't need the tunnels to be L2, you can use IPIP (ipencap) instead of EoIP, saving a few bytes of the packet size as ipencap has smaller headers than EoIP. So it may compensate for the extra headers of (ML)PPP.

I understand there is no such thing in MikroTik as MLPPP, and I have to do it manually. I have a lot of questions, because I think there is no material online which would cover my use-case.
  • So how shall my L2TP tunnels look like on initiator and responder side (connect to, allow fast path, use ipsec, user)?
  • I guess original responder shall be the L2TP server, and initiator shall be the L2TP client. Is this assumption correct?
  • Shall I still use "manual" IPsec tunnels for this artificial MLPPP with multiple L2TPs?
  • How can I "connect" IPsec tunnels (previously they were connected via IP address to the EoIPs) to L2TP connections? By doing the same with IPIPs?
  • What kind of mangles shall I create to distribute traffic between L2TPs? Like, defining routes, and use some IP header field (DSCP)?
  • Both IPIP and L2TP client has an option to specify IPsec secret. As I have manually created IPsec tunnels, I'm assuming I don't need to set these, is this correct?
  • So IPsecs are built up, they will encapsulate IPIP, and with IPIP addresses I have to create L2TP connections?

Thank you very much, I'm very interested if this can help me getting a bit further :)
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Split traffic then merge

Thu Dec 01, 2022 8:52 am

I understand there is no such thing in MikroTik as MLPPP, and I have to do it manually.
This sounds like an understandable misunderstanding :) MLPPP indeed stands for "Multi-Link PPP", but nevertheless it can be used even on a single link. The benefit of using MLPPP on a single link is that it allows to split large payload packets into multiple transport ones that do not exceed a certain size. So instead of fragmentation of the transport packets at IP level, which can cause packet loss because non-first fragments are treated badly in some parts of the internet, the payload packets get split. So the packet rate still doubles like in case of IP level fragmentation, but other negative effects of fragmentation are suppressed.

In routerOS, a real multi-link operation is only supported for PPPoE client, which isn't helpful for your use case.

So the first thing I would try would be to use a single L2TP/IPsec tunnel with MLPPP enabled; to enable MLPPP, you have to set, at both the client and the server, max-mtu and max-mru allowed for the transport packets to something like 1300 (it can be fine-tuned later but for most traffic a small max-mtu doesn't make much difference as it doesn't matter whether you split a 1500-byte payload packet into a 1000-byte one and a 500-byte one or into a 1499-byte one and a 1-byte one) and mrru to 1504.

If that is not sufficient (the per-flow bandwidth limitation is still in place), you need to distribute the transport packets of the same L2TP flow among multiple unerlying IPIP tunnels, each of them encrypted using its own IPsec tunnel, same like we did with EoIP. You can keep the multi-EoIP setup in place, add the multi-IPIP one to it (using the same transport addresses like the EoIP tunnels do), and add routing tables and mangle rules to evenly distribute the traffic between the L2TP client and the L2TP server among those IPIP tunnels. You cannot use ECMP (multiple gateways for the same route) as route caching would cause all packets to use the same IPIP tunnel.
I have a lot of questions, because I think there is no material online which would cover my use-case.
  • So how shall my L2TP tunnels look like on initiator and responder side (connect to, allow fast path, use ipsec, user)?
    There will be just a single L2TP tunnel even if distribution of the traffic over multiple IPIP tunnels will be used. You will have to link the L2TP client to some local address c.c.c.c at router C - client (by setting src-address=c.c.c.c), and let it connect to some local address s.s.s.s at router S - server; at router C, the routing to s.s.s.s will be controlled by the routing tables and mangle rules, and so will be the routing to c.c.c.c at router S. There is no need to assign an address to each of the IPIP tunnel interfaces, I even prefer to assign s.s.s.s and c.c.c.c to port-less bridge interfaces than to assign them to any of the IPIPs. The user and password of the /interface l2tp-client row on C must match the name and password of a /ppp secret row on S.
  • I guess original responder shall be the L2TP server, and initiator shall be the L2TP client. Is this assumption correct?
    Technically it is not mandatory in this setup with the IPIP tunnels sandwiched between L2TP and IPsec, but I would also colocate L2TP server with IPsec responder.
  • Shall I still use "manual" IPsec tunnels for this artificial MLPPP with multiple L2TPs?
    Yes, as said above, keep the "manual" IPsec tunnels you have, and just add IPIP tunnels in parallel to the EoIP ones (you can remove the EoIP ones in future if this approach proves better).
  • How can I "connect" IPsec tunnels (previously they were connected via IP address to the EoIPs) to L2TP connections? By doing the same with IPIPs?
    There will be no direct relationship between the IPsec and the L2TP. The IPsec encrypts the transport packets of IPIP that flow between their transport addresses; the payload of IPIP will be the transport of L2TP, with an unrelated set of addresses (this is mandatory - c.c.c.c and s.s.s.s must not match the IPsec policies)
  • What kind of mangles shall I create to distribute traffic between L2TPs? Like, defining routes, and use some IP header field (DSCP)?
    Just as many routing tables as there are IPsec-encrypted IPIP tunnels at each device, with a single default route each, each using another IPIP tunnel interface as gateway. And then, mangle rules in chain output will assign the corresponding routing marks to packets from c.c.c.c to s.s.s.s (at C) and from s.s.s.s to c.c.c.c (at S) using the nth match, to distribute the LT2P transport traffic evenly:
    /ip route
    add routing-mark=via-ipip1 gateway=ipip1
    add routing-mark=via-ipip2 gateway=ipip2
    add routing-mark=via-ipip3 gateway=ipip3
    /ip firewall mangle
    add chain=output src-address=!c.c.c.c action=accept
    add chain=output nth=3,1 action=mark-routing new-routing-mark=via-ipip1
    add chain=output nth=2,1 action=mark-routing new-routing-mark=via-ipip2
    add chain=output action=mark-routing new-routing-mark=via-ipip3
  • Both IPIP and L2TP client has an option to specify IPsec secret. As I have manually created IPsec tunnels, I'm assuming I don't need to set these, is this correct?
    You even must not set these, as it would create another set of peers and policies that could interfere with the manually configured ones.
  • So IPsecs are built up, they will encapsulate IPIP, and with IPIP addresses I have to create L2TP connections?
    As stated above, do not link the c.c.c.c/32 and s.s.s.s/32 addresses to the IPIP interfaces, use some empty bridges for that - again, it's just for clarity reasons, technically you can attach them to any of the IPIP tunnel interfaces.
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Thu Dec 01, 2022 11:44 am

Wow, thank you, this is thorough (again)! I did even have to let my coffee work to process :)

If that is not sufficient (the per-flow bandwidth limitation is still in place), you need to distribute the transport packets of the same L2TP flow among multiple unerlying IPIP tunnels...
Yes, limitation is still in place.

There will be just a single L2TP tunnel even if distribution of the traffic over multiple IPIP tunnels will be used. (...)
There is no need to assign an address to each of the IPIP tunnel interfaces, I even prefer to assign s.s.s.s and c.c.c.c to port-less bridge interfaces than to assign them to any of the IPIPs.

There will be no direct relationship between the IPsec and the L2TP. The IPsec encrypts the transport packets of IPIP that flow between their transport addresses; the payload of IPIP will be the transport of L2TP, with an unrelated set of addresses (this is mandatory - c.c.c.c and s.s.s.s must not match the IPsec policies)

As stated above, do not link the c.c.c.c/32 and s.s.s.s/32 addresses to the IPIP interfaces, use some empty bridges for that - again, it's just for clarity reasons, technically you can attach them to any of the IPIP tunnel interfaces.

I'm a bit lost on these:
  • no need to assign an address to each of the IPIP tunnel interfaces?
  • technically you can attach them to any of the IPIP tunnel interfaces?
  • IPIP tunnels shall use the very same addresses as EoIPs do, right? So EoIP tunnels will remain intact (usable), but due to routing, packets will follow a different path via IPIPs?

Thank you!
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Split traffic then merge

Thu Dec 01, 2022 4:31 pm

I'm a bit lost on these:
The payload packet enters a tunnel interface; the tunneling process encapsulates the payload packet into a transport one and sends it to the remote-address configured for that tunnel interface; the source address is chosen by routing if no local-address is specified, or the local-address is used no matter what the routing suggests.

But the addresses I am talking about, c.c.c.c and s.s.s.s ones, are addresses of the IPIP payload, which is the L2TP transport. An L3 tunnel (point-to-point) interface does not necessarily need an IP address attached to it - if you tell a route that its gateway is the tunnel interface, routing will send packets that match that route's destination via that interface. On the other hand, these addresses must be assigned to some interface at their respective routers, to allow the routers to use them as their own ones. So to avoid confusion half a year later, I suggest to create dedicated bridge interfaces.

The other way round would be to attach an individual /32 address and network to each of the IPIP interfaces, and set the gateways of routes to the network addresses.

And yes, EoIP tunnels will remain usable, but routing will make sure that packets between c.c.c.c and s.s.s.s will run via IPIP tunnels (unless you'd choose c.c.c.c and s.s.s.s from the same subnet and attach them to the bond interfaces using the EoIP ones).
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Thu Dec 01, 2022 9:45 pm

Hi!

Thanks. I'm just started to make this happen, but already stucked:
You will have to link the L2TP client to some local address c.c.c.c at router C - client (by setting src-address=c.c.c.c), and let it connect to some local address s.s.s.s at router S - server; at router C, the routing to s.s.s.s will be controlled by the routing tables and mangle rules, and so will be the routing to c.c.c.c at router S.
(...)
I would also colocate L2TP server with IPsec responder
Shall I create one "L2TP Client" Interface in both initiator and responder routers? So "L2TP Server Binding" is not necessary in either router? On my Tik, "L2TP Server Binding doesn't have an option to set an address:

Image

Even, for "L2TP Client" I don't seem to have src-address, but I have Connect-to:

Image

IPIP tunnels anyway are already setup, and running. Strangely, at C (client, initiator), IPIPs are getting 1442 as MTU, however on S (server, responder), they are having 1434. I can't seem to be able to change it though.
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Thu Dec 01, 2022 10:23 pm

Ah, I haven't realized that "L2TP Server" and "L2TP Server Binding" are different things.

Now what you wrote, makes sense to me, I'll try it tomorrow.
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Thu Dec 01, 2022 10:44 pm

Okay, something I miss:

"You will have to link the L2TP client to some local address c.c.c.c at router C - client (by setting src-address=c.c.c.c)"

What do you mean here by "link"? I tried to set an address to L2TP Client (/ip address), but it's an invalid interface, I can't set its address.

Moreover, in PPP, I have both "Secrets" and "L2TP Secrets". Shouldn't I use "L2TP Secrets" to let this L2TP connect?

Currently I added a few test items in both C and S:
  • Virtual bridge in C: address: 100.100.103.2
  • Virtual bridge in S: address: 100.100.103.1
  • Route In C: 100.100.103.1 gw: ipip1
  • Route In S: 100.100.103.2 gw: ipip1
  • No mangles as of now
  • L2TP Client in C: connect to: 100.100.103.1
Ipip is running, but L2TP can't connect (it stays in "Connecting..." status).

What do I miss? :?
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Split traffic then merge

Fri Dec 02, 2022 8:59 am

At the server side, forget about L2TP server bindings and L2TP secrets, use just regular PPP secrets, the server binding will be created dynamically. You can create a static binding between the (user)name of the secret and the server binding interface name later if needed.

At the client side, if the src-address field is missing in the L2TP Client settings, you either have an outdated RouterOS or an outdated Winbox:
l2tpclient.png
In RouterOS, it has been added somewhere between 6.45 and 6.47; in Winbox, I've got no idea as I rarely use it.
You do not have the required permissions to view the files attached to this post.
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Fri Dec 02, 2022 2:52 pm

Hi!

Yes, indeed, I was a bit out-dated (6.46.xy), now updated to latest and I have the src-address option at the client, and L2TP is up and running now :)

Thank you!

I'll continue with mangles, but those will be easy now.
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Fri Dec 02, 2022 6:41 pm

Hm.

Why is "Actual MTU" not respecting "Max MTU"?

Image
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Split traffic then merge

Fri Dec 02, 2022 7:18 pm

Because max-mtu has a different meaning if MLPPP is enabled.
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Fri Dec 02, 2022 8:48 pm

Unfortunately I can't make this work. Not even the L2TP:

If I route packets to this "running" l2tp gateway, those packets are not arriving on the other end of the L2TP.

So packet is going from C-> 100.100.103.2 -> IPIP1(C) -> IPIP1(S) -> 100.100.103.1(S) -> some resource at S, but there are no packets in S on the L2TP interface.

If I set the local and remote addresses of the secret, it is "kindof" working, however the l2tp is hanging up at every 20-30s.

So what I don't get:
  • Can I still mark my packets I wish to push through this L2TP tunnel in "prerouting" chain with routing-mark "route-through-l2tp"?
  • Output chain happens later than preroute chain, so can I apply the output mangles to packets which already has this "route-through-l2tp" routing-mark?
  • If I set local and remote addresses of the PPP secret in S, it creates dynamic routes for 100.100.103.2 via l2tp, which is wrong, as I won't be able to round-rubin the route through IPIPs.
  • If I don't set these addresses, there is no connection between the 2 Tiks, they can't even ping the other half (C can't ping S with 100.100.103.1, and S can't ping C with 100.100.103.2).
So somewhat I'm missing, but can't seem to solve it.

As far as I understand:

In client router: My payload -> mangle (mark routing) -> marked payload's gw is l2tp -> l2tp's dst-addr's gw is ipip1 -> IPsec -> INTERNET, Is this correct?

In server router: INTERNET -> IPsec -> ipip1 -> How this will know that the payload is l2tp?
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Split traffic then merge

Fri Dec 02, 2022 10:39 pm

You've said the /interface l2tp-client was running, haven't you? If so, /interface l2tp-server print at the server side should show an <l2tp-username> interface as well.

You have to set the remote-address and local-address on the /ppp secret row, but you have to use yet another pair of addresses than the c.c.c.c and s.s.s.s used to establish the L2TP tunnel. And there seems to still remain some misunderstanding - it is correct that there is only a single L2TP tunnel. The round-robin distribution will be applied to the L2TP transport packets, not to the L2TP payload ones.

A packet that passes through an output chain is a packet sent by the router itself; such packets never pass through prerouting. And vice versa, packets that came from the outside do pass through prerouting but never pass through output.

But once a payload packet gets encapsulated into a transport one, the transport one is a separate packet from the point of view of the firewall.

So the actual payload packet arrives via an Ethernet interface, gets handled by prerouting, filter and postrouting, and "leaves" via L2TP. L2TP encapsulates it into a transport packet, which is sent by the router itself, so it passes through output and postrouting and "leaves" via IPIP. IPIP encapsulates it into its own transport packet, which is also sent by the router itself, passes through output and postrouting, and gets caught by the IPsec policy and encapsulated again.

Of course you can (and have to) assign routing-mark values to packets you want to send via the L2TP tunnel (unless you would want almost everything to go through there, or just a few destination subnets no matter the source).
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Sat Dec 03, 2022 12:57 pm

You've said the /interface l2tp-client was running, haven't you? If so, /interface l2tp-server print at the server side should show an <l2tp-username> interface as well.
Yes, it was up, but with some invalid (or empty) IP addresses, as I haven't set those remote-address and local-address in the secret - my bad, I have never used L2TP with MikroTik (yet).

You have to set the remote-address and local-address on the /ppp secret row, but you have to use yet another pair of addresses than the c.c.c.c and s.s.s.s used to establish the L2TP tunnel. And there seems to still remain some misunderstanding - it is correct that there is only a single L2TP tunnel. The round-robin distribution will be applied to the L2TP transport packets, not to the L2TP payload ones.

A packet that passes through an output chain is a packet sent by the router itself; such packets never pass through prerouting. And vice versa, packets that came from the outside do pass through prerouting but never pass through output.

But once a payload packet gets encapsulated into a transport one, the transport one is a separate packet from the point of view of the firewall.

So the actual payload packet arrives via an Ethernet interface, gets handled by prerouting, filter and postrouting, and "leaves" via L2TP. L2TP encapsulates it into a transport packet, which is sent by the router itself, so it passes through output and postrouting and "leaves" via IPIP. IPIP encapsulates it into its own transport packet, which is also sent by the router itself, passes through output and postrouting, and gets caught by the IPsec policy and encapsulated again.

Of course you can (and have to) assign routing-mark values to packets you want to send via the L2TP tunnel (unless you would want almost everything to go through there, or just a few destination subnets no matter the source).
Thank you very much for your patience, I now understand fully this concept.

Now everything is up and running, however there is not too much of performance difference compared to Bonding/EoIPs. Transfer speed (measured with speedtest.net) almost seems decreasing as I increase the IPIPs, ie. with a single IPIP it reached ~21MBps (Note: ISP's limitation is only activated after a few minutes of constant transfer, so this test was not yet affected), then I added 2 more IPIPs, and transfer decreased to somewhere below 10MBps (but this can be due to daylight timings: I tested in the morning with one IPIP, by that time it was 5am at S, and I tested with more IPIPs when it was already 8 or 9am at S - who knows).

However it works as it should, so every packet reaches now its destination, I don't have any NATting or routing problem now. Thank you!

I have only two questions left: is it possible that creating more transports (IPIPs) for L2TP cause lower passthrough between 2 routers than if L2TP has only one transport IPIP? Maybe it's due to mangles? CPU load doesn't reach even 25% though.

It seems the best result can be achieved by using as less transport as possible (2), however they seem can't provide the latency needed for streaming video needs.

Normally the stream I wish to watch is having a huge first impact (keyframe), then transport is relaxed for a few seconds, then another huge impact (next keyframe) on transport. However if transport is not fast enough (both in terms of bandwidth and latency), video will stall, as it's not getting the next frame in proper time.

I have no affect on the stream itself (i.e. I can't change it, but another codec would be much more suitable for this tunnel :( ), but I have another Tik at another location, which is not behind NAT so I have an idea (question2): if due to this stream's behavior, ISP limitation might be not an issue here; what would be the fastest, most reactive connection between two Tiks if they are:
Server (Video stream comes through) -- MT1 -- direct connection to internet -- ISP2 -- MT2 -- Client (wish to watch the stream)

No need to be secured, I wish to use this purely for video passthrough, as fast and responsive as possible.

Thank you!
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Split traffic then merge

Sat Dec 03, 2022 3:57 pm

there is not too much of performance difference compared to Bonding/EoIPs
I wasn't expecting the throughput to be higher - my sole expectation from this approach was that a single stream connection would be distributed among multiple UDP sessions at the lowest transport level (which is the same for bonding of EoIP and for L2TP over IPIP), and that the payload would have a full 1500-byte MTU without IP fragments at the lowest transport level. L2TP is not famous for its speed, especially on your relatively weak hardware.

is it possible that creating more transports (IPIPs) for L2TP cause lower passthrough between 2 routers than if L2TP has only one transport IPIP? Maybe it's due to mangles?
It is possible but strange. The amount of packet handling doesn't depend on the number of IPIP channels used. And mangling is a relatively lightewight operation as compared to encapsulation of payload packets into L2TP transport ones.

if due to this stream's behavior, ISP limitation might be not an issue here; what would be the fastest, most reactive connection between two Tiks if they are:
Without encryption and without NAT traversal, IPIP is the simplest, and therefore fastest, encapsulation method. If you need NAT traversal, you have to use L2TP or IPsec (SSTP can do that too but for some reason it is way slower than L2TP). For IPsec, you can set enc-algorithms and auth-algorithms to null in the phase 2 proposal if you don't care about security, so it should be as fast as a hardware-accelerated encryption would be on the same hardware.
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Tue Dec 06, 2022 11:19 am

Thank you, sindy!

It's always a pleasure learning new things :)

My video stream has been "kindof" solved by transcoding it on-the-fly to a better codec (h265), which reduced the bandwidth requirement, and can utilize the earlier created EoIP tunnels perfectly.

Sometimes we have to think "out-of-the-box", to have the right solution :)
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Sun Feb 19, 2023 10:44 am

Hi!

I'm facing two issues with this configuration currently, and I can't debug what's the heck. Everything was worked perfectly, but I had to reboot the "initiator" router (due to other reasons).

After initiator TIK rebooted, tunnel is not building up correctly, to be more exact:
  • IPSec tunnels in both TIKs are "established": so I think all ports, NATs are configured correctly (TBH why they wouldn't, I just rebooted the router)
  • IPSec tunnels in initiator has "Rx Packets" but no "Tx Packets" (i.e. all "Tx Packets" are 0)
  • IPSec tunnels in responder has "Tx Packets" but no "Rx Packets" (i.e. all "Rx Packets" are 0) -> this seems that responder is OK, but initiator is not (makes sense, as I rebooted the initiator)
  • EoIP tunnels are in "RS" state in initiator, but they are in "S" state in responder (i.e. no EoIP tunnels are usable -> hence tunnel is down)

IPSec in initiator also has a problem: I tried disabling all IPSec policies except one, and then I killed all the active connections.
What I'd expect after this: 1 IPSec to become "established". However, all (even disabled) policies are coming back as "established", with only one having "PH2 Total = 1" and "Rx bytes".

Do you have any recommendation on what could I try? It really seems my IPSec configuration is not going to survive a reboot for some weird reason.
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Sun Feb 19, 2023 11:01 am

I sorted it out: TIK ran out of free space (3% based on "Files" in winbox). After removal of a few stuff managed to get 9% free, and rebooted, and now all tunnels are properly creating.
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Split traffic then merge

Sun Feb 19, 2023 11:18 am

I'm afraid it is actually unrelated. Being out of RAM could be related (although it usually causes a reboot quite soon), being out of file space is not really likely.

What often happens is that after reboot of the initiator or some NATing router on the path between the initiator and the responder, the packets from the initiator start arriving to the responder from a different socket than before. The responder handles them because it ignores the source socket and only checks the SPI, but it sends the response packets to the previously learned socket (which also keeps the eventual tracked connection alive in its own firewall), so the response packets never reach the initiator as the NAT on its end of the path does not let them through.

But your case seems different as you've seen only Rx packets and no Tx ones. I'd have to see that live.
 
danergo
Member Candidate
Member Candidate
Topic Author
Posts: 131
Joined: Tue Dec 24, 2019 8:49 pm

Re: Split traffic then merge

Sun Feb 19, 2023 11:38 am

Yes I guess you are right, free space is increased by removing massive amount of configuration done recently, which might caused high RAM usage.

What was interesting though, after tunnel was up and running, router could manage these additional configuration without problems, but it was not able to properly reboot with them.

I believe RAM consumption during boot is a little higher than normally and it caused problems with building up the IPSec tunnels properly.

Although your point with different routes makes sense too, but in this particular case that was not the problem.

Anyway, I'm happy that it's sorted out, I could even reach higher throughput (this time from initiator to responder) than before.

Thank you!

Who is online

Users browsing this forum: Bing [Bot], DanMos79, GoogleOther [Bot] and 60 guests