Split traffic then merge

RouterOS General
1

I’d like to solve a strange ISP limitation.

Let’s consider this topology:

Server -- MT1 -- ISP1 -- inland internet -- ISP2 -- MT2 -- Client

In this topology, if I run iperf3, which is a network bandwidth tester, bandwidth from server->client can easily reach the uplink bandwidth limit which is currently 25MBps.

However if the client is abroad:

Server -- MT1 -- ISP1 -- abroad internet -- ISPAbroad -- MT2 -- Client

I can only measure 5MBps from the server->client. But only if I measure with a single thread.

If I measure with parallel option of iperf3 (this way it creates multiple connections), the bandwidth can reach the uplink limits (25MBps) even through the abroad ISP.

So technically I have the bandwidth, but not on a single connection.

Is there a way to use those MT1 and MT2 to defeat this limitation?

I want to use a single TCP traffic from the server, then MT1 shall split the traffic into multiple concurrent connections, and then MT2 shall reassemble them and for the client it would seem as if traffic was never splitted.

What do you think? Is this something achievable with MikroTik? (As the bandwidth is quite low compared to processing power, I don’t see any reasons why it should not be possible, just I don’t know which direction should I go).

Thank you!

How to create multichannel VPN tunnel?
Setting up IKEv2 VPN Server behind NAT
2

Try setting up a GRE tunnel between the two if both have public IP addresses. If it works, it could be that the ISP won’t limit GRE per connection. If it doesn’t work, the only way to split a single TCP flow among several paths is to use bonding in balance-rr mode, but bonding can only bond together L2 interfaces, so it needs EoIP tunnels. But EoIP is seen to the outside world as plain GRE, so even packets belonging to different EoIP tunnels will most likely be seen by the ISP’s limiter as belonging to the same flow. However, there is the GRE ID field which Mikrotik misuses, so you can give it a try.

If it doesn’t help, or if GRE doesn’t pass through the ISPs at all (beware, there are issues with GRE handling in Mikrotik’s firewall in current ROS versions), or because one of the Mikrotiks doesn’t have a public IP on itself, you would have to establish multiple SSTP tunnels between the Tiks, and run an EoIP tunnel over each of them. This would make sure that each EoIP would be seen as a different TCP flow, but the overhead of the nested encapsulations would be terrible and TCP over TCP is also a headache source once packets get lost. Multiple L2TP tunnels won’t help as the client side uses port 1701 as well so a single flow again. On the other hand, if the limitation only affects TCP connections, a single L2TP tunnel might hide the TCP from the ISP.

3

can only measure 5MBps from the server->client. But only if I measure with a single thread.

If I measure with parallel option of iperf3 (this way it creates multiple connections), the bandwidth can reach the uplink limits (25MBps) even through the abroad ISP.

.
little bit unclear how long you did the test ? … maybe only the policy-engine of your fancy provider and/or your fancy country*, would have needed a litlle more time to compute you are trying to cheat ?!
.
so while iperf3 is threading ( in network terms better multiplexing ) over different ports … I would prefer a bonding interface over multiple open-vpn tunnels :exclamation:
.
guess iperf is using udp per default ( not 100% pos.) … MTik open-vpn is udp-too … guess you would need different ppp-profiles to have multiple tunnel-ends on the same ip’s ??!
.
… figure it out before a warlord overthrowes your current gouvernment and you need a complete new approach :sunglasses:
.
[*recently found my own ass a banana-republican]

4

Iperf does tcp by default.
And it does not do on different ports because I manually needed to create a port forward and give this port as parameter to iperf.
So it goes on the same port but multiple “routes”.

For how long? I tried 10s, 30s, 300s, and 40mins.
All measurements were the same. Ie. 1 thread 5mbps, 8thread 25mbps.

5

Neither of them have public address at the moment.
Both sits behind a nat, but I can bring both to public with dmz. However I have no idea what these 3rd party isp routers’ dmzs allow.

Extra info: I already tried PPTP, but same limitation happened.
As I needed to manually forward a port for iperf measurement, I think ISP limits only single tcp connections. Multiple connections on the same tcp port can go higher.

Would this change anything?

6

As you have tried PPTP and it was throttled, it means that GRE is throttled too (PPTP uses GRE as transport).

What I haven’t realized yesterday is that with plain IPsec with exchange-mode=ike2, you can use different ports at both ends for each connection relatively easily, and it will use UDP also for transport packets rather than naked ESP, as due to the port forwarding necessary to differentiate the ports there will be NAT (well, OK, NPT) in the path. The rest of the concept remains unchanged - /interface bonding in balance-rr mode, bonding together five or six eoip tunnels, each transported using its own IPsec peer.

So I’d start with a single plain IPsec IKEv2 connection, to see whether UDP is getting throttled as well. If it is, you’d have to create multiple IPsec initiators at one end, and a single responder on the other, but at the responder side, you’d port-forward several outside UDP ports to 4500; on the initiator side, you’d set the these ports as port parameters of /ip ipsec peer rows. send-initial-contact must be set to no at all peers involved, otherwise it won’t work. To make it even more stealth, you can use src-nat rules at the initiator side: chain=output dst-address=ip.of.responder.peer protocol=udp dst-port=one-of-the-remote-ports action=src-nat to-ports=some-random-port.

To make each of the EoIP tunnels use its dedicated IPsec tunnel, you have to create several local IP addresses at one device (I’d recommend the initiator to keep things a bit simpler, as the policies at initiator side will be configured manually and the responder will mirror them using a template) and link the EoIP tunnels at this end to those auxiliary addresses. The IPsec policies associated to the peers would have to tunnel between one of these addresses each at one side and the single address at the other one.

7

.
sorry to interfere only for beeing so nitpicky … I have to give the tcp-default …
but … mutiple “routes” ? … nöh
.

39494
39500
39498
39496

.

root@badger:~# iperf -c  192.168.67.140 -P 4
------------------------------------------------------------
Client connecting to 192.168.67.140, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.100.78 port 39494 connected with 192.168.67.140 port 5001
[  6] local 192.168.100.78 port 39500 connected with 192.168.67.140 port 5001
[  5] local 192.168.100.78 port 39498 connected with 192.168.67.140 port 5001
[  4] local 192.168.100.78 port 39496 connected with 192.168.67.140 port 5001
[ ID] Interval       Transfer     Bandwidth
[  6]  0.0-10.0 sec   286 MBytes   240 Mbits/sec
[  5]  0.0-10.0 sec   218 MBytes   183 Mbits/sec
[  3]  0.0-10.0 sec   219 MBytes   183 Mbits/sec
[  4]  0.0-10.0 sec   395 MBytes   331 Mbits/sec
[SUM]  0.0-10.0 sec  1.09 GBytes   936 Mbits/sec
root@badger:~#
8

I’m not sure that I understand correctly what you are trying to mean here, sorry.

9

Thank you! This seems a bit complicated for me from the first sight but I like MTs very much and always up for learning new things.

So, you’re suggesting me to first creat an IPsec tunnel and retry the iperf tests, am I right?
Creating IPsec tunnel would be good based on this example: https://wiki.mikrotik.com/wiki/Manual:IP/IPsec#Site_to_Site_IPsec_tunnel?


10

I’m not sure that I understand correctly

.
just a side-degression in iperf … please proceed

11

Unfortunately, none of the examples matches the needs (NAT at both ends, one with port forwarding) exactly. The example you have chosen is OK as a base, but

  • use exchange-mode=ikev2 (at peers at both ends)
  • at one machine (the one which you’ve configured as the PPTP client, as it is probably easier for you to configure port forwarding for the other one), set the peer’s address to the public IP behind which the other machine is NATed, and in the identity, set my-id=fqdn:some.name
  • at the other machine, set the peer’s address to 0.0.0.0/0, and in the identity, set remote-id=some.name, and on the router between it and the internet, set port-forwarding of UDP port 4500 (unless you already use 1:1 NAT there)
12

Thanks!

I have started to create the IPsec tunnel, and I’d appreciate if you could validate this before I run into a problems that’s caused by myself:

Server side MikroTik’s name: server
Server side NAT router of ISP: isp_server
Client side MikroTik’s name: client
Client side NAT router of ISP: isp_client

Topology:

server - isp_server(a.b.c.d) - WAN - isp_client - client

Now I created the profiles, proposals, peers and indentities on both parties:

[server] > ip ipsec profile add dh-group=modp2048 enc-algorithm=aes-128 name ike1-site2
[server] > ip ipsec proposal add enc-algorithms=aes-128-cbc name=ike1-site2 pfs-group=modp2048
[server] > ip ipsec peer add address=0.0.0.0/0 name=ike1-site2 profile=ike1-site2 exchange-mode=ike2 send-initial-contact=no
[server] > ip ipsec identity add peer=ike1-site2 secret=thisisnotasecurepsk remote-id=fqdn:some.name

[client] > ip ipsec profile add dh-group=modp2048 enc-algorithm=aes-128 name ike1-site2
[client] > ip ipsec proposal add enc-algorithms=aes-128-cbc name=ike1-site2 pfs-group=modp2048
[client] > ip ipsec peer add address=a.b.c.d/32 name=ike1-site2 profile=ike1-site2 exchange-mode=ike2 send-initial-contact=no
[client] > ip ipsec identity add peer=ike1-site2 secret=thisisnotasecurepsk my-id=fqdn:some.name

At this point there is no tunnel created (yet). I think I also need to set the policy, am I right?
Are the above configuration correct until this point?

Note: on isp_server, UDP4500 is forwarded to server (MTik).

Thank you!

13

I don’t remember whether peers connect if no policy refers to them, but I think they should. I forgot to tell you that you have to set passive=yes at the peer which has address=0.0.0.0/0. This may cause some error (you cannot initiate connection to 0.0.0.0/0) so the peer may be inactive.


Yes, but there is more to the policies.

As you’ve decided to set up your own proposals instead of using the default ones, you’ll have to add something more at the responder (server) side:
/ip ipsec policy group add name=ike1-site2
/ip ipsec policy add group=ike1-site2 template=yes proposal=ike1-site2

On the initiator (client) side, create some policy not conflicting with any address you use for starters, something like src-address=1.2.3.4 dst-address=5.6.7.8 peer=ike1-site2 tunnel=yes proposal=. On the responder (server) side, add generate-policy=port-strict policy-template-group=ike1-site2 to the identity, so that the responder would generate a mirror policy to what the initiator suggests its own one from the template with the proper proposal.

If these policies get up at both ends, the IPsec setup as such works, and you can start setting up the auxiliary addresses for the EoIP tunnels, and policies for interconnecting these addresses. The policy above is intended only to test the IPsec without breaking anything.

14

Okay, thank you, I’m here now:

On the server, passive=yes was set (I think that was automatical).

“On the responder (server) side, add generate-policy=port-strict policy-template-group=ike1-site2 to the identity” - OK, done this.

Moreover did these on the server:

[server] > ip ipsec policy group add name=ike1-site2
[server] > ip ipsec policy add group=ike1-site2 template=yes proposal=ike1-site2

On the client I’ve done this:

[client] > ip ipsec policy group add name=ike1-site2
[client] > ip ipsec policy add group=ike1-site2 src-address=1.2.3.4 dst-address=5.6.7.8 peer=ike1-site2 tunnel=yes proposal=ike1-site2

Also I did set policy-template-group=ike1-site2 on the client’s identity (it was set to defaults before).

I think the tunnel is still not running (there are no active peers on either MTiks).

Those src and dst addresses needs to be in MTik’s LAN range? Or they can be random (they are random now).
Server and client MTik has different subnets for their LAN. Don’t really know if this is important now though.

15

Don’t I need to add accept rule for UDP4500 on input chain (server side)?

16

Maybe. Mikrotik keeps improving things, I may have missed that.


The policy groups are relevant only where policies are generated from templates, so on client side adding them does no harm but is useless.


Stupid question - are incoming connections to UDP port 4500 permitted in chain=input of /ip firewall filter at the server Tik?


The idea was to first try with a policy which matches on addresses which don’t exist in the Mikrotiks, so that it wouldn’t steal any traffic by mistake (IPsec policies override even routing to connected subnets).
Since there are different LAN subnets (which is in general good), you can use a policy interconnecting these subnets if you don’t need to set up any firewall rules between them; if you do, let’s skip it for the moment, we’ll eventually add the firewall rules first and policies next.

17

Okay, that was the problem, accepting 4500UDP @server Tik eventually let the IPsec to stand up.

One quick question: now that I opened UDP4500, anyone knows my IP (and preshared key + sets the correct ID for identity) can create an IPsec tunnel to my MT (As long as server’s identity is set for matching by remote id)?

Also, there is a new policy on the server (responder): <5.6.7.8:0->1.2.3.4:0>, it seems readonly, I guess this is good and valid and comes from the initiator.

So now if I’d create some nat rules I could access responder’s lan from initiator, right?

Thanks very much. How shall I go forward? Firewall rules or EoIP tunnel(s)?

18

Correct.


Correct. It proves that the complete setup works, and you can now replace this test policy at initiator by some more useful one(s).


Yes, depending on the existing setup, some exceptions from src-nat/masquerade may be necessary to make the added policy client.lan.subnet->server.lan.subnet see the packets. But you don’t need this policy if you take the EoIP over IPsec way.


I think the initial idea was to first check whether UDP (IPsec transport packets) is also throttled?

But it’s up to you of course.
To transport the EoIP using the IPsec, I’d recommend to create, at the initiator, an /interface bridge without any member ports, attach to it a private /32 IP address outside the LAN subnets of both the initiator and the responder, and create a policy with this address as src-address and the responder’s WAN IP as dst-address. A mirror policy will be added at the responder. It should not be necessary to disable and re-enable the peers to that the additional policy would come up.

Once you can see the policy up, add an action=accept protocol=gre ipsec-policy=in,ipsec rule to chain=input of /ip firewall filter at both machines, and it must be before the “drop invalid” one (there’s currently a bug regarding GRE handling in firewall; EoIP uses GRE protocol). Once this is done, you can add the EoIP interfaces, with properly set local and remote addresses, at both ends (the tunnel-id must be the same at both ends). You can then attach IP addresses from the same subnet to their ends, and you should be able to use the IP address of the far end as a gateway for a route to the far end LAN subnet (symmetrically at both machines of course) so that you could test the TCP client/server connection this way, and check the speed.

The EoIP interfaces are interfaces like any other, so depending on how the firewall looks like, you may have to add permissive rules to chain=forward of /ip firewall filter at both ends to make the client-server connection possible.

If the speed is OK, that’s it; but since it wasn’t OK using PPTP, I assume UDP will be throttled too, so the next step would be to add a second IPsec tunnel using a different UDP port on at least one end.

Out of curiosity, can you reveal what countries are we talking about?

19

What is this bridge used for? (I gave it the 10.0.0.1/32 address)


What shall be the local and remote address of EoIPs at initiator and responder?

Now my policies look like this:

initator: 10.0.0.1 -> WAN IP of responder (192.168.100.100)
responder (mirrored): WAN IP of responder (192.168.100.100) -> 10.0.0.1

I tried adding EoIPs like this:

initiator: local IP=10.0.0.1, remote IP=192.168.100.100
responder: local IP=192.168.100.100, remote IP=10.0.0.1

Then attached one IP of local subnets to both of them respectively, then tried to ping from one Tik the other, but it did not go through.
Some firewall rules might still be needed, could you please confirm those upper questions for me?

Thank you!

20

Just for clarity of the configuration. It is actually not important to which particular interface a “virtual” IP address is attached, but if you attach it to a dedicated interface which is always up because it inherits nothing from any physical one, it becomes always up no matter what, plus it is clearly logically separated from the other interfaces. And an empty bridge is the only L2 virtual interface you can create and it stays up.

If you ever get to using several EoIP tunnels in parallel, you’ll need one such address at initiator side for each tunnel, but all of them may be configured on the same interface.


This is correct.


This is wrong. Imagine the two EoIP interfaces as two Ethernet interfaces interconnected using a cable, so you have to assign to each of them an address from the same subnet so that they could ping each other without any route. A /30 is enough, but it must not collide with any other subnet used at either site (not just machine).