Fasttrack encypted connections the Piggyback way (test)

RouterOS General
1

I have been bussy with IKEv2 connections the last few days and now all is working I was disappointed the my RB760iGS only managed to do 70-90 Mbit/s due to networking an firewalling task being taking all the CPU of Core 0 while the others are almost idling.

I am thinking and going to setup in a moment a two RB760iGS in line where my current one will doe the PPoE connection to my ISP and the encypted traffic (IKEv2) and my spare/backup one will do the tasks of my current one except for the PPPoE and encypting. It will do the rest what my current one did.

What do I want to archive is that can use Fasttracking again so that routed traffic to outside and back is fast and that the spare router takes care of being a helper.

I need some help because routing is a not a strong point of me…yet. The IKEv2 connections get routed to the PPPEoE connection by default and have each a own dynamic internal address in the 10.x.x.x range. So I am thinking to have the PPPoE/encypting router also the 10.0.0.0/8 or /9 as internal network to have no NAT needed on encryption and only the PPPoE is Natted.

Having one big network can give problems because all the exit points of the IKEv2 can see each other.

Any pointer/help to make this a easy as possible and of course a faster encryption without the network/firewall slowing it down?

2

Sorry to disappoint you, but the sad truth is that the firewall processing and the matching to traffic selectors of IPsec policies are tightly linked together. So it is possible to selectively exclude only the traffic which needs to get encrypted from fasttracking, so that the rest could still be fasttracked, but if you want almost all your traffic to go via IPsec, selective fasttracking doesn’t help you much.

The fact that the transport packets will be fasttracked on another device doesn’t make them be handled faster on the one which encrypts them, so you would not gain much. What might make some sense is to run the PPPoE client on one device and the encryption on the other one.

3

Thanks Sindy and I was this afternoon ofline to test it so I did not see your reply earlier. I had the PPPoE running and changed my settings but I could not get any traffic to the “PPPoE” router so I still know nothing. I had to discover that you have to use a bridge to even have an IP on ether2 visible to the other side so that took too much time.

So why my assumption? This because when I run the IKEv2 client on my computer through the router I get about 150Mbits and I just tried with Fastracking on UDP4500 and then I got about 200Mbps through the RB750iGS. So I could double or even triple my speed by splitting up the networking/fire-walling from the encrypting and PPPoE.

Later this week I have an other go and step one is just the PPPoE separate and if that works starting IKEv2 connection starting a one and working up to four a five at the same time.

I have to first find out how just get a working connection between the two routers. I create a 10.0.0.2 on ether1 and changed the NAT-SRC to that IP and completed the routing to the 10.0.0.0 network. The PPPoE router had on ether2 10.0.0.1 and that started to work when I created a bridge. The PPPoE goes over a fiber connect the SFP.

4

If your IKEv2 client is running on the PC, the UDP transport of the encrypted data becomes a plaintext transit traffic for the router connecting that PC to the rest of the world, so fasttracking that traffic makes sense if the router doesn’t have enough CPU to handle the forwarding and firewalling.

If your IKEv2 client is running on the router itself, the UDP transport of the encrypted data also becomes a plaintext traffic for the router, but it is a locally originated&terminated one, not a transit one, so it cannot be fasttracked on that same router.

Maybe better attach a drawing so that it becomes clear what you want to set up and what will be the roles of the devices in the chain.

5

Sigh…so after a lot of time spending to learn how to setup routing and understanding how I can provide a fixed src-nat address I have it working.

On the router (one) having the PPPoE connection now is also the IKEV2 connections and the automatic src-nat generated by mode-config using an address list works.

I am using now the dreaded ‘double NAT’ and on router 1 the generated src-nat for IKEv2. On router 2 sits filter/nat/mangle/raw. Using now an ABC range of addresses, C for internal network, A for router-router communication and B to encypt traffic routing.

My connection speed increased from about 60-70Mbits to 120-130 Mbits by and load on one core on router 1 of 100% due to networking…NAT and PPPoE
Router 2 runs a load of about 40% so a HUGE improvement to the 99% before.

On router 2 I do in Mangle splitting up the traffic for the IKv2 connection in several Routing Mark’s and then Nat I set the src-address needed for the IKEv2 src-address Natting on Router 1. In IP-Route I made several lines and created for each IKv2 connection a address in IP-Address.

You need two routers in cascade for this. I have looked at Bridge but then he IKEv2 would still be running on router 2.

So now first some backups of the configs so that can recreate the setup swiftly.

I thought first to us action=route but I could not get that working.

update: I have now taken away the switch I had to use to be able to connect to both routers at the same time through Winbox and the speed gained a other 10Mbits and is now steady on 150Mbits. The CPU load is very nice split over the processor cores and the all get about 50% load at that speed and for a 760iGS that is not fast-tracked traffic.

6

OK, so I have completely misunderstood your intentions how to distribute the tasks between the boxes from the OP. Thumbs up for the idea. I nevertheless wonder whether a loop through the same router using an IPIP tunnel, where one pass through the firewall would only care about IPsec policy matching and the other pass would only care about the actual firewalling (so fasttracking could be used), would spread the load between the two CPU threads or still one thread would be doing everything and the rest would be idling.

7

Well, that suggestion was relevant in the context of one CPU thread being loaded at 100 % and the others idling as you’ve stated here, not the whole machine running at 100 % as you’ve stated in the 6.45 beta topic. If we talk about 100 % of all CPUs, organizing the 2nd pass through the firewall makes no sense of course. If it’s still true that you have some idling CPU threads and you want to let them do the job of the other router, the instruction how to set up the IPIP “loopback” tunnel is here in chapter Implementation Details.

But the key question is whether the IPIP transport packets are handed over via some RAM queue so their “transmission” and “reception” can be handled by different threads, or whether the same thread will always handle both ends of the tunnel. So it may happen that you build the tunnel and find out that all the load is still concentrated in the same CPU thread.

8

I tried to build IPIP on the single router but I did not manage to get it working. The example in the wiki seems to not do what I see on my router.Thanks for the link and I will see if that is working. I already overheated my brain serveral time in the past week.

If I can get it to work then Mikrotik can implement in a client so sharing load on CPU and use several connections at the same time.

Thanks for your help in this.

9

hmmm reading it I going to put it in a third blank. RB750Gr2 to see how it works and my live boxes are to complicated now to fit in in one time.

During testing IPIP I noticed that in connections only one line appeared of the four expected that stated the searched dynamic IP of the IKEv2 connection which matches the the marker src-address in mode config.
I thought success but I could not pass any traffic looking in torch.

10

I have made ‘profile’ screenshots when the router(s) are loaded and doing encrypting:

no picture present

IKEv2 in cascade setup of a box doing the PPPoE and IKEv2. There i NAT running on the box for the IKEv2.

no picture present

At the same time the other router doing filtering/nat/mangle/raw

no picture present

11

The width of the lane between the switch chip and the CPU is not limiting you if all the WAN traffic (which in your test case means the encrypted traffic) flows through the SFP and all the LAN (non-encrypted) one through the electrical ports - you cannot push more than 1 Gbit/s through the SFP lane so the 1 Gbit/s of the remaining lane to the switch chip is sufficient. Similarly, if you use a single electrical port as WAN and the SFP is not installed, you can use half of the aggregate bandwidth of the lanes for WAN and the other half for LAN. As soon as you start thinking about distributing traffic among multiple gigabit WAN ports, the lane capacity always limits you severely.

Looking at the CPU loads: the summary CPU load of the “outer” machine (PPPoE + encryption) is 188,5 %, and the summary CPU load of the “inner” machine (firewall optimised using fasttracking) is 93,5 %. So theoretically there is still some “space” left even if all the tasks from the inner router migrate to the outer machine, so it is worth trying whether the same packet will be processed by different CPU thread before and after the pass through the IPIP tunnel.

12

I had not yet used Fasttracking and the next two are 'profiles’fastracked using outer (GW) and inner (filter/nat/mangle/raw).

Inner fasttracked:

no picture present

And for comparison non encrypting:

no picture present]

And as second comparison non encrypting on a standalone router with PPPoE:

no picture present

13

I have put it into my router and I don’t know where to enter the entry point of the IKEv2 server. The IP I get is 10.40.8.178 as example. The IP 1.2.3.4 addresses in NAT I replaced by the IP address used as gateway of the PPPoE.

I don’t any traffic going through the three NAT lines of you. BTW I get an extra NAT line if I use mode config if that points to an address list with IP addresses. That NAT line will contain the entry point of the IKEv2 and the src-address with match it.

14

It sounds as if you’ve copied too much of the L2TP server side setup - in fact it is only the IPIP tunnel which needs to be copied. I’ll have a look later this week, it just dawned on me that in order that the whole setup made sense, the src-nat handling must be done in the pass through the firewall where the fasttracking is active, but all those src-nated packets which evade the fasttracking will be stolen by the IPsec policy already during that pass. So we will end up with 1 % of packets being stolen before making it to the IPIP tunnel and the rest after passing through it.

15

Thanks looking forward to it.

With IKEv2 I need to know the which IP entry point is given and ROS knows it but having no script on start / change I can’t automate it. When using mode config + address list I get a NAT line at the top src-natting the new address of the entry point for encrypting. If not set in mode config then no NAT line is generated.

Having your setup in the router, I tested the plain throughput and saw that I had 100Mbits higher speed when using no fast tracking (250–>350Mbits) and when using fast tracking the other cores got also loaded.

16

I’m in a state of light shock… I’ve reset a hAP ac² to defaults and built an IKEv2 IPsec peer & identity, and on the remote peer, I’ve created an identity referring to a mode-config with address=192.168.209.1 and split-include=0.0.0.0/0. So at the initiator side, it looks as follows:

[me@MyTik] > ip ipsec export
...
/ip ipsec mode-config
set [ find default=yes ] src-address-list=mode-config-list
/ip ipsec peer
add address=192.168.10.84/32 exchange-mode=ike2 name=peer1
/ip ipsec identity
# Suggestion to use stronger pre-shared key or different authentication method
add generate-policy=port-strict mode-config=request-only peer=peer1 secret=averysecureone
[me@MyTik] > ip firewall address-list print
Flags: X - disabled, D - dynamic
 #   LIST                            ADDRESS                                              CREATION-TIME        TIMEOUT
 0   mode-config-list                192.168.99.0/24                                      may/27/2019 21:04:34
[me@MyTik] > ip address print
Flags: X - disabled, I - invalid, D - dynamic
 #   ADDRESS            NETWORK         INTERFACE
 0   ;;; defconf
     192.168.99.1/24    192.168.99.0    bridge
 1 D 192.168.10.83/24   192.168.10.0    ether1
 2 D 192.168.209.1/32   192.168.209.1   ether1
[me@MyTik] > ip ipsec policy print
Flags: T - template, X - disabled, D - dynamic, I - invalid, A - active, * - default
 0 T * group=default src-address=::/0 dst-address=::/0 protocol=all proposal=default template=yes

 1  DA  src-address=192.168.209.1/32 src-port=any dst-address=0.0.0.0/0 dst-port=any protocol=all action=encrypt level=unique
       ipsec-protocols=esp tunnel=yes sa-src-address=192.168.10.83 sa-dst-address=192.168.10.84 proposal=default ph2-count=1

I have intentionally not changed the default firewall rules, so the chain=forward action=fasttrack-connection in /ip firewall filter was there. So I’ve expected to see the usual behaviour where every Nth packet of a fasttracked connection will not be fasttracked, and these Nth packets will be caught by the IPsec policy and delivered, while the actually fasttracked ones will take the default route to nowhere (if I disable the peer, the connected laptop in 192.168.99.0/24 can get nowhere as the default route’s gateway is the LAN bridge) as they will miss the IPsec policy due to fasttracking.

To my surprise, the packet count in fasttracked connections at the “client” side is equal to the one at the “server” side. So even actually fasttracked packets get caught by the policy and delivered via the SA:

client:

[me@MyTik] > ip firewall connection print where dst-address~"216.58.201.78:443"
Flags: E - expected, S - seen-reply, A - assured, C - confirmed, D - dying, F - fasttrack, s - srcnat, d - dstnat
 #          PR.. SRC-ADDRESS           DST-ADDRESS           TCP-STATE   TIMEOUT     ORIG-RATE REPL-RATE ORIG-PACKETS REPL-PACKETS
 0  SAC Fs  tcp  192.168.99.254:57468  216.58.201.78:443     established 23h59m46s        0bps      0bps        2 654        4 019

server (where fasttracking is off):

[me@vpn-server] > ip firewall connection print where dst-address~"216.58.201.78:443"
Flags: E - expected, S - seen-reply, A - assured, C - confirmed, D - dying, F - fasttrack, s - srcnat, d - dstnat
 #          PR.. SRC-ADDRESS           DST-ADDRESS           TCP-STATE   TIMEOUT     ORIG-RATE REPL-RATE ORIG-PACKETS REPL-PACKETS
 0  SAC  s  tcp  192.168.209.1:57468   216.58.201.78:443     established 23h59m43s        0bps      0bps        2 648        4 019

So given that even the actually fasttracked packets are caught by the IPsec policy, there is no way to separate the “real” firewall processing from the IPsec policy matching by insertion of an IPIP hop between the two, therefore there is no way to eventually split these two stages of processing of the same packet which would eventually lead to each stage to be executed by another CPU core/thread.

I wonder whether it is an intentional change in 6.44 or something else, and what’s the impact on performance.

So what remains at your side is to replicate this setup and see the difference in throughput between fasttracking off and fasttracking on when PPPoE, IPsec and firewalling is done on a single machine. My home uplink of 20 Mbit/s is not suitable for any stress testing.

BTW, I suspect that the fact that the IPsec processing is stuck to one thread/core is intentional. The receiving side is not happy when the packets arrive in swapped order, and to ensure that they don’t, all of them have to be processed by the same thread on the sending side. So it is well possible that with several SAs you’d have a higher overall throughput, but your application scenario doesn’t leave any space for several SAs.

17

Thanks Sindy and it a pity that it did not work as expected.

Did you try using different ports as you control the client and the server? When I use IKEv2 I don’t activate notracking for now.

Tested it with one active IKEv2 connection active and still one core was loaded up…general observation is that the policy on download is to use one core and when it hits 100% to use more cores, when when uploading it seems to switch earlier to more cores/threads. This occurs on the PPPoE and IKEv2 and probably also applying to other traffic/interfaces.

Going to run your config later to see how it goes.

This is running only one IKEv2 connection:
ikev-single-instance.JPG
This was uploading with IKEv2 and L2TP/IPSEC simultaneous and then the cores are more loaded:
ikev-l2tp.JPG

18

So after giving up on running it on router I returned to using two routers to be able to use Mangle + PCC to distribute traffic over several IKEv2 and L2TP/IPSEC connections. Also activated fastracking for the first NAT on the ‘inner’ router which was a bit of hustle. I had made a jump to two chains, one IKEv2 and the other for L2TP/IPSEC.

After trying different setups the one with one chain containing both VPN connections. First the L2TP/IPSEC which are connection marked at the top to have better overview in the connections screen. Then I mark the connection for IKEv2 and then I apply fasttrack to new connections. I use source as separator in PCC and in this setup all connections are kept neatly apart and there is no cross-overs between the two VPN’s. The L2TP would be killed as soon it was fasttracked and the IKEv2 does not suffer from that because it is processed at the second router (GW/PPPoE/IKEv2)

I disabled IPv6 for now, because I don’t use it and support in ROS for that is still in development.

The next step is to do away with the first NAT for IKEv2 and route it directly. There is a NAT the GW router because IKEv2 generates a dynamic NAT line for each IKEv2 connection.

Speed of the IKEv2 doubled by moving it to the GW router.