Problem with RBM33g and two independent IPsec tunnels [SOLVED]

Allow me to ask you for help. I need to create two independent IPsec tunnels between RBM33g with two LTE modules with dynamic natted IPs and Strongswan server with two IPs:

RBM33g + 2x LTE modules: lte1 - WAN20: dynamic natted IP, lte2 - WAN21: dynamic natted IP

Strongswan server: WAN1, WAN2: static IPs

WAN20…WAN1
10.20.10.2/32 <======================================> 10.20.10.1/32

WAN21… WAN2
10.21.10.2/32 <======================================> 10.21.10.1/32

On lte1 and lte2 interfaces I have the DHCP clients with the following scripts:


    :local rmark "WAN21"
    :local count [/ip route print count-only where comment="WAN21"]
    :if ($bound=1) do={
        :if ($count = 0) do={
            /ip route add gateway=$"gateway-address" comment="WAN21" routing-mark=$rmark vrf-interface=lte2
            /ip ipsec peer set [find comment=lte2] local-address=$"lease-address"
        } else={
            :if ($count = 1) do={
                :local test [/ip route find where comment="WAN21"]
                :if ([/ip route get $test gateway] != $"gateway-address") do={
                    /ip route set $test gateway=$"gateway-address"
                }
            } else={
                :error "Multiple routes found"
            }
        }
    } else={
        /ip route remove [find comment="WAN21"]
    }
}

and

{
    :local rmark "WAN20"
    :local count [/ip route print count-only where comment="WAN20"]
    :if ($bound=1) do={
        :if ($count = 0) do={
            /ip route add gateway=$"gateway-address" comment="WAN20" routing-mark=$rmark vrf-interface=lte1
            /ip ipsec peer set [find comment=lte1] local-address=$"lease-address"
        } else={
            :if ($count = 1) do={
                :local test [/ip route find where comment="WAN20"]
                :if ([/ip route get $test gateway] != $"gateway-address") do={
                    /ip route set $test gateway=$"gateway-address"
                }
            } else={
                :error "Multiple routes found"
            }
        }
    } else={
        /ip route remove [find comment="WAN20"]
    }
}

The mangle rules are the following:


[admin@MikroTik] > ip firewall mangle print
Flags: X - disabled, I - invalid, D - dynamic 
 0    chain=output action=mark-connection new-connection-mark=ipsec20 passthrough=yes protocol=udp 
      dst-address=87.236.194.190 dst-port=500,4500 log=no log-prefix="" 
 1    chain=output action=mark-connection new-connection-mark=ipsec20 passthrough=yes 
      protocol=ipsec-esp dst-address=87.236.194.190 log=no log-prefix="" 
 2    chain=output action=mark-routing new-routing-mark=WAN20 passthrough=no connection-mark=ipsec20 
      log=no log-prefix="" 
 3    chain=output action=mark-connection new-connection-mark=ipsec21 passthrough=yes protocol=udp 
      dst-address=87.236.194.191 dst-port=500,4500 
 4    chain=output action=mark-connection new-connection-mark=ipsec21 passthrough=yes 
      protocol=ipsec-esp dst-address=87.236.194.191 
 5    chain=output action=mark-routing new-routing-mark=WAN21 passthrough=no connection-mark=ipsec21
        log=no log-prefix=""

After LTE connections are established the DHCP states are the following:


[admin@MikroTik] > ip dhcp-client print detail
Flags: X - disabled, I - invalid, D - dynamic 
 0   ;;; lte2
     interface=lte2 add-default-route=no use-peer-dns=yes use-peer-ntp=yes 
     dhcp-options=hostname,clientid 
     status=bound address=100.89.166.102/30 gateway=100.89.166.101 dhcp-server=100.89.166.101 
     primary-dns=194.228.211.33 secondary-dns=160.218.161.60 expires-after=5d22h57m15s 
 1   ;;; lte1
     interface=lte1 add-default-route=no use-peer-dns=yes use-peer-ntp=yes 
     dhcp-options=hostname,clientid 
     status=bound address=100.126.31.94/30 gateway=100.126.31.93 dhcp-server=100.126.31.93 
     primary-dns=93.153.117.49 secondary-dns=93.153.117.17 expires-after=5d22h54m6s

Created routes are:


[admin@MikroTik] > ip route print detail
Flags: X - disabled, A - active, D - dynamic, 
C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, 
B - blackhole, U - unreachable, P - prohibit 
 0 A S  ;;; WAN20
        dst-address=0.0.0.0/0 gateway=100.126.31.93 gateway-status=100.126.31.93 reachable via  lte1 
        distance=1 scope=30 target-scope=10 routing-mark=WAN20 vrf-interface=lte1 
 1 A S  ;;; WAN21
        dst-address=0.0.0.0/0 gateway=100.89.166.101 
        gateway-status=100.89.166.101 reachable via  lte2 distance=1 scope=30 target-scope=10 
        routing-mark=WAN21 vrf-interface=lte2 
 2 A S  dst-address=0.0.0.0/0 gateway=lte2 gateway-status=lte2 reachable distance=1 scope=30 
        target-scope=10 
 3   S  dst-address=0.0.0.0/0 gateway=lte1 gateway-status=lte1 reachable distance=1 scope=30 
        target-scope=10 
 4  DC  dst-address=10.20.10.0/24 pref-src=10.20.10.2 gateway=ether1 gateway-status=ether1 unreachable 
        distance=255 scope=10 
 5  DC  dst-address=10.21.10.0/24 pref-src=10.21.10.2 gateway=ether2 gateway-status=ether2 unreachable 
        distance=255 scope=10 
 6 ADC  dst-address=100.89.166.100/30 pref-src=100.89.166.102 gateway=lte2 
        gateway-status=lte2 reachable distance=0 scope=10 
 7 ADC  dst-address=100.126.31.92/30 pref-src=100.126.31.94 gateway=lte1 gateway-status=lte1 reachable 
        distance=0 scope=10

Regarding the established IPsec tunnels the situation is the following:


 
[admin@MikroTik] > ip ipsec policy print detail      
Flags: T - template, X - disabled, D - dynamic, I - invalid, A - active, * - default 
 0 T * group=default src-address=::/0 dst-address=::/0 protocol=all proposal=default template=yes 
 1  A  ;;; lte1
       peer=peer1 tunnel=yes src-address=10.20.10.2/32 src-port=any dst-address=10.20.10.1/32 dst-port=any protocol=all action=encrypt level=require ipsec-protocols=esp 
       sa-src-address=100.126.31.94 sa-dst-address=87.236.194.190 proposal=default ph2-count=1 
 2  A  peer=peer2 tunnel=yes src-address=10.21.10.2/32 src-port=any dst-address=10.21.10.1/32 dst-port=any protocol=all action=encrypt level=require ipsec-protocols=esp 
       sa-src-address=100.89.166.102 sa-dst-address=87.236.194.191 proposal=default ph2-count=1
       
[admin@MikroTik] > ip ipsec peer print detail        
Flags: X - disabled, D - dynamic, R - responder 
 0     ;;; lte2
       name="peer2" address=87.236.194.191/32 local-address=100.89.166.102 profile=default exchange-mode=ike2 send-initial-contact=yes 
 1     ;;; lte1
       name="peer1" address=87.236.194.190/32 local-address=100.126.31.94 profile=default exchange-mode=ike2 send-initial-contact=yes 
       
[admin@MikroTik] > ip ipsec active-peer print detail
Flags: R - responder, N - natt-peer 
 0 R  id="87.236.194.190" local-address=100.126.31.94 port=4500 remote-address=87.236.194.190 port=4500 state=established side=responder uptime=1h27m38s last-seen=1m7s 
      ph2-total=1 
 1 R  id="87.236.194.191" local-address=100.89.166.102 port=4500 remote-address=87.236.194.191 port=4500 state=established side=responder uptime=1h24m30s last-seen=1m51s 
      ph2-total=1
      
[admin@MikroTik] > ip ipsec installed-sa print detail
Flags: H - hw-aead, A - AH, E - ESP 
 0  E spi=0x4794381 src-address=87.236.194.190:4500 dst-address=100.126.31.94:4500 state=mature auth-algorithm=sha1 enc-algorithm=null 
      auth-key="9099f83ec63f6a7f6e7651f372fdb765c536b83b" enc-key="" addtime=aug/21/2019 11:23:52 expires-in=13m37s add-lifetime=24m9s/30m12s current-bytes=132710 
      current-packets=1981 replay=128 
 1  E spi=0xC1442435 src-address=100.126.31.94:4500 dst-address=87.236.194.190:4500 state=mature auth-algorithm=sha1 enc-algorithm=null 
      auth-key="87fbc5bda95894df11024e1263a469c3708d5a28" enc-key="" addtime=aug/21/2019 11:23:52 expires-in=13m37s add-lifetime=24m9s/30m12s current-bytes=133112 
      current-packets=1987 replay=128 
 2  E spi=0x721BBAC src-address=87.236.194.191:4500 dst-address=100.89.166.102:4500 state=mature auth-algorithm=sha1 enc-algorithm=null 
      auth-key="d779407fab6aaf8d03207291ad73a47b14a890e7" enc-key="" addtime=aug/21/2019 11:27:03 expires-in=16m49s add-lifetime=24m10s/30m13s current-bytes=106530 
      current-packets=1590 replay=128 
 3  E spi=0xCCE212D1 src-address=100.89.166.102:4500 dst-address=87.236.194.191:4500 state=mature auth-algorithm=sha1 enc-algorithm=null 
      auth-key="9e4447eb829256d786855ef4d33c066f3c08691b" enc-key="" addtime=aug/21/2019 11:27:03 expires-in=16m49s add-lifetime=24m10s/30m13s current-bytes=107602 
      current-packets=1606 replay=128

From server side I can ping 10.20.10.2 and 10.21.10.2 IPs without problem and pings are going through appropriate tunnels - the following connections are created:


[admin@MikroTik] > ip firewall connection print detail
Flags: E - expected, S - seen-reply, A - assured, C - confirmed, D - dying, F - fasttrack, s - srcnat, d - dstnat 
 2  S C     protocol=icmp src-address=10.20.10.1 dst-address=10.20.10.2 reply-src-address=10.20.10.2 reply-dst-address=10.20.10.1 icmp-type=8 icmp-code=0 icmp-id=11224 
            timeout=9s orig-packets=51 orig-bytes=4 284 orig-fasttrack-packets=0 orig-fasttrack-bytes=0 repl-packets=51 repl-bytes=4 284 repl-fasttrack-packets=0 
            repl-fasttrack-bytes=0 orig-rate=1344bps repl-rate=1344bps 
 3  S C     protocol=icmp src-address=10.21.10.1 dst-address=10.21.10.2 reply-src-address=10.21.10.2 reply-dst-address=10.21.10.1 icmp-type=8 icmp-code=0 icmp-id=11226 
            timeout=9s orig-packets=34 orig-bytes=2 856 orig-fasttrack-packets=0 orig-fasttrack-bytes=0 repl-packets=34 repl-bytes=2 856 repl-fasttrack-packets=0 
            repl-fasttrack-bytes=0 orig-rate=672bps repl-rate=672bps

My problem is that I am not able to ping server from RMB33g side. When I ping the 10.20.10.1 and 10.21.10.1 from router side the following connections are created:


[admin@MikroTik] > ip firewall connection print detail
Flags: E - expected, S - seen-reply, A - assured, C - confirmed, D - dying, F - fasttrack, s - srcnat, d - dstnat 
 2    C     protocol=icmp src-address=100.89.166.102 dst-address=10.20.10.1 reply-src-address=10.20.10.1 reply-dst-address=100.89.166.102 icmp-type=8 icmp-code=0 
            icmp-id=10753 timeout=9s orig-packets=52 orig-bytes=2 600 orig-fasttrack-packets=0 orig-fasttrack-bytes=0 repl-packets=0 repl-bytes=0 
            repl-fasttrack-packets=0 repl-fasttrack-bytes=0 orig-rate=400bps repl-rate=0bps 
 3    C     protocol=icmp src-address=100.89.166.102 dst-address=10.21.10.1 reply-src-address=10.21.10.1 reply-dst-address=100.89.166.102 icmp-type=8 icmp-code=0 
            icmp-id=10753 timeout=9s orig-packets=37 orig-bytes=1 850 orig-fasttrack-packets=0 orig-fasttrack-bytes=0 repl-packets=0 repl-bytes=0 
            repl-fasttrack-packets=0 repl-fasttrack-bytes=0 orig-rate=400bps repl-rate=0bps

So, icmp src-address and reply-dst-addreses are incorrect for both pings they are the same and it is IP address of lte2 interface form DHCP instead of right IPs. I suppose it is connected with routing and mangling but unfortunately I am not experienced on these fields :frowning: For me is interesting that when I am pinging server from router with ping source address option all works well and the right connections are created


[admin@MikroTik] > ping 10.20.10.1 src-address=10.20.10.2
  SEQ HOST                                     SIZE TTL TIME  STATUS                                                                                                      
    0 10.20.10.1                                 56  64 32ms 
    1 10.20.10.1                                 56  64 37ms 
    2 10.20.10.1                                 56  64 36ms 
    sent=3 received=3 packet-loss=0% min-rtt=32ms avg-rtt=35ms max-rtt=37ms
     
[admin@MikroTik] > ping 10.21.10.1 src-address=10.21.10.2  
  SEQ HOST                                     SIZE TTL TIME  STATUS                                                                                                      
    0 10.21.10.1                                 56  64 35ms 
    1 10.21.10.1                                 56  64 36ms 
    2 10.21.10.1                                 56  64 36ms 
    sent=3 received=3 packet-loss=0% min-rtt=35ms avg-rtt=35ms max-rtt=36ms 

[admin@MikroTik] > ip firewall connection print detail   
Flags: E - expected, S - seen-reply, A - assured, C - confirmed, D - dying, F - fasttrack, s - srcnat, d - dstnat 
 2  S C     protocol=icmp src-address=10.20.10.2 dst-address=10.20.10.1 reply-src-address=10.20.10.1 reply-dst-address=10.20.10.2 icmp-type=8 icmp-code=0 icmp-id=11777 
            timeout=9s orig-packets=60 orig-bytes=3 000 orig-fasttrack-packets=0 orig-fasttrack-bytes=0 repl-packets=60 repl-bytes=3 000 repl-fasttrack-packets=0 
            repl-fasttrack-bytes=0 orig-rate=800bps repl-rate=800bps 
 3  S C     protocol=icmp src-address=10.21.10.2 dst-address=10.21.10.1 reply-src-address=10.21.10.1 reply-dst-address=10.21.10.2 icmp-type=8 icmp-code=0 icmp-id=11777 
            timeout=9s orig-packets=18 orig-bytes=900 orig-fasttrack-packets=0 orig-fasttrack-bytes=0 repl-packets=18 repl-bytes=900 repl-fasttrack-packets=0 
            repl-fasttrack-bytes=0 orig-rate=800bps repl-rate=400bps

I hope with your help I will be able to modify my misconfigured router to achieve full bidirectional traffic in both IPsec tunnels :sunglasses:

For your help thank you in advance.

Unfortunately till present I have obtained no reply. So allow me to ask you the following simple query:

Is it possible to create two independent IPsec/IKEv2 tunnels with full bidirectional traffic between

LAN1 <=> (WAN1 dynamic, natted IP) <==> (WAN10, public, static IP) <=> LAN10
LAN2 <=> (WAN2 dynamic, nattedI P) <==> (WAN10, public, static IP) <=> LAN10

on one routerboard with two LTE modules? Certainly it is not a problem with using two routerboards and every routerboard with one dynamic natted WAN (LTE module) but when I tried to do it with one board with two LTE modules I have been unsuccessful the result is described in my previous post.

Thank you in advance.

It’s too late down here so that I could analyse it completely, however as you can ping when you set the src-address manually, you can as well add dedicate routes with pref-src to do this automatically for any traffic towards the remote IPs:

/ip route
add dst-address=10.20.10.1/32 pref-src=10.20.10.2/32 gateway=lte1
add dst-address=10.21.10.1/32 pref-src=10.21.10.2/32 gateway=lte1

The actual gateway in these routes is not important as the IPsec policy will steal the packet anyway, the packets just have to be routed “somewhere” so that the IPsec policy could intercept them.

Dear sindy, with your routes all is working like a charm and I can continue with my experiment with true bonding on four LTE channels :slight_smile:

Here is the short video of my experimental equipment :smiley: :sunglasses: It consists of 4xRBM33g, 1xhEX ,1xRB260GS switch and is connected to Strongswan server via 4 EoIP tunnels and 5 IPsec/IKEv2 tunnels. To this equipment are connected two cameras and PC is connected to hEX and via another IPsec tunnel to Strongswan server with RADIUS as a client. Bonding mode is balance-rr with ARP monitoring

https://www.dropbox.com/s/bdpl2jhqpdwbx6a/img_3419.mov?dl=0

Now I can use only two RBM33g boards instead of four :slight_smile: My only problem now are the EoIP tunnels that are unreliable and sometimes must be reset on server side. In my opinion this is because the EoIP protocol is not implemented in Linux kernel and I have to use some implementation of this protocol from github https://github.com/Nat-Lab/eoip . It is a pitty that the GRETAP protocol that supports Ethernet over GRE and is implemented in Linux kernel since 2.6.28 is not implemented in ROS. Because RouterOS is Linux based, this feature is probably already in RouterOS kernel, so it should be very easy for Mikrotik developers to implement it and it will be very helpfull to make ethernet bridges between Linux and RouterOS.

Many thanks for your help once more.