[L2TP+IPSec] Client to Client Connectivity

nomatter · Sun Oct 14, 2018 8:46 pm

Hello Everyone!

I have the following network with l2tp/ipsec

Server Router - 10.50.0.10
private address - 192.168.2.254

Client2 Router - 10.50.0.11
private address - 192.168.128.254

Client1 Router - 10.50.0.13
private address - 192.168.5.254

I have connectivity between each of the clients (and hosts behind them) and a server:
i can ping from 192.168.2.1 to 192.168.128.1 and vice versa
the same for other client 192.168.2.1 to 192.168.5.1 and vice versa

but i have no connectivity between clients: i can not ping 10.50.0.13 from 10.50.0.11 and vice versa.

I have configured routes on each router but it's not the cause.
Server

Code: Select all


 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE



 8 A S  ;;; PPP to client2  network

        192.168.5.0/24             10.50.0.10        10.50.0.13                1

 9 A S  ;;; PPP - to client1 network

        192.168.128.0/24         10.50.0.10        10.50.0.11                1

Client1

Code: Select all


 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE



 8 A S  ;;; PPP to server  network

        192.168.2.0/24             10.50.0.13        10.50.0.10                1

 9 A S  ;;; PPP - to client2 network

        192.168.128.0/24         10.50.0.13        10.50.0.10                1

Client2

Code: Select all


 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE



 8 A S  ;;; PPP to server  network

        192.168.2.0/24             10.50.0.11        10.50.0.10                1

 9 A S  ;;; PPP - to client1 network

        192.168.5.0/24         10.50.0.11        10.50.0.10                     1

10.50.0.XX are virtual addresses for the tunnels.
I have public IP only for the server. Clients are behind the ISP NAT.

One important detail here:
at first i have configured only l2tp (without ipsec) and everything worked: there was a connectivity between all of the routers and hosts behind them
(client2 <->client1, client2<->server, client1 <->server)
also there was a NAT rule on each router:
chain=srcnat action=masquerade out-interface=all-ppp log=no log-prefix=""

as part of ipsec configuration i've disabled that rule on each router.
Maybe this caused the issue. but I was told that this rule needs to be disabled and i should use NAT traversal checkbox when configuring IPsec peers.
so now i have client2<->server, client1 <->server but NO client2<->client1 connectivity

I'm definitely missing something in configuration of IPSEC.

Thank you for your attention
Best regards!

sindy · Sun Oct 14, 2018 9:09 pm

You need an exception from a src-nat or masquerade rule for the traffic to be tunnelled using a plain IPsec, because in such setup, ipsec policies cherry-pick packets they like after the NAT handling has been done. In your case, where you use IPsec to carry only the L2TP tunnel and route the site-to-site traffic via that tunnel, this is not necessary.

However, I cannot see how the src-nating of connections between the clients, which you have disabled, should help it work, given that the routes you've published are correct (provided that 10.50.0.x are addresses assigned to the L2TP tunnel interfaces of the clients and the server, not their WAN addresses).

So I suspect you have configured the IPsec to steal too much traffic. Can you post your configuration exports following the guideline in my automatic signature?

nomatter · Sun Oct 14, 2018 10:01 pm

Hi sindy!

Thank you for your reply!

Here are exports:
Server

# sep/01/2018 18:48:38 by RouterOS 6.34.4
# software id = S8B2-VCX2
#
/interface bridge
add comment=wifi+ethernet name=LAN
/interface wireless
set [ find default-name=wlan1 ] band=2ghz-b/g/n disabled=no mode=ap-bridge ssid=Intornet
/interface ethernet
set [ find default-name=ether5 ] comment=port5 name=AM
set [ find default-name=ether2 ] comment="port2 - master" name=LAN1
set [ find default-name=ether3 ] comment=port3 master-port=LAN1 name=LAN2
set [ find default-name=ether4 ] comment=port4 master-port=LAN1 name=LAN3
set [ find default-name=ether1 ] comment=port1 mac-address=60:67:20:E8:52:E0 name=WAN
/interface pppoe-client
add add-default-route=yes disabled=no interface=WAN max-mru=1480 max-mtu=1480 mrru=1600 name=ISP \
    use-peer-dns=yes user=ISPuser
/interface l2tp-server
add name=gl user=gl
add name=radiologist user=radiologist
add name=radiologist2 user=radiologist2
/ip neighbor discovery
set AM comment=port5
set LAN1 comment="port2 - master"
set LAN2 comment=port3
set LAN3 comment=port4
set WAN comment=port1
set LAN comment=wifi+ethernet
/interface wireless security-profiles
set [ find default=yes ] authentication-types=wpa2-psk eap-methods="" group-ciphers=tkip,aes-ccm mode=\
    dynamic-keys unicast-ciphers=tkip,aes-ccm
/ip pool
add name=dhcp_pool1 ranges=192.168.1.10-192.168.1.50
/ip dhcp-server
add address-pool=dhcp_pool1 disabled=no interface=LAN lease-time=1d name=dhcp
/ppp profile
add change-tcp-mss=yes name=remote_office use-compression=yes use-encryption=yes
/interface bridge port
add bridge=LAN interface=LAN1
add bridge=LAN interface=wlan1
/interface l2tp-server server
set authentication=mschap2 default-profile=remote_office enabled=yes
/ip address
add address=192.168.1.254/24 interface=LAN network=192.168.1.0
add address=109.86.12.192/24 disabled=yes interface=WAN network=109.86.12.0
add address=192.168.2.254/24 interface=AM network=192.168.2.0
/ip dhcp-server lease
add address=192.168.1.102 mac-address=1C:CD:E5:52:0F:18 server=dhcp
add address=192.168.1.101 client-id=1:60:67:20:e8:52:e0 mac-address=60:67:20:E8:52:E0 server=dhcp
add address=192.168.1.103 mac-address=D8:6C:02:9C:74:DE server=dhcp
/ip dhcp-server network
add address=192.168.1.0/24 gateway=192.168.1.254
/ip firewall filter
add chain=input in-interface=WAN protocol=icmp
add chain=input dst-port=1701 protocol=udp
add chain=input dst-port=500 protocol=udp
add chain=input dst-port=4500 protocol=udp
add chain=input protocol=ipsec-esp
add action=drop chain=input in-interface=WAN
# radiologist not ready
add action=drop chain=forward in-interface=radiologist out-interface=LAN
add action=drop chain=forward disabled=yes in-interface=gl out-interface=LAN
add action=drop chain=forward disabled=yes in-interface=LAN out-interface=gl
add action=drop chain=forward disabled=yes in-interface=AM out-interface=LAN
add action=drop chain=forward in-interface=radiologist2 out-interface=LAN
add action=drop chain=forward in-interface=LAN out-interface=radiologist2
# radiologist not ready
add action=drop chain=forward in-interface=LAN out-interface=radiologist
/ip firewall nat
add action=masquerade chain=srcnat out-interface=WAN
add action=masquerade chain=srcnat out-interface=ISP
/ip ipsec peer
add address=0.0.0.0/0 enc-algorithm=aes-256,aes-192,aes-128 exchange-mode=main-l2tp generate-policy=port-strict
/ip ipsec policy
set 0 disabled=yes
add dst-address=192.168.5.0/24 sa-dst-address=10.50.0.13 sa-src-address=10.50.0.10 src-address=192.168.2.0/24 \
    tunnel=yes
add dst-address=192.168.128.0/24 sa-dst-address=10.50.0.11 sa-src-address=10.50.0.10 src-address=192.168.2.0/24 \
    tunnel=yes
add dst-address=192.168.3.0/24 sa-dst-address=10.50.0.12 sa-src-address=10.50.0.10 src-address=192.168.2.0/24 \
    tunnel=yes
/ip route
add disabled=yes distance=1 gateway=109.86.12.254
add comment="PPP - To radiologist network" distance=1 dst-address=192.168.3.0/24 gateway=10.50.0.12 pref-src=\
    10.50.0.10
add comment="PPP to radilogist2 (kv office) network" distance=1 dst-address=192.168.5.0/24 gateway=10.50.0.13
add comment="PPP - to gl network" distance=1 dst-address=192.168.128.0/24 gateway=10.50.0.11
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www disabled=yes
set ssh disabled=yes
set api disabled=yes
set winbox address=192.168.1.0/24
set api-ssl disabled=yes
/ppp secret
add local-address=10.50.0.10 name=gl profile=remote_office remote-address=10.50.0.11 service=l2tp
add local-address=10.50.0.10 name=radiologist profile=remote_office remote-address=10.50.0.12 service=l2tp
add local-address=10.50.0.10 name=radiologist2 profile=remote_office remote-address=10.50.0.13 service=l2tp
/system clock
set time-zone-autodetect=no
/system leds
set 0 interface=wlan1
/system routerboard settings
set protected-routerboot=disabled
/tool traffic-monitor
add disabled=yes interface=gl name=tmon1 threshold=0 traffic=received
add disabled=yes interface=radiologist name=tmon2 threshold=0

_______________________________________________________________________

Client1

# aug/30/2018 10:07:53 by RouterOS 6.34.4
# software id = ZUNB-FW0U
#
/interface bridge
add comment=wifi+ethernet name=LAN
/interface wireless
set [ find default-name=wlan1 ] disabled=no mode=ap-bridge ssid=AM
/interface ethernet
set [ find default-name=ether1 ] comment="port1 - master" name=LAN1
set [ find default-name=ether2 ] comment=port2 master-port=LAN1 name=LAN2
set [ find default-name=ether3 ] comment=port3 master-port=LAN1 name=LAN3
set [ find default-name=ether4 ] comment=port4 master-port=LAN1 name=LAN4
set [ find default-name=ether5 ] comment=port5 mac-address=D4:BE:D9:50:B9:7F \
    name=WAN
/interface l2tp-client
add allow=mschap2 connect-to=SERVER_PUBLIC_IP disabled=no mrru=1600 name=gl \
    user=gl
/ip neighbor discovery
set LAN1 comment="port1 - master"
set LAN2 comment=port2
set LAN3 comment=port3
set LAN4 comment=port4
set WAN comment=port5
set LAN comment=wifi+ethernet
/interface wireless security-profiles
set [ find default=yes ] authentication-types=wpa2-psk eap-methods="" \
    group-ciphers=tkip,aes-ccm mode=dynamic-keys unicast-ciphers=tkip,aes-ccm
add authentication-types=wpa2-psk eap-methods="" group-ciphers=tkip,aes-ccm \
    management-protection=allowed mode=dynamic-keys name=wifi_guest \
    supplicant-identity="" unicast-ciphers=tkip,aes-ccm
/interface wireless
add comment="guest access" disabled=no keepalive-frames=disabled mac-address=\
    02:00:00:AA:00:00 master-interface=wlan1 multicast-buffering=disabled name=\
    wlan2 security-profile=wifi_guest ssid=AM_Guest wds-cost-range=\
    0 wds-default-cost=0
/interface wireless manual-tx-power-table
set wlan2 comment="guest access"
/interface wireless nstreme
set *9 comment="guest access"
/ip neighbor discovery
set wlan2 comment="guest access"
/ip ipsec policy group
add name=remote_office
/ip pool
add name=dhcp_pool1 ranges=192.168.128.10-192.168.128.100
add name=dhcp_pool2 ranges=192.168.17.200-192.168.17.220
/ip dhcp-server
add address-pool=dhcp_pool1 disabled=no interface=LAN lease-time=1d name=dhcp1
add address-pool=dhcp_pool2 disabled=no interface=wlan2 lease-time=1h name=\
    dhcp2
/interface bridge port
add bridge=LAN interface=LAN1
add bridge=LAN interface=wlan1
/ip address
add address=192.168.128.254/24 interface=LAN network=192.168.128.0
add address=192.168.17.1/24 interface=wlan2 network=192.168.17.0
/ip dhcp-client
add default-route-distance=0 dhcp-options=hostname,clientid disabled=no \
    interface=WAN
/ip dhcp-server network
add address=192.168.17.0/24 gateway=192.168.17.1
add address=192.168.128.0/24 gateway=192.168.128.254
/ip firewall address-list
add address=192.168.0.5 list="no internet"
/ip firewall filter
add action=drop chain=forward disabled=yes out-interface=WAN src-address=\
    192.168.128.5
add chain=input dst-port=1701 protocol=udp
add chain=input dst-port=4500 protocol=udp
add chain=input dst-port=500 protocol=udp
add chain=input protocol=ipsec-esp
add action=drop chain=forward comment="drop guest wlan from local wlan" \
    dst-address=192.168.128.0/24 src-address=192.168.17.0/24
add action=drop chain=input comment="drop incoming connections" in-interface=\
    WAN
/ip firewall nat
add action=masquerade chain=srcnat out-interface=WAN
add action=masquerade chain=srcnat disabled=yes out-interface=gl
/ip ipsec peer
add address=10.50.0.10/32 enc-algorithm=aes-256,aes-192,aes-128 \
    generate-policy=port-strict policy-template-group=remote_office
/ip ipsec policy
set 0 disabled=yes
add dst-address=192.168.2.0/24 sa-dst-address=10.50.0.10 sa-src-address=\
    10.50.0.11 src-address=192.168.128.0/24 tunnel=yes
/ip route
add comment="to PACS network" distance=1 dst-address=192.168.2.0/24 gateway=\
    10.50.0.10 pref-src=10.50.0.11
add comment="to radiologist network" distance=1 dst-address=192.168.3.0/24 \
    gateway=10.50.0.10 pref-src=10.50.0.11
add comment="to radiologist2 (kv office) network" distance=1 dst-address=\
    192.168.5.0/24 gateway=10.50.0.10 pref-src=10.50.0.11
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www disabled=yes
set ssh disabled=yes
set api disabled=yes
set api-ssl disabled=yes
/system clock
set time-zone-name=Europe/kv
/system leds
set 0 interface=wlan1
/system routerboard settings
set protected-routerboot=disabled

____________________________________________________
Client2

# aug/31/2018 02:26:20 by RouterOS 6.41.3
# software id = FEB8-Z5JJ
#
# model = 951Ui-2HnD
# serial number = 8D00089CDC82
/interface bridge
add fast-forward=no name=AM
add fast-forward=no name=LAN
/interface ethernet
set [ find default-name=ether4 ] comment=port4 name=AM1
set [ find default-name=ether5 ] comment=port5 name=AM2
set [ find default-name=ether3 ] comment=port3 name=AM3
set [ find default-name=ether2 ] comment=port2 name=LAN1
set [ find default-name=ether1 ] comment=port1 name=WAN
/interface l2tp-client
add allow=mschap2 comment="l2tp to PACS" connect-to=SERVER_PUBLIC_IP disabled=no \
    mrru=1600 name=PPP_to_Pacs user=radiologist2
/interface wireless
set [ find default-name=wlan1 ] disabled=no mode=ap-bridge name=\
    AM-wlan ssid=AM-guest
/interface wireless security-profiles
set [ find default=yes ] authentication-types=wpa2-psk eap-methods="" mode=\
    dynamic-keys supplicant-identity=MikroTik
add authentication-types=wpa2-psk eap-methods="" management-protection=allowed \
    mode=dynamic-keys name=profile-guest-wifi supplicant-identity=""
add authentication-types=wpa2-psk eap-methods="" management-protection=allowed \
    mode=dynamic-keys name=AM-corp supplicant-identity=""
/interface wireless
add disabled=no keepalive-frames=disabled mac-address=CE:2D:E0:BE:6A:66 \
    master-interface=AM-wlan multicast-buffering=disabled name=\
    AM-corp security-profile=AM-corp ssid=AM \
    wds-cost-range=0 wds-default-cost=0 wps-mode=disabled
/ip ipsec policy group
add name=remote_office
/ip pool
add name=dhcp_pool1 ranges=192.168.6.10-192.168.6.35
add name=guest-wi-fi-pool ranges=192.168.17.10-192.168.17.25
add name=dhcp_pool5 ranges=192.168.6.10-192.168.6.25
/ip dhcp-server
add address-pool=dhcp_pool5 disabled=no interface=LAN lease-time=8h name=dhcp1
/interface bridge port
add bridge=LAN interface=LAN1
add bridge=AM interface=AM3
add bridge=AM interface=AM1
add bridge=AM interface=AM2
add bridge=LAN interface=AM-wlan
add bridge=AM interface=AM-corp
/ip address
add address=192.168.5.254/24 interface=AM network=192.168.5.0
add address=192.168.6.254/24 interface=LAN network=192.168.6.0
/ip dhcp-client
add dhcp-options=hostname,clientid disabled=no interface=WAN
/ip dhcp-server network
add address=192.168.6.0/24 dns-server=8.8.8.8 gateway=192.168.6.254
/ip firewall filter
add action=accept chain=input in-interface=WAN protocol=icmp
add action=accept chain=input dst-port=1701 protocol=udp src-port=""
add action=accept chain=input dst-port=500 protocol=udp
add action=accept chain=input dst-port=4500 protocol=udp
add action=accept chain=input protocol=ipsec-esp
add action=drop chain=input in-interface=WAN
add action=drop chain=forward in-interface=LAN out-interface=AM
add action=drop chain=forward in-interface=AM out-interface=LAN
/ip firewall nat
add action=masquerade chain=srcnat out-interface=WAN
add action=masquerade chain=srcnat disabled=yes out-interface=PPP_to_Pacs
/ip ipsec peer
add address=10.50.0.10/32 dh-group=modp1024 enc-algorithm=\
    aes-256,aes-192,aes-128 exchange-mode=main-l2tp generate-policy=port-strict \
    policy-template-group=remote_office
/ip ipsec policy
set 0 disabled=yes
add dst-address=192.168.2.0/24 sa-dst-address=10.50.0.10 sa-src-address=\
    10.50.0.13 src-address=192.168.5.0/24 tunnel=yes
add dst-address=192.168.128.0/24 sa-dst-address=10.50.0.11 sa-src-address=\
    10.50.0.13 src-address=192.168.5.0/24 tunnel=yes
/ip route
add comment="to PACS network" distance=1 dst-address=192.168.2.0/24 gateway=\
    10.50.0.10
add comment="to radiologist network" distance=1 dst-address=192.168.3.0/24 \
    gateway=10.50.0.10 pref-src=10.50.0.13
add comment="to gl network" distance=1 dst-address=192.168.128.0/24 \
    gateway=10.50.0.10
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www disabled=yes
set ssh disabled=yes
set api disabled=yes
set api-ssl disabled=yes
/system clock
set time-zone-autodetect=no

sindy · Sun Oct 14, 2018 10:59 pm

Yes, you've set it up a bit upside down as compared to what is used normally: instead of using an IPsec tunnel to deliver L2TP packets as the L2TP/IPsec is standardized, you've used an L2TP tunnel to deliver IPsec packets. So you have chosen the worst from both worlds.

Now, your IPsec policies only properly cover the traffic between each client and the central server, not the traffic between the clients.

The clients only have the server as a peer, so the policy's sa-dst-address can only be the server's one. So in your client configuration, the policy with dst-address=192.168.128.0/24 sa-dst-address=10.50.0.11 sa-src-address=10.50.0.13 src-address=192.168.5.0/24 tunnel=yes will never come up because there is no /ip ipsec peer with address=10.50.0.11 in that configuration. Normally, packets matching that policy's traffic selector should be delivered using the regular routes, i.e. via the L2TP tunnel without encryption, because the policy is not active so it should be ignoring them, but you're lucky and it doesn't work for some reason.

So you have several possibilities:

make it the usual L2TP/IPSec way, i.e. to set up IPsec connections between the routers only to carry the L2TP packets and use the existing normal routes for everything except the L2TP transport packets,
leave out the L2TP completely and set the IPsec and policies properly, which involves up to as many policies as destination networks on each client, and as many policies as clients on the server (which gives you larger MTU of the payload)
fix the current setup which basically means to do the same as above except that you lose some MTU to the L2TP overhead

Which way you want to continue?

nomatter · Sun Oct 14, 2018 11:28 pm

Thank you, sindy!

My final purpose is to have secure connection between remote networks. Actually, I'm not expert in networking so my idea was that i setup L2TP tunnels as transport means and IPSec should encrypt the traffic.

I have only 1 Public address for the server and no possibilities to obtain public addresses for the clients.
Also another difficulty in this case that is really problematic to be on-site of 2 clients so I have to choose an option that will allow me to do as much work as possible remotely without engaging other people.( I mean it would be ideal to setup client routers without dropping internet connection on their sides or have the ability to reconnect quickly without much assistance from local personnel.)

But main purpose, nevertheless - I want to make it properly and securely.
As I understand the option 1 in your reply fits this the most? Am I right?
Which of those would you advise in this case?

sindy · Mon Oct 15, 2018 11:43 am

I'm not expert in networking

Well, if networking is not your major and you got this far with IPsec, chapeau.

I have only 1 Public address for the server and no possibilities to obtain public addresses for the clients.

That's quite common, and plain IPsec can also deal with it. That's from where came the advice of the other guy that you should activate NAT traversal support in IPsec configuration, as he apparently also didn't even dream that you could be transporting IPsec over L2TP (so no NAT traversal necessary as it has been already handled by the L2TP layer in that case).

Also another difficulty in this case that is really problematic to be on-site of 2 clients so I have to choose an option that will allow me to do as much work as possible remotely without engaging other people.( I mean it would be ideal to setup client routers without dropping internet connection on their sides or have the ability to reconnect quickly without much assistance from local personnel.)

Well, I understand your current status to be that you can access the client sites from the server one using the L2TP tunnels even if you disable the IPsec peer on the central site, am I right? If so, all the rest can be done remotely, with a little help of Safe Mode. If you switch it on, a snapshot of the configuration is taken, and if you later lock yourself out of the router by some configuration change, which means that the management session stops working, the router detects that in less than a minute and restores the snapshot. When you get past the critical point and the management connection still works, you can switch safe mode off and then eventually switch it on again for another batch of potentially dangerous changes. It is in fact useful to build a habit of doing everything in safe mode, but you have to switch it off and on again from time to time so that you wouldn't lose too many of your changes if you lock yourself out.

But main purpose, nevertheless - I want to make it properly and securely.

The bad news here is that you have already made a serious mistake - as I understand it, the client devices are already at their respective sites, so they are currently accessible using plain L2TP, which means that they are accessible not only to you but to anyone who is able to break its authentication which is too weak for today's network jungle. And it uses no encryption whatsoever so even without breaking into your routers, anyone along the path between the routers can sniff the plaintext contents of the data unless the applications which talk together use their own encryption (like TLS).

As I understand the option 1 in your reply fits this the most? Am I right?
Which of those would you advise in this case?

Option 1 (encrypt L2TP connections using IPsec) means exactly that, adding encryption to the already working setup, the price to pay is that you have two tunneling protocols stacked on top of each other, so less space for the passenger data in each transport packet.

Option 2 keeps only IPsec tunneling in place, which means you save the overhead of the L2TP and have more space in the transport packets for the payload.

In more complex network with dynamic routing protocols, or where you need L2 transparency, you have no other choice but Option 1 (where the transport packets of L2TP or some other tunneling protocol are encrypted using IPsec) despite its drawbacks, but in your case with no chance for redundant paths between two routers, I would myself prefer Option 2. But the choice is yours and Option 1 may be faster to set up. Different parts of your existing configuration may be reused for each of the two options.

A note regarding your firewall configuration - it may be a matter of personal opinion, but I strongly prefer to build the firewall as "deny everything except what you know you need to permit" (this is what you currently have in the input chain) rather than "permit everything except what you know you need to deny" (which is what you currently have in the forward chain). The difference is that if you forget to permit something in the deny-all-but-exceptions variant, your legal users will let you know quickly, whilst if you forget to deny something in the permit-all-but-exceptions variant, your illegal users will let you know never (well, some of them do, asking for a ransom). Here is a supercharged introduction to firewall.

nomatter · Mon Oct 15, 2018 12:43 pm

Thank you, sindy for your detailed explanation. It is much appreciated!

Well, I understand your current status to be that you can access the client sites from the server one using the L2TP tunnels even if you disable the IPsec peer on the central site, am I right?

Yes, you're absolutely right.

The bad news here is that you have already made a serious mistake - as I understand it, the client devices are already at their respective sites, so they are currently accessible using plain L2TP, which means that they are accessible not only to you but to anyone who is able to break its authentication which is too weak for today's network jungle. And it uses no encryption whatsoever so even without breaking into your routers, anyone along the path between the routers can sniff the plaintext contents of the data unless the applications which talk together use their own encryption (like TLS).

Yes, I understand this. That's why I've started all this with IPsec.

Thank you for the Safe mode. I didn't know about it. I'll google how to use it properly.

Option 1 (encrypt L2TP connections using IPsec) means exactly that, adding encryption to the already working setup, the price to pay is that you have two tunneling protocols stacked on top of each other, so less space for the passenger data in each transport packet.

Is this guide ok for implementing option 1 (Basic L2TP/IpSec setup chapter)?
https://wiki.mikrotik.com/wiki/Manual:Interface/L2TP

Option 2 keeps only IPsec tunneling in place, which means you save the overhead of the L2TP and have more space in the transport packets for the payload.

Does the chapter Road Warrior setup with Mode Conf fit my situation? Will I have connectivity between clients in that case?
https://wiki.mikrotik.com/wiki/Manual:IP/IPsec

Thank you for your advice with firewall- i'll work on that for sure.

sindy · Mon Oct 15, 2018 2:10 pm

Yes, I understand this. That's why I've started all this with IPsec.

What I had in mind is that after such an extended period of exposing some attack surface to the internet, you cannot be 100% sure that someone hasn't made use of it in the meantime. So to be totally sure, you would have to netinstall all the routers involved.

Is this guide ok for implementing option 1 (Basic L2TP/IpSec setup chapter)?
https://wiki.mikrotik.com/wiki/Manual:Interface/L2TP

Yes, except that as you enjoy the luxury of having a Mikrotik at both ends, I would strongly recommend to use IKEv2, which requires to set up the IPsec layer manually at both the client and the server.

Does the chapter Road Warrior setup with Mode Conf fit my situation? Will I have connectivity between clients in that case?
https://wiki.mikrotik.com/wiki/Manual:IP/IPsec

Mode Config doesn't fit here as you need a LAN to LAN connection for which Mode Config is not the most suitable tool, and you don't need to assign the tunnel address to the client from server side as you have a limited number of static clients (although they are behind NAT). So basically you only need the server to dynamically create policies at its side to reflect those requested by clients, because the sa-dst-address are unknown to the server in advance, until the client connects. And for security reasons (stolen device) you also need to use one of the authentication modes where the clients have individual secrets or certificates; if you don't happen to have an external RADIUS server at hand on the server site, this boils down to auth-mode=pre-shared-key-xauth.

The connectivity between the clients is ensured by means of the policies. So one possibility is to set up the policies like 192.168.my.subnet to 192.168.whatever, which means a single policy at each client (actually, two, will explain later) and one policy per each client at the server. Another possibility is to set as many policies at each client as the remote subnets the client needs to get to; this means, in your actual case with one central site and three remote ones, three policies per client (unless there is a pair of clients doesn't need to talk to each other).

The above is equal to having a single route to 192.168.0.0/16 vs. having several individual routes to 192.168.remote.subnetX/mask where you wouldn't use IPsec.

With normal routes, the exception for local traffic is automatic, as there is always a "connected" route to local LAN subnet, which constitutes an exception from the 192.168.whatever destination; with IPsec policies, you have to place such exception into the policy list manually:

/ip ipsec policy add src-address=192.168.my.subnet/24 dst-address=192.168.my.subnet/24 action=none
/ip ipsec policy add src-address=192.168.my.subnet/24 dst-address=192.168.0.0/16 action=encrypt sa-src-address=0.0.0.0 sa-dst-address=the.server.public.ip level=unique
And unlike with routes where the prefix length od the dst-address automatically defines the search order, so declaration order is irrelevant, the policies' traffic selectors consist of several fields so the search order strictly follows the declaration order and it is your responsibility to arrange the policies properly.

One more bad news: to be able to use IKEv2 (which is believed to be more secure than its predecessor IKE) to establish the IPsec tunnels, you'll have to upgrade all your devices to 6.43.2.

nomatter · Mon Oct 15, 2018 5:17 pm

Seems i'm kinda lost

As I understand what i need to do for option 1 (L2TP/IPSEC) is the following:
1. Revert all my manual IPSec confugurations on all routers. So there is only default (from factory) ipsec config.
2. Perform IPSEC configuration as described here https://wiki.mikrotik.com/wiki/Manual:Interface/L2TP
IpSec Config
On server side:

/ip ipsec proposal
set [ find default=yes ] enc-algorithms=3des,aes-128,aes-192,aes-256
/ip ipsec peer
add generate-policy=port-strict hash-algorithm=sha1 nat-traversal=yes secret=test123456

RouterOS as client:

/ip ipsec proposal
set [ find default=yes ] enc-algorithms=aes-128
/ip ipsec peer
add address=SERVER_PUBLIC_IP_ADDR hash-algorithm=sha1 nat-traversal=yes secret=test123456

3. Add ipsec policies on each client:

3.1
/ip ipsec policy add src-address=192.168.128.0/24 dst-address=192.168.128.0/24 action=none
/ip ipsec policy add src-address=192.168.128.0/24 dst-address=192.168.0.0/16 action=encrypt sa-src-address=0.0.0.0 sa-dst-address=the.server.public.ip level=unique

3.2
/ip ipsec policy add src-address=192.168.5.0/24 dst-address=192.168.5.0/24 action=none
/ip ipsec policy add src-address=192.168.5.0/24 dst-address=192.168.0.0/16 action=encrypt sa-src-address=0.0.0.0 sa-dst-address=the.server.public.ip level=unique

4. Rules like that can be disabled
chain=srcnat action=masquerade out-interface=all-ppp log=no log-prefix=""

5. Routes can be left as is

3 A S ;;; to Server network
192.168.2.0/24 10.50.0.13 10.50.0.10 1
7 A S ;;; to Client1 network
192.168.128.0/24 10.50.0.13 10.50.0.10 1

Did I get it correct or messed up everything?

sindy · Mon Oct 15, 2018 7:34 pm

1. Revert all my manual IPSec confugurations on all routers. So there is only default (from factory) ipsec config.

Yes.

2. Perform IPSEC configuration as described here https://wiki.mikrotik.com/wiki/Manual:Interface/L2TP
IpSec Config on server side:

/ip ipsec proposal
set [ find default=yes ] enc-algorithms=3des,aes-128,aes-192,aes-256
/ip ipsec peer
add generate-policy=yes hash-algorithm=sha1 nat-traversal=yes secret=test123456

Seems the manual isn't updated carefully... in part of it, generate-policy=port-strict or port-override is already mentioned, but in the example, yes has remained. I have no clue what is correct in 6.34.1, so expect surprises. You want either yes or port-strict, whichever is permitted in your version. Also, add send-initial-contact=no to the peer parameters at the server.

RouterOS as client:

/ip ipsec proposal
set [ find default=yes ] enc-algorithms=aes-128
/ip ipsec peer
add address=SERVER_PUBLIC_IP_ADDR hash-algorithm=sha1 nat-traversal=yes secret=test123456

Almost correct. The problem is that the WAN IP of the client may change over time, so you have to cheat and choose some static address on the machine. It will be NATed anyway so it doesn't matter if you NAT it one more time locally. So you have to add local-address=192.168.x.1 (if that's the own address of that Mikrotik in its LAN subnet) to peer's configuration. It would be a non-issue if you would be using a tunnel mode of the SA but tunnel mode costs some extra packet space.

Add send-initial-contact=no to the peer parameters at the client.

3. Add ipsec policies on each client:

3.1
/ip ipsec policy add src-address=192.168.128.0/24 dst-address=192.168.128.0/24 action=none
/ip ipsec policy add src-address=192.168.128.0/24 dst-address=192.168.0.0/16 action=encrypt sa-src-address=0.0.0.0 sa-dst-address=the.server.public.ip level=unique

3.2
/ip ipsec policy add src-address=192.168.5.0/24 dst-address=192.168.5.0/24 action=none
/ip ipsec policy add src-address=192.168.5.0/24 dst-address=192.168.0.0/16 action=encrypt sa-src-address=0.0.0.0 sa-dst-address=the.server.public.ip level=unique

The above would be true for Option 2.

For Option 1, i.e. to transport only the L2TP packets, you need to set the policies like this:
/ip ipsec policy add src-address=192.168.x.1 dst-address=SERVER_PUBLIC_IP_ADDR protocol=udp dst-port=1701 tunnel=no
The src-address is the same local IP of the client Mikrotik like used as local-address on the peer.
With tunnel=no, src-address and sa-src-address are the same so only src-address is configured; same for dst.

4. Rules like that can be disabled
chain=srcnat action=masquerade out-interface=all-ppp log=no log-prefix=""

Not "like that" - this one has to be disabled, but the usual masquerade rule
action=masquerade chain=srcnat out-interface=WAN
must stay.

5. Routes can be left as is
3 A S ;;; to Server network
192.168.2.0/24 10.50.0.13 10.50.0.10 1
7 A S ;;; to Client1 network
192.168.128.0/24 10.50.0.13 10.50.0.10 1

Correct.

Do it in steps.

Be sure you use encrypted connection to the remote site (ssh, WinBox or https - no telnet, no plain http). You'll be setting the ipsec pre-shared key over the net and you don't want it to be sniffed on the path between the devices.
First configure only the peers, and after a short while, /ip ipsec remote peer print should show you the peer in state established - one at each client, three at the server if you do all clients at the same time. As no policy will be active yet, the L2TP connection will still run without encryption.
Then, add the policy at the client, with dst-port=33333 instead of 1701, to see whether it becomes active, a mirror policy with a D (dynamic) sign will appear at the server, and at both, you'll see a pair of /ip ipsec installed-sa.
If all of the above happens, you may disable the policy, modify its dst-port value to 1701, and re-enable it. Your management connection should survive, but now it should now run encrypted. I cannot exclude that it may fail or start behaving weird due to change of MTU on the path, but don't panic and just open a new one if that happens.
if you cannot connect to the client for more than a minute, simply disable the peer at the server. The IPsec connections will go down and so will the policy (after 30 minutes at worst) and you'll be able to connect to the client again via the unencrypted tunnel.
Once you succeed in this with all three clients, your next step will be to add ipsec-policy=in,ipsec to the /ip firewall filter rule chain=input ... dst-port=1701 on the server. This is to prevent unencrypted L2TP connections from establishing in future, only packets which emerge from an SA are accepted.

With the configuration above:

you are using IKE(v1) exchange mode main; the ultimate goal should be to switch over to IKEv2 which requires an upgrade of all devices
you are not using the strongest available encryption and authentication (I would use sha-256 and aes-256 everywhere, in peer settings as well as the proposals, and at least modp3072 as pfs-group in /ip ipsec proposal and as dh-group in /ip ipsec peer. This can be done already before you start, always at both ends (client & server).
you have a single common IPsec secret for all clients; the L2TP credentials are only useful for client distinction, not for security. So if someone gets hold of one of the client routers, you cannot cut him off without cutting off the remaining client routers. To change that, you would have to set auth-mode of the peers to pre-shared-key-xauth, and do the following:
- in clients' peer configurations, set passive=no and some xauth-login and xauth-password values
- in server's peer configuration, set passive=yes and add each client's xauth-login and xauth-password as name and password in /ip ipsec user
This is on top of the L2TP credentials. Ideally, the IPsec pre-shared secret as well as the IPsec xauth-passwords should be radomly generated 128-byte strings, i.e. 256 hexadecimal symbol strings with 0x at the beginning. The usernames for L2TP and IPsec may be the same, the passwords for L2TP are probably redundant but I don't know whether it is possible not to use them at all.

nomatter · Tue Oct 16, 2018 11:32 pm

Hi sindy!

First of all thank you for babysitting with me! I can not imagine how much time it would take to setup all this without your help.

So after 2 evenings trying here are my results:

First configure only the peers, and after a short while, /ip ipsec remote peer print should show you the peer in state established - one at each client, three at the server if you do all clients at the same time. As no policy will be active yet, the L2TP connection will still run without encryption.

Succeeded with peers.
I'm able to see remote peers: 2 on the server and 1 for each client. they have status "established"

Then, add the policy at the client, with dst-port=33333 instead of 1701, to see whether it becomes active, a mirror policy with a D (dynamic) sign will appear at the server, and at both, you'll see a pair of /ip ipsec installed-sa.

On one client (6.41.3 Router OS) all happens like you say. Policy becomes active and mirror policy appears on the server. SAs installed.
---
On other one (6.34.1 Router OS) i can not see whether it's active or not. And no mirror policy generated on server. No installed SAs. So here is the end of story for this client.

ip ipsec policy print 
Flags: T - template, X - disabled, D - dynamic, I - inactive, * - default 
 0 T * group=default src-address=::/0 dst-address=::/0 protocol=all proposal=default template=yes 

 1     src-address=192.168.128.254/32 src-port=any dst-address=SERVER_PUBLIC_IPaddr/32 dst-port=33333 protocol=udp 
       action=encrypt level=require ipsec-protocols=esp tunnel=no sa-src-address=192.168.128.254 
       sa-dst-address=SERVER_PUBLIC_IPaddr proposal=default priority=0

a lot of messages like this in logs of client's server
22:30:25 ipsec,error phase1 negotiation failed due to time up 192.168.128.254[500]
<=>SERVER_PUBLIC_IPaddr[500] 868bacdc9dab1caa:0000000000000000

Remote peer - server still present.

If all of the above happens, you may disable the policy, modify its dst-port value to 1701, and re-enable it. Your management connection should survive, but now it should now run encrypted. I cannot exclude that it may fail or start behaving weird due to change of MTU on the path, but don't panic and just open a new one if that happens.
if you cannot connect to the client for more than a minute, simply disable the peer at the server. The IPsec connections will go down and so will the policy (after 30 minutes at worst) and you'll be able to connect to the client again via the unencrypted tunnel.

After I change dst-port in policy connection gets lost. I'm unable to reconnect and just peer disabling on server helps. After 30min I'm able to connect again to client via winbox.
Even after reenabling peer policy at this client does not become active

Flags: T - template, X - disabled, D - dynamic, I - invalid, A - active, 
* - default 
 0 T * group=default src-address=::/0 dst-address=::/0 protocol=all 
       proposal=default template=yes 

 1     src-address=192.168.5.254/32 src-port=any dst-address=SERVER_PUBLIC_IPaddr/32 
       dst-port=1701 protocol=udp action=encrypt level=require 
       ipsec-protocols=esp tunnel=no proposal=default ph2-count=0

Of course this relates to that client which was able to connect at previous step using 33333 dst port.

Here are my ipsec exports:

Server

/ip ipsec proposal
set [ find default=yes ] auth-algorithms=sha256 enc-algorithms=aes-256-cbc pfs-group=modp3072
/ip ipsec peer
add address=0.0.0.0/0 auth-method=pre-shared-key-xauth dh-group=modp3072 enc-algorithm=aes-256 generate-policy=\
    port-strict hash-algorithm=sha256 passive=yes send-initial-contact=no
/ip ipsec user
add name=User1
add name=User2

----

  ip ipsec policy print 
Flags: T - template, X - disabled, D - dynamic, I - inactive, * - default 
 0 T * group=default src-address=::/0 dst-address=::/0 protocol=all proposal=default template=yes 

 1  D  src-address=SERVER_PUBLIC_IPaddr/32 src-port=33333 dst-address=Client1_detected_IPaddr/32 dst-port=any protocol=udp 
       action=encrypt level=require ipsec-protocols=esp tunnel=no sa-src-address=SERVER_PUBLIC_IPaddr 
       sa-dst-address=Client1_detected_IPaddr priority=2 

 2  D  src-address=SERVER_PUBLIC_IPaddr/32 src-port=1701 dst-address=Client1_detected_IPaddr/32 dst-port=any protocol=udp 
       action=encrypt level=require ipsec-protocols=esp tunnel=no sa-src-address=SERVER_PUBLIC_IPaddr 
       sa-dst-address=Client1_detected_IPaddr priority=2

Client1 (the one which passing step with dst-port 33333) -6.41.3

/ip ipsec proposal
set [ find default=yes ] auth-algorithms=sha256 enc-algorithms=aes-256-cbc \
    pfs-group=modp3072
/ip ipsec peer
add address=SERVER_PUBLIC_IPaddr/32 auth-method=pre-shared-key-xauth dh-group=modp3072 \
    enc-algorithm=aes-256 hash-algorithm=sha256 local-address=192.168.5.254 \
    send-initial-contact=no xauth-login=User1
/ip ipsec policy
add dst-address=SERVER_PUBLIC_IPaddr/32 dst-port=1701 protocol=udp src-address=\
    192.168.5.254/32

Client 2 ( the one on which just peer stage finishes successfully) -6.34.1

/ip ipsec proposal
set [ find default=yes ] auth-algorithms=sha256 enc-algorithms=aes-256-cbc pfs-group=modp3072
/ip ipsec peer
add address=SERVER_PUBLIC_IPaddr/32 auth-method=pre-shared-key-xauth dh-group=modp3072 enc-algorithm=aes-256 \
    hash-algorithm=sha256 local-address=192.168.128.254 send-initial-contact=no
/ip ipsec policy
add dst-address=SERVER_PUBLIC_IPaddr/32 dst-port=33333 protocol=udp sa-dst-address=SERVER_PUBLIC_IPaddr sa-src-address=\
    192.168.128.254 src-address=192.168.128.254/32

Not "like that" - this one has to be disabled, but the usual masquerade rule
action=masquerade chain=srcnat out-interface=WAN
must stay.

Disabled all-ppp rules.
The one you've mentioned is present

Also one detail to mention ( I don't know is this necessary information in this case or not)
Server connects to Internet as PPPoE client so I have NAT rule on it:
add action=masquerade chain=srcnat out-interface=ISP

where ISP is

/interface pppoe-client> print 
Flags: X - disabled, R - running 
 0  R name="ISP" max-mtu=1480 max-mru=1480 mrru=1600 interface=WAN user="user" password="passwd" 
      profile=default keepalive-timeout=60 service-name="" ac-name="" add-default-route=yes 
      default-route-distance=0 dial-on-demand=no use-peer-dns=yes allow=pap,chap,mschap1,mschap2

sindy · Wed Oct 17, 2018 12:14 am

Hm, it's most likely related to the cheating. The policy matches on the private src-address (192.168.5.254), but the /interface l2tp-client is sending the L2TP packets with a source address of the interface through which the default route goes, which is the WAN one (PPPoE in this case but that's not important).

So do the following:
/ip firewall nat print where chain=srcnat
/ip firewall nat add place-before=0 dst-address=SERVER_PUBLIC_IPaddr/32 protocol=udp dst-port=1701 action=src-nat to-addresses=192.168.5.254

This should make the L2TP transport packets (but not the IPsec transport packets into which the former ones get encapsulated) be src-nat'ed to the address which the policy will match.

But before you do the above, change the policy's dst-port back to 33333, enable it and look whether there are ports (:4500) on the src-address and dst-address shown by /ip ipsec installed-sa print, or whether there is just the IP address with no :port following it. If the ports are there, add the NAT rule as above, change the dst-port in the policy to 1701 and activate it again, it should work this time.

nomatter · Wed Oct 17, 2018 12:33 am

But before you do the above, change the policy's dst-port back to 33333, enable it and look whether there are ports (:4500) on the src-address and dst-address shown by /ip ipsec installed-sa print, or whether there is just the IP address with no :port following it. If the ports are there, add the NAT rule as above, change the dst-port in the policy to 1701 and activate it again, it should work this time.

Ports were there.

Added rule (it asked for chain so put "srcnat")

After disabling policy, changing dst port to 1701 and enabling connection was lost.
So seems like I need to disable peer again on the server to connect to that client.

sindy · Wed Oct 17, 2018 12:37 am

Hm, if only I knew whether it is related to your old RouterOS version or to some conceptual mistake.

While waiting for those 30 minutes, can you see the /ip ipsec installed-sa print to count packets and bytes in either direction? Or just post the output here after replacing the public IP of the server, the keys are temporary so no risk in pasting them.

nomatter · Wed Oct 17, 2018 12:45 am

Here is the output from server.

0 E spi=0xCCDB3C7 src-address=Client1_detected_IPaddr:4500 dst-address=SERVER_PUBLIC_IPaddr:4500 state=mature auth-algorithm=sha256 
     enc-algorithm=aes-cbc auth-key="61de260702cbcbe2fbb52e6ca4a2a64e579437e9f23079b4ffa7707e77c3f576" 
     enc-key="dc53f229b347fc78192fbea1227632dced32a1a2a2e972b5338bb8b8b36e4bf8" add-lifetime=24m/30m replay=128 

 1 E spi=0xBC0461A src-address=Client1_detected_IPaddr:4500 dst-address=SERVER_PUBLIC_IPaddr:4500 state=mature auth-algorithm=sha256 
     enc-algorithm=aes-cbc auth-key="8a5b6a642272f8eea1627c6a0cc570b627ae44e3ca2d4948632bc27f9ebe5523" 
     enc-key="88c08d89d126be6605a98de87e088aec715348698b2afb0c44c9d122b1424b4a" add-lifetime=24m/30m replay=128

sindy · Wed Oct 17, 2018 1:25 am

Show me the current /ip firewall nat export, /ip ipsec export , and /ip address export from the client... (mind the public IP of the server). As there was no packet shown in either direction of the installed-sa, it seems that the client is not sending which is strange.

nomatter · Wed Oct 17, 2018 3:46 pm

Here are exports

# oct/09/2016 13:53:13 by RouterOS 6.41.3
# software id = FEB8-Z5JJ
#
# model = 951Ui-2HnD
# serial number = 8D00089CDC82
/ip firewall nat
add action=src-nat chain=srcnat dst-address=SERVER_PUBLIC_IPaddr dst-port=1701 \
    protocol=udp to-addresses=192.168.5.254
add action=masquerade chain=srcnat out-interface=WAN
add action=masquerade chain=srcnat disabled=yes out-interface=PPP_to_Pacs


/ip ipsec proposal
set [ find default=yes ] auth-algorithms=sha256 enc-algorithms=aes-256-cbc \
    pfs-group=modp3072
/ip ipsec peer
add address=SERVER_PUBLIC_IPaddr/32 auth-method=pre-shared-key-xauth dh-group=\
    modp3072 enc-algorithm=aes-256 hash-algorithm=sha256 local-address=\
    192.168.5.254 secret="secret" send-initial-contact=no \
    xauth-login=User1 xauth-password=passwd
/ip ipsec policy
add dst-address=SERVER_PUBLIC_IPaddr/32 dst-port=1701 protocol=udp src-address=\
    192.168.5.254/32

	
	/ip address
add address=192.168.5.254/24 interface=AM network=192.168.5.0
add address=192.168.6.254/24 interface=LAN network=192.168.6.0

---
I've found this warning on wiki page (https://wiki.mikrotik.com/wiki/Manual:IP/IPsec):

Warning: Ipsec is very sensitive to time changes. If both ends of the IpSec tunnel are not synchronizing time equally(for example, different NTP servers not updating time with the same timestamp), tunnels will break and will have to be established again.

Could this be the cause of fails? I've noticed that i have different time on all three routers involved (due to numerous restore configs procedures)

sindy · Wed Oct 17, 2018 4:10 pm

Leaving aside certificate validity as we don't use certificates here, abrupt jumps in time may cause ipsec connections to drop and re-establish, but an established installed-sa not transporting data is unlikely to be related to difference in absolute time between the peers.

There is nothing in your export which would explain why it doesn't work the expected way. I'll have to spend some time fiddling with it in the lab before I'll be able to tell you whether it needs a small adjustment or whether there is some conceptual gap in it.

nomatter · Wed Oct 17, 2018 4:53 pm

Thank you sindy!

Will wait.

I hope that it's not related to different (and outdated) Router OS versions. But it would be strange as It was possible to implement IPSec on exactly the same software ( wrongly, sure, but still more or less working).

But as I understand you, I will need to update them anyway.
Should I expect complete configuration loss while upgrading from 6.34 to current version?
Should configuration persist while upgrading from 6.41?

sindy · Wed Oct 17, 2018 5:24 pm

You should upgrade in any case as many security vulnerabilities have been patched since 6.34.

The change from 6.34 to 6.43 is quite a big jump, and there was a change of bridge handling between 6.40.x and 6.41, so I would upgrade to 6.40.9 in the first step, and then to 6.43.2, always saving a backup and an export fo a file on flash (I don't know you device models so it may differ) and downloading them to your PC before upgrading. And of course you should start from the client which can be accessed with least headache. Chances for something to go wrong are low but not zero. Don't forget to upgrade also the firmware after each RouterOS upgrade.

nomatter · Wed Oct 17, 2018 10:18 pm

Got it.

If there is some possibility of a failure it will definitely happen

So I need to plan visit to the site to be 100% sure that i''ll have router configured the same day when upgrade takes place. The problem is it's in another town.
So this goal will be reached in the future.

---

Maybe there is something wrong with L2TP setup if you say that configs i've provided seems ok at first sight?

---
UPDATE
You would not believe me but for 1 client it's working!
And the magic is - I did nothing for that.
Yesterday in the end I made backups of current configs and rolled back to just L2TP.

And today I restored them and it started to work immediately.

So One client is ok now! this is the one with 6.41.3 Router OS version for which server was able to generate mirror dynamic policy.

So only one client is left for now.
It is able to establish peer connection but policy is not active and no mirror policy on server

Here are exports

/ip firewall nat export 
add action=src-nat chain=srcnat dst-address=SERVER_PUBLIC_IPaddr dst-port=1701 protocol=udp to-addresses=192.168.128.254
add action=masquerade chain=srcnat out-interface=WAN
add action=masquerade chain=srcnat disabled=yes out-interface=PPP_to_Pacs

/ip ipsec proposal
set [ find default=yes ] auth-algorithms=sha256 enc-algorithms=aes-256-cbc pfs-group=modp3072
/ip ipsec peer
add address=SERVER_PUBLIC_IPaddr/32 auth-method=pre-shared-key-xauth dh-group=modp3072 enc-algorithm=aes-256 \
    hash-algorithm=sha256 local-address=192.168.128.254 secret="secret" send-initial-contact=no
/ip ipsec policy
add dst-address=SERVER_PUBLIC_IPaddr/32 dst-port=1701 protocol=udp sa-dst-address=SERVER_PUBLIC_IPaddr sa-src-address=\
    192.168.128.254 src-address=192.168.128.254/32
	
	
/ip address
add address=192.168.128.254/24 interface=LAN network=192.168.128.0
add address=192.168.17.1/24 interface=wlan2 network=192.168.17.0

Seems like the same as for Successful Client.
So no idea why one is ok and another not.

The only difference I'm able to notice is remote port for this client:

ip ipsec remote-peers print 
0 local-address=SERVER_PUBLIC_IPaddr port=4500 remote-address=Client1_detected_IPaddr port=4500 state=established side=responder established=8m23s 
1 local-address=SERVER_PUBLIC_IPaddr port=4500 remote-address=Client2_detected_IPaddr port=28307 state=established side=responder established=15m50s

sindy · Thu Oct 18, 2018 1:25 am

I've realized when starting to put it together in the lab that the firewall is causing trouble, because it remembers the old L2TP connection without the NAT, but then I got stuck into some other trouble here. So in your case, a restart or timing out of the L2TP connection has caused the connection in the firewall to expire, so the new one got established already NATed, and that was it. But it also means that the successful NAT could have actually killed the normal connection without IPsec, I'm actually surprised that it didn't. But there is no way to modify the policy to avoid this

Can you reveal your time zone?

nomatter · Thu Oct 18, 2018 1:36 am

Can you reveal your time zone?

Do you mean my real world one?
it's GMT+3

sindy · Fri Oct 19, 2018 12:28 am

The way L2TP/IPsec is standardized, it should use "transport" mode of IPsec, where only the payload following the IP header is encrypted (tunneled). Or, from another perspective, the transport packet inherits the IP header from the transported packet as the latter is converted into the former.

The "tunnel" mode of IPsec encapsulates also the IP header of the transported packet, which means that less space in the transport packet remains available for the useful payload, so using this mode to deliver L2TP packets would be a waste of packet space.

But for /interface l2tp-client, you cannot specify the local address on the Mikrotik to be used to send the L2TP packets from. The source address of these packets is assigned as a secondary result of their routing - first the routing determines the outgoing interface for the packet, and based on that the IP address of that interface is used as that packet's source address, unless another one is specified using the pref-src parameter of the route. The NAT handling may further change the source address before the packet is actually sent, and after that the IPsec policy may steal the packet and send it using the security association, but the initial assignment of the address is done as described above.

The problem which we are addressing is that the IP address of your WAN is a dynamic one, and we can only specific a static src-address and sa-src-address for a manually configured policy. There are several other possibilities but each has its own drawback:

the IPsec policy (along with the peer) can be created dynamically if /interface l2tp-client is configured with use-ipsec=yes, but in that case, you cannot choose anything but the shared secret, so IKE (v1) with some default settings is always used in this case
the IPsec policy can also be created dynamically if you use the auth-mode=pre-shared-key-xauth and mode-config=request-only settings on the statically configured /ip ipsec peer, because in this case, the server assigns yet another IP address to the client; however, this way only a tunnel mode policy can be created, so we'd lose some packet bytes and gain nothing in return
the configuration of the statically configured policy can be updated each time the WAN interface gets a new IP address, as RouterOS allows to run a script each time an address is assigned to a PPP interface, but it requires scripting and rewrites the configuration on the flash disk

.

So my action=src-nat ... dst-port=1701 rule overwrites the source address of the L2TP packets from the address currently assigned to the WAN interface to the static one we've chosen for the policy's src-address, but as the rule cannot be activated and deactivated dynamically depending on whether the IPsec policy is active or not, it acts even if the tunnel is down, which means that the ISP should normally drop such packet. Which further means that it is not possible to safely switch between plain L2TP and L2TP over IPsec, as the first way needs this rule not to be there while the second one requires it.

However, the behaviour you have seen was different, right? Even after you've inserted the NAT rule, the connection continued to work on plain L2TP but stopped working once IPsec went up. The reason is that NAT processing is executed for all packets of a connection, but the decision whether to activate the NAT processing for the connection at all and what address to use is taken when handling the initial packet of each connection and never changed throughout the lifetime of the connection. The firewall sees an L2TP session as a normal UDP connection, so as long as packets in at least one direction keep coming at least once in three minutes, it considers all of them as belonging to still the same connection. And the firewall doesn't care what happens with the packets later. So once you've activated the policy, it couldn't see any packets matching its traffic selector, as the packets continued to be sent with the WAN interface address as source, and thus they took the usual path directly via WAN. What I still don't get is why they stopped being received at the server as there, the mirror policy should have matched the answers, but the two installed-sa weren't showing packets in any direction.

So all in all, once you set up everything this way and reboot the client machine, it should always come up properly, but the transition from one approach to the other one is the tricky part of the operation. So you may need to write one simple script which will wait for 30 minutes and then disable the policy and the NAT rule and reboot the router, and schedule this script to run at startup. Then, you would create an even simpler script which would immediately enable the policy and the NAT rule and reboot the router, and schedule it to run in 5 minutes from the time you do it.

The idea is that if you get access after the first reboot, you can stop the script started right at boot during the 30 minutes it does nothing; if you don't, it will disable the policy and rule and the changes and reboot again.

You need a script even for the first reboot because you need to reboot right after enabling the policy, and if you issue /system reboot from a manually entered line, it always asks for confirmation, but it would never be delivered as by activating the policy you lose access. If run in non-interactive mode, such as if called within a script which is started using the scheduler, /system reboot does not ask for confirmation and reboots the machine immediately.

nomatter · Fri Oct 19, 2018 12:05 pm

Hi sindy!

The way L2TP/IPsec is standardized, it should use "transport" mode of IPsec, where only the payload following the IP header is encrypted (tunneled). Or, from another perspective, the transport packet inherits the IP header from the transported packet as the latter is converted into the former.

The "tunnel" mode of IPsec encapsulates also the IP header of the transported packet, which means that less space in the transport packet remains available for the useful payload, so using this mode to deliver L2TP packets would be a waste of packet space.

But for /interface l2tp-client, you cannot specify the local address on the Mikrotik to be used to send the L2TP packets from. The source address of these packets is assigned as a secondary result of their routing - first the routing determines the outgoing interface for the packet, and based on that the IP address of that interface is used as that packet's source address, unless another one is specified using the pref-src parameter of the route. The NAT handling may further change the source address before the packet is actually sent, and after that the IPsec policy may steal the packet and send it using the security association, but the initial assignment of the address is done as described above.

So if i setup L2TP client like that (specifying Public ip address of the server)

interface l2tp-client
add allow=mschap2 connect-to=SERVER_PUBLIC_IP disabled=no mrru=1600 name=gl \
    user=gl

and on server there is secret for user "gl" where local(10.50.0.10) and remote(10.50.0.11) address is specified

Plain L2TP connection is established.

Then I'm adding adding ipsec setup to client

/ip ipsec proposal
set [ find default=yes ] auth-algorithms=sha256 enc-algorithms=aes-256-cbc pfs-group=modp3072
/ip ipsec peer
add address=SERVER_PUBLIC_IPaddr/32 auth-method=pre-shared-key-xauth dh-group=modp3072 enc-algorithm=aes-256 \
    hash-algorithm=sha256 local-address=192.168.128.254 secret="secret" send-initial-contact=no
/ip ipsec policy
add dst-address=SERVER_PUBLIC_IPaddr/32 dst-port=1701 protocol=udp sa-dst-address=SERVER_PUBLIC_IPaddr sa-src-address=\
    192.168.128.254 src-address=192.168.128.254/32

SO ipsec connection must be established.
And L2TP packets must flow inside that connection and NOT vice versa.

But I don't understand to what connection router gives priority. So which one (L2TP or IPSEC) is established first.
And how I can be sure that I have first off all IPSEC connection which carries out L2TP traffic.
And question in my head arises for what do i need l2tp tunnel itself then.
Sorry if these are noob questions.

By the way exports above are from client which is unable to install SAs

However, the behaviour you have seen was different, right? Even after you've inserted the NAT rule, the connection continued to work on plain L2TP but stopped working once IPsec went up.

Plain L2tp went down exactly after I've changed port 33333 to 1701 in IPsec policy on the client.
Nevertheless there were 2 policies generated on the server one with 33333 port and another with 1701.
Adding NAT Rule didn't change anything for that moment.

But when I've restored exactly this config all was fine immediately ( so perhaps this rule solved the issue as it was present in the beginning this time)

So my action=src-nat ... dst-port=1701 rule overwrites the source address of the L2TP packets from the address currently assigned to the WAN interface to the static one we've chosen for the policy's src-address, but as the rule cannot be activated and deactivated dynamically depending on whether the IPsec policy is active or not, it acts even if the tunnel is down, which means that the ISP should normally drop such packet. Which further means that it is not possible to safely switch between plain L2TP and L2TP over IPsec, as the first way needs this rule not to be there while the second one requires it.

But I don't need to switch between plain l2tp and l2tp over ipsec. I need to have only secure connections.
And for now I can switch easily just by restoring configs which is fine for me. But it is useful only on this development phase. Because I can setup something only in the evening and need to have all the day some kind of connection between sites anyway.
Once I have all the sites with ipsec working properly i will never switch to plain l2tp.

So all in all, once you set up everything this way and reboot the client machine, it should always come up properly, but the transition from one approach to the other one is the tricky part of the operation. So you may need to write one simple script which will wait for 30 minutes and then disable the policy and the NAT rule and reboot the router, and schedule this script to run at startup. Then, you would create an even simpler script which would immediately enable the policy and the NAT rule and reboot the router, and schedule it to run in 5 minutes from the time you do it.

The idea is that if you get access after the first reboot, you can stop the script started right at boot during the 30 minutes it does nothing; if you don't, it will disable the policy and rule and the changes and reboot again.

You need a script even for the first reboot because you need to reboot right after enabling the policy, and if you issue /system reboot from a manually entered line, it always asks for confirmation, but it would never be delivered as by activating the policy you lose access. If run in non-interactive mode, such as if called within a script which is started using the scheduler, /system reboot does not ask for confirmation and reboots the machine immediately.

I don't understand why do I need any script in this case.
If I already have proper IPsec connection.

My only concern is what will happen if server router or client router will go down due to power failure. And after powering up will I have everything working?
If answer is yes, than I don't get why those scripts are needed.

-------
And another question which I don't get completely.
Why the second client can not activate ipsec policy with exactly the same setup. i need to have all of the clients connections secured. other way it does not make sense.
And plain l2tp works all the time for this client. Does this mean that knowing public server's ip and L2TP username and password someone can connect any time. Just traffic would be unencrypted.
I suppose ( and there can not be any other answer) that some different setting hides somewhere but I'm unable to get it. I did setup both clients performing each step one by one in exactly the same way.

sindy · Fri Oct 19, 2018 2:12 pm

And L2TP packets must flow inside that connection and NOT vice versa.

But I don't understand to what connection router gives priority. So which one (L2TP or IPSEC) is established first.
And how I can be sure that I have first off all IPSEC connection which carries out L2TP traffic.
And question in my head arises for what do i need l2tp tunnel itself then.
Sorry if these are noob questions.

They are not noob. The answer is that there is neither priority nor coordination, L2TP will keep trying to connect no matter whether IPsec is already running or not, but

as soon as the IPsec policy becomes active, it starts stealing the L2TP packets and transporting them using its SA. This is true if the source address of the L2TP packets is the one of the WAN and the policy's src-address matches that WAN address; if the source address of L2TP packets and the policy's src-address is our internal private address, the packets may nevertheless get through to the server before the policy gets up because the ISP will NAT them, but the responses won't make it back to the right address as the ISP doesn't have a route to that address (or it does but in his network, that private address belongs to some other host). So by using the private address, you prevent L2TP from working without the IPsec.
once all your clients are migrated, you may prevent L2TP connections incoming without IPsec from establishing by adding ipsec-policy=in,ipsec to the action=accept protocol=udp dst-port=1701 ... rule in /ip firewall filter. But it actually requires to re-arrange the firewall a bit more than this to make it really secure because with only that rule, once the L2TP connection establishes, it can continue even if the policy goes down later on.

By the way exports above are from client which is unable to install SAs

Do the "working" and "non-working" clients run the same RouterOS version? If not, I'd attribute that to incompatibility between the server and client versions and consider dealing with ut a waste of time, with one exception - if both clients are hidden behind the same public IP as seen from the server perspective. If this is the case, it will be easier to use Option2, because the modifications to the server configuration which permit to overcome the "multiple clients behind same NAT" problem are quite complex.

Plain L2tp went down exactly after I've changed port 33333 to 1701 in IPsec policy on the client.
Nevertheless there were 2 policies generated on the server one with 33333 port and another with 1701.

This sounds as if something didn't work well when handling the policy renegotiation at the server.

Adding NAT Rule didn't change anything for that moment.
But when I've restored exactly this config all was fine immediately ( so perhaps this rule solved the issue as it was present in the beginning this time)

I've tried to explain the above in my previous post - change (addition) of a NAT rule doesn't affect already ongoing connections which, if newly established, would be handled by that changed (added) rule.

But I don't need to switch between plain l2tp and l2tp over ipsec. I need to have only secure connections.
And for now I can switch easily just by restoring configs which is fine for me. But it is useful only on this development phase. Because I can setup something only in the evening and need to have all the day some kind of connection between sites anyway.
Once I have all the sites with ipsec working properly i will never switch to plain l2tp.

The sole idea behind the scripts is to be able to do the one-time switchover from plaintext to IPsec-encrypted L2TP knowing that you won't have to mount the horse if the switchover fails.

Configuration restore is another way of doing the same what the plaintext->encrypted script does (enable nat rule and 1701 policy and restart), but the revertion script (encrypted->plaintext) is a way to avoid driving to client site. But if you get to client site anyway before attempting the plaintext->encrypted transition, then yes, the scripts are useless. In my case, a visit to client site may involve a 1500 km flight to a different country, so I prefer to use methods allowing to recover from a mistake automatically.

My only concern is what will happen if server router or client router will go down due to power failure. And after powering up will I have everything working?
If answer is yes, than I don't get why those scripts are needed.

As explained above.

Why the second client can not activate ipsec policy with exactly the same setup. i need to have all of the clients connections secured. other way it does not make sense.

See above.

Does this mean that knowing public server's ip and L2TP username and password someone can connect any time. Just traffic would be unencrypted.

Yes it does. That's why we need to prevent L2TP without IPsec from establishing.

I suppose ( and there can not be any other answer) that some different setting hides somewhere but I'm unable to get it. I did setup both clients performing each step one by one in exactly the same way.

See above.

nomatter · Sat Oct 20, 2018 1:54 pm

Now I understand what you'd told me before.

Do the "working" and "non-working" clients run the same RouterOS version? If not, I'd attribute that to incompatibility between the server and client versions and consider dealing with ut a waste of time, with one exception - if both clients are hidden behind the same public IP as seen from the server perspective. If this is the case, it will be easier to use Option2, because the modifications to the server configuration which permit to overcome the "multiple clients behind same NAT" problem are quite complex.

Working client - 6.41.3
Non-working client - 6.34.4
Server - 6.34.4

Clients have different public IP when server router detects connection from them.
Working client's ip is the same all the time.
Non-working client's ip keeps changing all the time. I suppose that ISP of non-working client is behind the NAT off bigger ISP (could be the reason?)

So today I've decided to make things simpler just to be able to get more details while investigating the issue with second client.

What I did:
1. Reverted all three routers to plain l2tp state.
2. Activated "use ipsec" in settings of l2tp server.
3. Activated "use ipsec" on both clients.
4. Regarding nat Rules: on this stage only 1 rule is present on all three routers:
add action=masquerade chain=srcnat out-interface=WAN

The behavior of the system is the following:
1. Working client connected again. SA installed and there is traffic movement in both directions. So even after this kind of setup everything is fine again.
---
2. Non-working client tries to connect. I'm keeping to receive endless messages in server log

13:36:54 l2tp,info first L2TP UDP packet received from 193...
13:37:28 l2tp,info first L2TP UDP packet received from 193...
13:38:02 l2tp,info first L2TP UDP packet received from 193...
13:38:36 l2tp,info first L2TP UDP packet received from 193...
13:39:10 l2tp,info first L2TP UDP packet received from 193...
13:39:45 l2tp,info first L2TP UDP packet received from 193...
13:40:19 l2tp,info first L2TP UDP packet received from 193...

and in client's log

13:46:23 l2tp,ppp,info gl: terminating... - session closed 
13:46:23 l2tp,ppp,info gl: disconnected 
13:46:33 l2tp,ppp,info gl: initializing... 
13:46:33 l2tp,ppp,info gl: connecting... 
13:46:57 l2tp,ppp,info gl: terminating... - session closed 
13:46:57 l2tp,ppp,info gl: disconnected 
13:47:07 l2tp,ppp,info gl: initializing... 
13:47:07 l2tp,ppp,info gl: connecting... 
13:47:31 l2tp,ppp,info gl: terminating... - session closed 
13:47:31 l2tp,ppp,info gl: disconnected 
13:47:41 l2tp,ppp,info gl: initializing... 
13:47:41 l2tp,ppp,info gl: connecting...

3. Now ipsec policies on both non working client and server are generated dynamically.
4.SA on non working client are trying to install but don't last long:
and one important notice
from client to server I see some current bytes but in other direction (from server to client) always zero.

Maybe ISP of non-working client is causing this weird behavior, what do you think?

sindy · Sat Oct 20, 2018 3:16 pm

Working client - 6.41.3
Non-working client - 6.34.4
Server - 6.34.4

It does sound weird that a client running a different version than a server works better with said server than a client running the same version as the server, but I still deem it possible that something may not work perfect in 6.34.4 and having 6.41.3 only on client side may be sufficient to resolve that.

Clients have different public IP when server router detects connection from them.

So we can forget about the problem of multiple clients seen behind the same public address by the server. Good.

Non-working client's ip keeps changing all the time. I suppose that ISP of non-working client is behind the NAT off bigger ISP (could be the reason?)

There may be many reasons why a connection initiated by a client comes from another public IP each time. But the simplest explanation is that a large number of clients running on private WAN addresses are NATed to a much smaller pool of public addresses so each new pinhole is allocated on a randomly chosen address from the pool, without keeping any record of previous assignments.

3. Now ipsec policies on both non working client and server are generated dynamically.

Just FYI, both are created dynamically but each of them by a completely different mechanism:

on client, it is the l2tp configuration handler which dynamically creates both the peer and the policy based on the configuration of /interface l2tp-client (use-ipsec, ipsec-secret). This is something RouterOS - specific.
on server, the peer is also dynamically created by the l2tp configuration handler (so RouterOS - specific again), but the policy is generated as a mirror of the one indicated by the client, which is one of standard variants of IPsec negotiation.

from client to server I see some current bytes but in other direction (from server to client) always zero.

Can you post /ip ipsec policy print and /ip ipsec installed-sa print from the server in this state (systematically replacing each occurrence of the same public IP by same meaningful string)?

Maybe ISP of non-working client is causing this weird behavior, what do you think?

It doesn't sound likely to me. Yes, I've seen cases where plain ESP packets were dropped between peers in ISP's network, but in your case, the ESP gets encapsulated into UDP because one of the peers is NATed, which also causes the control communication to move from initial UDP connection between ports 500 to the same UDP connection on ports 4500 which is used for the UDP-encapsulated ESP transport packets - this is how the NAT traversal works. So unless the ISP uses some very sophisticated approach to drive you mad, it is not very likely that they would be selectively filtering some packets from a given UDP connection and let others from the same connection go through. So my bet is on wrong mirroring of the policy suggested by the client on server side, causing the L2TP packets not to be stolen by the policy and be sent directly. And as the pinhole in the ISP's NAT is open for the UDP-encapsulated ESP (towards server port 4500) but not for the plaintext UDP-transported L2TP packets (towards server port 1701), the ISP's firewall doesn't let the plaintext server->client packets through. And the creation of the dynamic policy at the server may depend on the way how the client suggests it, which could be the reason why a 6.41.3 client performs better against a 6.34.4 server than a 6.34.4 client. Seeing a log from the server (/system logging add topics=ipsec,!packet) might help confirm or reject these assumptions. If posting the log, don't forget to apply an IP address substitution consistent with the one used above for policy and installed-sa.

nomatter · Sat Oct 20, 2018 5:53 pm

It does sound weird that a client running a different version than a server works better with said server than a client running the same version as the server

Yes, very weird. I thought it's more likely to have troubles with newer version of RouterOS but not the same.

I still deem it possible that something may not work perfect in 6.34.4 and having 6.41.3 only on client side may be sufficient to resolve that

I will definitely update all routers. But I think this will happen in a month. So it would be very nice to have at least something before that.

ip ipsec policy print
Flags: T - template, X - disabled, D - dynamic, I - inactive, * - default 
 0 T * group=group1 src-address=::/0 dst-address=::/0 protocol=all 
       proposal=default template=yes 

 1  D  src-address=SERVER_PUBLIC_IPaddr/32 src-port=1701 dst-address=NONWorkingClient_detected_IPaddr/32 
       dst-port=55978 protocol=udp action=encrypt level=require 
       ipsec-protocols=esp tunnel=no sa-src-address=SERVER_PUBLIC_IPaddr 
       sa-dst-address=NONWorkingClient_detected_IPaddr priority=2 

 2  D  src-address=SERVER_PUBLIC_IPaddr/32 src-port=1701 dst-address=WorkingClient_detected_IPaddr/32 
       dst-port=any protocol=udp action=encrypt level=require 
       ipsec-protocols=esp tunnel=no sa-src-address=SERVER_PUBLIC_IPaddr 
       sa-dst-address=WorkingClient_detected_IPaddr priority=2 

 3  D  src-address=SERVER_PUBLIC_IPaddr/32 src-port=1701 dst-address=NONWorkingClient_detected_IPaddr/32 
       dst-port=59278 protocol=udp action=encrypt level=require 
       ipsec-protocols=esp tunnel=no sa-src-address=SERVER_PUBLIC_IPaddr 
       sa-dst-address=NONWorkingClient_detected_IPaddr priority=2

ip ipsec installed-sa print 
Flags: A - AH, E - ESP 
 0 E spi=0x6D433DB src-address=WorkingClient_detected_IPaddr:4500 dst-address=SERVER_PUBLIC_IPaddr:4500 state=mature auth-algorithm=sha1 enc-algorithm=aes-cbc 
     auth-key="245d3658f242e9630577c9534247f4e1dee94906" enc-key="a5cc16f952fe6ec6df54bcfe7de724c494d8753e780df68ce117ad6bffeb9322" addtime=oct/20/2018 16:50:54 
     expires-in=20m56s add-lifetime=24m/30m current-bytes=11826 replay=128 

 1 E spi=0x48FDAA4 src-address=SERVER_PUBLIC_IPaddr:4500 dst-address=WorkingClient_detected_IPaddr:4500 state=mature auth-algorithm=sha1 enc-algorithm=aes-cbc 
     auth-key="3b7d017bdfdb5a3f9f818e4af15c0d64310a53fa" enc-key="2414be5fcb3aabd4f024662ea42868c946d8b68a299bc3ea885398ad9db79330" addtime=oct/20/2018 16:50:54 
     expires-in=20m56s add-lifetime=24m/30m current-bytes=1982 replay=128 

 2 E spi=0xD64FA20 src-address=NONWorkingClient_detected_IPaddr:59278 dst-address=SERVER_PUBLIC_IPaddr:4500 state=mature auth-algorithm=sha1 enc-algorithm=aes-cbc 
     auth-key="d949adf5c16b689602451e79a8275d37c7459d06" enc-key="062c2b26d8cab2570a3afc81a863e76825c752288567c7ab9f452a237ab31d6f" addtime=oct/20/2018 16:59:36 
     expires-in=29m38s add-lifetime=24m/30m current-bytes=432 replay=128 

 3 E spi=0x670CF89 src-address=SERVER_PUBLIC_IPaddr:4500 dst-address=NONWorkingClient_detected_IPaddr:59278 state=mature auth-algorithm=sha1 enc-algorithm=aes-cbc 
     auth-key="e51a8080982fa1d590cbf84ad0e564bbe393e09a" enc-key="42b59e8b7273016516317bf7f965f5bf9adce162a1ab9042403926969d240bcd" add-lifetime=24m/30m replay=128

sindy · Sat Oct 20, 2018 7:05 pm

First - if you are concerned about privacy, edit your previous post and remove the log from there. I wrote topics=ipsec,!packet for a reason, and even if the packet dumps would not be there, some rows in the log reveal the IP addresses in hexadecimal.

Other than that - yes, there is some mess.

When you look for differences between policies and SAs of the working and the non-working client, you can see dst-port=any for the working one and dst-port=59278 (i.e. a particular value) for the non-working one. As the policy at server side is a mirror of the policy indicated by the client (which, from the point of view of IPsec, has been defined externally). So this is one clear difference between 6.41.3 and 6.34.1.

What is much worse is that the dst-port in the policy has the the same value like the port at the client side address in the SAs relevant to this policy. This is a clear indication of something being wrong.

The difference between transport and tunnel mode I've explained earlier only goes as deep as the IP header, not the UDP header. So there is no reason why the client-facing (dst) port in the traffic selector of the policy should be the same like the one from which the UDP-encapsulated ESP is coming. So although the client's ISP has src-nated the source port 4500 used by the IPsec stack to 59278, the source port used by the L2TP stack on the client side is always 1701 (although the server accepts packets coming with different source addresses).

So it seems that handling of the NAT traversal when creating a dynamic policy is broken in 6.34.1 (which makes me curious how it can work with the working client which is also NATed, otherwise the installed-sa would show only addresses without ports). When you were using the manual configuration, the src-port on client side policy was not specified which actually means src-port=any and it didn't help.

So all in all, I'm afraid you'll have to upgrade the server router ASAP to move forward.

nomatter · Sat Oct 20, 2018 7:25 pm

First - if you are concerned about privacy, edit your previous post and remove the log from there. I wrote topics=ipsec,!packet for a reason, and even if the packet dumps would not be there, some rows in the log reveal the IP addresses in hexadecimal.

oops, thank you.

So all in all, I'm afraid you'll have to upgrade the server router ASAP to move forward.

Ok, so what are the chances that i'll lose wan connection when upgrading from 6.34.4 to 6.40.9? and then from 6.40.9 to the latest version.
If my configuration with WAN and LAN persists then it's ok even if i lose everything else.

I can have a Teamviewer connection to the remote site. and do the upgrade using it. All I need that one of the pcs is able to connect to the router and router is able to provide Internet access.
but if I lose internet connection game over effect will be devastating. as the site will be at least few days broken
and someone will need to change all the plans and urgently go there.

sindy · Sat Oct 20, 2018 7:46 pm

I was originally thinking that all your devices are in a single city. If you can't get on site to upgrade, I'm afraid it would be too risky. You could use teamviewer over mobile internet, but if things go terribly wrong, you may have to netinstall which seems to be impossible using teamviewer as rumour has it that only a single network interface must be permitted on the Windows PC so that netinstall would work. So one extra device, upgraded and configured on your table and then sent to the destination site by s-mail, is the next best option after being physically present on site.

nomatter · Sat Oct 20, 2018 8:07 pm

I've just upgraded server-router. But it is 2 meters behind me:)

From 6.34.4 to 6.43.4 in single step. Firmware also.
Seems like configuration is ok.

Can we say that everything will be like that for client? Or there is still chance that something will go wrong? and this experiment does not mean anything.

sindy · Sat Oct 20, 2018 8:19 pm

A success at one place doesn't guarantee success on another place, Murphy's laws work everywhere. I've myself only had an issue once, when upgrading between release candidate versions, but other people did report bricked machines when upgrading between adjacent stable versions. So if it involves driving or sending other people on place out of schedule, I would not take the risk.

However, if it was the server in that star configuration you have described, this upgrade should be sufficient to make the non-working client work, in worst case with a little help of manual configuration on the server side.

nomatter · Sat Oct 20, 2018 9:59 pm

A success at one place doesn't guarantee success on another place, Murphy's laws work everywhere

Absolutely agree.

But I've decided to play russian roulette.
And luckily (this time) upgrade on client finished successfully.

So some problems were solved and some new arrived

1. When going through simple "use ipsec" option on clients and server everything is ok ( on both clients)

2. When performing setup we're discussing:
2.1. Policy on upgraded client become active. Respective mirror policy appears on server.
2.2. SAs are installed. There are current bytes in client to server direction. But no current bytes in server to client direction

2.3 L2TP connection is unable to establish

sindy · Sat Oct 20, 2018 10:18 pm

2.2. SAs are installed. There are current bytes in client to server direction. But no current bytes in server to client direction
2.3 L2TP connection is unable to establish

2.3 is a consequence of 2.2, but it is a puzzle to me.

Again, show me the /ip ipsec policy print and /ip ipsec installed-sa print in this state, from both ends (as I understand, you can get to the client even while the tunnel is down, as you have the PC with Teamviewer there). It would be fine also to have the l2tp log (disable the /system logging item for ipsec and add another one, /system logging add topics=l2tp,!packet) from the server side.

In any case, having L2TP encrypted using IKE(v1)-established SAs is much better than having them not encrypted at all.

nomatter · Sat Oct 20, 2018 10:44 pm

Server:

ip ipsec policy print 
Flags: T - template, X - disabled, D - dynamic, I - invalid, A - active, * - default 
 0 T * group=default src-address=::/0 dst-address=::/0 protocol=all proposal=default template=yes 

 1  DA  src-address=SERVER_PUBLIC_IPaddr/32 src-port=1701 dst-address=WorkingClient_detected_IPaddr/32 dst-port=any protocol=udp 
       action=encrypt level=unique ipsec-protocols=esp tunnel=no proposal=default ph2-count=1 

 2  DA  src-address=SERVER_PUBLIC_IPaddr/32 src-port=1701 dst-address=NONWorkingClient_detected_IPaddr/32 dst-port=any protocol=udp 
       action=encrypt level=unique ipsec-protocols=esp tunnel=no proposal=default ph2-count=1

ip ipsec installed-sa print 
Flags: H - hw-aead, A - AH, E - ESP 
 0  E spi=0xEF16AF7 src-address=WorkingClient_detected_IPaddr:4500 dst-address=SERVER_PUBLIC_IPaddr:4500 state=mature auth-algorithm=sha25>
      enc-algorithm=aes-cbc enc-key-size=256 
      auth-key="feb76457f67477ed415faaf9568000db0b54f5ece05cd5839e49cb4f48c20f7d" 
      enc-key="ee88825c462ef91280cac2dbe273ea47474a8d049937a964ffea54ae8ca40dc8" addtime=oct/20/2018 19:11:01 
      expires-in=11m26s add-lifetime=24m/30m current-bytes=24072 current-packets=221 replay=128 

 1  E spi=0x7BA5FC5 src-address=SERVER_PUBLIC_IPaddr:4500 dst-address=WorkingClient_detected_IPaddr:4500 state=mature auth-algorithm=sha25>
      enc-algorithm=aes-cbc enc-key-size=256 
      auth-key="4f92da8ef9dc2e790358c7af2e9a0f8547e673a16294c88ed3d01b06117373ee" 
      enc-key="9a210eedc49607033e846cd416cd21b9e4441dad432838ee9ccc94dde4cab1d3" addtime=oct/20/2018 19:11:01 
      expires-in=11m26s add-lifetime=24m/30m current-bytes=3612 current-packets=56 replay=128 

 2  E spi=0x8444769 src-address=NONWorkingClient_detected_IPaddr:35570 dst-address=SERVER_PUBLIC_IPaddr:4500 state=mature 
      auth-algorithm=sha256 enc-algorithm=aes-cbc enc-key-size=256 
      auth-key="fd1fef57ba81cb46c0d7f39ca63a389e447bc87a9c3caf3405342fcc62c8e81b" 
      enc-key="8b1d74d3acfcbc6d364a6ae12ff183460284cc7022cc62c1bb68c24d30463015" addtime=oct/20/2018 19:15:11 
      expires-in=15m36s add-lifetime=24m/30m current-bytes=16308 current-packets=151 replay=128 

 3  E spi=0x19E600A src-address=SERVER_PUBLIC_IPaddr:4500 dst-address=NONWorkingClient_detected_IPaddr:35570 state=mature 
      auth-algorithm=sha256 enc-algorithm=aes-cbc enc-key-size=256 
      auth-key="821b5b1fa80a245ae026a0b52792d392a151e49920d005ffc2550f25b6337991" 
      enc-key="076f8cf4412e05d27a0d0d52f423d01d748caed5dc0b211e8fdf30f03b132fda" add-lifetime=24m/30m replay=128

	  19:33:38 l2tp,info first L2TP UDP packet received from 192.168.128.254 
19:33:38 l2tp,debug tunnel 88 entering state: wait-ctl-conn 
19:33:54 l2tp,debug tunnel 88 received no replies, disconnecting 
19:33:54 l2tp,debug tunnel 88 entering state: dead 
19:34:12 l2tp,info first L2TP UDP packet received from 192.168.128.254 
19:34:12 l2tp,debug tunnel 89 entering state: wait-ctl-conn 
19:34:28 l2tp,debug tunnel 89 received no replies, disconnecting 
19:34:28 l2tp,debug tunnel 89 entering state: dead 
19:34:46 l2tp,info first L2TP UDP packet received from 192.168.128.254 
19:34:46 l2tp,debug tunnel 90 entering state: wait-ctl-conn

-------------------------------------------------------------------
Client

ip ipsec policy print 
Flags: T - template, X - disabled, D - dynamic, I - invalid, A - active, * - default 
 0 T * group=default src-address=::/0 dst-address=::/0 protocol=all proposal=default template=yes 

 1  A  src-address=192.168.128.254/32 src-port=any dst-address=SERVER_PUBLIC_IPaddr/32 dst-port=1701 protocol=udp 
 action=encrypt level=require ipsec-protocols=esp tunnel=no proposal=default ph2-count=1

ip ipsec installed-sa print 
Flags: H - hw-aead, A - AH, E - ESP 
 0  E spi=0x19E600A src-address=SERVER_PUBLIC_IPaddr:4500 dst-address=192.168.128.254:4500 state=mature auth-algorithm=sha256 enc-algorithm=aes-cbc enc-key-size=256 
      auth-key="821b5b1fa80a245ae026a0b52792d392a151e49920d005ffc2550f25b6337991" enc-key="076f8cf4412e05d27a0d0d52f423d01d748caed5dc0b211e8fdf30f03b132fda" 
      add-lifetime=24m/30m replay=128 

 1  E spi=0x8444769 src-address=192.168.128.254:4500 dst-address=SERVER_PUBLIC_IPaddr:4500 state=mature auth-algorithm=sha256 enc-algorithm=aes-cbc enc-key-size=256 
      auth-key="fd1fef57ba81cb46c0d7f39ca63a389e447bc87a9c3caf3405342fcc62c8e81b" enc-key="8b1d74d3acfcbc6d364a6ae12ff183460284cc7022cc62c1bb68c24d30463015" 
     addtime=oct/20/2018 22:15:11 expires-in=13m11s add-lifetime=24m/30m current-bytes=19332 current-packets=179 replay=128

UPDATE

When i changed on server and both clients the mode from main to IKE2 working client started to behave the same as non working - no current packet from server to client when watching SAs

sindy · Sat Oct 20, 2018 11:11 pm

I'm lost. Policies at both ends say tunnel=no, but nevertheless your l2tp log at server side says l2tp,info first L2TP UDP packet received from 192.168.128.254. This does not happen in my lab, where the client is behind a (double) NAT, and the same message in the server's l2tp log reports the address closest to the server as the l2tp source.

The rest is a consequence, the server sends a response to 192.168.128.254 which of course doesn't match the policy, so the responses don't go back via the tunnel.

nomatter · Sun Oct 21, 2018 12:17 am

I don't know maybe i'm already really tired and start to mess things up but
if i use simple option "use ipsec" than 1 client works and other no, if i setup things like you've proposed another client works and the previous one no.

For now I've done some hybrid setup where one client is using "use ipsec" and the second one normal setup - but even I don't like it

Now all of them have the same Router Os version (6.43.4) and same firmware version.

Really frustrated - things seems to follow some strange logic which i'm not able to get.

sindy · Sun Oct 21, 2018 12:35 am

In the meantime I've found in my lab setup that the action=src-nat rule is not sufficient to make it work; instead, I had to use a dedicated route to the server IP at the client with pref-src=the.private.ip.used because the l2tp-client uses the address internally and if the src-nat rule changes it, part of the sub-protocols of L2TP doesn't work (which is weird by its own). But this has nothing to do with what happens in your case, where the server reports the packets from the client to come from the private address used at the client. Can you show me what does /ip firewall connection prinit where dst-address~":(4500|1701) show when the manual configuration as per my proposal is active?

BTW, if you don't reboot both ends when changing configuration, disable the l2tp interface at client side for 4 minutes, change the configuration during those 4 minutes, and then enable the l2tp interface again. The UDP connections in the firewall's connection tracker need 3 minutes to time out.

I wIll respond tomorrow.

nomatter · Sun Oct 21, 2018 2:24 am

Here they are:
(Hope image is visible enough . Because code tags destroyed all the formatting)

sindy · Sun Oct 21, 2018 1:06 pm

Show me the complete configuration of the non-working client, using /export hide-sensitive verbose, and also /ip route print, when the manual configuration of IPsec is present but the policy is disabled. I have an impression that there is something wrong with the routing, and thus the client sends the IPsec packets towards the public IP of the server through the L2TP tunnel, which is the only way I can imagine how the private address of the client could be seen as packet source address of the server for packets towards SERVER_PUBLIC_IPaddr:4500.

Or just check whether put [/interface l2tp-client get [find] add-default-route] doesn't return true; if it does, do /interface l2tp-client set [find] add-default-route=no.

nomatter · Sun Oct 21, 2018 2:34 pm

put [/interface l2tp-client get [find] add-default-route]

Returns False.

Here is export from non-working client

/export hide-sensitive verbose        
# aug/29/2018 19:50:26 by RouterOS 6.43.4
# software id = ZUNB-FW0U
#
# model = 951G-2HnD
# serial number = 642E06875CDF
/interface bridge
add admin-mac=6C:3B:6B:C2:61:F7 ageing-time=5m arp=enabled arp-timeout=auto auto-mac=no comment=wifi+ethernet dhcp-snooping=no disabled=no fast-forward=no forward-delay=15s igmp-snooping=no \
    max-message-age=20s mtu=auto name=LAN priority=0x8000 protocol-mode=rstp transmit-hold-count=6 vlan-filtering=no
/interface ethernet
set [ find default-name=ether1 ] advertise=10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full arp=enabled arp-timeout=auto auto-negotiation=yes bandwidth=unlimited/unlimited comment=\
    "port1 - master" disabled=no full-duplex=yes l2mtu=1598 loop-protect=default loop-protect-disable-time=5m loop-protect-send-interval=5s mac-address=6C:3B:6B:C2:61:F7 mtu=1500 name=LAN1 \
    orig-mac-address=6C:3B:6B:C2:61:F7 rx-flow-control=off speed=100Mbps tx-flow-control=off
set [ find default-name=ether2 ] advertise=10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full arp=enabled arp-timeout=auto auto-negotiation=yes bandwidth=unlimited/unlimited comment=port2 \
    disabled=no full-duplex=yes l2mtu=1598 loop-protect=default loop-protect-disable-time=5m loop-protect-send-interval=5s mac-address=6C:3B:6B:C2:61:F8 mtu=1500 name=LAN2 orig-mac-address=\
    6C:3B:6B:C2:61:F8 rx-flow-control=off speed=100Mbps tx-flow-control=off
set [ find default-name=ether3 ] advertise=10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full arp=enabled arp-timeout=auto auto-negotiation=yes bandwidth=unlimited/unlimited comment=port3 \
    disabled=no full-duplex=yes l2mtu=1598 loop-protect=default loop-protect-disable-time=5m loop-protect-send-interval=5s mac-address=6C:3B:6B:C2:61:F9 mtu=1500 name=LAN3 orig-mac-address=\
    6C:3B:6B:C2:61:F9 rx-flow-control=off speed=100Mbps tx-flow-control=off
set [ find default-name=ether4 ] advertise=10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full arp=enabled arp-timeout=auto auto-negotiation=yes bandwidth=unlimited/unlimited comment=port4 \
    disabled=no full-duplex=yes l2mtu=1598 loop-protect=default loop-protect-disable-time=5m loop-protect-send-interval=5s mac-address=6C:3B:6B:C2:61:FA mtu=1500 name=LAN4 orig-mac-address=\
    6C:3B:6B:C2:61:FA rx-flow-control=off speed=100Mbps tx-flow-control=off
set [ find default-name=ether5 ] advertise=10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full arp=enabled arp-timeout=auto auto-negotiation=yes bandwidth=unlimited/unlimited comment=port5 \
    disabled=no full-duplex=yes l2mtu=1598 loop-protect=default loop-protect-disable-time=5m loop-protect-send-interval=5s mac-address=D4:BE:D9:50:B9:7F mtu=1500 name=WAN orig-mac-address=\
    6C:3B:6B:C2:61:FB rx-flow-control=off speed=100Mbps tx-flow-control=off
/queue interface
set LAN queue=no-queue
/interface ethernet switch
set 0 cpu-flow-control=yes mirror-source=none mirror-target=none name=switch1
/interface ethernet switch port
set 0 default-vlan-id=auto vlan-header=leave-as-is vlan-mode=disabled
set 1 default-vlan-id=auto vlan-header=leave-as-is vlan-mode=disabled
set 2 default-vlan-id=auto vlan-header=leave-as-is vlan-mode=disabled
set 3 default-vlan-id=auto vlan-header=leave-as-is vlan-mode=disabled
set 4 default-vlan-id=auto vlan-header=leave-as-is vlan-mode=disabled
set 5 default-vlan-id=auto vlan-header=leave-as-is vlan-mode=disabled
/interface list
set [ find name=all ] comment="contains all interfaces" exclude="" include="" name=all
set [ find name=none ] comment="contains no interfaces" exclude="" include="" name=none
set [ find name=dynamic ] comment="contains dynamic interfaces" exclude="" include="" name=dynamic
/interface lte apn
set [ find default=yes ] add-default-route=yes apn=internet default-route-distance=2 name=default use-peer-dns=yes
/interface wireless security-profiles
set [ find default=yes ] authentication-types=wpa2-psk disable-pmkid=no eap-methods="" group-ciphers=tkip,aes-ccm group-key-update=5m interim-update=0s management-protection=disabled mode=dynamic-keys \
    mschapv2-username="" name=default radius-called-format=mac:ssid radius-eap-accounting=no radius-mac-accounting=no radius-mac-authentication=no radius-mac-caching=disabled radius-mac-format=\
    XX:XX:XX:XX:XX:XX radius-mac-mode=as-username static-algo-0=none static-algo-1=none static-algo-2=none static-algo-3=none static-sta-private-algo=none static-transmit-key=key-0 supplicant-identity=\
    MikroTik tls-certificate=none tls-mode=no-certificates unicast-ciphers=tkip,aes-ccm
add authentication-types=wpa2-psk disable-pmkid=no eap-methods="" group-ciphers=tkip,aes-ccm group-key-update=5m interim-update=0s management-protection=allowed mode=dynamic-keys mschapv2-username="" \
    name=wifi_guest radius-called-format=mac:ssid radius-eap-accounting=no radius-mac-accounting=no radius-mac-authentication=no radius-mac-caching=disabled radius-mac-format=XX:XX:XX:XX:XX:XX \
    radius-mac-mode=as-username static-algo-0=none static-algo-1=none static-algo-2=none static-algo-3=none static-sta-private-algo=none static-transmit-key=key-0 supplicant-identity="" tls-certificate=\
    none tls-mode=no-certificates unicast-ciphers=tkip,aes-ccm
/interface wireless
set [ find default-name=wlan1 ] adaptive-noise-immunity=none allow-sharedkey=no ampdu-priorities=0 amsdu-limit=8192 amsdu-threshold=8192 antenna-gain=0 area="" arp=enabled arp-timeout=auto band=2ghz-b/g \
    basic-rates-a/g=6Mbps basic-rates-b=1Mbps bridge-mode=enabled channel-width=20mhz compression=no country=no_country_set default-ap-tx-limit=0 default-authentication=yes default-client-tx-limit=0 \
    default-forwarding=yes disable-running-check=no disabled=no disconnect-timeout=3s distance=dynamic frame-lifetime=0 frequency=2412 frequency-mode=manual-txpower frequency-offset=0 guard-interval=any \
    hide-ssid=no ht-basic-mcs=mcs-0,mcs-1,mcs-2,mcs-3,mcs-4,mcs-5,mcs-6,mcs-7 ht-supported-mcs=\
    mcs-0,mcs-1,mcs-2,mcs-3,mcs-4,mcs-5,mcs-6,mcs-7,mcs-8,mcs-9,mcs-10,mcs-11,mcs-12,mcs-13,mcs-14,mcs-15,mcs-16,mcs-17,mcs-18,mcs-19,mcs-20,mcs-21,mcs-22,mcs-23 hw-fragmentation-threshold=disabled \
    hw-protection-mode=none hw-protection-threshold=0 hw-retries=7 interworking-profile=disabled keepalive-frames=enabled l2mtu=1600 mac-address=6C:3B:6B:C2:61:FC max-station-count=2007 mode=ap-bridge \
    mtu=1500 multicast-buffering=enabled multicast-helper=default name=wlan1 noise-floor-threshold=default nv2-cell-radius=30 nv2-downlink-ratio=50 nv2-mode=dynamic-downlink nv2-noise-floor-offset=\
    default nv2-qos=default nv2-queue-count=2 nv2-security=disabled nv2-sync-secret="" on-fail-retry-time=100ms preamble-mode=both radio-name=6C3B6BC261FC rate-selection=advanced rate-set=default \
    rx-chains=0,1 scan-list=default secondary-channel="" security-profile=default ssid=AM station-bridge-clone-mac=00:00:00:00:00:00 station-roaming=enabled supported-rates-a/g=\
    6Mbps,9Mbps,12Mbps,18Mbps,24Mbps,36Mbps,48Mbps,54Mbps supported-rates-b=1Mbps,2Mbps,5.5Mbps,11Mbps tdma-period-size=2 tx-chains=0,1 tx-power-mode=default update-stats-interval=disabled vlan-id=1 \
    vlan-mode=no-tag wds-cost-range=50-150 wds-default-bridge=none wds-default-cost=100 wds-ignore-ssid=no wds-mode=disabled wireless-protocol=any wmm-support=disabled wps-mode=push-button
add area="" arp=enabled arp-timeout=auto bridge-mode=enabled comment="guest access" default-ap-tx-limit=0 default-authentication=yes default-client-tx-limit=0 default-forwarding=yes \
    disable-running-check=no disabled=no hide-ssid=no interworking-profile=disabled keepalive-frames=disabled l2mtu=1600 mac-address=02:00:00:AA:00:00 master-interface=wlan1 max-station-count=2007 mode=\
    ap-bridge mtu=1500 multicast-buffering=disabled multicast-helper=default name=wlan2 security-profile=wifi_guest ssid=AM_Guest station-bridge-clone-mac=00:00:00:00:00:00 station-roaming=\
    enabled update-stats-interval=disabled vlan-id=1 vlan-mode=no-tag wds-cost-range=0 wds-default-bridge=none wds-default-cost=0 wds-ignore-ssid=no wds-mode=disabled wmm-support=disabled wps-mode=\
    push-button
/interface wireless manual-tx-power-table
set wlan1 manual-tx-powers="1Mbps:17,2Mbps:17,5.5Mbps:17,11Mbps:17,6Mbps:17,9Mbps:17,12Mbps:17,18Mbps:17,24Mbps:17,36Mbps:17,48Mbps:17,54Mbps:17,HT20-0:17,HT20-1:17,HT20-2:17,HT20-3:17,HT20-4:17,HT20-5:17\
    ,HT20-6:17,HT20-7:17,HT40-0:17,HT40-1:17,HT40-2:17,HT40-3:17,HT40-4:17,HT40-5:17,HT40-6:17,HT40-7:17"
set wlan2 comment="guest access"
/interface wireless nstreme
set wlan1 disable-csma=no enable-nstreme=no enable-polling=yes framer-limit=3200 framer-policy=none
set wlan2 comment="guest access"
/ip dhcp-client option
set clientid_duid code=61 name=clientid_duid value="0xff\$(CLIENT_DUID)"
set clientid code=61 name=clientid value="0x01\$(CLIENT_MAC)"
set hostname code=12 name=hostname value="\$(HOSTNAME)"
/ip hotspot profile
set [ find default=yes ] dns-name="" hotspot-address=0.0.0.0 html-directory=hotspot html-directory-override="" http-cookie-lifetime=3d http-proxy=0.0.0.0:0 login-by=cookie,http-chap name=default \
    rate-limit="" smtp-server=0.0.0.0 split-user-domain=no use-radius=no
/ip hotspot user profile
set [ find default=yes ] add-mac-cookie=yes address-list="" idle-timeout=none !insert-queue-before keepalive-timeout=2m mac-cookie-timeout=3d name=default !parent-queue !queue-type shared-users=1 \
    status-autorefresh=1m transparent-proxy=no
/ip ipsec mode-config
set [ find default=yes ] name=request-only responder=no
/ip ipsec peer profile
set [ find default=yes ] dh-group=modp3072 dpd-interval=2m dpd-maximum-failures=5 enc-algorithm=aes-256 hash-algorithm=sha256 lifetime=1d name=default nat-traversal=yes proposal-check=obey
/ip ipsec policy group
set [ find default=yes ] name=default
/ip ipsec proposal
set [ find default=yes ] auth-algorithms=sha256 disabled=no enc-algorithms=aes-256-cbc lifetime=30m name=default pfs-group=modp3072
/ip pool
add name=dhcp_pool1 ranges=192.168.128.10-192.168.128.100
add name=dhcp_pool2 ranges=192.168.17.200-192.168.17.220
/ip dhcp-server
add address-pool=dhcp_pool1 authoritative=after-2sec-delay bootp-support=static disabled=no interface=LAN lease-script="" lease-time=1d name=dhcp1 use-radius=no
add address-pool=dhcp_pool2 authoritative=after-2sec-delay bootp-support=static disabled=no interface=wlan2 lease-script="" lease-time=1h name=dhcp2 use-radius=no
/ppp profile
set *0 address-list="" !bridge !bridge-horizon !bridge-path-cost !bridge-port-priority change-tcp-mss=yes !dns-server !idle-timeout !incoming-filter !insert-queue-before !interface-list !local-address \
    name=default on-down="" on-up="" only-one=default !outgoing-filter !parent-queue !queue-type !rate-limit !remote-address !session-timeout use-compression=default use-encryption=default use-mpls=\
    default use-upnp=default !wins-server
set *FFFFFFFE address-list="" !bridge !bridge-horizon !bridge-path-cost !bridge-port-priority change-tcp-mss=yes !dns-server !idle-timeout !incoming-filter !insert-queue-before !interface-list \
    !local-address name=default-encryption on-down="" on-up="" only-one=default !outgoing-filter !parent-queue !queue-type !rate-limit !remote-address !session-timeout use-compression=default \
    use-encryption=yes use-mpls=default use-upnp=default !wins-server
/interface l2tp-client
add add-default-route=no allow=mschap2 allow-fast-path=no connect-to=SERVER_PUBLIC_IPaddr dial-on-demand=no disabled=no keepalive-timeout=60 max-mru=1450 max-mtu=1450 mrru=1600 name=gl profile=\
    default-encryption use-ipsec=no user=gl
/queue interface
set gl queue=no-queue
/queue type
set 0 kind=pfifo name=default pfifo-limit=50
set 1 kind=pfifo name=ethernet-default pfifo-limit=50
set 2 kind=sfq name=wireless-default sfq-allot=1514 sfq-perturb=5
set 3 kind=red name=synchronous-default red-avg-packet=1000 red-burst=20 red-limit=60 red-max-threshold=50 red-min-threshold=10
set 4 kind=sfq name=hotspot-default sfq-allot=1514 sfq-perturb=5
set 5 kind=pcq name=pcq-upload-default pcq-burst-rate=0 pcq-burst-threshold=0 pcq-burst-time=10s pcq-classifier=src-address pcq-dst-address-mask=32 pcq-dst-address6-mask=128 pcq-limit=50KiB pcq-rate=0 \
    pcq-src-address-mask=32 pcq-src-address6-mask=128 pcq-total-limit=2000KiB
set 6 kind=pcq name=pcq-download-default pcq-burst-rate=0 pcq-burst-threshold=0 pcq-burst-time=10s pcq-classifier=dst-address pcq-dst-address-mask=32 pcq-dst-address6-mask=128 pcq-limit=50KiB pcq-rate=0 \
    pcq-src-address-mask=32 pcq-src-address6-mask=128 pcq-total-limit=2000KiB
set 7 kind=none name=only-hardware-queue
set 8 kind=mq-pfifo mq-pfifo-limit=50 name=multi-queue-ethernet-default
set 9 kind=pfifo name=default-small pfifo-limit=10
/queue interface
set LAN1 queue=only-hardware-queue
set LAN2 queue=only-hardware-queue
set LAN3 queue=only-hardware-queue
set LAN4 queue=only-hardware-queue
set WAN queue=only-hardware-queue
set wlan1 queue=wireless-default
set wlan2 queue=wireless-default
/routing bgp instance
set default as=65530 client-to-client-reflection=yes !cluster-id !confederation disabled=no ignore-as-path-len=no name=default out-filter="" redistribute-connected=no redistribute-ospf=no \
    redistribute-other-bgp=no redistribute-rip=no redistribute-static=no router-id=0.0.0.0 routing-table=""
/routing ospf instance
set [ find default=yes ] disabled=no distribute-default=never !domain-id !domain-tag in-filter=ospf-in metric-bgp=auto metric-connected=20 metric-default=1 metric-other-ospf=auto metric-rip=20 \
    metric-static=20 !mpls-te-area !mpls-te-router-id name=default out-filter=ospf-out redistribute-bgp=no redistribute-connected=no redistribute-other-ospf=no redistribute-rip=no redistribute-static=no \
    router-id=0.0.0.0 !routing-table !use-dn
/routing ospf area
set [ find default=yes ] area-id=0.0.0.0 disabled=no instance=default name=backbone type=default
/snmp community
set [ find default=yes ] addresses=::/0 authentication-protocol=MD5 encryption-protocol=DES name=public read-access=yes security=none write-access=no
/system logging action
set 0 memory-lines=1000 memory-stop-on-full=no name=memory target=memory
set 1 disk-file-count=2 disk-file-name=log disk-lines-per-file=1000 disk-stop-on-full=no name=disk target=disk
set 2 name=echo remember=yes target=echo
set 3 bsd-syslog=no name=remote remote=0.0.0.0 remote-port=514 src-address=0.0.0.0 syslog-facility=daemon syslog-severity=auto syslog-time-format=bsd-syslog target=remote
/user group
set read name=read policy=local,telnet,ssh,reboot,read,test,winbox,password,web,sniff,sensitive,api,romon,tikapp,!ftp,!write,!policy,!dude skin=default
set write name=write policy=local,telnet,ssh,reboot,read,write,test,winbox,password,web,sniff,sensitive,api,romon,tikapp,!ftp,!policy,!dude skin=default
set full name=full policy=local,telnet,ssh,ftp,reboot,read,write,policy,test,winbox,password,web,sniff,sensitive,api,romon,dude,tikapp skin=default
/caps-man aaa
set called-format=mac:ssid interim-update=disabled mac-caching=disabled mac-format=XX:XX:XX:XX:XX:XX mac-mode=as-username
/caps-man manager
set ca-certificate=none certificate=none enabled=no package-path="" require-peer-certificate=no upgrade-policy=none
/caps-man manager interface
set [ find default=yes ] disabled=no forbid=no interface=all
/certificate settings
set crl-download=yes crl-store=system crl-use=yes
/interface bridge port
add auto-isolate=no bpdu-guard=no bridge=LAN broadcast-flood=yes disabled=no edge=auto fast-leave=no frame-types=admit-all horizon=none hw=yes ingress-filtering=no interface=LAN1 internal-path-cost=10 \
    learn=auto multicast-router=temporary-query path-cost=10 point-to-point=auto priority=0x80 pvid=1 restricted-role=no restricted-tcn=no tag-stacking=no trusted=no unknown-multicast-flood=yes \
    unknown-unicast-flood=yes
add auto-isolate=no bpdu-guard=no bridge=LAN broadcast-flood=yes disabled=no edge=auto fast-leave=no frame-types=admit-all horizon=none ingress-filtering=no interface=wlan1 internal-path-cost=10 learn=\
    auto multicast-router=temporary-query path-cost=10 point-to-point=auto priority=0x80 pvid=1 restricted-role=no restricted-tcn=no tag-stacking=no trusted=no unknown-multicast-flood=yes \
    unknown-unicast-flood=yes
add auto-isolate=no bpdu-guard=no bridge=LAN broadcast-flood=yes disabled=no edge=auto fast-leave=no frame-types=admit-all horizon=none hw=yes ingress-filtering=no interface=LAN2 internal-path-cost=10 \
    learn=auto multicast-router=temporary-query path-cost=10 point-to-point=auto priority=0x80 pvid=1 restricted-role=no restricted-tcn=no tag-stacking=no trusted=no unknown-multicast-flood=yes \
    unknown-unicast-flood=yes
add auto-isolate=no bpdu-guard=no bridge=LAN broadcast-flood=yes disabled=no edge=auto fast-leave=no frame-types=admit-all horizon=none hw=yes ingress-filtering=no interface=LAN3 internal-path-cost=10 \
    learn=auto multicast-router=temporary-query path-cost=10 point-to-point=auto priority=0x80 pvid=1 restricted-role=no restricted-tcn=no tag-stacking=no trusted=no unknown-multicast-flood=yes \
    unknown-unicast-flood=yes
add auto-isolate=no bpdu-guard=no bridge=LAN broadcast-flood=yes disabled=no edge=auto fast-leave=no frame-types=admit-all horizon=none hw=yes ingress-filtering=no interface=LAN4 internal-path-cost=10 \
    learn=auto multicast-router=temporary-query path-cost=10 point-to-point=auto priority=0x80 pvid=1 restricted-role=no restricted-tcn=no tag-stacking=no trusted=no unknown-multicast-flood=yes \
    unknown-unicast-flood=yes
/interface bridge settings
set allow-fast-path=yes use-ip-firewall=no use-ip-firewall-for-pppoe=no use-ip-firewall-for-vlan=no
/ip firewall connection tracking
set enabled=auto generic-timeout=10m icmp-timeout=10s tcp-close-timeout=10s tcp-close-wait-timeout=10s tcp-established-timeout=1d tcp-fin-wait-timeout=10s tcp-last-ack-timeout=10s \
    tcp-max-retrans-timeout=5m tcp-syn-received-timeout=5s tcp-syn-sent-timeout=5s tcp-time-wait-timeout=10s tcp-unacked-timeout=5m udp-stream-timeout=3m udp-timeout=10s
/ip neighbor discovery-settings
set discover-interface-list=!dynamic
/ip settings
set accept-redirects=no accept-source-route=no allow-fast-path=yes arp-timeout=30s icmp-rate-limit=10 icmp-rate-mask=0x1818 ip-forward=yes max-neighbor-entries=8192 route-cache=yes rp-filter=no \
    secure-redirects=yes send-redirects=yes tcp-syncookies=no
/interface detect-internet
set detect-interface-list=none internet-interface-list=none lan-interface-list=none wan-interface-list=none
/interface l2tp-server server
set allow-fast-path=no authentication=pap,chap,mschap1,mschap2 caller-id-type=ip-address default-profile=default-encryption enabled=no keepalive-timeout=30 max-mru=1450 max-mtu=1450 max-sessions=\
    unlimited mrru=disabled one-session-per-host=no use-ipsec=no
/interface ovpn-server server
set auth=sha1,md5 cipher=blowfish128,aes128 default-profile=default enabled=no keepalive-timeout=60 mac-address=FE:EB:9F:52:1B:9F max-mtu=1500 mode=ip netmask=24 port=1194 require-client-certificate=no
/interface pptp-server server
set authentication=mschap1,mschap2 default-profile=default-encryption enabled=no keepalive-timeout=30 max-mru=1450 max-mtu=1450 mrru=disabled
/interface sstp-server server
set authentication=pap,chap,mschap1,mschap2 certificate=none default-profile=default enabled=no force-aes=no keepalive-timeout=60 max-mru=1500 max-mtu=1500 mrru=disabled pfs=no port=443 tls-version=any \
    verify-client-certificate=no
/interface wireless align
set active-mode=yes audio-max=-20 audio-min=-100 audio-monitor=00:00:00:00:00:00 filter-mac=00:00:00:00:00:00 frame-size=300 frames-per-second=25 receive-all=no ssid-all=no
/interface wireless cap
set bridge=none caps-man-addresses="" caps-man-certificate-common-names="" caps-man-names="" certificate=none discovery-interfaces="" enabled=no interfaces="" lock-to-caps-man=no static-virtual=no
/interface wireless sniffer
set channel-time=200ms file-limit=10 file-name="" memory-limit=10 multiple-channels=no only-headers=no receive-errors=no streaming-enabled=no streaming-max-rate=0 streaming-server=0.0.0.0
/interface wireless snooper
set channel-time=200ms multiple-channels=yes receive-errors=no
/ip accounting
set account-local-traffic=no enabled=no threshold=256
/ip accounting web-access
set accessible-via-web=no address=0.0.0.0/0
/ip address
add address=192.168.128.254/24 disabled=no interface=LAN network=192.168.128.0
add address=192.168.17.1/24 disabled=no interface=wlan2 network=192.168.17.0
/ip cloud
set ddns-enabled=no update-time=yes
/ip cloud advanced
set use-local-address=no
/ip dhcp-client
add add-default-route=yes default-route-distance=1 dhcp-options=hostname,clientid disabled=no interface=WAN use-peer-dns=yes use-peer-ntp=yes
/ip dhcp-server config
set store-leases-disk=5m
/ip dhcp-server network
add address=192.168.17.0/24 caps-manager="" dhcp-option="" dns-server="" gateway=192.168.17.1 ntp-server="" wins-server=""
add address=192.168.128.0/24 caps-manager="" dhcp-option="" dns-server="" gateway=192.168.128.254 ntp-server="" wins-server=""
/ip dns
set allow-remote-requests=no cache-max-ttl=1w cache-size=2048KiB max-concurrent-queries=100 max-concurrent-tcp-sessions=20 max-udp-packet-size=4096 query-server-timeout=2s query-total-timeout=10s \
    servers=""
/ip firewall address-list
add address=192.168.0.5 disabled=no list="no internet"
/ip firewall filter
add action=drop chain=forward !connection-bytes !connection-limit !connection-mark !connection-nat-state !connection-rate !connection-state !connection-type !content disabled=yes !dscp !dst-address \
    !dst-address-list !dst-address-type !dst-limit !dst-port !fragment !hotspot !icmp-options !in-bridge-port !in-interface !ingress-priority !ipsec-policy !ipv4-options !layer7-protocol !limit log=no \
    log-prefix="" !nth !out-bridge-port out-interface=WAN !p2p !packet-mark !packet-size !per-connection-classifier !port !priority !protocol !psd !random !routing-mark !routing-table src-address=\
    192.168.128.5 !src-address-list !src-address-type !src-mac-address !src-port !tcp-flags !tcp-mss !time !ttl
add action=accept chain=input !connection-bytes !connection-limit !connection-mark !connection-nat-state !connection-rate !connection-state !connection-type !content disabled=no !dscp !dst-address \
    !dst-address-list !dst-address-type !dst-limit dst-port=1701,500,4500 !fragment !hotspot !icmp-options !in-bridge-port !in-interface !ingress-priority !ipsec-policy !ipv4-options !layer7-protocol \
    !limit log=no log-prefix="" !nth !out-bridge-port !out-interface !p2p !packet-mark !packet-size !per-connection-classifier !port !priority protocol=udp !psd !random !routing-mark !routing-table \
    !src-address !src-address-list !src-address-type !src-mac-address !src-port !tcp-flags !tcp-mss !time !ttl
add action=accept chain=input !connection-bytes !connection-limit !connection-mark !connection-nat-state !connection-rate !connection-state !connection-type !content disabled=no !dscp !dst-address \
    !dst-address-list !dst-address-type !dst-limit !dst-port !fragment !hotspot !icmp-options !in-bridge-port !in-interface !ingress-priority !ipsec-policy !ipv4-options !layer7-protocol !limit log=no \
    log-prefix="" !nth !out-bridge-port !out-interface !p2p !packet-mark !packet-size !per-connection-classifier !port !priority protocol=ipsec-esp !psd !random !routing-mark !routing-table !src-address \
    !src-address-list !src-address-type !src-mac-address !src-port !tcp-flags !tcp-mss !time !ttl
add action=drop chain=forward comment="drop guest wlan from local wlan" !connection-bytes !connection-limit !connection-mark !connection-nat-state !connection-rate !connection-state !connection-type \
    !content disabled=no !dscp dst-address=192.168.128.0/24 !dst-address-list !dst-address-type !dst-limit !dst-port !fragment !hotspot !icmp-options !in-bridge-port !in-interface !ingress-priority \
    !ipsec-policy !ipv4-options !layer7-protocol !limit log=no log-prefix="" !nth !out-bridge-port !out-interface !p2p !packet-mark !packet-size !per-connection-classifier !port !priority !protocol !psd \
    !random !routing-mark !routing-table src-address=192.168.17.0/24 !src-address-list !src-address-type !src-mac-address !src-port !tcp-flags !tcp-mss !time !ttl
add action=drop chain=input comment="drop incoming connections" !connection-bytes !connection-limit !connection-mark !connection-nat-state !connection-rate !connection-state !connection-type !content \
    disabled=no !dscp !dst-address !dst-address-list !dst-address-type !dst-limit !dst-port !fragment !hotspot !icmp-options !in-bridge-port in-interface=WAN !ingress-priority !ipsec-policy !ipv4-options \
    !layer7-protocol !limit log=no log-prefix="" !nth !out-bridge-port !out-interface !p2p !packet-mark !packet-size !per-connection-classifier !port !priority !protocol !psd !random !routing-mark \
    !routing-table !src-address !src-address-list !src-address-type !src-mac-address !src-port !tcp-flags !tcp-mss !time !ttl
/ip firewall nat
add action=src-nat chain=srcnat dst-address=SERVER_PUBLIC_IPaddr dst-port=1701 protocol=udp to-addresses=192.168.128.254 !to-ports
add action=masquerade chain=srcnat !connection-bytes !connection-limit !connection-mark !connection-rate !connection-type !content disabled=no !dscp !dst-address !dst-address-list !dst-address-type \
    !dst-limit !dst-port !fragment !hotspot !icmp-options !in-bridge-port !in-interface !ingress-priority !ipsec-policy !ipv4-options !layer7-protocol !limit log=no log-prefix="" !nth !out-bridge-port \
    out-interface=WAN !packet-mark !packet-size !per-connection-classifier !port !priority !protocol !psd !random !routing-mark !routing-table !src-address !src-address-list !src-address-type \
    !src-mac-address !src-port !tcp-mss !time !to-addresses !to-ports !ttl
add action=masquerade chain=srcnat !connection-bytes !connection-limit !connection-mark !connection-rate !connection-type !content disabled=yes !dscp !dst-address !dst-address-list !dst-address-type \
    !dst-limit !dst-port !fragment !hotspot !icmp-options !in-bridge-port !in-bridge-port-list !in-interface !in-interface-list !ingress-priority !ipsec-policy !ipv4-options !layer7-protocol !limit log=\
    no log-prefix="" !nth !out-bridge-port !out-bridge-port-list out-interface=gl !out-interface-list !packet-mark !packet-size !per-connection-classifier !port !priority !protocol !psd !random \
    !routing-mark !routing-table !src-address !src-address-list !src-address-type !src-mac-address !src-port !tcp-mss !time !tls-host !to-addresses !to-ports !ttl
/ip firewall service-port
set ftp disabled=no ports=21
set tftp disabled=no ports=69
set irc disabled=no ports=6667
set h323 disabled=no
set sip disabled=no ports=5060,5061 sip-direct-media=yes sip-timeout=1h
set pptp disabled=no
set udplite disabled=no
set dccp disabled=no
set sctp disabled=no
/ip hotspot service-port
set ftp disabled=no ports=21
/ip hotspot user
set [ find default=yes ] comment="counters and limits for trial users" disabled=no name=default-trial
/ip ipsec peer
add address=SERVER_PUBLIC_IPaddr/32 auth-method=pre-shared-key-xauth disabled=no exchange-mode=main generate-policy=no local-address=192.168.128.254 policy-template-group=default profile=default \
    send-initial-contact=no xauth-login=gl
/ip ipsec policy
set 0 disabled=no dst-address=::/0 group=default proposal=default protocol=all src-address=::/0 template=yes
add action=encrypt disabled=no dst-address=SERVER_PUBLIC_IPaddr/32 dst-port=1701 ipsec-protocols=esp level=require proposal=default protocol=udp src-address=192.168.128.254/32 src-port=any tunnel=no
/ip ipsec user settings
set xauth-use-radius=no
/ip proxy
set always-from-cache=no anonymous=no cache-administrator=webmaster cache-hit-dscp=4 cache-on-disk=no cache-path=web-proxy enabled=no max-cache-object-size=2048KiB max-cache-size=unlimited \
    max-client-connections=600 max-fresh-time=3d max-server-connections=600 parent-proxy=:: parent-proxy-port=0 port=8080 serialize-connections=no src-address=::
/ip route
add !bgp-as-path !bgp-atomic-aggregate !bgp-communities !bgp-local-pref !bgp-med !bgp-origin !bgp-prepend !check-gateway comment="to PACS network" disabled=no distance=1 dst-address=192.168.2.0/24 \
    gateway=10.50.0.10 pref-src=10.50.0.11 !route-tag !routing-mark scope=30 target-scope=10
add !bgp-as-path !bgp-atomic-aggregate !bgp-communities !bgp-local-pref !bgp-med !bgp-origin !bgp-prepend !check-gateway comment="to radiologist network" disabled=no distance=1 dst-address=192.168.3.0/24 \
    gateway=10.50.0.10 pref-src=10.50.0.11 !route-tag !routing-mark scope=30 target-scope=10
add !bgp-as-path !bgp-atomic-aggregate !bgp-communities !bgp-local-pref !bgp-med !bgp-origin !bgp-prepend !check-gateway comment="to radiologist2 (kv office) network" disabled=no distance=1 \
    dst-address=192.168.5.0/24 gateway=10.50.0.10 pref-src=10.50.0.11 !route-tag !routing-mark scope=30 target-scope=10
/ip service
set telnet address="" disabled=yes port=23
set ftp address="" disabled=yes port=21
set www address="" disabled=yes port=80
set ssh address="" disabled=yes port=22
set www-ssl address="" certificate=none disabled=yes port=443
set api address="" disabled=yes port=8728
set winbox address="" disabled=no port=8291
set api-ssl address="" certificate=none disabled=yes port=8729
/ip smb
set allow-guests=yes comment=MikrotikSMB domain=MSHOME enabled=no interfaces=all
/ip smb shares
set [ find default=yes ] comment="default share" directory=/pub disabled=no max-sessions=10 name=pub
/ip smb users
set [ find default=yes ] disabled=no name=guest read-only=yes
/ip socks
set connection-idle-timeout=2m enabled=no max-connections=200 port=1080
/ip ssh
set always-allow-password-login=no forwarding-enabled=no host-key-size=2048 strong-crypto=no
/ip traffic-flow
set active-flow-timeout=30m cache-entries=32k enabled=no inactive-flow-timeout=15s interfaces=all
/ip traffic-flow ipfix
set bytes=yes dst-address=yes dst-address-mask=yes dst-mac-address=yes dst-port=yes first-forwarded=yes gateway=yes icmp-code=yes icmp-type=yes igmp-type=yes in-interface=yes ip-header-length=yes \
    ip-total-length=yes ipv6-flow-label=yes is-multicast=yes last-forwarded=yes nat-dst-address=yes nat-dst-port=yes nat-src-address=yes nat-src-port=yes out-interface=yes packets=yes protocol=yes \
    src-address=yes src-address-mask=yes src-mac-address=yes src-port=yes tcp-ack-num=yes tcp-flags=yes tcp-seq-num=yes tcp-window-size=yes tos=yes ttl=yes udp-length=yes
/ip upnp
set allow-disable-external-interface=no enabled=no show-dummy-rule=yes
/mpls
set dynamic-label-range=16-1048575 propagate-ttl=yes
/mpls interface
set [ find default=yes ] disabled=no interface=all mpls-mtu=1508
/mpls ldp
set distribute-for-default-route=no enabled=no hop-limit=255 loop-detect=no lsr-id=0.0.0.0 path-vector-limit=255 transport-address=0.0.0.0 use-explicit-null=no
/port firmware
set directory=firmware ignore-directip-modem=no
/ppp aaa
set accounting=yes interim-update=0s use-circuit-id-in-nas-port-id=no use-radius=no
/radius incoming
set accept=no port=3799
/routing bfd interface
set [ find default=yes ] disabled=no interface=all interval=0.2s min-rx=0.2s multiplier=5
/routing mme
set bidirectional-timeout=2 gateway-class=none gateway-keepalive=1m gateway-selection=no-gateway origination-interval=5s preferred-gateway=0.0.0.0 timeout=1m ttl=50
/routing rip
set distribute-default=never garbage-timer=2m metric-bgp=1 metric-connected=1 metric-default=1 metric-ospf=1 metric-static=1 redistribute-bgp=no redistribute-connected=no redistribute-ospf=no \
    redistribute-static=no routing-table=main timeout-timer=3m update-timer=30s
/snmp
set contact="" enabled=no engine-id="" location="" trap-community=public trap-generators=temp-exception trap-target="" trap-version=1
/system clock
set time-zone-autodetect=yes time-zone-name=Europe/...
/system clock manual
set dst-delta=+00:00 dst-end="jan/01/1970 00:00:00" dst-start="jan/01/1970 00:00:00" time-zone=+00:00
/system identity
set name=MikroTik
/system leds
set 0 disabled=no interface=wlan1 leds=wlan-led type=wireless-status
/system leds settings
set all-leds-off=never
/system logging
set 0 action=memory disabled=no prefix="" topics=info
set 1 action=memory disabled=no prefix="" topics=error
set 2 action=memory disabled=no prefix="" topics=warning
set 3 action=echo disabled=no prefix="" topics=critical
/system note
set note="" show-at-login=yes
/system ntp client
set enabled=no primary-ntp=0.0.0.0 secondary-ntp=0.0.0.0 server-dns-names=""
/system resource irq
set 0 cpu=auto
set 1 cpu=auto
set 2 cpu=auto
set 3 cpu=auto
/system routerboard settings
set auto-upgrade=no boot-device=nand-if-fail-then-ethernet boot-protocol=bootp force-backup-booter=no protected-routerboot=disabled reformat-hold-button=20s reformat-hold-button-max=10m silent-boot=no
/system upgrade mirror
set check-interval=1d enabled=no primary-server=0.0.0.0 secondary-server=0.0.0.0 user=""
/system watchdog
set auto-send-supout=no automatic-supout=yes no-ping-delay=5m ping-timeout=1m watch-address=none watchdog-timer=yes
/tool bandwidth-server
set allocate-udp-ports-from=2000 authenticate=yes enabled=yes max-sessions=100
/tool e-mail
set address=0.0.0.0 from=<> port=25 start-tls=no user=""
/tool graphing
set page-refresh=300 store-every=5min
/tool mac-server
set allowed-interface-list=all
/tool mac-server mac-winbox
set allowed-interface-list=all
/tool mac-server ping
set enabled=yes
/tool romon
set enabled=no id=00:00:00:00:00:00
/tool romon port
set [ find default=yes ] cost=100 disabled=no forbid=no interface=all
/tool sms
set allowed-number="" auto-erase=no channel=0 port=none receive-enabled=no
/tool sniffer
set file-limit=1000KiB file-name="" filter-cpu="" filter-direction=any filter-interface="" filter-ip-address="" filter-ip-protocol="" filter-ipv6-address="" filter-mac-address="" filter-mac-protocol="" \
    filter-operator-between-entries=or filter-port="" filter-stream=no memory-limit=100KiB memory-scroll=yes only-headers=no streaming-enabled=no streaming-server=0.0.0.0
/tool traffic-generator
set latency-distribution-max=100us measure-out-of-order=yes stats-samples-to-keep=100 test-id=0
/user aaa
set accounting=yes default-group=read exclude-groups="" interim-update=0s use-radius=no

 ip route print  
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit 
 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 0 ADS  0.0.0.0/0                          100.101.0.1               1
 1 ADC  100.101.0.0/17     100.101.0.11    WAN                       0
 2   S  ;;; to server network
        192.168.2.0/24     10.50.0.11      10.50.0.10                1
 3   S  ;;; to another network (currently not needed)
        192.168.3.0/24     10.50.0.11      10.50.0.10                1
 4   S  ;;; to another client network
        192.168.5.0/24     10.50.0.11      10.50.0.10                1
 5  DC  192.168.17.0/24    192.168.17.1    wlan2                   255
 6 ADC  192.168.128.0/24   192.168.128.254 LAN                       0

sindy · Sun Oct 21, 2018 4:47 pm

I have no clue what happens there. Nothing in the client configuration explains to me why the IPsec packets from the client should get to the server with a private source address as the /ip firewall connection print shows.

The client gets its WAN address from the ISP from the RFC6598 shared address space (100.64.0.0/10) and the server sees is as NONWorkingClient_detected_IPaddr, so a NAT takes place there, so even if packets with source address 192.168.128.254 would miraculously bypass the action=masquerade out-interface=WAN rule and be sent out through the WAN without getting src-NATed, they should be unable to reach the server without getting src-NATed by the ISP's NAT device, unless there is a misconfiguration on this device too and it src-NATs only addresses from the 100.64.0.0/10 range and lets the rest pass through unchanged.

IPsec transport packets are normally ignored by IPsec policies to prevent routing loops, but again, even if this protection failed, the policy only takes care about packets with dst-port=1701, so how it should steal packets with dst-port=4500 is a mystery too.

The way how IPsec works it is impossible to distinguish incoming packets coming via an interface already in plaintext from those which came via that interface in ESP and were decrypted from it, both the transport packet and the payload packets appear there and sometimes in wrong order. This, however, does not happen for outbound packets, where only the encrypted one is captured.

So I would first run /tool sniffer on the WAN of the client to see whether eventually some packets with src-address=192.168.128.254 can be seen there. The sequence of steps would be:

disable the peer
enable the policy
disable the L2TP tunnel
wait 4 minutes
/tool sniffer set file-limit=8000KiB file-name=ipsec-start.pcap filter-cpu="" filter-direction=any filter-interface=bridge filter-ip-address="" filter-ip-protocol=udp filter-ipv6-address="" filter-mac-address="" filter-mac-protocol="" filter-operator-between-entries=and filter-port="" filter-stream=no memory-limit=100KiB memory-scroll=no only-headers=no streaming-enabled=no streaming-server=0.0.0.0
/tool sniffer start
enable the peer and wait until the peer and policy become active
tool sniffer stop

Now, check whether any packets have been sniffed with the private src-address:

/tool sniffer packet print where src-address~"192.168.128.254".

If not, download and save the .pcap file and do the following:

/tool sniffer set file-name=l2tp-start.pcap
/tool sniffer start
enable the /interface l2tp-client and wait for 15 seconds
tool sniffer stop

And here again check whether any packets with the private src-address have been sniffed, using the same command as above.

nomatter · Sun Oct 21, 2018 5:41 pm

Ok, will do that.

But for now to have it working i did the following.

I've removed local address 192.168.128.254 from peer.
I've changed srs-addr and sa-src-addr to address which is assigned by ISP. Of course if this address will change ipsec will go down.
Disabled rule chain=srcnat action=src-nat to-addresses=192.168.128.254 protocol=udp dst-address=SERVER_PUBLIC_IPaddr dst-port=1701 log=no log-prefix=""

Don't understand why it needs to be changed that way if other client is ok with previous setup.

This is not robust setup and seems too fragile.

sindy · Sun Oct 21, 2018 6:30 pm

As said, the very goal of those dances with drums (i.e. using the private address for peer and policy) is to be independent of the IP address currently assigned by the DHCP client while still being able to choose the IPsec parameters freely instead of getting the "l2tp default" ones assigned by the system and to use the transport mode of IPsec.

Before testing with the static IPsec configuration, add the following route:
/ip route add
dst-address=SERVER_PUBLIC_IPaddr/32 gateway=8.8.8.8 target-scope=30 pref-src=192.168.128.254

And keep the action=src-nat ... dst-port=1701 rule in /ip firewall nat in place and enabled.

The purpose of that weird route is to misuse the recursive next-hop search mechanism to avoid the need to adjust the gateway parameter of an individual route with pref-src each time the DHCP renewal eventually changes the address of the WAN gateway. This way, the route will actually use the WAN gateway address from the DHCP but will find it recursively via a static address which doesn't need to be changed.

nomatter · Sun Oct 21, 2018 11:40 pm

SO adding this

/ip route add
dst-address=SERVER_PUBLIC_IPaddr/32 gateway=8.8.8.8 target-scope=30 pref-src=192.168.128.254

And keep the action=src-nat ... dst-port=1701 rule in /ip firewall nat in place and enabled.

solved the issue.

Thank you, sir! You know how to do it!

Can i consider this as good result?

As you told before adding rule
ipsec-policy=in,ipsec to the /ip firewall filter rule chain=input ... dst-port=1701 on the server
is enough to finish?
To change to IKE2 exchange mode is it enough just to change it in all peers settings? Now i see that everything is not so simple

sindy · Sun Oct 21, 2018 11:58 pm

SO adding this
...
solved the issue.

I'd love if it was that simple, there must have been something else which caused the IPsec transport packets to loop somehow. Giving the connection tracker the 4-minute rest time between changes is the key I guess. The added route with pref-src only made LT2P process start using the right address on client side, it has not fixed the problem of server L2TP packets making it to the server still with a private source IP.

Can i consider this as good result?

If you can reboot the server and all clients come up again, and if you can reboot a client and it comes up again, then yes, you can consider it a good result.

As you told before adding rule
ipsec-policy=in,ipsec to the /ip firewall filter rule chain=input ... dst-port=1701 on the server
is enough to finish?

This prevents new L2TP connections from getting established without IPsec. It may not be enough to break an already established connection if IPsec fails, so better check that.

Also, remove the rules add action=accept chain=input disabled=no dst-port=1701,500,4500 protocol=udp from the clients' firewalls - the clients will always initiate the connections so no need that incoming connections to clients would be permitted from outside. Do that even though all the clients are behind a NAT - you never know what other users in their WAN subnets may try to do.

To change to IKE2 exchange mode is it enough just to change it in all peers settings? Now i see that everything is not so simple

Yes, it should be enough - exchange-mode=ike2, auth-mode=pre-shared-key-xauth etc. as described above.

nomatter · Mon Oct 22, 2018 12:27 am

I'd love if it was that simple, there must have been something else which caused the IPsec transport packets to loop somehow. Giving the connection tracker the 4-minute rest time between changes is the key I guess. The added route with pref-src only made LT2P process start using the right address on client side, it has not fixed the problem of server L2TP packets making it to the server still with a private source IP.

So I need to revert this routing rule and do the procedure described above with the sniffer?

BTW, with this rule I don't see private address in server logs, and see detected public one.
Not sure does it mean anything.

If you can reboot the server and all clients come up again, and if you can reboot a client and it comes up again, then yes, you can consider it a good result.

Yes, connection comes up after reboot.

Also, remove the rules add action=accept chain=input disabled=no dst-port=1701,500,4500 protocol=udp from the clients' firewalls - the clients will always initiate the connections so no need that incoming connections to clients would be permitted from outside. Do that even though all the clients are behind a NAT - you never know what other users in their WAN subnets may try to do.

Ok

Yes, it should be enough - exchange-mode=ike2, auth-mode=pre-shared-key-xauth etc.

tried to do that but both clients failed to connect. the same story with current bytes. no current bytes in server to client direction.
Maybe dynamically created policies on the server need to expire first?

----
Also wanted to ask is there a proper way to test these secured channels (i hope). To be sure that they are really secured more or less?

sindy · Mon Oct 22, 2018 12:42 am

So I need to revert this routing rule and do the procedure described above with the sniffer?

BTW, with this rule I don't see private address in server logs, and see detected public one.
Not sure does it mean anything.

Sure it does mean something - in particular that it works the way which was planned from the beginning

. The private source address seen in the server logs was an unexplained anomaly which prevented it from working, but it is not possible that it would be fixed by adding that route at client. I'd not do the sniffing and would be happy that it works as is - as you've checked that it comes up after the reboots, I'd attribute the mysterious problems to non-expired connections on firewall.

tried to do that but both clients failed to connect. the same story with current bytes. no current bytes in server to client direction.
Maybe dynamically created policies on the server need to expire first?

If you disable the peers at both the clients and the server, the policies should go down. Then the 4-minute quarantine, and then enable the peers again after switching over to ike2, first on server, then on clients.

nomatter · Mon Oct 22, 2018 1:11 am

Sure it does mean something - in particular that it works the way which was planned from the beginning . The private source address seen in the server logs was an unexplained anomaly which prevented it from working, but it is not possible that it would be fixed by adding that route at client. I'd not do the sniffing and would be happy that it works as is - as you've checked that it comes up after the reboots, I'd attribute the mysterious problems to non-expired connections on firewall.

Seems I've misunderstood your previous message. Now it's clear.

If you disable the peers at both the clients and the server, the policies should go down. Then the 4-minute quarantine, and then enable the peers again after switching over to ike2, first on server, then on clients.

Will do that tomorrow as one of the Teamviewer connections went down. I don't want to risk any more this weekend

I'll post the results tomorrow.

nomatter · Mon Oct 22, 2018 9:04 pm

If you disable the peers at both the clients and the server, the policies should go down. Then the 4-minute quarantine, and then enable the peers again after switching over to ike2, first on server, then on clients.

No luck with this.
After enabling peers none of the clients are able to connect.
Policies, remote peers, installed SAs - all of these are present.
But no current packets in server to client direction

sindy · Mon Oct 22, 2018 9:20 pm

Nor via reboot of the server followed by reboot of the clients?

nomatter · Mon Oct 22, 2018 9:41 pm

No luck after reboot also

sindy · Mon Oct 22, 2018 9:49 pm

That makes no sense to me... can you activate l2tp logging and show me the log? Disable the l2tp interface on client, then run, on server, /log print follow-only topics~"l2tp" file=l2tp-start, then enable l2tp interface on client and after 1 minute stop (Ctrl-C) the log print at server. Then download the file from server, clean up the public IPs and post it here.

nomatter · Mon Oct 22, 2018 10:18 pm

ok run

log print follow-only where topics~"l2tp" file=l2tp-start

but i'm unable to find results? where should this "l2tp-start" be?
maybe log was empty and therefore it's not present?

sindy · Mon Oct 22, 2018 11:18 pm

Can you see an enabled item topics=l2tp under /system logging? If not, /system logging add topics=l2tp,!packet and then repeat it from start, otherwise the file may be really empty, but in that case, the installed-sa from client to server should show no packets, is that the case?

nomatter · Mon Oct 22, 2018 11:28 pm

I don't know where file is but here is the log

log print follow-only where topics~"l2tp" file=l2tp-start2
20:22:14 l2tp,info first L2TP UDP packet received from Client1_detected_IPaddr 
20:22:14 l2tp,debug tunnel 28 entering state: wait-ctl-conn 
20:22:30 l2tp,debug tunnel 28 received no replies, disconnecting 
20:22:30 l2tp,debug tunnel 28 entering state: dead 
20:22:39 l2tp,info first L2TP UDP packet received from Client1_detected_IPaddr 
20:22:39 l2tp,debug tunnel 29 entering state: wait-ctl-conn 
20:22:55 l2tp,debug tunnel 29 received no replies, disconnecting 
20:22:55 l2tp,debug tunnel 29 entering state: dead 
20:23:03 l2tp,info first L2TP UDP packet received from Client1_detected_IPaddr 
20:23:03 l2tp,debug tunnel 30 entering state: wait-ctl-conn 
20:23:19 l2tp,debug tunnel 30 received no replies, disconnecting 
20:23:19 l2tp,debug tunnel 30 entering state: dead

the installed-sa from client to server should show no packets, is that the case?

No packets from server to client

sindy · Mon Oct 22, 2018 11:59 pm

That's totally weird... the log says the first packet has arrived but doesn't attempt to send anything, and makes an impression that only one has arrived.

If you disable the /interface l2tp-client everywhere for 4 minutes, then start it on one of the clients, and then run /ip firewall connection print interval=1s where dst-address~":1701" at both that client and the server, can you see the same number of orig-packets at both ends of the connection? And can you see any reply-packets at server side?

nomatter · Tue Oct 23, 2018 12:50 am

can you see the same number of orig-packets at both ends of the connection?

At first yes,
then connection on client reestablishes.
And starts increasing till 6 then drops.
On server number is the same and total story is like that:
C S
1 - 1
2 - 2
...
6 - 6
1 - 7
...
6-12
...
and so on

And can you see any reply-packets at server side?

yes

sindy · Tue Oct 23, 2018 1:01 am

Hm. In such case, please show me the /ip ipsec policy print and /ip ipsec installed-sa from both ends, I have a suspicion.

nomatter · Tue Oct 23, 2018 1:08 am

Server

ip ipsec policy print 
Flags: T - template, X - disabled, D - dynamic, I - invalid, A - active, * - default 
 0 T * group=default src-address=::/0 dst-address=::/0 protocol=all proposal=default template=yes 

 1  DA  src-address=SERVER_PUBLIC_IPaddr/32 src-port=1701 dst-address=Client1_detected_IPaddr/32 dst-port=4500 protocol=udp action=encrypt level=unique ipsec-protocols=esp tunnel=no 
       proposal=default ph2-count=1 


ip ipsec installed-sa print 
Flags: H - hw-aead, A - AH, E - ESP 
 0  E spi=0x16A63CE src-address=Client1_detected_IPaddr:4500 dst-address=SERVER_PUBLIC_IPaddr:4500 state=mature auth-algorithm=sha256 enc-algorithm=aes-cbc enc-key-size=256 
      auth-key="542b02e7f95002ccb412c52b60ab4dfa3131ad3c1a2e67d99e4ce148941c7761" enc-key="b15f654d0c4fd1e47741e4a3df6926f6117499d1260d31de93cafc96dd3cb0c3" 
      addtime=oct/22/2018 21:44:19 expires-in=10m50s add-lifetime=24m19s/30m24s current-bytes=22032 current-packets=204 replay=128 

 1  E spi=0x2343B96 src-address=SERVER_PUBLIC_IPaddr:4500 dst-address=Client1_detected_IPaddr:4500 state=mature auth-algorithm=sha256 enc-algorithm=aes-cbc enc-key-size=256 
      auth-key="f209c7767d95593f4edf3fa26d71bc26f5f07c7636418784085343443da682f7" enc-key="a9ce10508aabab06f3604431458974cd60cb978cd81234e0619ceb12e93e499b" 
      add-lifetime=24m19s/30m24s replay=128

----
Client

 ip ipsec policy print 
Flags: T - template, X - disabled, D - dynamic, I - invalid, A - active, * - default 
 0 T * group=default src-address=::/0 dst-address=::/0 protocol=all proposal=default template=yes 

 1  A  src-address=192.168.5.254/32 src-port=any dst-address=SERVER_PUBLIC_IPaddr/32 dst-port=1701 protocol=udp action=encrypt level=require ipsec-protocols=esp 
       tunnel=no proposal=default ph2-count=1 
	   
	   
 ip ipsec installed-sa print 
Flags: H - hw-aead, A - AH, E - ESP 
 0  E spi=0x2343B96 src-address=SERVER_PUBLIC_IPaddr:4500 dst-address=192.168.5.254:4500 state=mature auth-algorithm=sha256 enc-algorithm=aes-cbc enc-key-size=256 
      auth-key="f209c7767d95593f4edf3fa26d71bc26f5f07c7636418784085343443da682f7" enc-key="a9ce10508aabab06f3604431458974cd60cb978cd81234e0619ceb12e93e499b" 
      add-lifetime=24m4s/30m6s replay=128 

 1  E spi=0x16A63CE src-address=192.168.5.254:4500 dst-address=SERVER_PUBLIC_IPaddr:4500 state=mature auth-algorithm=sha256 enc-algorithm=aes-cbc enc-key-size=256 
      auth-key="542b02e7f95002ccb412c52b60ab4dfa3131ad3c1a2e67d99e4ce148941c7761" enc-key="b15f654d0c4fd1e47741e4a3df6926f6117499d1260d31de93cafc96dd3cb0c3" 
      addtime=oct/17/2018 19:29:07 expires-in=8m10s add-lifetime=24m4s/30m6s current-bytes=24624 current-packets=228 replay=128

and now (few minutes later) there are no polices on server and installed-sa

sindy · Tue Oct 23, 2018 1:16 am

Gents in Riga are poking fun on us... the auto-generated policy at server side has dst-port=4500 which is a nonsense, the client has src-port=any.

What is the peer configuration at server (/ip ipsec peer export verbose)? If there is generate-policy=port-strict, try to change it to port-override and disable/re-enable the peer, then try again and check how the dynamically added policy looks in that case. If there already is port-override, try port-strict.

nomatter · Tue Oct 23, 2018 1:22 am

Gents in Riga are poking fun on us...

If there is generate-policy=port-strict, try to change it to port-override and disable/re-enable the peer, then try again and check how the dynamically added policy looks in that case

port-strict

/ip ipsec peer profile
set [ find default=yes ] dh-group=modp3072 dpd-interval=2m dpd-maximum-failures=5 enc-algorithm=aes-256 hash-algorithm=sha256 lifetime=1d name=default nat-traversal=yes \
    proposal-check=obey
/ip ipsec peer
add address=0.0.0.0/0 auth-method=pre-shared-key-xauth disabled=no exchange-mode=ike2 generate-policy=port-strict passive=yes policy-template-group=default profile=\
    default secret="secret" send-initial-contact=no

sindy · Tue Oct 23, 2018 1:32 am

OK, and how does it look like with port-override?

nomatter · Tue Oct 23, 2018 1:38 am

ip ipsec policy print 
Flags: T - template, X - disabled, D - dynamic, I - invalid, A - active, * - default 
 0 T * group=default src-address=::/0 dst-address=::/0 protocol=all proposal=default template=yes 

 1  DA  src-address=SERVER_PUBLIC_IPaddr/32 src-port=any dst-address=Client1_detected_IPaddr/32 dst-port=any protocol=udp action=encrypt level=unique ipsec-protocols=esp tunnel=no 
       proposal=default ph2-count=1

and both clients... are working.

man, if i stood in front of you, i would applaud!

sindy · Tue Oct 23, 2018 1:54 am

Hehe... Do you live in the capital of your country or in the small town at the north-east? We might have a beer together (well, some fruit juice in my case, but the principle is the same) next time I'll come to the capital.

nomatter · Tue Oct 23, 2018 7:51 am

We might have a beer together (well, some fruit juice in my case, but the principle is the same) next time I'll come to the capital.

Yes, sure. I owe you one, my friend.

Do you live in the capital of your country or in the small town at the north-east?

In the capital.
Just curious, how have you found out? From the logs in which I'd posted ip addresses in hex?

sindy · Tue Oct 23, 2018 8:51 am

Just curious, how have you found out? From the logs in which I'd posted ip addresses in hex?

Exactly. The rest were just little supplementary confirmations. Don't worry, nothing in your English gave me a clue

nomatter · Wed Oct 24, 2018 11:18 pm

BTW, want to ask one more thing

Do I need to change anything in setup discussed if i want to add Windows 10 or MAC Os client?

The idea is to be able to make service connection for network (or devices behind router) maintenance from the Internet.

sindy · Thu Oct 25, 2018 12:21 am

Well... first, the embedded clients in Win and Android only support L2TP with IKE(v1), and only support plain IKEv2 (without L2TP) using certificates (i.e. pre-shared keys are not supported for IKEv2). I have no clue how is it with Apple devices.

So if you don't mind setting up an IKE(v1) peer, with exchange-mode=main-l2tp, in parallel to the one with exchange-mode=ike2, it is technically possible, but in such case all the fiddling with L2TP over IKEv2 was kind of waste of time as you'd keep the IKE(v1) access open to the world.

Otherwise, you can follow the road warrior settings for IKEv2 on the manual page, which also tells you how to create a home-made certificate signed by your own certification authority and export it for import on Windows. But while I think that if you set a mode-config on the existing peer, the mode-config will only be actually used if the client requires it, which the RouterOS clients would not do so they wouldn't be affected, you'd have to switch the auth-method from pre-shared-key-xauth to rsa-signature, i.e. you'd have to create the certificates also for the Mikrotik clients. IKEv2 initial request can be distinguished from IKE(v1) one so two peers with otherwise same settings may coexist and not interfere with each other, but the authentication method isn't clear from the initial packet so two exhange-mode=ike2 peers with different auth methods cannot coexist on same IP address and port. So if you would want to keep the site to site network on pre-shared key authentication rather than a certificate one, or if the mode-config use doesn't actually depend on whether the initiator asks for it or not, obsucre methods might be used to resolve this, like making use of the fact that Windows send even the IKEv2 initial request to port 500 while Mikrotik sends it to 4500 straight away, which gives you a chance to use firewall address lists to redirect incoming requests from Windows to a peer running on an additional IP address internally, but I don't know how Apple devices behave in this regard.

Or you may use SSTP (which requires certificates anyway), or you may use SSH with tunneling, but I'm not sure how any of these goes along with Apple.

nomatter · Thu Oct 25, 2018 10:29 pm

So if you don't mind setting up an IKE(v1) peer, with exchange-mode=main-l2tp, in parallel to the one with exchange-mode=ike2, it is technically possible, but in such case all the fiddling with L2TP over IKEv2 was kind of waste of time as you'd keep the IKE(v1) access open to the world.

absolutely agree.

So as I understand there are 2 options:
1. Switch to certificates exchange mode. And create new peer with special profile as I think there are some special settings needed for windows(for example) - dh group 3072 seems like is not supported by win10.
2. Setup an SSTP server in parallel to existing setup. actually i can not explain this properly but i like it more. i like the idea that i would have special separate setup for site-to-site and another one for other purposes. Or this sounds like the same story as having second IKE1 peer?

sindy · Thu Oct 25, 2018 10:51 pm

I don't feel competent to compare security of IPsec and SSTP, but haven't found anything regarding SSTP's vulnerabilities than this. From other than security perspective, SSTP, like every other VPN which uses TCP transport, is not the best choice in environments where packet loss is an issue, but for occasional management access it should be sufficient.

nomatter · Fri Oct 26, 2018 11:11 pm

Thank you sindy! Now I have the direction.
I think this topic can be closed.

For those who don't want to read everything and have similar issues
1. Initial setup is incorrect.
2.Needed to update all routers to current version of Router OS
3. The whole configuration of IPSec is described here and here
4. Needed to setup ipsec on the server like this

/ip ipsec peer profile
set [ find default=yes ] dh-group=modp3072 enc-algorithm=aes-256 hash-algorithm=sha256
/ip ipsec proposal
set [ find default=yes ] auth-algorithms=sha256 enc-algorithms=aes-256-cbc pfs-group=modp3072
/ip ipsec peer
add address=0.0.0.0/0 auth-method=pre-shared-key-xauth exchange-mode=ike2 generate-policy=port-override passive=yes send-initial-contact=no
/ip ipsec user
add name=User1
add name=User2

Many thanks to sindy! I couldn't have done it without your help. I've learned a lot last two exciting (

) weeks. thank you for your patience and detailed explanations!

nomatter · Sun Nov 04, 2018 1:01 pm

But no, should return to this

I've tried to work on firewall part:
and decided to start from defconf so that's what i have now:

 ip firewall filter print 
Flags: X - disabled, I - invalid, D - dynamic 
 0  D ;;; special dummy rule to show fasttrack counters
      chain=forward action=passthrough 

 1    ;;; defconf: accept established,related,untracked
      chain=input action=accept connection-state=established,related,untracked 

 2    ;;; defconf: drop invalid
      chain=input action=drop connection-state=invalid 

 3    ;;; defconf: accept ICMP
      chain=input action=accept protocol=icmp 

 4    ;;; accept in ipsec policies
      chain=input action=accept protocol=udp port=1701 log=no log-prefix="" ipsec-policy=in,ipsec 

 5    ;;; accept from office
      chain=input action=accept protocol=udp src-address=Client1_detected_IPaddr dst-port=4500 log=no log-prefix="" 

 6    ;;; defconf: drop all not coming from LAN
      chain=input action=drop in-interface-list=!Local log=no log-prefix="" 

 7 X  ;;; defconf: accept in ipsec policy
      chain=forward action=accept log=no log-prefix="" ipsec-policy=in,ipsec 

 8 X  ;;; defconf: accept out ipsec policy
      chain=forward action=accept log=no log-prefix="" ipsec-policy=out,ipsec 

 9    ;;; defconf: fasttrack
      chain=forward action=fasttrack-connection connection-state=established,related 

10    ;;; defconf: accept established,related, untracked
      chain=forward action=accept connection-state=established,related,untracked 

11    ;;; defconf: drop invalid
      chain=forward action=drop connection-state=invalid 

12    ;;; defconf:  drop all from WAN not DSTNATed
      chain=forward action=drop connection-state=new connection-nat-state=!dstnat in-interface=WAN

What is confusing me:
1. Rule 4 doesn't change anything. it could be disabled and nothing changes. So How can I be sure that only encrypted traffic enters the router.
2. If i don't have Rule 5 one of the clients l2tp connection is dropped out by Rule 6.
3. I think that I don't need Rule 12 as i'm connected to internet as pppoe client via WAN interface. And, in general, how to properly adjust rules taking this into account? I think input chain is ok as everything is dropped except from LAN. Not sure regarding forward chain.

sindy · Sun Nov 04, 2018 2:17 pm

The whole concept of the default firewall builds on connection tracking (or, more exactly, connection tracking has been created to enable such a concept), which means that most rules deal with initial packets of new connections, thus deciding whether a new connection as a whole will be permitted or not, while most packets belonging to any of the already established connections are handled by a single rule in the beginning of each chain - action=accept connection-state=established,related.

So if you think rule 4 doesn't change anything, switch the clients off for 10 minutes and then try to re-connect them while rule 4 is disabled - it won't succeed. Same case for rule 5 - I guess if you switch off for 10 minutes the client which now doesn't need that rule to work, it will be unable to connect too once you connect it again.

UDP connections normally time out 3 minutes after last packet seen by the connection tracker, however both L2TP and IPsec use keepalive mechanisms which don't give up on a single fault, hence I suggest 10 minutes although less may be actually sufficient. You can remove connections manually, but it won't help much as the next outgoing keepalive packet from the L2TP or IPsec process running on server will create a new connection (with swapped src-address and dst-address as compared to the original one) unless you install a chain=output connection-state=new protocol=udp src-port=1700,4500 action=drop rule preventing that, so you'd have to also disable the /ip ipsec peer and the /interface l2tp-server server to get rid of the keepalives. So a 10-minute coffee break is the lazy man's choice if the sole purpose is to check whether what I write is true

Regarding point 3., RouterOS provides some useful extensions to the commonly known netfilter capabilities configured using iptables, allowing you to use "zone-based" firewall rules. You have the (src|dst)-address-list and (in|out)-interface-list match conditions, allowing you to refer to otherwise unrelated addresses and subnets in a single rule, and to refer to "all WAN", "all VPN" or "all LAN for client group X" interfaces in a single rule. So in the default firewall, WAN is normally referred to as an in-interface-list, and it is your responsibility to make pppoe-out1 (or whatever name you gave it) a member of that interface-list if you haven't used the QuickSet to configure the PPPoE as WAN.

Also, the IP firewall only cares about the interfaces with IP address attached to them. So although the frame carrying the IP packet physically comes in via e.g. ether1 which is the carrier interface of /interface pppoe-client, this fact is irrelevant for the IP firewall which can see the pppoe-out1 as the source of the IP packet. And if there was an IP address attached to ether1 in addition to its function as pppoe-out1's carrier interface, only packets coming directly (i.e. in frames with ethertype 0x800) to it would match in-interface=ether1 whereas those with ethertype=0x8864 (PPPoE session stage) would still only match in-interface=pppoe-out1.

nomatter · Sun Nov 04, 2018 4:57 pm

So a 10-minute coffee break is the lazy man's choice if the sole purpose is to check whether what I write is true

Hmm, i always forget about these "keepalives".
I've just disabled the rule and disabled/enabled peer on server to test is it possible to come up with rule disabled.

Same case for rule 5 - I guess if you switch off for 10 minutes the client which now doesn't need that rule to work, it will be unable to connect too once you connect it again.

I don't understand why do I need this rule if i already have rule 4. I've tried to add 4500,500 ports to rule 4 but it doesn't help till ipsec-policy=in,ipsec is present.
Also I need to specify detected public ip of the client as src-address in rule 5 which is not good.

So in the default firewall, WAN is normally referred to as an in-interface-list, and it is your responsibility to make pppoe-out1 (or whatever name you gave it) a member of that interface-list

Yes, I've noticed interface-list and changed just to one interface - WAN as i wasn't sure how to handle pppoe-out properly.
So i just need to add pppoe-out to interface-list which should contain ether1 (to which ISP cable is connected) and pppoe-out?

So although the frame carrying the IP packet physically comes in via e.g. ether1 which is the carrier interface of /interface pppoe-client, this fact is irrelevant for the IP firewall which can see the pppoe-out1 as the source of the IP packet

Or adding pppoe-out won't change anything?
I don't understand how should i secure my ISP connection. It seems like forward chain is open for everything which is ready to come along ISP cable

Maybe something similar to this is needed? (with chain-forward)

;;; defconf: drop all not coming from LAN
chain=input action=drop in-interface-list=!Local log=no log-prefix=""

Or maybe input chain is also leaky like a sieve?

sindy · Sun Nov 04, 2018 6:28 pm

I don't understand why do I need this rule if i already have rule 4. I've tried to add 4500,500 ports to rule 4 but it doesn't help till ipsec-policy=in,ipsec is present.

Help against what exactly?
The whole concept is the following:

on the HQ machine with static public IP address (server), permit initial packets of connections to UDP dst-port 4500 from anywhere without a limitation by ipsec-policy=in,ipsec because the IPsec control and transport packets must be able to come directly from the internet; on the client machine this rule may not be there because the client initiates the connection and by default (empty chain=output) it can initiate any connection it wants to. The security on port 4500 is provided by the IPsec daemon, not by the firewall.
on the server, permit initial L2TP packets (to UDP dst-port 1701) only if they come in encrypted inside an IPsec tunnel, hence you specify an additional match condition ipsec-policy=in,ipsec. Unlike other types of tunnels, IPsec creates no virtual interface, so the in-interface of the decrypted and decapsulated packets is the same like the one of the transport packets inside which the former ones arrive. So you need the ipsec-policy matcher to distinguish between the decapsulated payload (ipsec-policy=in,ipsec) and the transport packets (ipsec-policy=in,none)

So rule 4 prevents plaintext initial L2TP packets from being accepted. Rule 5 permits IPsec control and transport packets, and it depends on how much you trust the IPsec daemon and your passwords on one hand, and on how reliably you can anticipate from which ranges the public address to which your clients' addresses are NATed on the other hand, whether you restrict the source addresses only to subnets from which you expect legitimate IPsec connections to come or whether you keep it open to the world.

I mention only UDP port 4500 above because when using IKEv2, Mikrotik acting as IPsec initiator (client) doesn't use UDP port 500 at all. With IKE(v1) and/or some other-than-Mikrotik clients, same treatment must be given to UDP port 500 like to port 4500.

Yes, I've noticed interface-list and changed just to one interface - WAN as i wasn't sure how to handle pppoe-out properly.
So i just need to add pppoe-out to interface-list which should contain ether1 (to which ISP cable is connected) and pppoe-out?

Even more than that - you can have only pppoe-out in the list to which you refer in the firewall using in-interface-list when you want to permit/deny packets from WAN. Once again, unless ether1 has its own IP address, there is no way how an IP packet could be seen by IP firewall as coming in via ether1, because without any IP configuration attached to ether1, any eventual frame carrying an IP packet (i.e. one with ethertype 0x800) is dropped already by the kernel if it arrives to ether1, so it never makes it to the IP firewall.

I don't understand how should i secure my ISP connection. It seems like forward chain is open for everything which is ready to come along ISP cable

I keep recommending to build the firewalls as "drop everything except what you know you need to permit" rather than "permit everything except what you know you need to drop". With the first approach, if you forget to permit something, your legitimate users start yelling on you quickly; with the second one, your illegitimate users will be happy but never let you know.

So in your case, in chain=input, the very last rule should be just action=drop with no additional conditions; but before you activate that rule, make sure that you have placed the exceptions you need, at first place the access to management services (ssh, winbox, www-ssl aka https) from a list of permitted IP addresses and/or interfaces.

What Mikrotik does in the default configuration is that they combine two last rules in chain=input (namely, action=accept in-interface-list=LAN followed by action=drop), by a single rule with the same effect: action=drop in-interface-list=!LAN. This is possible due to the fact that the default handling of the packet, i.e. the one which takes place if the packet didn't match to any rule in the chain, is "accept".

Regarding chain=forward, even if there is a NAT on the WAN side, packets coming in via WAN port with destination IP address matching one of the LAN hosts would be delivered to the host. Therefore, the last rule saying action=drop in-interface-list=WAN connection-nat-state=!dstnat again effectively substitutes the following three rules:
action=accept in-interface-list=WAN connection-nat-state=dstnat
action=accept in-interface-list=!WAN
action=drop

So with this three-in-one rule in place, the clients on LAN can initiate connections towards servers accessible via WAN, but clients in the internet can only initiate connections towards servers in LAN if you create a dst-nat (port forwarding) rule in chain=dstnat of /ip firewall nat - of course provided that the magic action=accept connection-state=established,related rule is the first one in chain=forward of /ip firewall filter.

In your case, the virtual interfaces created for the L2TP tunnels are not added to any interface list, so rules matching in-interface-list=!WAN or in-interface-list=!LAN will match on them, while rules matching in-interface-list=LAN or in-interface-list=WAN will not.

You can also use /ppp profile to specify an interface list to which the server-side L2TP interface should be added once it appears after successful client authentication.

And of course, if you have a single WAN interface on a router, it doesn't really matter whether you put it to an interface list named WAN and refer to it "symbolically" using in-interface-list, or whether you modify the firewall rules to use in-interface=the-actual-name-of-the-wan-interface.

nomatter · Sun Nov 04, 2018 7:42 pm

I understand the concept of ip firewall filter defconf more or less in general but when i need to adapt it to my real case that's where problems start

And of course getting access to Internet as pppoe client brings me to the edge of my poor networking knowledge.

Help against what exactly?

I mean that I thought adding dst-port=4500 to Rule 4 should replace Rule 5.

Is there any way to work around src-address in Rule 5? I don't want to have any surprises if ISP changes that address. And of course it will happen in the moment when i'm not able to help right away

So in your case, in chain=input, the very last rule should be just action=drop with no additional conditions; but before you activate that rule, make sure that you have placed the exceptions you need, at first place the access to management services (ssh, winbox, www-ssl aka https) from a list of permitted IP addresses and/or interfaces.

What Mikrotik does in the default configuration is that they combine two last rules in chain=input (namely, action=accept in-interface-list=LAN followed by action=drop), by a single rule with the same effect: action=drop in-interface-list=!LAN. This is possible due to the fact that the default handling of the packet, i.e. the one which takes place if the packet didn't match to any rule in the chain, is "accept".

Sorry, didn't get that.
Do I need to change anything in last rule of input chain? as i understand it is ok like it is, right?
or it's better to do something like that
chain=input action=accept dst-address=192.168.1.254(Router) dst-port=8291 src-address=192.168.1.1(for example)
chain=input action=drop

Regarding chain=forward, even if there is a NAT on the WAN side, packets coming in via WAN port with destination IP address matching one of the LAN hosts would be delivered to the host. Therefore, the last rule saying action=drop in-interface-list=WAN connection-nat-state=!dstnat again effectively substitutes the following three rules:

SO all i need to do is change in-interface from ether1 to pppoe-out1, right?

tdw · Sun Nov 04, 2018 8:54 pm

I don't feel competent to compare security of IPsec and SSTP, but haven't found anything regarding SSTP's vulnerabilities than this. From other than security perspective, SSTP, like every other VPN which uses TCP transport, is not the best choice in environments where packet loss is an issue, but for occasional management access it should be sufficient.

SSTP is certainly much simpler to deploy when the endpoints don't have static IPs and you have NAT issues with multiple clients at the same location - certainly it is the only secure choice for Mikrotik site-to-site if IPsec isn't an option as both PPTP and L2TP with MPPE (suffer from password extraction from MSCHAPv2 handshake), plus the Mikrotik OpenVPN client (see https://janis-streib.de/2018/04/11/mikr ... -security/), are insecure.

In addition to your comments about TCP transport, there is a lot of overhead (PPP inside SSTP inside TLS) which Mikrotiks don't have hardware acceleration for, and as long as you use MSCHAPv2 for the PPP authentication to mitigate against some attack scenarios it should be secure, assuming people can't compromise the TLS session. If you need any light bed-time reading https://winprotocoldoc.blob.core.window ... -SSTP].pdf

sindy · Sun Nov 04, 2018 9:17 pm

I mean that I thought adding dst-port=4500 to Rule 4 should replace Rule 5.

It cannot. You must keep the permissive rule (now 5) for dst-port=4500 free of the ipsec-policy=in,ipsec condition in order to allow the IPsec connection establish, and you must keep the ipsec-policy=in,ipsec condition in the permissive rule (now 4) for dst-port=1701 in order to prevent plaintext (not encrypted) L2TP conections from establishing.

Is there any way to work around src-address in Rule 5? I don't want to have any surprises if ISP changes that address. And of course it will happen in the moment when i'm not able to help right away

The src-address condition can be simply left out from the rule, in that case anyone can attempt to set up an IPsec connection and the IPsec authentication must reject such attempt. Or you can find out the whole IPv4 range allocated to the ISP of the client and permit that whole range which certainly reduces the attack surface, yet if by some miracle the ISP would get this range extended in future (most likely scenario is a merge with another ISP), you might find your client disconnected.

Sorry, didn't get that.
Do I need to change anything in last rule of input chain? as i understand it is ok like it is, right?
or it's better to do something like that
chain=input action=accept dst-address=192.168.1.254(Router) dst-port=8291 src-address=192.168.1.1(for example)
chain=input action=drop

A couple of months ago I would have told you "keep the default as it prevents access from the internet (via WAN) and as Winbox is secure enough". Today I cannot say this because some vulnerabilities of Winbox have been discovered, and although the discovered ones have already been fixed, you cannot be 100 % sure some yet undiscovered ones don't exist while Mikrotik became an attractive target given its market share, and as multi-environment malware exists which can infect a Windows machine on your LAN and from there attack your Mikrotik. So yes, restricting access to management services even from LAN to just a couple of IP addresses, or even dedicating one ethernet interface for management, enhances the device security.

SO all i need to do is change in-interface from ether1 to pppoe-out1, right?

Right.

nomatter · Mon Nov 05, 2018 9:35 pm

Thank you, tdw!

Thank you, sindy! Now it's clear.
Just one thing is a mystery for me - why rule 5 affects just Client1?
Looking at connections I see that src-port for Client2 is different

#          PR.. SRC-ADDRESS                      DST-ADDRESS             
 0  SAC     udp  Client2_detected_IPaddr:1701    SERVER_PUBLIC_IPaddr:1701
 1  SAC     udp  Client1_detected_IPaddr:4500    SERVER_PUBLIC_IPaddr:4500
 2  SAC     udp  Client2_detected_IPaddr:48703   SERVER_PUBLIC_IPaddr:4500
 3  SAC     udp  Client1_detected_IPaddr:1701    SERVER_PUBLIC_IPaddr:1701

But there is dst-port in rule 5 so i thought it should be essential for both clients not just for one.

;;; accept from office
      chain=input action=accept protocol=udp src-address=Client1_detected_IPaddr dst-port=4500 log=no log-prefix=""

What does it mean that rule 5 doesn't affect Client2? of course if i delete src-address from it
( it is that client for which there was need to add route /ip route add
dst-address=SERVER_PUBLIC_IPaddr/32 gateway=8.8.8.8 target-scope=30 pref-src=192.168.128.254 )

sindy · Mon Nov 05, 2018 10:03 pm

The remark about deleting the src-address from the rule has confused me a bit.

But if there is just a missing negation somewhere in that statement as I suppose, and if the actual behaviour is that if you disable rule 5, disable the peer on both client1 and client2, and give the firewall 10 minutes to "cool down", only client2 can reconnect after those 10 minutes whereas client1 cannot, there are the following possibilities:

a rule in /ip firewall raw sets packets from client2's public address not to be connection tracked; if so, rule 1 lets them in as they match connection-state=untracked
the packets from that client come in via an interface which doesn't match in-interface-list=!Local so rule 6 doesn't drop them and thus they get accepted by default
disabling the peer doesn't remove the installed-sa which, in case of IKEv2, is much more likely than the two explanations above. So after disabling the peers on clients, check whether the policies are still active on clients, and if they are, disable them too which should destroy the associated installed-sa. And after the 10 minutes of cooldown, check the list of connections with dst-address~":4500" on server one more time to see that it is really empty. IKEv2 uses dst-port 4500 to initiate the management connection so if the running installed-sa keeps it open, the 10 minutes are not enough, and even 30 may not be if the SA rekeys internally, not using the control connection (this is how it works on Cisco, I'm not sure whether it is mandatory or optional in KIEv2 to do it this way).

If the explanation above proves true, we'll have another problem - how is it possible that client1 doesn't exhibit the same behaviour

nomatter · Mon Nov 05, 2018 10:25 pm

The remark about deleting the src-address from the rule has confused me a bit.

Yeah, when I read it again it confuses me too. It's evening so it's hard to be precise for me

I mean that if everything was the same for both clients rule 5 would affect both of them - disabling rule 5 drops l2tp connection in 30-40 secs, enabling it brings it back almost immediately. (like it is for client 1 now)

But if there is just a missing negation somewhere in that statement as I suppose, and if the actual behaviour is that if you disable rule 5, disable the peer on both client1 and client2, and give the firewall 10 minutes to "cool down", only client2 can reconnect after those 10 minutes whereas client1 cannot

I think i will need parallel Teamviewer connection for that. To test it without additional headache. This will take me couple of days to choose the right moment

sindy · Mon Nov 05, 2018 10:34 pm

disabling rule 5 drops l2tp connection in 30-40 secs, enabling it brings it back almost immediately.

This is even more incomprehensible to me. Once any connection establishes, rule 1 should be sufficient to keep it up, i.e. disabling rule 5 should not affect an already established connection from client 1, only prevent a new one from establishing if the existing one would time out. So when you disable rule 5, does the connection from client 1 to port 4500 disappear from the connection list?

nomatter · Tue Nov 06, 2018 8:02 am

No, both clients remain:
After enabling rule connection 2 appears for 5 secs.

  /ip firewall connection print where dst-address~":4500"
Flags: E - expected, S - seen-reply, A - assured, C - confirmed, D - dying, F - fasttrack, s - srcnat, d - dstnat 
 #          PR.. SRC-ADDRESS                                   DST-ADDRESS                       TCP-STATE   TIMEOUT     
 0  SAC     udp  Client1_detected_IPaddr:4500         SERVER_PUBLIC_IPaddr:4500                                 2m57s          
 1  SAC     udp  Client2_detected_IPaddr:48703        SERVER_PUBLIC_IPaddr:4500                                 2m58s          
 2    C     udp  Client1_detected_IPaddr:1294         SERVER_PUBLIC_IPaddr:4500                                 5s

Seems like rule 5 is needed to maintain this connection 2. As it appears periodically in connection list even if i don't touch rule 5

sindy · Tue Nov 06, 2018 10:27 am

Connection 2 in your previous post has a 5 s timeout because it got no response from the server, so it has not reached the "stream" (bi-directional) state:

[me@MyTik] > ip firewall connection tracking print
               udp-timeout: 10s
        udp-stream-timeout: 3m

However, it is not clear to me from where it appears, given that there already is an active connection from Client 1. So the ISP's NAT at client side behaves funny, or more likely there is some other device behind that NAT which attempts to connect to our server, or the IPsec stack on Client 1 is attempting to establish another connection which is not very likely.

So at this moment I'd sniff packets coming from Client1_detected_IPaddr at server side to port 4500, and packets sent to SERVER_PUBLIC_IPaddr:4500 at client side, and compare the captures to see whether the packets belonging to Connection 2 above are sent by the Client 1 itself or come from elsewhere. Or maybe start from a simpler step, just watch the connections on Client 1. If you say that Connection 2 pops up at server side right after you enable Rule 5, the attempts must be there all the time, so if the ghost connection originates from or passes through Client 1, you'll see it among its connections in addition to the working one.

nomatter · Tue Nov 06, 2018 10:07 pm

Or maybe start from a simpler step, just watch the connections on Client 1. If you say that Connection 2 pops up at server side right after you enable Rule 5, the attempts must be there all the time, so if the ghost connection originates from or passes through Client 1, you'll see it among its connections in addition to the working one.

Connection 2 pops up not only after enabling Rule 5 - it pops up periodically all the time. But after enabling Rule 5 it pops up immediately so it somehow triggers it( i think it is one of the reasons not only)

And i've never seen it on client side. It is always like this:

/ip firewall connection print where dst-address~":4500"
Flags: E - expected, S - seen-reply, A - assured, C - confirmed, D - dying, F - fasttrack, s - srcnat, d - dstnat 
 #          PR.. SRC-ADDRESS           DST-ADDRESS           TCP-STATE   TIMEOUT     
 0  SAC  s  udp  192.168.5.254:4500    SERVER_PUBLIC_IPaddr:4500                 2m58s

while on server:

 ip firewall connection print where dst-address~"(:4500)" 
Flags: E - expected, S - seen-reply, A - assured, C - confirmed, D - dying, F - fasttrack, s - srcnat, d - dstnat 
 #          PR.. SRC-ADDRESS                        DST-ADDRESS                 TCP-STATE   TIMEOUT    
 0  SAC     udp  Client1_detected_IPaddr:4500    SERVER_PUBLIC_IPaddr:4500                 2m58s           
 1  SAC     udp  Client2_detected_IPaddr:48703   SERVER_PUBLIC_IPaddr:4500                 2m57s            
 2    C     udp  Client1_detected_IPaddr:1303    SERVER_PUBLIC_IPaddr:4500                 8s

sindy · Tue Nov 06, 2018 10:30 pm

Hm... if nothing appears at client side, you'd really have to sniff at server side to see what it really is - packets belonging to Connection 0 misrouted by ISP or some attempt to connect from another source behind the same public IP. But if it keeps popping up even if Rule 5 is disabled, I don't understand it either, as in such case no new connections should appear. So once the source port of Connection 2 would change, Connection 2 should disappear.

If you disable the peer on Client 1, does Connection 2 disappear along with Connection 0 or does it continue re-appearing?

nomatter · Tue Nov 06, 2018 11:15 pm

Connection 0 and 1 remain all the time. Even after disabling l2tp server and ipsec peer on server after 10 mins.

So to prevent connection 2:
1. I disable peer on server. conn 0 and 1 still remain
Or
2. I disable peer on corresponding client. conn 0 and 1 still remain
or
3. Disable Rule 5 but as result l2tp connection drops. so it's somehow similar to p.2

sindy · Tue Nov 06, 2018 11:31 pm

If disabling Rule 5 causes the L2TP to stop working although Connection 0 remains active, I'm afraid that the ISP is guilty, sending part of the packets from client to server using another path through their network than the initial ones for some weird reason, so they come to you from a different source port but as they bear a known SPI, they are handled.

I'd have to see the capture from server side to be able to confirm this. Have you ever worked with Wireshark?

nomatter · Wed Nov 07, 2018 4:31 pm

I looked through CCNA prep materials last year (just for self-education not for real exam preparation) and it was used in a few labs.

I need to recall all this stuff...

From mikrotik side as i understand i can apply filtering using /tool sniffer ?

sindy · Wed Nov 07, 2018 4:38 pm

At mikrotik you can filter what you sniff (to save space on device by not capturing not interesting packets) but you can watch the captured packets only up to L4 (UDP, TCP port) which is not deep enough for the purpose. So you need to capture to a file and then download the file and open it using Wireshark to see what's really going on there.

I've got an invitation to Skype from an unknown person in your area, is it you?

Other than that, if you don't feel like reading Wireshark, I can send my phone number to you using UDP.

nomatter · Wed Nov 07, 2018 6:18 pm

I've got an invitation to Skype from an unknown person in your area, is it you?

Nope.
I don't even know how to get your skype id

maybe i need to have some number of posts to see forum profile's details

I can send my phone number to you using UDP.

Sorry, didn't get this. Is that some secret technology?

I'll try to get Wireshark working for me - it's quite interesting.
Should i look for Connection 2 only? and for how long do i need to sniff to get relevant results?

sindy · Wed Nov 07, 2018 7:16 pm

I don't even know how to get your skype id maybe i need to have some number of posts to see forum profile's details

Nope, I haven't set any contact info in the profile, that's why I was so surprised by this coincidence. Personal mail service is disabled on the forum probably because otherwise Mikrotik support staff and some knowledgeable users would be flooded with messages.

Sorry, didn't get this. Is that some secret technology?

Well, in a way

If you know someone's public IP, you can send packets to it, and either put the information into the packets or use just port numbers to deliver information consisting of only digits. The second approach doesn't require anything else but Mikrotik at both ends of the line, with packet contents it would take more effort at both sending and receiving side.

Should i look for Connection 2 only? and for how long do i need to sniff to get relevant results?

Well, I would sniff for some ten minutes, with filter-ip-address=the.public.ip.of.client.1 filter-ip-protocol=udp filter-operator-between-entries=and filter-direction=any (and the rest of filter-xxx empty).

What I suspect is that the NAT at ISP side works weird and although it delivers packets which arrive from server to client through the initial pinhole, it creates new pinholes for the opposite direction, or that some weird way the packets from the client manage to come via different interfaces or, even worse, there is some crazy double-encapsulation.

Therefore, I would look

whether packets received from source port of Connection 0 to port 4500 exist at all (there may actually be none some time after the connection has established)
whether packets to port 4500 received within Connection 2 have the same source MAC address like packets received within Connection 0 if the latter ones are seen, or have their source MAC address matches the one to which those belonging to Connection 0 are sent (which tells us whether they come in via the same interface or not)
whether there is only a single source port of Connection 2 or whether it changes over time
whether packets coming via Connection 0 (if existent) and Connection 2 have the same SPI, or whether packets coming via two subsequent instances of Connection 2 (differing by source port) have the same SPI
whether packets coming via Connection 2 aren't much smaller than packets coming via Connection 0 and at the same time, which could indicate double encapsulation

RouterOS unfortunately doesn't use pcapng format for sniffing so it is not possible to identify the in-interface from the capture file.

nomatter · Wed Nov 07, 2018 11:10 pm

whether packets received from source port of Connection 0 to port 4500 exist at all (there may actually be none some time after the connection has established)

yes, exist

whether packets to port 4500 received within Connection 2 have the same source MAC address like packets received within Connection 0 if the latter ones are seen, or have their source MAC address matches the one to which those belonging to Connection 0 are sent (which tells us whether they come in via the same interface or not)

I don't see MAC-addr for unknown reason - just all zeros

whether there is only a single source port of Connection 2 or whether it changes over time

The same. at least for these 10 minutes taken for observation. Because yesterday - there was another one in connection list.

whether packets coming via Connection 0 (if existent) and Connection 2 have the same SPI, or whether packets coming via two subsequent instances of Connection 2 (differing by source port) have the same SPI

all where SPI are present belong to conn 2
and all these NAT-keepalives and "informational" to conn 0

whether packets coming via Connection 2 aren't much smaller than packets coming via Connection 0 and at the same time, which could indicate double encapsulation

912 bits - ESP
344 bits - UDPencap
1264 - ISAKMP
screenshot above as the reference. pattern is the same for all observed time period.

Well, in a way If you know someone's public IP, you can send packets to it, and either put the information into the packets or use just port numbers to deliver information consisting of only digits. The second approach doesn't require anything else but Mikrotik at both ends of the line, with packet contents it would take more effort at both sending and receiving side.

Quite impressive

sindy · Wed Nov 07, 2018 11:54 pm

The screenshot is fine but it doesn't show the UDP source ports and ESP sequence numbers. Choose any of the ESP packets, and in the dissection pane, click open the UDP layer, right-click on UDP source port and choose Apply as Column from the context menu. Then, click-open the Encapsulated Security Payload layer and apply the ESP Sequence field as column. Next, choose Resize Columns from the View menu of Wireshark, move the two new columns before the Info one, and take the same screenshot again where you can see a mix of source ports from Connection 0 and Connection 2.

As for the size comparison, it doesn't seem to me that ESP packets would be coming encapsulated into one another.

You get mac address 00:00:00:00:00:00 because you sniff on PPPoE interface which is an L3 one so the packets begin from the IP layer so there is no MAC address in them.

nomatter · Thu Nov 08, 2018 10:56 am

Good morning sindy!

sindy · Thu Nov 08, 2018 12:39 pm

Hm, so in client->server direction, one pinhole (seen as Connection 0 at server side) carries the overhead (NAT keepalive, dead peer detection) whilst the other one (seen as Connection 2 at your side) does the "real job" of carrying encrypted data; in the opposite direction, everything goes inside Connection 0 and it obviously arrives to the client, otherwise the whole thing would not run. Given that the NAT keepalive packets are smaller than the ESP ones while the DPD ones are bigger, the ISP's NAT device most likely does not look at size but checks the UDP payload - the payload of the NAT keepalive packets is a single byte 0xff and the first four bytes of the DPD payload are 0x0, "non-ESP mark"; ESP packets have the SPI value at this position.

The behaviour is so unusual that I'd take a pcap simultaneously at client and server side and check that both the packets coming via Connection 0 and the packets coming via Connection 2 to the server are seen within the same connection on the client side, but I can not imagine any other explanation than a hyperactive ISP's NAT device.

So much regarding the analysis. Regarding what to do with it, I'd say not too much can be done or is necessary - rule 5 must be there anyway to permit packets from clients to get in for cases where the client goes down for a while and then restarts, and as discussed earlier, the only question is whether you keep the rule open to the world or whether you want to restrict it to the IP address ranges assigned to the ISPs serving your clients.

So you may talk to the ISP and ask them whether they are aware of what's going on, and maybe get some explanation, but it won't make any real difference if they fix it, for the reason stated above.

nomatter · Thu Nov 08, 2018 3:41 pm

So you may talk to the ISP and ask them whether they are aware of what's going on, and maybe get some explanation, but it won't make any real difference if they fix it, for the reason stated above.

Yes, i don't see any difference too. And I'm 100% sure they will not fix or change anything.
Thank you for your help!

Meanwhile, today i realized that rule 5 is needed for another client also.

There was no electricity in the building where server-router is and after it was turned on again (~after 1.5 hours) second client was unable to connect as there was only client 1 source address specified in rule 5.

Who is online