IKEv2 - Road Warrior (NAT Workaround)

frontdist · Sat Oct 06, 2018 11:29 pm

Hello,

I am looking for some feedback from the community to see if I am totally out to lunch or not with something that I am trying to work out...

Here is the situation: I do some work for a security contractor, and they have a number of clients who are looking to get access to their IP cameras but there is a bit of a problem... a number of these customers are in a rural area, and are served by a notoriously terrible service provider, and they have no ability to get Static IP or IPv6 addresses. The WISP uses multiple layers of NAT, and it makes it impossible to have an ingress path or do any port forwarding.

I have a public subnet at a datacenter where I am proposing that I set up a "headend" VPN concentrator which a number of remote Mtik Hex units can act as initiators/road warriors to dial in, and then route a public address to the remote device to hand off to the network cameras.

I can easily set this up with PPTP or L2TP, however there is a problem with that... one of the very well known limitations is that with NAT, there is an issue having multiple clients connect from the same Public IP due to the way the NAT-T is handled. This brings me to using IKEv2 as a solution...

What I would like to do is to create a IKEv2 road warrior configuration, and then run GRE over that connection to have a routable interface that I can use to pass traffic to and from, or run OSPF, etc over. I have read through the wiki configuration for IKEv2 road warrior a hundred times, and I was able to get the "datacenter" and "remote" device to connect to each other and set up the correct security associations, but I can't seem to pass traffic over it and was hoping someone could clue me in, or provide additional suggestions as to where I should be looking.

I believe that with IKEv2 connections, it's not necessarily a "tunnel" the way GRE, L2TP, PPTP are so it doesn't present an interface on either side, but that traffic passes as a result of policy matching, which is why I wanted to run GRE over it so I could direct traffic to an interface and "route"

Here are some of the issues to resolve:
- Passing traffic from one side of the security association to the other
- Establishing GRE tunnel over the IPSEC (can I build it between loopback/bridge interfaces?)
- How do I specify which end device gets which IP address from the IPSEC connection (can I specify by certificate? Individual mode configs?)
- I am trying to avoid SSTP and OpenVPN due to the TCP limitations, and am trying to use IKEv2 because of NAT - is there another solution that I should be looking at?

Ideally, I am going to have 10-15 clients connected, each having a public address routed to them, which it will DMZ to the NVR (or I can port forward as appropriate), and be able to remote manage the device, or anything else I need to do on that end. I am looking for a solution that is stable for weak connections, doesn't require user intervention, and can handle the idiocy of multiple NAT layers.

Thanks!

sindy · Mon Oct 08, 2018 10:10 pm

Using pure IPsec (IKEv2), you can use /ip ipsec user to configure username, "password" and IP address in a similar way as with /ppp secret for ppp interfaces if you use pre-shared key & xauth authentication mode (it doesn't work with certificates). However, there is no script you could associate with the event of IPsec connection to come up, so I'm not sure whether assigning the IP address to the clients from the server side brings any advantage over "hardcoding" it at client side.

frontdist · Mon Oct 08, 2018 11:56 pm

Using pure IPsec (IKEv2), you can use /ip ipsec user to configure username, "password" and IP address in a similar way as with /ppp secret for ppp interfaces if you use pre-shared key & xauth authentication mode (it doesn't work with certificates). However, there is no script you could associate with the event of IPsec connection to come up, so I'm not sure whether assigning the IP address to the clients from the server side brings any advantage over "hardcoding" it at client side.

Thanks...

I have no problem hard coding the client address, as basically I am just looking for an endpoint for my GRE tunnel (within IPSEC), so that I can route whatever I need to the other end. I am getting myself confused with this and it's not going well.

I have a working IKEv2 point connection set up, and from the road-warrior side I can connect to a machine on the "headend" side, but I can't seem to be able to establish a GRE tunnel from one end to another.

I will post my relevant configs when I have an opportunity.

sindy · Tue Oct 09, 2018 1:15 am

You need to assign unique private addresses to the client routers (best attached to a bridge with no member ports) and set the GRE tunnels between an address on the server router and these addresses, using the IPsec in tunnel mode. So the IPsec policy transports GRE between the private address at the client side and the address on the server side. The rest is routed via the GRE. You can use IPsec policies alone but it is easy to get lost.

frontdist · Wed Oct 10, 2018 6:46 am

You need to assign unique private addresses to the client routers (best attached to a bridge with no member ports) and set the GRE tunnels between an address on the server router and these addresses, using the IPsec in tunnel mode. So the IPsec policy transports GRE between the private address at the client side and the address on the server side. The rest is routed via the GRE. You can use IPsec policies alone but it is easy to get lost.

I didn't think I could choose a tunnel/transport mode when the tunnel was IKEv2?

Am I able to preselect a remote address from the setup on the remote end? It doesn't NEED to be dynamically assigned, because it never has to change, and I will be setting it up manually. The problem is I didn't think that I could statically assign the address because I am using a dynamic policy on one end of the SA. I did try messing around with a GRE tunnel by setting it up as a bridge interface, but didn't get anywhere there either.

Unlike a site to site tunnel, I am trying to set this up as a "road warrior" style because the remote endpoints are always going to have dynamic public IP's, and are always going to be behind at least 2 layers of NAT.

Config for the head-end:

# oct/09/2018 23:44:46 by RouterOS 6.43.1
# software id = 
#
# model = RouterBOARD 1100x4
# serial number = 
/interface bridge
add fast-forward=no name=IPSec_Bridge
add fast-forward=no name=loopback_test
/interface gre
add !keepalive name=gre-tunnel1 remote-address=192.168.77.3
/interface list
add name=WAN
add name=LAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip ipsec policy group
add name=rw-policies
/ip ipsec proposal
add name=rw-proposal pfs-group=none
/ip pool
add name=dhcp ranges=192.168.88.100-192.168.88.110
add name=rw-pool ranges=192.168.77.2-192.168.77.5
/ip dhcp-server
add address-pool=dhcp disabled=no interface=ether2 name=dhcp1
/ip ipsec mode-config
add address-pool=rw-pool address-prefix-length=32 name=rw-conf split-include=192.168.89.0/24,192.168.90.0/24
/interface list member
add interface=ether1 list=WAN
add interface=ether2 list=LAN
add interface=ether3 list=LAN
/ip address
add address=210.xxx.xxx.98/30 interface=ether1 network=210.xxx.xxx.96
add address=192.168.90.10/24 interface=IPSec_Bridge network=192.168.90.0
add address=192.168.89.1/24 interface=ether2 network=192.168.89.0
add address=172.30.30.1 interface=loopback_test network=172.30.30.1
/ip dhcp-server network
add address=192.168.88.0/24 gateway=192.168.88.1 netmask=24
/ip dns
set servers=8.8.8.8
/ip firewall filter
add action=accept chain=forward dst-address=192.168.89.0/24 src-address=192.168.88.0/24
add action=accept chain=forward dst-address=192.168.88.0/24 src-address=192.168.89.0/24
/ip firewall nat
add action=accept chain=srcnat dst-address=192.168.88.0/24 src-address=192.168.89.0/24
add action=masquerade chain=srcnat out-interface-list=WAN
/ip ipsec peer
add auth-method=rsa-signature certificate=server1 exchange-mode=ike2 generate-policy=port-strict mode-config=rw-conf passive=yes policy-template-group=rw-policies
/ip ipsec policy
add dst-address=192.168.77.0/24 group=rw-policies proposal=rw-proposal src-address=0.0.0.0/0 template=yes
/ip route
add distance=1 gateway=210.xxx.xxx.97
add distance=1 dst-address=210.xxx.xxx.104/32 gateway=*F
/system clock
set time-zone-name=America/Toronto
/system routerboard settings
set silent-boot=no

Remote config to come...

sindy · Wed Oct 10, 2018 4:05 pm

I didn't think I could choose a tunnel/transport mode when the tunnel was IKEv2?

Transport/tunnel mode is a choice related to ESP used to transport the tunnelled data; both IKE and IKEv2 are used only to set up the AH or ESP transport and negotiate the mode.

But once at least one of the peers is behind a NAT, i.e. if peer A can see packets coming from peer B with a different source address than peer B's actual one, both peers need to encapsulate ESP into UDP, as ESP doesn't support the concept of ports, so at maximum one plain ESP connection per a pair of local+remote IP addresses is possible. So two clients connecting to the same server IP from behind the same NAT IP would be unable to work without the UDP encapsulation of the ESP.

But the source address of the plain ESP pcaket decapsulated from the UDP is inherited from the UDP, and in transport mode the source address of the payload packet decrypted from the ESP packet is also inherited from the ESP, so you could still have only one such client if you would use transport mode.

So here the need for tunnel mode comes into play, where the IPIP encapsulation is used inside the ESP, so the original source address of the payload survives. But if it is not a public one, there is a chance that it will be in conflict with some other private address, so they need to be coordinated between the clients and the server.

So in your scenario where all the clients come from behind the same NAT address:

at server side,
- you use a peer with address=0.0.0.0/0 but it must use one of the authentication methods which allow to use something else than remote IP address to choose a local context for each connection. In case of IKEv2 implementation on Mikrotik, you can only use auth-method=pre-shared-key-xauth for this. Mikrotik currently doesn't support choosing of local context up to client's certificate. You don't need to use mode-config if you assign the private IP (or range) to each client statically in its own configuration.
- to adjust to an unknown-in-advance remote address for the UDP-encapsulated ESP (policy's sa-dst-address) you have to set generate-policy=port-strict on the peer and let the actual address be determined and used during the IKEv2 negotiation. But it is strongly recommended to provide a dedicated restricted policy template for each of the clients so that one of the clients could not steal the show by redirecting all the server's traffic to itself.
at client side, you configure the peer to represent the server, i.e. with address=the.ip.of.the.server, and you configure the policy statically and set level=unique. So when the control connection establishes, the client suggests a policy and the server looks for a mirror one among its templates; if it can find exactly one, it uses it, otherwise the connection fails.

So if you could coordinate the private ranges of the clients and you wouldn't have a mental problem with using traffic selectors (policies) instead of regular routing, you might skip the GRE layer. But if you insist on GRE, the policy may be configured for just a single address at client side, and the GRE tunnel must be attached to this address.

So the complete encapsulation is then IP->UDP->ESP->IPIP->GRE.

I did try messing around with a GRE tunnel by setting it up as a bridge interface, but didn't get anywhere there either.

This cannot work. A gre interface is an L3 one so it cannot be a member port of a bridge - only L2 interfaces can be bridge ports. But you can attach addresses from different subnets to the two ends of a gre tunnel and it will still work if the routing configuration is appropriate. So on server side, you set /ip address add interface=gre-1 address=a.a.a.a/32 network=b.b.b.b/32; on client side, you set /ip address add interface=gre-1 address=b.b.b.b/32 network=a.a.a.a/32. If the server receives a packet for b.b.b.b, it sends it via gre-1 thanks to the dynamic route created. But to make sure that the server receives the packet, you need to configure routing accordingly before the server. So if b.b.b.b/32 is in server's WAN subnet, you have to set arp=proxy-arp on server's WAN interface; if it is not in server's WAN subnet, you have to advertise it using BGP or RIP dependning on what the ISP tells you, or ask for a static route.

frontdist · Sat Oct 13, 2018 2:56 am

This is incredible information, thank you for the time you put into your response.

I have been too busy with other projects this week, but I am going to lab this tomorrow and see where I get.

Once I do, I will post relevant configs and results for everyone to take a look at.

Thanks!

frontdist · Sat Oct 13, 2018 8:09 pm

I have a version of it working... it's incredibly slow, but it's working.

Will post the configs when I have a chance to see if it can be further diagnosed.

The "client" end will have a /30 on a LAN port, and the security camera NVR will be connected to that.

I am using mangle to force the client IP (non routable) over the GRE tunnel to the head-end MTik, and from there I am doing src and dst NAT to a static public IP from my routable /24.

Unfortunately I am getting at most 10 Mbps, and both ends have 300/300 fibre connections, so I am not sure what I am doing to make a mess of it. Both ends have hardware IPsec (client side is a Hex, head-end side is a brand new RB1100AHx4).

Hopefully the configs can be used to help diagnose it.

frontdist · Sun Oct 14, 2018 2:07 am

I have run into an additional issue here and am looking for help getting around it.

I have posted the configs below for two of the routers (one client, one server) and am running into an issue when I have more than one client behind the same IP address. I am getting phase1 to come up between both client routers and the server, but only one will establish phase2 at a time. I have to disconnect one of the client routers to get the other one to establish a connection and pass traffic.

For my setup so far, what I have done is first create an IPSEC connection between the two ends, and using a private /32 at each end of the connection. From there, I created a GRE tunnel between the two points, and then used mangle to mark packets from a LAN IP (on the client side), so it was routed over the GRE tunnel, and then I did src-nat and dst-nat on the server side router to translate the private IP to a different public IP from another block. For each client "router" only one LAN port will be active, and it will hand out a single address (LAN port will be on a /30) to a single machine. Each client router will therefore have a translated public IP at the server side Mtik.

Is there a better way of doing this?

Also, I am having a terrible time with throughput and latency. I have played a bit with MSS clamping in mangle on both ends, but I can't seem to fine tune this to my liking. Throughput is pretty bad considering there is a 300/300 connection at each end. Granted this is only a test environment, but I want to make sure I'm not doing anything in my config that would negatively impact performance.

Here is the current configuration:

Server

# oct/13/2018 18:53:32 by RouterOS 6.43.1
# software id = 
#
# model = RouterBOARD 1100x4
# serial number = 
/interface bridge
add fast-forward=no name=IPSec_Bridge
add fast-forward=no name=loopback_test
/interface gre
add !keepalive local-address=172.20.20.1 name=gre-tunnel1 remote-address=172.20.20.2
/interface list
add name=WAN
add name=LAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip ipsec peer profile
set [ find default=yes ] enc-algorithm=aes-256,aes-128
/ip ipsec policy group
add name=rw-policies
add name=11B9
add name=EC5B
/ip ipsec proposal
set [ find default=yes ] enc-algorithms=aes-256-cbc,aes-128-cbc pfs-group=none
/ip pool
add name=rw-pool ranges=192.168.77.2-192.168.77.5
/ip dhcp-server
add address-pool=dhcp disabled=no interface=ether2 name=dhcp1
/ip ipsec mode-config
add address-pool=rw-pool address-prefix-length=32 name=rw-conf split-include=192.168.89.0/24,192.168.90.0/24
/interface list member
add interface=ether1 list=WAN
add interface=ether2 list=LAN
add interface=ether3 list=LAN
/ip address
add address=210.xxx.xxx.98/30 interface=ether1 network=210.xxx.xxx.96
add address=192.168.89.1/24 interface=ether2 network=192.168.89.0
add address=172.30.30.1 interface=loopback_test network=172.30.30.1
add address=192.168.90.10 interface=IPSec_Bridge network=192.168.90.10
add address=172.20.20.1 interface=gre-tunnel1 network=172.20.20.1
/ip dhcp-server network
add address=192.168.88.0/24 gateway=192.168.88.1 netmask=24
/ip dns
set servers=8.8.8.8
/ip firewall filter
add action=accept chain=forward dst-address=192.168.89.0/24 src-address=192.168.88.0/24
add action=accept chain=forward dst-address=192.168.88.0/24 src-address=192.168.89.0/24
/ip firewall mangle
add action=change-mss chain=forward new-mss=1240 out-interface=gre-tunnel1 passthrough=yes protocol=tcp tcp-flags=syn
/ip firewall nat
add action=accept chain=srcnat dst-address=192.168.88.0/24 src-address=192.168.89.0/24
add action=accept chain=srcnat dst-address=172.20.20.2 src-address=172.20.20.1
add action=src-nat chain=srcnat src-address=10.0.0.2 to-addresses=210.xxx.xxx.104
add action=accept chain=srcnat disabled=yes dst-address=10.0.0.0/24 src-address=192.168.89.0/24
add action=dst-nat chain=dstnat dst-address=210.xxx.xxx.104 to-addresses=10.0.0.2
add action=masquerade chain=srcnat disabled=yes
/ip ipsec peer
add auth-method=rsa-signature certificate=server1 disabled=yes exchange-mode=ike2 generate-policy=port-strict mode-config=rw-conf passive=yes policy-template-group=\
    rw-policies
add address=0.0.0.0/0 auth-method=pre-shared-key-xauth exchange-mode=ike2 generate-policy=port-strict my-id=fqdn:server.11b9 passive=yes policy-template-group=11B9 secret=\
    secret send-initial-contact=no
# This entry is unreachable
add address=0.0.0.0/0 auth-method=pre-shared-key-xauth exchange-mode=ike2 generate-policy=port-strict my-id=fqdn:server.ec5b passive=yes policy-template-group=EC5B secret=\
    secret send-initial-contact=no
/ip ipsec policy
set 0 disabled=yes
add dst-address=192.168.88.0/24 src-address=192.168.89.0/24 template=yes
add dst-address=172.20.20.2/32 group=11B9 src-address=172.20.20.1/32 template=yes
add dst-address=172.20.20.6/32 group=EC5B src-address=172.20.20.5/32 template=yes
/ip ipsec user
add name=user1 password=Password1
add name=user2 password=Password2
/ip route
add distance=1 gateway=210.xxx.xxx.97
add distance=1 dst-address=10.0.0.0/29 gateway=gre-tunnel1
add distance=1 dst-address=172.20.20.2/32 gateway=gre-tunnel1
add distance=1 dst-address=192.168.88.0/24 gateway=gre-tunnel1
/ip service
set telnet disabled=yes
set ftp disabled=yes
set ssh disabled=yes
/system clock
set time-zone-name=
/system routerboard settings
set silent-boot=no
/tool bandwidth-server
set authenticate=no

frontdist · Sun Oct 14, 2018 2:07 am

Client

# oct/13/2018 18:51:18 by RouterOS 6.43.2
# software id = 
#
# model = RouterBOARD 750G r3
# serial number = 
/interface ethernet
set [ find default-name=ether1 ] speed=100Mbps
set [ find default-name=ether2 ] speed=100Mbps
set [ find default-name=ether3 ] disabled=yes speed=100Mbps
set [ find default-name=ether4 ] disabled=yes speed=100Mbps
set [ find default-name=ether5 ] disabled=yes speed=100Mbps
/interface gre
add !keepalive local-address=172.20.20.2 name=gre-tunnel1 remote-address=172.20.20.1
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip hotspot profile
set [ find default=yes ] html-directory=flash/hotspot
/ip ipsec peer profile
set [ find default=yes ] enc-algorithm=aes-256,aes-128
/ip ipsec policy group
add name=11B9
/ip ipsec proposal
set [ find default=yes ] enc-algorithms=aes-256-cbc,aes-128-cbc pfs-group=none
/ip pool
add name=dhcp ranges=10.0.0.5-10.0.0.6
/ip dhcp-server
add address-pool=dhcp disabled=no name=defconf
/ip neighbor discovery-settings
set discover-interface-list=LAN
/interface list member
add interface=ether2 list=LAN
add interface=ether1 list=WAN
add interface=gre-tunnel1 list=LAN
/ip address
add address=10.0.0.1/29 comment=defconf interface=ether2 network=10.0.0.0
add address=172.20.20.2 interface=gre-tunnel1 network=172.20.20.2
/ip dhcp-client
add comment=defconf dhcp-options=hostname,clientid disabled=no interface=ether1
/ip dhcp-server network
add address=10.0.0.0/29 comment=defconf gateway=10.0.0.1 netmask=29
/ip dns
set allow-remote-requests=yes
/ip dns static
add address=10.0.0.1 name=router.lan
/ip firewall filter
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=drop chain=input comment="defconf: drop all not coming from LAN" in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=established,related
add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf:  drop all from WAN not DSTNATed" connection-nat-state=!dstnat connection-state=new in-interface-list=WAN
/ip firewall mangle
add action=change-mss chain=forward new-mss=1330 passthrough=yes protocol=tcp tcp-flags=syn
add action=mark-routing chain=prerouting new-routing-mark=GRE passthrough=yes src-address=10.0.0.2
/ip firewall nat
add action=accept chain=srcnat disabled=yes dst-address=172.20.20.1 src-address=172.20.20.2
add action=accept chain=srcnat out-interface=gre-tunnel1 src-address=10.0.0.2
add action=accept chain=srcnat disabled=yes out-interface=gre-tunnel1 src-address=10.0.0.1
/ip ipsec peer
add address=210.xxx.xxx.98/32 auth-method=pre-shared-key-xauth exchange-mode=ike2 generate-policy=port-strict my-id=fqdn:client.11b9 policy-template-group=11B9 secret=secret \
    xauth-login=user2 xauth-password=Password2
/ip ipsec policy
add dst-address=172.20.20.1/32 level=unique sa-dst-address=210.xxx.xxx.98 sa-src-address=0.0.0.0 src-address=172.20.20.2/32 tunnel=yes
/ip route
add distance=1 gateway=gre-tunnel1 routing-mark=GRE
add distance=1 dst-address=192.168.89.0/24 gateway=gre-tunnel1
/system clock
set time-zone-name=America/Toronto
/system resource irq rps
set ether1 disabled=no
set ether2 disabled=no
set ether3 disabled=no
set ether4 disabled=no
set ether5 disabled=no
/system routerboard settings
set silent-boot=no
/tool bandwidth-server
set authenticate=no
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN

frontdist · Sun Oct 14, 2018 3:40 am

I am able to occasionally get both connected at the same time, but it is inconsistent, and I am seeing this warning as well:

Unreachable.jpg

sindy · Sun Oct 14, 2018 11:55 am

You have misunderstood how the /ip ipsec peer configuration works.

When an initial packet comes from the remote initiator, its source IP address and the exchange mode are matched against the address and exchange-mode items of all rows under /ip ipsec peer for exact match of exchange mode and best match of address. Best match of address means that the row with longest address prefix (mask) is chosen if more than one match. So if there are three peers with exchange-mode=ike2 and address values 0.0.0.0/0, 100.100.100.0/24, and 100.100.100.5/32, the third one will be chosen if the initial IKEv2 packet comes from 100.100.100.5, the second one will be chosen it the packet comes from 100.100.100.6, and the first one will be chosen if the packet comes from 100.100.99.3.

If some other local-address other than the default 0.0.0.0 (meaning any local address of the routerboard) is specified for the row in /ip ipsec peer, it is also taken into account.

This is the stage where the rest of the configuration elements is chosen. So all actual remote peers whose initial packets match the same row in /ip ipsec peer

must be distinguished from one another by their ID, which is looked up as the name item under /ip ipsec user (if auth-method=pre-shared-key-xauth) or in an external user database using RADIUS (if auth-mode=eap-radius or rsa-signature-hybrid),
must make do with the same settings of the responder peer, namely the profile, generate-policy, and policy-group.

In your configuration, you have set two /ip ipsec peer rows with same address and exchange-mode, which means only the one of them declared first is ever used. What should have warned you was that there is no remote-id parameter of the peer nor any link between the /ip ipsec user items and the /ip ipsec peer items.

Now regarding the assignment of addresses and matching policies to the initiators (clients). As a single /ip ipsec peer row is used, at responder (server) side, to create several distinct connections to actual remote peers whose addresses are unknown in advance, the policies for them must be dynamically generated as the sa-dst-address of the policy must be determined when the remote peer comes up. So the responder peer configuration must be set to generate-policy=port-strict or port-override.

The IP address of the client can either be configured statically at the client (or an existing one can be used) or the server can assign it using the mode-config procedure, and so can be the subnets which the client can reach via the tunnel. In the first case, the policy (or multiple policies) are configured statically at the client as well, because all of their parameters (src-address, dst-address, sa-src-address, sa-dst-address) are known in advance. In the second case, also generate-policy at the client must be set to port-strict or port-override.

If you insist on running the GRE tunnels to clients, I'd recommend to set the address and policies statically at each client device, given that you probably cannot fully automate the link between the public IP you assign to a particular client and the client's location, so you cannot create 20 identical client configurations and only set individual user names and passwords for them. You need the link between the user name and the public address tunneled to it as well, and this link goes via the private address which the client will use and to which the public one will be tunneled via the GRE.

But if you can wrap your head around the idea of using no GRE at all, you can assign the public IPs directly using mode-config if you set these addresses in the /ip ipsec user at server side. In that case, you have to assign a mode-config profile to the /ip ipsec peer row at both the client and server side, whereas at client side it would be a modified version of the default request-only one and at server side it would be one created for the use case.

The easiest way how to configure the policy templates at responder side is to set a single policy template in the group, saying src-address=0.0.0.0/0 dst-address=0.0.0.0/0, but it is dangerous as it would let a misconfigured peer steal the whole responder's traffic by setting src-address=0.0.0.0/0 in its own policy. The safest approach is to create the exact policies for each client as templates. Or you can simply create a single template with as narrow src-address and dst-address subnets as sufficient for it to work (so if you decide to assign the public addresses directly using mode-config, you would set the template to src-address=0.0.0.0/0 dst-address=the.public.subnet.you.use/its-mask).

On the client side, the policy template's src-address and dst-address may stay the default ones.

One important moment is encryption. Do you really need it, given that you'd only encrypt the path from the client to your server, and from there on the data would flow as the camera has sent them anyway? If you don't, you can set enc-algorithms=null in the proposals at both sides, and thus save some CPU (unless authentication cannot be hardware assisted if encryption algorithm is null, I've never tried that). If you use ssh, https, or winbox to connect to the clients via the tunnel, you don't need the IPsec to encrypt the management access either.

On the other hand, you do want the highest available protection against someone connecting instead of your client. So use strong encryption alogorithms in /ip ipsec peer profile, and use a hex-encoded 128-byte random string (i.e. 256 hex symbols) as the /ip ipsec peer secret and other such strings as /ip ipsec user password of the individual clients.

If you choose the way of direct assignment of the public IP addresses to the client devices, there is one more extra you need to cover, which is the port forwarding to the cameras in situation where the public IP is unknown in advance. The dst-nat rules have to match on in-interface=the-wan-if-name ipsec-policy=in,ipsec so that they would only be applied on traffic coming in via the IPsec tunnel.

frontdist · Sun Oct 14, 2018 2:08 pm

Thanks Sindy, I see what you mean now and it make a lot more sense to me...

In the case of avoiding GRE, part of the problem is that I don't know how to make the client camera use the IPSEC address and tunnel as its default gateway for outbound connections...

The cameras we use have an "automatic NAT traversal" that talks to a mediator server on the outside, but with this particular WISP they have such aggressive NAT policies that the connection often gets killed and client can't access the cameras from the app.

One solution is obviously to DST-NAT the cameras using the IPSEC mode-config address as you alluded to, but is there a way to force the client device to have it's 0.0.0.0/0 gateway go over the IPSEC tunnel rather than out the 0.0.0.0/0 at ether1 which is the WAN source for the client Mikrotik? That's partly why I was using the GRE tunnel, because I could mark the packet and send it over a specific tunnel.

Thanks!

sindy · Sun Oct 14, 2018 3:26 pm

The cameras we use have an "automatic NAT traversal" that talks to a mediator server on the outside, but with this particular WISP they have such aggressive NAT policies that the connection often gets killed and client can't access the cameras from the app.

Three possibilities of "automatic NAT traversal" exist:

the server device can use external servers to determine how the NAT behaves and adjust its own behaviour accordingly, but this only works with UDP and only in a limited number of cases and only with really simple NATs,
the server device sets up a permanent connection to an external server and the clients connect to that external server which forwards the connections to the server device behind the NAT (so the client must provide some other, application-layer identifier to tell the external server which server device it wishes to connect to),
the server device uses UPnP protocol to talk to the NATing router and asks it to create dst-nat rules for it on its internet-facing interface, but in this case several devices of the same type fight for the same ports on a single common public address

So I'm afraid that if the cameras need more than a single incoming TCP connection to work, you may have a tough time making all that work with several cameras behind a single IP, and that the "aggresivity" of the ISP's NAT is not your biggest problem. Frequent changes of the public IP of the clients may be a problem of course, but it remains a problem even with the IPsec workaround, because it takes some time to the peers to notice that the tunnel stopped working and to re-establish it. So expect gaps in accessibility of the cameras even if the rest runs smoothly.

One solution is obviously to DST-NAT the cameras using the IPSEC mode-config address as you alluded to, but is there a way to force the client device to have it's 0.0.0.0/0 gateway go over the IPSEC tunnel rather than out the 0.0.0.0/0 at ether1 which is the WAN source for the client Mikrotik? That's partly why I was using the GRE tunnel, because I could mark the packet and send it over a specific tunnel.

I didn't know that it's the cameras what actively establishes outbound connections (I'm not a remote surveillance specialist), so I was thinking in the old-fashioned "make my server on private LAN accessible from outside" way.

So to make the cameras' outgoing connections use the IPsec tunnel with no GRE, you have to src-nat all these outgoing connections to the address assigned to the local end of the IPsec tunnel. The IPsec policies inspect the outgoing packets after the firewall handling, including NAT, has been done. See the details in last-but-one paragraph of this post and/or in the related part of the manual.

If the cameras use UPnP, you will have to use the GRE tunnel because UPnP needs to have an external interface defined, and plain IPsec tunnel doesn't provide any interface (the IP address provided by mode-config is levitating in the air or is attached to a seemingly randomly chosen existing interface, I don't know). But from what you have written it doesn't seem likely to me that the cameras use UPnP.

frontdist · Sun Oct 14, 2018 4:04 pm

I do expect some gaps in coverage, but it is better than the current situation, which is that the cameras are visible for 30 mins to a few hours after they are booted, then disappear until something is done.

Yes, I could set them up on an automatic reboot scheduler, but they take a while to come up, and the footage is stored locally anyways, so I wouldn't want them to reboot in the middle of a triggered incident.

I have things up and running now using the same Peer selector 0.0.0.0/0, same group and policy as I discovered my error thanks to your helpful posts.

I now have two "client" devices connecting to the server from behind the same NAT, and was able to either provide an address via mode-config (with a template on both sides), or statically assign a policy on the client side and put a generic template (covering the intended ranges) on the server side. I was also able to use null/null, though I don't see much of an improvement in throughput.

I am seeing a strange issue once the clients connect though, the first client is fine, and I see traffic passing on both sides of the SA that is created, however when the second client connects, I only see traffic coming from the "client" side of the SA, I don't see any return traffic from the server side of the SA. Not sure what that is about, but possibly with how my home router is handling the two connections, not sure?

I am going to continue along the GRE path, I can force the cameras to use the tunnel as a default gateway, which will allow one of two use cases here... Either I can allow them to contact the mediator and the client can connect with the app, or in the case where static addressing and PAT is required, I can provide a static address on the server side and DST-NAT it to the IP handed off at the "client" side Mikrotik.

I also am having an issue with the MTU... I had two "client" routers set up exactly the same, and GRE tunnels the same, but one had a MTU of 1280 and the other one was 1370 on the server side (from PMTU). On the client side, they were both the same (from PMTU), but different than both the displayed server side numbers. It doesn't make much sense to me why that is happening, and it's making it a challenge to set up correct MSS values.

Also having a strange issue when I do internet speed tests... If directly ping the server being used, I get something reasonable 7ms, etc. however when I use speedtest.net the ping is showing over 1000ms, and although I get a "download" result, the upload is just a flat line and never works. All firewall options were disabled and still an issue. Not sure what that's about... obviously I can upload because I can browse...

Luckily this isn't going to be tunneling the entire connection for the client (only the cameras), but I would like it to work optimally.

sindy · Sun Oct 14, 2018 5:00 pm

I now have two "client" devices connecting to the server from behind the same NAT, and was able to either provide an address via mode-config (with a template on both sides), or statically assign a policy on the client side and put a generic template (covering the intended ranges) on the server side. I was also able to use null/null, though I don't see much of an improvement in throughput.

I've made a test, if you set enc-algorithms to null, the hardware acceleration doesn't work although the auth-algorithms is one of those supported by hardware acceleration. So it only makes sense to use null encryption where none of the peer devices can benefit from hardware acceleration; where both ends support it, choose the strongest auth-algorithm and enc-algorithm combination which both ends support and forget about it.

I am seeing a strange issue once the clients connect though, the first client is fine, and I see traffic passing on both sides of the SA that is created, however when the second client connects, I only see traffic coming from the "client" side of the SA, I don't see any return traffic from the server side of the SA. Not sure what that is about, but possibly with how my home router is handling the two connections, not sure?

When you /ip ipsec installed-sa print on the server, can you see the port number in the dst-address and src-address of the SAs? If not, it means that the NAT discovery has failed and that the peers use plain ESP for the transported data, which means that the two destinations cannot be distinguished from one another by anything else than the SPI. Another explanation could be that your policies overlap. So show me your /ip ipsec policy print and /ip ipsec installed-sa print from the server in this situation and be careful when obfuscating (i.e. modify only the "most significant = first" two bytes of the IP addresses so that the context doesn't get lost.

I am going to continue along the GRE path, I can force the cameras to use the tunnel as a default gateway

The default gateway for the camera itself is always the client Mikrotik. The default gateway for the traffic from the cameras on the client Mikrotik is what you need tto affect, and if you have a GRE tunnel, you use the GRE tunnel interface name or the network address from the IP configuration (somehow misleadingly called /ip address) attached to that interface as a gateway in the default route with the routing-mark. But without the GRE tunnel, you put the address pool range you use to assign addresses to the cameras onto an /ip firewall address-list to which the action=src-nat rule, dynamically created by the IPsec astro clock once the SA establishes, refers. So any outgoing connection from any of these addresses will be src-nated to the address provided by mode-config, and this is the reason why the policy will match and steal them. So instead of the routing-mark, you use the src-address assigned by the src-nat rule to choose the outgoing path.

in the case where static addressing and PAT is required, I can provide a static address on the server side and DST-NAT it to the IP handed off at the "client" side Mikrotik.

The point is that with plain IPsec and mode-config, you don't do any dst-nat on the IPsec server. The server receives, via WAN, a packet for the address assigned to the IPsec client. It routes it according to the default route (which most likely would send it back to the WAN so you must make sure that a firewall doesn't drop such packet, which is otherwise a good practice), but at the last moment before the packet would be sent out the interface, the ipsec policy kicks in and steals it to deliver it via the SA to the client. In another words, the public address is not up on your IPsec server, it is up on the client. So you can filter the traffic for it on the server, but the dst-nat will be done on the client the way I've described above.

With GRE, you'd need to do the dst-nat twice - once from the public address to the private one from the range coordinated between the clients, and the second time on the client from the coordinated range address to the actual address of the camera, assuming that the local ranges may overlap between clients. If they don't and you can coordinate the address space between all clients, you have one headache less.

I also am having an issue with the MTU... I had two "client" routers set up exactly the same, and GRE tunnels the same, but one had a MTU of 1280 and the other one was 1370 on the server side (from PMTU). On the client side, they were both the same (from PMTU), but different than both the displayed server side numbers. It doesn't make much sense to me why that is happening, and it's making it a challenge to set up correct MSS values.

A lot of hidden magic happens internally, and sometimes there are bugs in it. Depending on the authentication algorithm used and the "plain ESP" or "ESP over UDP" transport mode chosen, the amount of overhead differs for the IPsec transport packets, leaving different space for the payload. I've seen someone complain here recently that for sha-256 used for authentication, the MTU calculations were correct while for sha-1 they were not, or vice versa. The GRE tunnel gets an information about MTU from the IPsec part, which gets it from the physical layer, etc. And if one element in the chain reports a wrong (too optimistic) value, you run into trouble. At least when everything gets correct, you should see equal values at/for all clients.

Also having a strange issue when I do internet speed tests... If directly ping the server being used, I get something reasonable 7ms, etc. however when I use speedtest.net the ping is showing over 1000ms, and although I get a "download" result, the upload is just a flat line and never works. All firewall options were disabled and still an issue. Not sure what that's about... obviously I can upload because I can browse...

The only thing which comes to my mind is that you have an action=fasttrack rule active without limitations at the server side or at the client side. Fasttracking is an optimization technique which is based on skipping some stages of packet processing for the bulk of packets belonging to already established connections, and these stages include not only firewall but also ipsec policy matching. So if you could reach 10 Mbit/s with fasttracking misrouting most of your traffic into a blackhole, chapeau

frontdist · Sun Oct 14, 2018 5:32 pm

You're a saint...

So, first with the fasttrack... When I said I had the firewall stuff disabled, I meant the block entries. I disabled fasttrack and all of a sudden I am getting almost line speed. I'll see what happens when I turn encryption back on now.

I didn't mean making the default gateway of the cameras the GRE endpoint, but routing them that way in the Mikrotik, I mis-wrote that as obviously that wouldn't work.

Thanks for the pointers on using mode-config and having it match policy.

Looking at the SA's, the first one used 4500, but the next one used 1024 as a port... They did both come up eventually, not sure why there was hesitancy at first. I spend a lot of time reading the forums, and sometimes I can't tell if something is or isn't working because I screwed up, or because they broke something in a firmware version that used to work on the last one, etc... I know Cisco, Juniper, etc have the same issues sometimes, but these seem to be particularly vulnerable...

sindy · Sun Oct 14, 2018 5:58 pm

So, first with the fasttrack... When I said I had the firewall stuff disabled, I meant the block entries. I disabled fasttrack and all of a sudden I am getting almost line speed. I'll see what happens when I turn encryption back on now.

The right thing to do is to add some match conditions to the action=fasttrack rule which will prevent it from matching on either direction of the traffic which must be seen by the ipsec policy, or to use /ip firewall raw rules to prevent this traffic from being connection-tracked at all (if you do that, you cannot use distinct connection-state and connection-nat-state values for that traffic elsewhere in /ip firewall, all of it matches to connection-state=untracked).

I didn't mean making the default gateway of the cameras the GRE endpoint, but routing them that way in the Mikrotik, I mis-wrote that as obviously that wouldn't work.

I know, I just like things to be written precisely because it avoids a lot of confusion.

Looking at the SA's, the first one used 4500, but the next one used 1024 as a port...

That's correct. Most NATs try to keep the original src-port when doing src-nat if possible, so the first connection comes from 4500 because that's what the client uses, for the second connection that port is already occupied for the same remote socket (ser.ver.ip:4500), so the NATing device has to choose another one.

I was asking only about the very presence of the port numbers, as their absence would have indicated that the transport was plain ESP.

They did both come up eventually, not sure why there was hesitancy at first. I spend a lot of time reading the forums, and sometimes I can't tell if something is or isn't working because I screwed up, or because they broke something in a firmware version that used to work on the last one, etc... I know Cisco, Juniper, etc have the same issues sometimes, but these seem to be particularly vulnerable...

If you haven't changed a single bit and it started working anyway, it's weird of course. What is your software version at both ends? There used to be an issue with SA key negotiation when pfs-group was different than none which has been fixed only very recently. Should this be the reason, you would see packets being counted on the sending SA on each device, but not being counted at the remote end of that SA. It had nothing to do with client and server roles, both directions were affected as the authentication of these packets was failing on receiving side due to the difference of keys which you could see if you had access to both the client and the server while the tunnel was down.

Again the wording is important here - you wrote

I only see traffic coming from the "client" side of the SA, I don't see any return traffic from the server side of the SA.

which implies that you were watching it at the server side. If exactly so, then the reason must different than suggested above; if you were actually observing the SA statistics at the client side, the reason suggested above may be the explanation.

frontdist · Tue Oct 16, 2018 12:15 am

Setup critique...

*apologies in advance, the [ c o d e ] tag isn't working for the second set for some reason???*

Here is what I have working so far... In most cases I will put a /30 on the LAN interface of the "client" router and simply provide a single address that goes through a src-nat at the "server/concentrator" end of things, however with remote management of the router, I would be able to adjust the size of the LAN subnet, the DHCP leases and on the server side make appropriate changes to the src-nat, and also do dst-nat if I wish, provided I know the client addresses of each camera (static). More likely scenario is to place a NVR or other router after the "client" Mikrotik, handing off a single address and allowing for that end of things to do firewalling and port forwarding as appropriate.

Reasons for the setup chosen:
- IKEv2 so that I could have multiple clients use the same source IP behind NAT because their WISP will not deploy IPv6, and refuses to give any static assignments on LTE (they are out of IPv4 space). Security was not a concern, however PPTP, and L2TP were unusable due to the NAT issue, and SSTP/OpenVPN were back-burnered in favour of using hardware encryption.
- GRE chosen to have flexibility over routing, possible expansion to OSPF or BGP if client base becomes too large for static routing maintenance.
- NAT performed at headend/concentrator router to make changes to client public addressing without having to access client router if offline, etc plus ability to share a single public IP for clients that don't require dedicated dst-nat support.
- No user access required to the "client" router. Two ports will be open on device (WAN and LAN), other ports blocked with label. Simple plug and play operation.

Problems to solve:
- Proper firewalling for client devices. These will not be connected to a "raw" internet connection, only behind another customer router so firewall only needs basic setup.
- Possibly making default route for clients "conditional" on the tunnel being up, though the clients can't see their cameras remotely now anyways, so not sure what it matters.
- Finding a way to give clients "local" access to their cameras (faster footage retrieval, browsing, etc) when on location without having to do static routing on their home router? Possible wireless router with dedicated ssid for management or direct connection to avoid being routed out and back in over the WAN.
- Ensure that I don't lock myself out from remote management. Management will have to be via GRE tunnel through the IPSEC. Currently the default firewall on the Hex devices block management attempts with the "block input chain not from (!LAN) LAN interface list" When this rule is disabled I am able to access the router remotely, when enabled I am blocked. I added the GRE interface to the "LAN" interface list, but that didn't work either. Only disabling the rule works. The access request is coming from a computer in the 192.168.89.0/24 subnet on the "concentrator" router. It's not a routing issue because disabling firewall rule allows it to work, but I must be misunderstanding how the packets are processed as part of the "input" chain.
- For the above, running "log" on the rule instead of "drop" results in all management connections (regardless of source address from the far side of the GRE tunnel), showing as if they are coming FROM the other side of the GRE tunnel, ever with a NAT accept rule in place on the server side preventing NAT from occurring between the management subnet (192.168.89.0/24) and the remote GRE endpoint, or remote client LAN IP.

Structure:
- 10.0.10.0/24 (in /32 allocations) used for "concentrator" side of GRE tunnel
- 10.0.11.0/24 (in /32 allocations) used for "client" side of GRE tunnel
- 10.0.101.0/24 to 10.0.199.0/24 used for "client" LAN side, however may be /30 or /29 as appropriate. However each client is separated by a full /24 for logical purposes.
- Mangle used to mark anything from client LAN subnet for routing over GRE tunnel

Here are the configs!

SERVER

# oct/15/2018 16:41:15 by RouterOS 6.43.2
# software id = 
#
# model = RouterBOARD 1100x4
# serial number = 
/interface gre
add !keepalive local-address=10.0.10.12 name=gre-62EC5B remote-address=10.0.11.12
add !keepalive local-address=10.0.10.1 name=gre-658B9B remote-address=10.0.11.1
add !keepalive local-address=10.0.10.11 name=gre-4911B9 remote-address=10.0.11.11
/interface list
add name=WAN
add name=LAN
add name=GRE-Out
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip ipsec peer profile
set [ find default=yes ] enc-algorithm=aes-256
add enc-algorithm=aes-256 hash-algorithm=sha256 name=profile1
/ip ipsec policy group
add name=FLS
/ip ipsec proposal
set [ find default=yes ] enc-algorithms=aes-256-cbc pfs-group=none
add auth-algorithms=sha256,sha1 enc-algorithms=aes-256-cbc,aes-128-cbc name=proposal1 pfs-group=none
/ip pool
add name=dhcp ranges=192.168.88.100-192.168.88.110
add name=rw-pool ranges=192.168.77.2-192.168.77.5
add name=mode-confid ranges=172.20.20.0/24
/ip dhcp-server
add address-pool=dhcp disabled=no interface=ether2 name=dhcp1
/ip ipsec mode-config
add address-pool=mode-confid name=cfg1 split-include=172.30.30.0/24
/interface list member
add interface=ether1 list=WAN
add interface=ether2 list=LAN
add interface=ether3 list=LAN
add interface=gre-658B9B list=GRE-Out
add interface=gre-62EC5B list=GRE-Out
add interface=gre-4911B9 list=GRE-Out
/ip address
add address=192.168.89.1/24 interface=ether2 network=192.168.89.0
add address=10.0.10.11 interface=gre-4911B9 network=10.0.10.11
add address=10.0.10.12 interface=gre-62EC5B network=10.0.10.12
add address=10.0.10.1 interface=gre-658B9B network=10.0.10.1
add address=210.xxx.xxx.230/30 comment="WAN PtP" interface=ether1 network=210.xxx.xxx.228
/ip dhcp-server network
add address=192.168.88.0/24 gateway=192.168.88.1 netmask=24
/ip dns
set servers=8.8.8.8
/ip firewall filter
add action=accept chain=forward dst-address=192.168.89.0/24 src-address=192.168.88.0/24
add action=accept chain=forward dst-address=192.168.88.0/24 src-address=192.168.89.0/24
/ip firewall mangle
add action=change-mss chain=forward new-mss=clamp-to-pmtu out-interface-list=GRE-Out passthrough=yes protocol=tcp tcp-flags=syn
add action=change-mss chain=forward disabled=yes new-mss=clamp-to-pmtu out-interface=gre-62EC5B passthrough=yes protocol=tcp tcp-flags=syn
add action=change-mss chain=forward disabled=yes new-mss=clamp-to-pmtu passthrough=yes protocol=tcp tcp-flags=syn
/ip firewall nat
add action=accept chain=srcnat dst-address=192.168.88.0/24 src-address=192.168.89.0/24
add action=accept chain=srcnat dst-address=172.20.20.2 src-address=172.20.20.1
add action=src-nat chain=srcnat src-address=10.0.101.0/29 to-addresses=210.xxx.xxx.128
add action=accept chain=srcnat disabled=yes dst-address=10.0.0.0/8 src-address=192.168.89.0/24
add action=dst-nat chain=dstnat dst-address=210.xxx.xxx.129 to-addresses=10.0.111.5
add action=masquerade chain=srcnat
/ip ipsec peer
add address=0.0.0.0/0 auth-method=pre-shared-key-xauth exchange-mode=ike2 generate-policy=port-strict passive=yes policy-template-group=FLS profile=profile1 secret=\
    SECRET_REDACTED send-initial-contact=no
/ip ipsec policy
set 0 disabled=yes
add dst-address=172.20.20.2/32 group=FLS proposal=proposal1 src-address=172.20.20.1/32 template=yes
add dst-address=172.20.20.6/32 group=FLS proposal=proposal1 src-address=172.20.20.5/32 template=yes
add dst-address=172.20.20.0/24 group=FLS proposal=proposal1 src-address=172.30.30.0/24 template=yes
add disabled=yes dst-address=172.20.20.10/32 group=FLS proposal=proposal1 src-address=172.30.30.10/32 template=yes
add dst-address=10.0.11.0/24 group=FLS proposal=proposal1 src-address=10.0.10.0/24 template=yes
/ip ipsec user
add name=658b9b password=REDACTED
add name=62ec5b password=REDACTED
add name=4911b9 password=REDACTED
/ip route
add distance=1 gateway=210.xxx.xxx.229
add distance=1 dst-address=10.0.11.1/32 gateway=gre-658B9B
add distance=1 dst-address=10.0.11.11/32 gateway=gre-4911B9
add distance=1 dst-address=10.0.11.12/32 gateway=gre-62EC5B
add distance=1 dst-address=10.0.101.0/29 gateway=gre-658B9B
add distance=1 dst-address=10.0.111.0/29 gateway=gre-4911B9
add distance=1 dst-address=10.0.112.0/30 gateway=gre-62EC5B
/ip service
set telnet disabled=yes
set ftp disabled=yes
set ssh disabled=yes
set www-ssl disabled=no
set api disabled=yes
set api-ssl disabled=yes
/system clock
set time-zone-name=
/system routerboard settings
set silent-boot=no
/tool bandwidth-server
set authenticate=no

CLIENT

# oct/15/2018 16:46:10 by RouterOS 6.43.2
# software id = 
#
# model = RouterBOARD 750G r3
# serial number = 
/interface bridge
add admin-mac=CC:2D:E0:65:8B:9C auto-mac=no comment=defconf name=bridge
/interface gre
add !keepalive local-address=10.0.11.1 name=gre-658B9B remote-address=10.0.10.1
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip ipsec peer profile
set [ find default=yes ] enc-algorithm=aes-256 hash-algorithm=sha256
/ip ipsec policy group
add name=FLS
/ip ipsec proposal
set [ find default=yes ] auth-algorithms=sha256 enc-algorithms=aes-256-cbc pfs-group=none
add enc-algorithms=aes-256-cbc name=proposal1 pfs-group=none
/ip pool
add name=dhcp ranges=10.0.101.2-10.0.101.6
/ip dhcp-server
add address-pool=dhcp disabled=no interface=bridge name=defconf
/interface bridge port
add bridge=bridge comment=defconf interface=ether2
/ip neighbor discovery-settings
set discover-interface-list=LAN
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1 list=WAN
add interface=gre-658B9B list=LAN
/ip address
add address=10.0.101.1/29 comment=defconf interface=ether2 network=10.0.101.0
add address=10.0.11.1 interface=gre-658B9B network=10.0.11.1
/ip dhcp-client
add comment=defconf dhcp-options=hostname,clientid disabled=no interface=ether1
/ip dhcp-server network
add address=10.0.101.0/29 comment=defconf gateway=10.0.101.1 netmask=30
/ip dns
set allow-remote-requests=yes
/ip dns static
add address=10.0.100.9 name=router.lan
/ip firewall filter
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=drop chain=input disabled=yes in-interface-list=!LAN log=yes
add action=accept chain=forward comment="defconf: accept in ipsec policy" ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" ipsec-policy=out,ipsec
add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf:  drop all from WAN not DSTNATed" connection-nat-state=!dstnat connection-state=new in-interface-list=WAN
/ip firewall mangle
add action=change-mss chain=forward new-mss=clamp-to-pmtu passthrough=yes protocol=tcp src-address=10.0.101.0/29 tcp-flags=syn
add action=mark-routing chain=prerouting new-routing-mark=GRE passthrough=yes src-address=10.0.101.0/29
/ip firewall nat
add action=accept chain=srcnat src-address=10.0.101.0/29
add action=masquerade chain=srcnat comment="defconf: masquerade" ipsec-policy=out,none out-interface-list=WAN
/ip ipsec peer
add address=210.xxx.xxx.230/32 auth-method=pre-shared-key-xauth exchange-mode=ike2 generate-policy=port-strict policy-template-group=FLS secret=REDACTED \
    xauth-login=658b9b xauth-password=REDACTED
/ip ipsec policy
add dst-address=10.0.10.1/32 level=unique sa-dst-address=210.xxx.xxx.230 sa-src-address=0.0.0.0 src-address=10.0.11.1/32 tunnel=yes
/ip route
add distance=1 gateway=gre-658B9B routing-mark=GRE
/system clock
set time-zone-name=
/system identity
set name=CC2DE0658B9B
/system routerboard settings
set silent-boot=no
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN

frontdist · Tue Oct 16, 2018 2:42 am

Disregard the note about remote management...

There was a bit of a NAT routing issue....

I realized I could do it one of two ways, either a static route to the management computer/winbox subnet, or a mangle rule applied to the output chain, with a routing mark that sent it via the GRE tunnel the same way I send other traffic. Both work, equally as well.

I have gone with the static route method for now as I feel like it is less processor intensive, and if I move towards a dynamic routing protocol in the future, I can install those routes dynamically instead of having to make a static entry.

frontdist · Tue Oct 16, 2018 10:27 pm

And some more to the story now, as I was trying a different approach to the GRE tunnels and having an issue with it.

Right now, and in the example config above for my GRE tunnel, I am using non-coincident addresses. In this config it is 10.0.10.1 on the "concentrator" side, and 10.0.11.1 on the "client" side. While this is possible, I would gather that it isn't totally correct. On each device, I assign the respective address, however it is a /32

What I believe would be the correct way to do it would be to actually have a /30 for the GRE link, for example 10.0.10.0/30, with the "concentrator" side being 10.0.10.1/30 and the "client" side being 10.0.10.2/30 however when I tried this configuration, it resulted in the GRE tunnel falling offline and a log message like the following:

"gre-62EC5B transmit loop detected, downing interface for 60 seconds"

The problem with doing it the way I have, is that it required a static route on the server side to reach the "client" device, and the route gateway is an interface instead of an address, and most importantly, to reach the client subnet I am required to put in the interface as a gateway, when in fact I should be able to put in the "next hop IP" which should be 10.0.10.2/30 if it was a proper point to point, but when I do that it becomes unreachable because I believe no next-hop lookup is performed on my static entry for the far side address (currently 10.0.11.1/32) because IT ALSO uses an interface as a gateway instead of an IP address.

Anyone know what I can do to fix this? I believe the gateway being an interface rather than a next hop is going to cause problems for me if I try to do any dynamic routing.

frontdist · Tue Oct 16, 2018 10:46 pm

NEVERMIND....

I have thought about this for 24 hours, and 5 minutes after I posted my comment I figured it all out. Once again, thanks goes to Sindy for the help.

I re-built the tunnels, this time instead of using the GRE interfaces as the IPSEC SA endpoints, I created bridge interfaces with the same IP addresses, created a GRE tunnel between the bridge interface IP's, and then put the respective sides of the /30 GRE Point to Point on each side of the tunnel (as addresses), and now I can make the remote subnets reachable via next hop/gateway address instead of interface.

A comment Sindy made one or two posts into this thread gave me everything I needed to know.

I hope my unabashed cries for help do someone else some good in the same situation.

Cheers!

sindy · Tue Oct 16, 2018 11:10 pm

Too late to comment on the GRE issue, plus I was unable yet to analyse your last-but-three post, but as you haven't explained the mistake in the GRE configuration, I'll explain it here for that "someone else in same situation".

In interface gre, the local-address and remote-address are addresses of devices between which the tunnel is established, and these were correct in that post, 10.0.10.11 and 10.0.11.11.

But the /ip address attached to the gre interface was wrong - it had the same value of network as of address, which is wrong:
add address=10.0.10.1 interface=gre-658B9B network=10.0.10.1

Normally, for /30 and shorter masks, the network value is address && mask.expanded.to.bits, but where address has a /32 mask, network is another address which, if used as a gateway of ip route, is used to identify the interface to which it is attached via the /ip address configuration element.

I suspect that you have changed the local-address and remote-address of /interface gre rather than address and network of the /ip address attached to it.

One formatting related one: the [code] tag needs a blank line in front and/or after if it behaves that weird way. So edit your post one more time, insert a blank line before the second [code], and it will start looking normal.

More than that, I was making use of this weird behaviour to highlight individual codewords in normal text, but had to stop that as other users have informed me that with non-default skin the [code] tag works always properly so the result was unreadable, like below:

... rather than

address

and

network

of the

/ip address

attached to it.

frontdist · Fri Oct 26, 2018 4:39 pm

Back at it again and I'm still having problems with multiple clients for some reason.

There doesn't seem to be any rhyme or reason as to why it works sometimes and not others. I have 5 Hex units, configured identically (except for IP parameters and IPSec user credentials), and sometimes they work and sometimes not.

Right now, I have 5 units that have attempted to connect from behind the same NAT, units 1, 2, and 4 connected (with "source" ports 4500, 1024, 1026) but unit 3 and 5 (source port 1025, 1027), has connected but I am getting a lack of traffic across the SA. Unit 3 has no traffic in either direction on the counters, and unit 5 is not receiving any traffic (concentrator not sending traffic according to SA counter).

I was under the impression that using IKEv2 and XAUTH between mikrotik units would allow multiple clients behind the same NAT IP, but something is going on here and causing difficulty...

Any thoughts? Configs to come.

sindy · Fri Oct 26, 2018 7:17 pm

First, does /ip ipsec installed-sa print show ip.add.re.ss:port as src-address and dst-address of all SAs or only ip.add.re.ss without port for some of them? From what you wrote it is not clear whether you've sourced the information about ports from /ip ipsec remote-peers print or from /ip ipsec installed-sa print.

Second, what are the RouterOS versions? Until 6.43.2, there was an issue that from time to time the SA keys didn't match between IKEv2 peers if pfs-group other than none was used in policy (phase2) proposal.

frontdist · Fri Oct 26, 2018 8:05 pm

PFS is set to none on every device... all devices on 6.43.4

One of the weird things that tends to happen when I am setting this up is something I see in my routes...

I have two routes I add to the client side:

0.0.0.0/0 gateway (concentrator side of GRE interface) with routing mark GRE and Distance 10
10.0.0.0/24 with the same gateway as above, no routing mark

The main 0.0.0.0/0 is a dynamic route added via DHCP when the WAN port is plugged into the upstream network.

On configurations where this isn't working properly, I get both routes (0/0 and 10.0.0.0/24) unreachable, however when I disable 0.0.0.0/0 the 10.0.0.0/24 becomes reachable. If I re-enable 0.0.0.0/0 (the one over GRE) then it flaps a few times and goes back to unreachable.

sindy · Fri Oct 26, 2018 8:14 pm

I need to see the configuration from the client and the server to be able to say more.

frontdist · Fri Oct 26, 2018 8:17 pm

Give me 5 minutes and I'll post server, working client and non-working client.

frontdist · Fri Oct 26, 2018 8:27 pm

Here are the configs...

Server

# oct/26/2018 10:21:46 by RouterOS 6.43.4
# software id = 
#
# model = RouterBOARD 1100x4
# serial number =
/interface bridge
add fast-forward=no name=bri-5E5DCB
add fast-forward=no name=bri-62EC5B
add fast-forward=no name=bri-658B9B
add fast-forward=no name=bri-658C27
add fast-forward=no name=bri-4911B9
/interface gre
add !keepalive local-address=10.0.10.13 name=gre-5E5DCB remote-address=10.0.11.13
add !keepalive local-address=10.0.10.12 name=gre-62EC5B remote-address=10.0.11.12
add !keepalive local-address=10.0.10.1 name=gre-658B9B remote-address=10.0.11.1
add !keepalive local-address=10.0.10.14 name=gre-658C27 remote-address=10.0.11.14
add !keepalive local-address=10.0.10.11 name=gre-4911B9 remote-address=10.0.11.11
/interface list
add name=WAN
add name=LAN
add name=GRE-Out
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip ipsec peer profile
set [ find default=yes ] enc-algorithm=aes-256
add enc-algorithm=aes-256,aes-128 hash-algorithm=sha256 name=profile1
/ip ipsec policy group
add name=FLS
/ip ipsec proposal
set [ find default=yes ] enc-algorithms=aes-256-cbc pfs-group=none
add auth-algorithms=sha256,sha1 enc-algorithms=aes-256-cbc,aes-128-cbc name=proposal1 pfs-group=none
/ip pool
add name=dhcp ranges=10.0.0.2-10.0.0.9
add name=rw-pool ranges=192.168.77.2-192.168.77.5
add name=mode-confid ranges=172.20.20.0/24
/ip dhcp-server
add address-pool=dhcp disabled=no interface=ether2 name=dhcp1
/routing bgp instance
set default as=65000 disabled=yes redistribute-other-bgp=yes router-id=10.255.255.0
/ip firewall connection tracking
set enabled=yes
/ip neighbor discovery-settings
set discover-interface-list=GRE-Out
/interface list member
add interface=ether1 list=WAN
add interface=ether2 list=LAN
add interface=ether3 list=LAN
add interface=gre-658B9B list=GRE-Out
add interface=gre-62EC5B list=GRE-Out
add interface=gre-4911B9 list=GRE-Out
add interface=gre-5E5DCB list=GRE-Out
add interface=gre-658C27 list=GRE-Out
/ip address
add address=10.0.0.1/24 comment="Management LAN" interface=ether2 network=10.0.0.0
add address=10.10.111.1/30 interface=gre-4911B9 network=10.10.111.0
add address=10.0.10.12 interface=bri-62EC5B network=10.0.10.12
add address=10.0.10.1 interface=bri-658B9B network=10.0.10.1
add address=210.xxx.xxx.230/30 comment="WAN PtP" interface=ether1 network=210.xxx.xxx.228
add address=10.10.101.1/30 interface=gre-658B9B network=10.10.101.0
add address=10.10.112.1/30 interface=gre-62EC5B network=10.10.112.0
add address=10.10.113.1/30 interface=gre-5E5DCB network=10.10.113.0
add address=10.0.10.13 interface=bri-5E5DCB network=10.0.10.13
add address=10.0.10.11 interface=bri-4911B9 network=10.0.10.11
add address=10.0.10.14 interface=bri-658C27 network=10.0.10.14
add address=10.10.114.1/30 interface=gre-658C27 network=10.10.114.0
/ip dhcp-server network
add address=10.0.0.0/28 dns-server=1.1.1.1,1.0.0.1 gateway=10.0.0.1 netmask=24
/ip dns
set servers=1.1.1.1,1.0.0.1
/ip firewall address-list
add address=10.0.0.0/24 list=admin_allow_in
add address=210.xxx.xxx.104 list=admin_allow_in
/ip firewall filter
add action=accept chain=input connection-state=established,related
add action=accept chain=input src-address-list=admin_allow_in
add action=accept chain=input protocol=icmp
add action=accept chain=input dst-port=4500 protocol=udp
add action=accept chain=input protocol=ipsec-esp
add action=accept chain=input protocol=ipsec-ah
add action=accept chain=input protocol=gre
add action=accept chain=forward disabled=yes dst-address=10.0.11.0/24 src-address=10.0.0.0/24
add action=drop chain=input dst-port=53 in-interface-list=WAN protocol=tcp
add action=drop chain=input dst-port=53 in-interface-list=WAN protocol=udp
add action=drop chain=input log=yes log-prefix=IPSEC
add action=accept chain=forward disabled=yes dst-address=192.168.88.0/24 src-address=192.168.89.0/24
/ip firewall mangle
add action=change-mss chain=forward new-mss=clamp-to-pmtu out-interface-list=GRE-Out passthrough=yes protocol=tcp tcp-flags=syn
add action=change-mss chain=forward disabled=yes new-mss=clamp-to-pmtu out-interface=gre-62EC5B passthrough=yes protocol=tcp tcp-flags=syn
add action=change-mss chain=forward disabled=yes new-mss=clamp-to-pmtu passthrough=yes protocol=tcp tcp-flags=syn
/ip firewall nat
add action=accept chain=srcnat disabled=yes dst-address=192.168.88.0/24 src-address=192.168.89.0/24
add action=accept chain=srcnat disabled=yes dst-address=172.20.20.2 src-address=172.20.20.1
add action=accept chain=srcnat disabled=yes dst-address=10.0.11.0/24 src-address=10.0.0.0/24
add action=src-nat chain=srcnat src-address=10.0.101.0/29 to-addresses=210.xxx.xxx.128
add action=src-nat chain=srcnat src-address=10.100.111.0/24 to-addresses=210.xxx.xxx.130
add action=dst-nat chain=dstnat dst-address=210.xxx.xxx.129 to-addresses=10.0.111.5
add action=masquerade chain=srcnat out-interface-list=WAN
add action=dst-nat chain=dstnat dst-address=210.xxx.xxx.131 dst-port=8000 protocol=tcp to-addresses=10.0.111.5 to-ports=8180
/ip ipsec peer
add address=0.0.0.0/0 auth-method=pre-shared-key-xauth exchange-mode=ike2 generate-policy=port-override local-address=210.xxx.xxx.230 passive=yes \
    policy-template-group=FLS profile=profile1 send-initial-contact=no
/ip ipsec policy
set 0 disabled=yes
add disabled=yes dst-address=172.20.20.2/32 group=FLS proposal=proposal1 src-address=172.20.20.1/32 template=yes
add disabled=yes dst-address=172.20.20.6/32 group=FLS proposal=proposal1 src-address=172.20.20.5/32 template=yes
add disabled=yes dst-address=172.20.20.0/24 group=FLS proposal=proposal1 src-address=172.30.30.0/24 template=yes
add disabled=yes dst-address=172.20.20.10/32 group=FLS proposal=proposal1 src-address=172.30.30.10/32 template=yes
add dst-address=10.0.11.0/24 group=FLS proposal=proposal1 src-address=10.0.10.0/24 template=yes
/ip ipsec user
add name=658b9b
add name=62ec5b
add name=4911b9
add name=5e5dcb
add name=658c27
/ip route
add distance=1 gateway=210.xxx.xxx.229
add distance=1 dst-address=10.100.101.0/29 gateway=10.10.101.2
add distance=1 dst-address=10.100.111.0/30 gateway=10.10.111.2
add distance=1 dst-address=10.100.112.0/30 gateway=10.10.112.2
add distance=1 dst-address=10.100.113.0/30 gateway=10.10.113.2
add distance=1 dst-address=10.100.114.0/30 gateway=10.10.114.2
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www disabled=yes
set ssh disabled=yes port=2200
set www-ssl certificate=server1 disabled=no
set api disabled=yes
set api-ssl disabled=yes
/routing bgp network
add disabled=yes network=10.0.0.0/24 synchronize=no
/routing bgp peer
add disabled=yes name=62EC5B remote-address=10.10.112.2 remote-as=65000 ttl=default
add disabled=yes name=658B9B remote-address=10.10.101.2 route-reflect=yes ttl=default
add disabled=yes name=4911B9 remote-address=10.10.111.2 remote-as=65000 ttl=default
add disabled=yes name=5E5DCB remote-address=10.10.113.2 remote-as=65000 ttl=default
add disabled=yes name=658C27 remote-address=10.10.114.2 remote-as=65000 ttl=default
/system clock
set time-zone-name=America/Los_Angeles
/system routerboard settings
set silent-boot=no
/tool bandwidth-server
set authenticate=no enabled=no
[admin@MikroTik] >

Working Client:

# oct/26/2018 13:20:05 by RouterOS 6.43.4
# software id =
#
# model = RouterBOARD 750G r3
# serial number =
/interface bridge
add fast-forward=no name=bri-4911B9
add admin-mac=64:D1:54:49:11:BA auto-mac=no comment=defconf name=bridge
/interface ethernet
set [ find default-name=ether3 ] disabled=yes
set [ find default-name=ether4 ] disabled=yes
set [ find default-name=ether5 ] disabled=yes
/interface gre
add !keepalive local-address=10.0.11.11 name=gre-4911B9 remote-address=10.0.10.11
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip hotspot profile
set [ find default=yes ] html-directory=flash/hotspot
/ip ipsec peer profile
set [ find default=yes ] enc-algorithm=aes-128 hash-algorithm=sha256
/ip ipsec proposal
set [ find default=yes ] auth-algorithms=sha256 enc-algorithms=aes-128-cbc pfs-group=none
/ip pool
add name=dhcp ranges=10.100.111.2
/ip dhcp-server
add address-pool=dhcp disabled=no interface=bridge name=defconf
/queue simple
add disabled=yes max-limit=5M/5M name=queue1 target=10.100.111.0/24
/routing bgp instance
set default as=65000 disabled=yes router-id=10.255.255.11
/interface bridge port
add bridge=bridge comment=defconf interface=ether2
add bridge=bridge comment=defconf interface=ether3
add bridge=bridge comment=defconf interface=ether4
add bridge=bridge comment=defconf interface=ether5
/ip neighbor discovery-settings
set discover-interface-list=LAN
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1 list=WAN
add interface=gre-4911B9 list=LAN
/ip address
add address=10.100.111.1/30 comment=defconf interface=ether2 network=10.100.111.0
add address=10.0.11.11 interface=bri-4911B9 network=10.0.11.11
add address=10.10.111.2/30 interface=gre-4911B9 network=10.10.111.0
/ip dhcp-client
add comment=defconf dhcp-options=hostname,clientid disabled=no interface=ether1
/ip dhcp-server network
add address=10.100.111.0/30 comment=defconf gateway=10.100.111.1 netmask=30
/ip dns
set allow-remote-requests=yes servers=1.1.1.1,1.0.0.1
/ip dns static
add address=10.100.111.1 name=router.lan
/ip firewall filter
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=drop chain=input comment="defconf: drop all not coming from LAN" in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" ipsec-policy=out,ipsec
add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf:  drop all from WAN not DSTNATed" connection-nat-state=!dstnat connection-state=new in-interface-list=WAN
/ip firewall mangle
add action=mark-routing chain=prerouting new-routing-mark=GRE passthrough=yes src-address=10.100.111.0/30
add action=change-mss chain=forward new-mss=clamp-to-pmtu out-interface=gre-4911B9 passthrough=yes protocol=tcp src-address=10.100.111.0/30 tcp-flags=syn
/ip firewall nat
add action=accept chain=srcnat comment="VPN NAT Bypass" out-interface=gre-4911B9 src-address=10.100.111.0/30
add action=masquerade chain=srcnat comment="defconf: masquerade" ipsec-policy=out,none out-interface-list=WAN
/ip ipsec peer
add address=210.xxx.xxx.230/32 auth-method=pre-shared-key-xauth exchange-mode=ike2 generate-policy=port-override xauth-login=4911b9
/ip ipsec policy
set 0 disabled=yes
add dst-address=10.0.10.11/32 level=unique sa-dst-address=210.xxx.xxx.230 sa-src-address=0.0.0.0 src-address=10.0.11.11/32 tunnel=yes
/ip route
add check-gateway=ping distance=10 gateway=10.10.111.1 routing-mark=GRE
add distance=1 dst-address=10.0.0.0/24 gateway=10.10.111.1
/ip service
set telnet disabled=yes
set ftp disabled=yes
set ssh disabled=yes
set api disabled=yes
set api-ssl disabled=yes
/routing bgp network
add network=10.100.111.0/30 synchronize=no
/routing bgp peer
add disabled=yes name=peer1 remote-address=10.10.111.1 remote-as=65000 ttl=default
/system clock
set time-zone-name=America/Toronto
/system identity
set name=FLS_01
/system routerboard settings
set silent-boot=no
/tool bandwidth-server
set enabled=no
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN

Non-Working Client:

# oct/26/2018 13:17:41 by RouterOS 6.43.4
# software id =
#
# model = RouterBOARD 750G r3
# serial number =
/interface bridge
add fast-forward=no name=bri-658B9B
add admin-mac=CC:2D:E0:65:8B:9C auto-mac=no comment=defconf name=bridge
/interface ethernet
set [ find default-name=ether3 ] disabled=yes
set [ find default-name=ether4 ] disabled=yes
set [ find default-name=ether5 ] disabled=yes
/interface gre
add !keepalive local-address=10.0.11.1 name=gre-658B9B remote-address=10.0.10.1
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip hotspot profile
set [ find default=yes ] html-directory=flash/hotspot
/ip ipsec peer profile
set [ find default=yes ] enc-algorithm=aes-128 hash-algorithm=sha256
/ip ipsec proposal
set [ find default=yes ] auth-algorithms=sha256 enc-algorithms=aes-128-cbc pfs-group=none
/ip pool
add name=dhcp ranges=10.100.101.2
/ip dhcp-server
add address-pool=dhcp disabled=no interface=bridge name=defconf
/interface bridge port
add bridge=bridge comment=defconf interface=ether2
add bridge=bridge comment=defconf interface=ether3
add bridge=bridge comment=defconf interface=ether4
add bridge=bridge comment=defconf interface=ether5
/ip neighbor discovery-settings
set discover-interface-list=LAN
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1 list=WAN
add interface=gre-658B9B list=LAN
/ip address
add address=10.100.101.1/29 comment=defconf interface=ether2 network=10.100.101.0
add address=10.0.11.1 interface=bri-658B9B network=10.0.11.1
add address=10.10.101.2/30 interface=gre-658B9B network=10.10.101.0
/ip dhcp-client
add comment=defconf dhcp-options=hostname,clientid disabled=no interface=ether1
/ip dhcp-server network
add address=10.100.101.0/29 comment=defconf gateway=10.100.101.1 netmask=29
/ip dns
set allow-remote-requests=yes
/ip dns static
add address=10.100.101.1 name=router.lan
/ip firewall filter
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=drop chain=input comment="defconf: drop all not coming from LAN" in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" ipsec-policy=out,ipsec
add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf:  drop all from WAN not DSTNATed" connection-nat-state=!dstnat connection-state=new in-interface-list=WAN
/ip firewall mangle
add action=mark-routing chain=prerouting new-routing-mark=GRE passthrough=yes src-address=10.100.101.0/29
add action=change-mss chain=forward new-mss=clamp-to-pmtu out-interface=gre-658B9B passthrough=yes protocol=tcp src-address=10.100.101.0/29 tcp-flags=syn
/ip firewall nat
add action=accept chain=srcnat out-interface=gre-658B9B src-address=10.100.101.0/29
add action=masquerade chain=srcnat comment="defconf: masquerade" ipsec-policy=out,none out-interface-list=WAN
/ip ipsec peer
add address=210.xxx.xxx.230/32 auth-method=pre-shared-key-xauth exchange-mode=ike2 generate-policy=port-override xauth-login=658b9b
/ip ipsec policy
set 0 disabled=yes
add dst-address=10.0.10.1/32 level=unique sa-dst-address=210.xxx.xxx.230 sa-src-address=0.0.0.0 src-address=10.0.11.1/32 tunnel=yes
/ip route
add check-gateway=ping distance=10 gateway=10.10.101.1 routing-mark=GRE
add distance=1 dst-address=10.0.0.0/24 gateway=10.10.101.1
/ip service
set telnet disabled=yes
set ftp disabled=yes
set ssh disabled=yes
set api disabled=yes
set api-ssl disabled=yes
/system clock
set time-zone-name=America/Toronto
/system identity
set name=CTL_01
/system routerboard settings
set silent-boot=no
/tool bandwidth-server
set enabled=no
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN

sindy · Fri Oct 26, 2018 9:05 pm

The only mistake I can see is that you have generate-policy=port-override on the intiator (client) peers where you at the same time have static policies (which are necessary). But as it works in some cases, it doesn't seem to be relevant.

I would attribute the flapping routes simply to the fact that the GRE tunnel is down on the non-working peers. When you disable the default route with routing-mark, which has check-gateway set to ping, the route to 10.0.0.0/24 which doesn't have check-gateway set probably recovers because the information about gateway unavailablility stops arriving (the default routing table is used to send the pings checking the gateway event though the route using that gateway has a routing-mark).

So post the /ip ipsec policy print and /ip ipsec installed-sa print from the server and the non-working client (replace the public IPs, the enc and auth keys are ephemeral so in 30 minutes they become useless for a hacker).

frontdist · Fri Oct 26, 2018 9:13 pm

Server Side:

The 50. IP address is real, but dynamic so I don't care...

[admin@MikroTik] > ip ipsec policy print 
Flags: T - template, X - disabled, D - dynamic, I - invalid, A - active, * - default 
 0 TX* group=default src-address=::/0 dst-address=::/0 protocol=all proposal=default template=yes 

 1 TX  group=FLS src-address=172.20.20.1/32 dst-address=172.20.20.2/32 protocol=all proposal=proposal1 template=yes 

 2 TX  group=FLS src-address=172.20.20.5/32 dst-address=172.20.20.6/32 protocol=all proposal=proposal1 template=yes 

 3 TX  group=FLS src-address=172.30.30.0/24 dst-address=172.20.20.0/24 protocol=all proposal=proposal1 template=yes 

 4 TX  group=FLS src-address=172.30.30.10/32 dst-address=172.20.20.10/32 protocol=all proposal=proposal1 template=yes 

 5 T   group=FLS src-address=10.0.10.0/24 dst-address=10.0.11.0/24 protocol=all proposal=proposal1 template=yes 

 6  DA  src-address=10.0.10.11/32 src-port=any dst-address=10.0.11.11/32 dst-port=any protocol=all action=encrypt level=unique ipsec-protocols=esp tunnel=yes 
       sa-src-address=210.xxx.xxx.230 sa-dst-address=50.100.53.111 proposal=proposal1 ph2-count=1 

 7  DA  src-address=10.0.10.12/32 src-port=any dst-address=10.0.11.12/32 dst-port=any protocol=all action=encrypt level=unique ipsec-protocols=esp tunnel=yes 
       sa-src-address=210.xxx.xxx.230 sa-dst-address=50.100.53.111 proposal=proposal1 ph2-count=1 

 8  DA  src-address=10.0.10.13/32 src-port=any dst-address=10.0.11.13/32 dst-port=any protocol=all action=encrypt level=unique ipsec-protocols=esp tunnel=yes 
       sa-src-address=210.xxx.xxx.230 sa-dst-address=50.100.53.111 proposal=proposal1 ph2-count=1 

 9  DA  src-address=10.0.10.14/32 src-port=any dst-address=10.0.11.14/32 dst-port=any protocol=all action=encrypt level=unique ipsec-protocols=esp tunnel=yes 
       sa-src-address=210.xxx.xxx.230 sa-dst-address=50.100.53.111 proposal=proposal1 ph2-count=1 

10  DA  src-address=10.0.10.1/32 src-port=any dst-address=10.0.11.1/32 dst-port=any protocol=all action=encrypt level=unique ipsec-protocols=esp tunnel=yes 
       sa-src-address=210.xxx.xxx.230 sa-dst-address=50.100.53.111 proposal=proposal1 ph2-count=1 
[admin@MikroTik] > ip ipsec installed-sa print 
Flags: H - hw-aead, A - AH, E - ESP 
 0 HE spi=0xC4C5D3C src-address=50.100.53.111:1027 dst-address=210.xxx.xxx.230:4500 state=mature auth-algorithm=sha256 enc-algorithm=aes-cbc enc-key-size=128 
      auth-key="1c37b709a3c4c5bdacabaab97dabc322010c392fc32235a4f3baa738112cf94f" enc-key="29e4249d45ddb533794cc0b749124d0d" addtime=oct/26/2018 10:57:49 expires-in=17m38s 
      add-lifetime=24m14s/30m18s current-bytes=7868 current-packets=88 replay=128 

 1 HE spi=0xF53E093 src-address=210.xxx.xxx.230:4500 dst-address=50.100.53.111:1027 state=mature auth-algorithm=sha256 enc-algorithm=aes-cbc enc-key-size=128 
      auth-key="f091236fbd08745385a5764e40e83f470e4117380a6d5bc686695abe95069703" enc-key="4437f221764da4583121eaebd6e6b49a" add-lifetime=24m14s/30m18s replay=128 

 2 HE spi=0x88651BB src-address=50.100.53.111:1025 dst-address=210.xxx.xxx.230:4500 state=mature auth-algorithm=sha256 enc-algorithm=aes-cbc enc-key-size=128 
      auth-key="781250cfe14e46431cf06a7cb1215d298ff06b4392122f613c7150c86859ea43" enc-key="7893908e6cf6d30bb88e7d7c7b5d2a2d" addtime=oct/26/2018 10:58:40 expires-in=18m18s 
      add-lifetime=24m5s/30m7s current-bytes=7468 current-packets=83 replay=128 

 3 HE spi=0x3B5739E src-address=210.xxx.xxx.230:4500 dst-address=50.100.53.111:1025 state=mature auth-algorithm=sha256 enc-algorithm=aes-cbc enc-key-size=128 
      auth-key="2a70f266cb2411766d20c78dddc31c8a10e848ef92ec718a201b491f8bf4677c" enc-key="bdc8ca10894bea48ecd13246c88bbc34" addtime=oct/26/2018 10:58:40 expires-in=18m18s 
      add-lifetime=24m5s/30m7s current-bytes=7492 current-packets=83 replay=128 

 4 HE spi=0x7A0DFE2 src-address=50.100.53.111:4500 dst-address=210.xxx.xxx.230:4500 state=mature auth-algorithm=sha256 enc-algorithm=aes-cbc enc-key-size=128 
      auth-key="50bfea5c5581a09389757e12750d663da1ec44496a8695f16c3028ffd39ca6ce" enc-key="c38231b37d534fac242d705670eba65c" addtime=oct/26/2018 10:58:43 expires-in=18m37s 
      add-lifetime=24m18s/30m23s current-bytes=345394 current-packets=364 replay=128 

 5 HE spi=0xCD64E82 src-address=50.100.53.111:1026 dst-address=210.xxx.xxx.230:4500 state=mature auth-algorithm=sha256 enc-algorithm=aes-cbc enc-key-size=128 
      auth-key="0fceca304d27ddd478fe6adb82f22d59af36b9b089b59e488e77b37896d33de8" enc-key="0788d7ef4e9c4c0eac7b98fbbc3f4455" addtime=oct/26/2018 10:58:46 expires-in=18m44s 
      add-lifetime=24m21s/30m27s current-bytes=1788 current-packets=12 replay=128 

 6 HE spi=0xF50BA7 src-address=210.xxx.xxx.230:4500 dst-address=50.100.53.111:4500 state=mature auth-algorithm=sha256 enc-algorithm=aes-cbc enc-key-size=128 
      auth-key="9c8dd11c24bb2ea5fca208006539a1f62ee65e827192079380cb295a65f239ef" enc-key="bd5b67aac9b95bca7feb4e1b1cc8c72b" addtime=oct/26/2018 10:58:48 expires-in=18m42s 
      add-lifetime=24m18s/30m23s current-bytes=154395 current-packets=593 replay=128 

 7 HE spi=0xC878F2F src-address=210.xxx.xxx.230:4500 dst-address=50.100.53.111:1026 state=mature auth-algorithm=sha256 enc-algorithm=aes-cbc enc-key-size=128 
      auth-key="034ed54a34932892542a14e0d43b6caa3221f66651637936bb7bfff0b08a5a53" enc-key="adf1ac6837dc54ab21ca9c32d12716d7" addtime=oct/26/2018 10:58:51 expires-in=18m49s 
      add-lifetime=24m21s/30m27s current-bytes=1812 current-packets=12 replay=128 

 8 HE spi=0x18EECE6 src-address=50.100.53.111:1024 dst-address=210.xxx.xxx.230:4500 state=mature auth-algorithm=sha256 enc-algorithm=aes-cbc enc-key-size=128 
      auth-key="1c9477171e95ab90c19b442d53cd7e6351eef50c76e9dfc0d9cebac0646bfeb8" enc-key="af7c121045c6dc150ee1e52c04b15568" addtime=oct/26/2018 10:59:10 expires-in=19m 
      add-lifetime=24m15s/30m19s current-bytes=7079 current-packets=79 replay=128 

 9 HE spi=0x4BB6F13 src-address=210.xxx.xxx.230:4500 dst-address=50.100.53.111:1024 state=mature auth-algorithm=sha256 enc-algorithm=aes-cbc enc-key-size=128 
      auth-key="385229a0cd747fa08bfb6e7153bdb0d966613b90e432b512011b65c0c4aa32e5" enc-key="2b517e1cd039569faa042e91392964c0" addtime=oct/26/2018 10:59:10 expires-in=19m 
      add-lifetime=24m15s/30m19s current-bytes=7252 current-packets=80 replay=128 

[admin@MikroTik] >

Client:

[admin@CTL_01] > ip ipsec policy print 
Flags: T - template, X - disabled, D - dynamic, I - invalid, A - active, * - default 
 0 TX* group=default src-address=::/0 dst-address=::/0 protocol=all proposal=default template=yes 

 1  A  src-address=10.0.11.1/32 src-port=any dst-address=10.0.10.1/32 dst-port=any protocol=all action=encrypt level=unique ipsec-protocols=esp tunnel=yes 
       sa-src-address=0.0.0.0 sa-dst-address=210.xxx.xxx.230 proposal=default ph2-count=1 
[admin@CTL_01] > ip ipsec installed-sa print 
Flags: H - hw-aead, A - AH, E - ESP 
 0 HE spi=0xF53E093 src-address=210.xxx.xxx.230:4500 dst-address=192.168.1.108:4500 state=mature auth-algorithm=sha256 enc-algorithm=aes-cbc enc-key-size=128 
      auth-key="f091236fbd08745385a5764e40e83f470e4117380a6d5bc686695abe95069703" enc-key="4437f221764da4583121eaebd6e6b49a" add-lifetime=24m2s/30m3s 
      replay=128 

 1 HE spi=0xC4C5D3C src-address=192.168.1.108:4500 dst-address=210.xxx.xxx.230:4500 state=mature auth-algorithm=sha256 enc-algorithm=aes-cbc enc-key-size=128 
      auth-key="1c37b709a3c4c5bdacabaab97dabc322010c392fc32235a4f3baa738112cf94f" enc-key="29e4249d45ddb533794cc0b749124d0d" addtime=oct/26/2018 13:57:49 
      expires-in=13m17s add-lifetime=24m2s/30m3s current-bytes=10464 current-packets=117 replay=128 
[admin@CTL_01] >

Right now, I have 4 units that are connected and working (all same config as the working config I posted above), and this one is NOT working.... All are from behind the same public IP and NAT.

You were spot-on about the ping check on only one of the static routes... with a ping check on the 10.0.0.0/24 it behaves in the same way.

What I was looking for was a way to send traffic out the WAN 0.0.0.0/0 if the GRE route was unavailable (i.e. the VPN server was offline, etc.), but it seems that the GRE tunnel stays up regardless of the tunnel actually working or not, so I am going to need another solution for this problem as well, unless it will actually work the way I am thinking it is supposed to.

sindy · Fri Oct 26, 2018 9:56 pm

Yeah, got it.

Before the IPsec policy comes into play, the GRE transport packets are routed via the default gateway, which sends them out the WAN interface, and there is the masquerade rule which src-nats them. And only after this has been done, the ipsec policy's traffic selectors look for their prey, and they ignore the src-nated packets.

But the /ip firewall nat rules are only inspected for the very first packet of each connection. So if the first packet of a GRE connection comes from the client, no NAT handling takes place for the remaining lifetime of the connection, so the packets from server to client have their normal source address (the local-address of the GRE tunnel) and the respective policy catches them. If the first ever packet goes from the server, it gets NATed and the policy ignores it, and as GRE keeps sending, the connection tracker timer for that connection doesn't expire between the attempts.

To check that, do /ip firewall connection print detail where protocol=gre and srcnat. To fix it, add an action=accept src-address=10.0.10.0/28 dst-address=10.0.11.0/28 rule to the top of chain=srcnat, or maybe adding ipsec-policy=out,none to the existing action=masquerade rule is sufficient (it seems to check whether a packet will match any policy later but the description isn't really clear).

However, after you do one of the above, you have to remove the existing connection from the tracker:
/ip firewall connection remove [find protocol=gre and srcnat]

frontdist · Fri Oct 26, 2018 10:16 pm

I added that src-nat rule, cleared the connection tracker and rebooted the router. Still no luck.

The masquerade rule already had the ipsec-policy=out,none attached to it.

What is so frustrating is that sometimes it works, sometimes it doesn't - and I can power down every single unit, reboot the server, and plug in the non-working unit only and it STILL won't connect, almost like there is some sort of latent memory of it not working the last time.

I also added the appropriate src-nat rule for the opposite direction on the server side and that didn't work either...

frontdist · Fri Oct 26, 2018 10:25 pm

I configured a 6th unit while we were going back and forth with this.

The 6th unit could NOT receive traffic back (SA counter 0) despite establishing phase2, when it was behind the same NAT as the other 4 working units, however when I gave it a unique public IP it COULD connect and pass traffic without issue. This was a direct public IP on the WAN interface (pppoe).

I took the non-working unit in question, and also tried to connect it directly to the internet (WAN port, pppoe), however it still did NOT work.

They are configured the same (the above two units), and I don't feel like I had this many issues on firmware 6.43.2 - is it possible that there is some sort of bug? I can't understand this behaviour where sometimes it works, sometimes it doesn't and it goes from one to the other without any config changes. It's frustrating.

EDIT: I just plugged the 6th unit back into the NAT network (with the other working 4), and now all of a sudden it works fine behind NAT. Still no joy with unit #5 (the one with the non-working config posted above).

Of note, the unit #6 doesn't have any of the src-nat rules for the GRE anchor IP addresses/subnets.

sindy · Fri Oct 26, 2018 10:59 pm

Show me /ip firewall connection print detail where protocol=gre from the server side (Ctrl-F for the public address before posting).

frontdist · Fri Oct 26, 2018 11:04 pm

Tried to downgrade to 6.43.2 and it refuses to work. Was hoping to try that as a solution, but I can't get a downgrade to work at all...

frontdist · Fri Oct 26, 2018 11:19 pm

Show me /ip firewall connection print detail where protocol=gre from the server side (Ctrl-F for the public address before posting).

Output:

[admin@MikroTik] > ip firewall connection print detail where protocol=gre 
Flags: E - expected, S - seen-reply, A - assured, C - confirmed, D - dying, F - fasttrack, s - srcnat, d - dstnat 
 0  SAC     protocol=gre src-address=10.0.11.14 dst-address=10.0.10.14 reply-src-address=10.0.10.14 reply-dst-address=10.0.11.14 gre-key=0 timeout=2m51s orig-packets=104 
            orig-bytes=9 355 orig-fasttrack-packets=0 orig-fasttrack-bytes=0 repl-packets=104 repl-bytes=9 385 repl-fasttrack-packets=0 repl-fasttrack-bytes=0 orig-rate=0bps 
            repl-rate=0bps 

 1  SAC     protocol=gre src-address=10.0.11.51 dst-address=10.0.10.51 reply-src-address=10.0.10.51 reply-dst-address=10.0.11.51 gre-key=0 timeout=2m54s orig-packets=103 
            orig-bytes=9 275 orig-fasttrack-packets=0 orig-fasttrack-bytes=0 repl-packets=88 repl-bytes=7 040 repl-fasttrack-packets=0 repl-fasttrack-bytes=0 orig-rate=0bps 
            repl-rate=0bps 

 2    C  s  protocol=gre src-address=10.0.10.1 dst-address=10.0.11.1 reply-src-address=10.0.11.1 reply-dst-address=210.xxx.xxx.230 gre-key=0 timeout=28s orig-packets=629 
            orig-bytes=53 908 orig-fasttrack-packets=0 orig-fasttrack-bytes=0 repl-packets=0 repl-bytes=0 repl-fasttrack-packets=0 repl-fasttrack-bytes=0 orig-rate=672bps 
            repl-rate=0bps 

 3  SAC     protocol=gre src-address=10.0.11.12 dst-address=10.0.10.12 reply-src-address=10.0.10.12 reply-dst-address=10.0.11.12 gre-key=0 timeout=2m52s orig-packets=104 
            orig-bytes=9 355 orig-fasttrack-packets=0 orig-fasttrack-bytes=0 repl-packets=104 repl-bytes=9 385 repl-fasttrack-packets=0 repl-fasttrack-bytes=0 orig-rate=0bps 
            repl-rate=0bps 

 4  SAC     protocol=gre src-address=10.0.11.11 dst-address=10.0.10.11 reply-src-address=10.0.10.11 reply-dst-address=10.0.11.11 gre-key=0 timeout=2m55s orig-packets=104 
            orig-bytes=9 355 orig-fasttrack-packets=0 orig-fasttrack-bytes=0 repl-packets=104 repl-bytes=9 385 repl-fasttrack-packets=0 repl-fasttrack-bytes=0 orig-rate=0bps 
            repl-rate=0bps

In the last few minutes, one that was online FLS_03 has gone offline and has a SA counter at 0 for server -> client traffic. No config changes were made.

sindy · Fri Oct 26, 2018 11:25 pm

Look at connection No. 2 in the list. It says reply-dst-address=210.xx.xx.230 (edit your post to obfuscate the leaked address) and it has the s mark (src-nat) which all the others don't.

So your "don't src-nat me!" rules on the server haven't prevented this from happening, so check them again (or post just /ip firewall nat export if you can't spot the issue).

frontdist · Fri Oct 26, 2018 11:35 pm

Output:

[admin@MikroTik] > ip firewall nat export  
# oct/26/2018 13:34:22 by RouterOS 6.43.4
# software id = 
#
# model = RouterBOARD 1100x4
# serial number = 
/ip firewall nat
add action=accept chain=srcnat disabled=yes dst-address=192.168.88.0/24 src-address=192.168.89.0/24
add action=accept chain=srcnat disabled=yes dst-address=172.20.20.2 src-address=172.20.20.1
add action=accept chain=srcnat dst-address=10.0.11.0/24 src-address=10.0.10.0/24
add action=accept chain=srcnat dst-address=10.0.10.0/24 src-address=10.0.11.0/24
add action=src-nat chain=srcnat src-address=10.0.101.0/29 to-addresses=210.xxx.xxx.128
add action=src-nat chain=srcnat src-address=10.100.111.0/24 to-addresses=210.xxx.xxx.130
add action=dst-nat chain=dstnat dst-address=210.xxx.xxx.129 to-addresses=10.0.111.5
add action=masquerade chain=srcnat out-interface-list=WAN
add action=dst-nat chain=dstnat dst-address=210.xxx.xxx.131 dst-port=8000 protocol=tcp to-addresses=10.0.111.5 to-ports=8180
[admin@MikroTik] >

sindy · Fri Oct 26, 2018 11:49 pm

Hm, lovely. No idea why

action=accept chain=srcnat dst-address=10.0.11.0/24 src-address=10.0.10.0/24
should not prevent
add action=masquerade chain=srcnat out-interface-list=WAN
from creating
src-address=10.0.10.1 dst-address=10.0.11.1 reply-src-address=10.0.11.1 reply-dst-address=210.xxx.xxx.230

Try adding protocol=!gre to the action=masquerade rule, but to me it is a clear bug (unless you've taken the effort to obfuscate even the private IPs and there is some typo in the rules) that the first action=accept rule has not stopped the connection from being masqueraded.

frontdist · Sat Oct 27, 2018 4:43 pm

Hi Sindy,

There was a transposition error in a most critical area...

One of the src-nat rules had an extra "0" in it which ruled it ineffective, once that was corrected and the connection tracking table cleared, things worked properly on a consistent basis. I think the biggest lesson here was about rooting out the cases in which a tunnel as trying to associate itself with a public IP address instead of the designated IPSec-captured private address. This was made worse by the "sometimes it worked, sometimes it didn't" state caused by the randomness of whether the ipsec connection was established first, or the gre tunnel tried to come up first - and the fact that it's no simple task clearing the connection tracking table.

Thanks again for all the help, I was making too many changes at the same time and lost track of everything. I hope this helps someone else!

Who is online