Route specific traffic through the VPN

Technetium · Sat Jun 08, 2019 4:08 pm

I have a network (192.168.1.XXX), connected to internet.
To access to an application i have to setup a VPN and route through the VPN tunnel the requests from local devices that are requiring for the VPN network IP (192.180.1.XXX)
How can i route only the requests to 192.180.1.1 through the VPN connection?

sindy · Sat Jun 08, 2019 5:12 pm

It depends on the VPN type. For most types of VPNs, including something-via-IPsec, you need just another route; for a policy-based IPsec, you need some route for the destination (the default one will do) and an /ip ipsec policy to intercept the packets already routed and deliver them to the destination via the security association.

Technetium · Sat Jun 08, 2019 5:38 pm

The VPN is a tunnel to reach a server that exposes a webserver, port 80,8080,443.
The client on my network had to digit on browser https://192.180.1.10 to use it.
Is an /ip ipsec policy setup sufficient ?

sindy · Sat Jun 08, 2019 5:48 pm

I can only repeat my question - what kind of tunnel in particular? Something-over-IPsec or just IPsec? Post the export of the configuration, see my automatic signature for a hint on anonymization.

Technetium · Sat Jun 08, 2019 6:49 pm

Just an IPSec (IKE -IPSec).

sindy · Sat Jun 08, 2019 7:10 pm

In that case yes, an /ip ipsec policy is normally enough. At the client side, it will have src-address=the.client.sub.net/mask dst-address=192.180.1.10, and at the server side, these items' values will be swapped. For a single protocol & port combination it makes sense to add these parameters to the policy; for several ports, it is better to use a policy not restricted to particular ports and if you want to block access to other ports than the listed ones, use regular /ip firewall filter rules for that.

But if you are not the administrator of the VPN "server" side and the server forces a policy to your Mikrotik as you have to use mode-config and policy-generate different from no to obtain an IP address, things may be different.

Technetium · Sat Jun 08, 2019 7:13 pm

I will try. I'm not the administrator of the server side.
The server side is managed by the company who sell the web application.
The VPN connect my router to them datacenter.

sindy · Sat Jun 08, 2019 8:01 pm

In that case I'm afraid I'll need to see the running configuration when the VPN is up - in particular, the result of /ip ipsec policy print. The IKE part (phase 1, the control connection of the IPsec tunnel) may be up but if you haven't configured the peer/identity (depending on RouterOS version) with mode-config and generate-policy properly, there may be no policy and the server may not accept a policy configured by you. The common issue is that if you connect to a remote network, the address at your side has to be unique in the remote network's context, which implies that it must be a public one (which should be unique by definition although nothing is impossible) or it must be assigned by them, which in turn makes it potentially conflicting with your network context. So the safest approach is that the server assigns you a public address from its own pool. And if it assigns you an address, you need that it assigns you also a policy to match that assigned address (which may not be the same every time you connect), and your /ip ipsec mode-config row must contain a name of an address-list which will be used in a dynamically added /ip firewall nat rule changing the source address of packets from hosts whose source-address matches that address-list so that the policy could match them and send them down the SA.

tharorris · Mon Jun 06, 2022 3:05 pm

Hello,

Can I give life to this thread by sharing my problem? I have two Mikrotik devices in different countries and they are connected together through IPsec. I want to route specific traffic (blocked sites) from one country to another country.

This are my settings in the one side.

/interface bridge
add admin-mac=C4:AD:34:D7:15:C2 auto-mac=no comment=defconf name=bridge
/interface ethernet
set [ find default-name=ether1 ] l2mtu=1500
set [ find default-name=ether4 ] l2mtu=1500
set [ find default-name=ether5 ] l2mtu=1500
/interface pppoe-client
add add-default-route=yes disabled=no interface=ether1 name=pppoe-out1 \
    use-peer-dns=yes user=v42049471
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/interface lte apn
set [ find default=yes ] ip-type=ipv4 use-network-apn=no
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip ipsec mode-config
set [ find default=yes ] connection-mark=via-ipsec
/ip ipsec peer
add address=HIDDEN exchange-mode=ike2 local-address=192.168.36.1 \
    name=Athens
/ip ipsec profile
set [ find default=yes ] dh-group=modp2048 enc-algorithm=aes-256 \
    hash-algorithm=sha256 lifetime=8h
/ip ipsec proposal
set [ find default=yes ] auth-algorithms=sha256 enc-algorithms=aes-256-cbc \
    lifetime=8h pfs-group=modp2048
/ip pool
add name=dhcp ranges=192.168.36.10-192.168.36.254
add name=vpn ranges=192.168.89.2-192.168.89.255
/ip dhcp-server
add address-pool=dhcp always-broadcast=yes interface=bridge lease-time=1h \
    name=defconf
/port
set 0 name=serial0
/ppp profile
set *FFFFFFFE dns-server=192.168.36.1 local-address=192.168.89.1 \
    remote-address=vpn
/routing bgp template
set default disabled=no output.network=bgp-networks
/routing ospf instance
add disabled=no name=default-v2
/routing ospf area
add disabled=yes instance=default-v2 name=backbone-v2
/interface bridge port
add bridge=bridge comment=defconf ingress-filtering=no interface=ether2
add bridge=bridge comment=defconf ingress-filtering=no interface=ether3
add bridge=bridge comment=defconf ingress-filtering=no interface=ether4
add bridge=bridge comment=defconf ingress-filtering=no interface=ether5
add bridge=bridge comment=defconf ingress-filtering=no interface=sfp1
/ip neighbor discovery-settings
set discover-interface-list=LAN
/ipv6 settings
set disable-ipv6=yes max-neighbor-entries=8192
/interface detect-internet
set detect-interface-list=all
/interface l2tp-server server
set use-ipsec=yes
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1 list=WAN
add interface=pppoe-out1 list=WAN
/interface ovpn-server server
set auth=sha1,md5
/interface sstp-server server
set default-profile=default-encryption
/ip address
add address=192.168.36.1/24 comment=defconf interface=bridge network=\
    192.168.36.0
/ip cloud
set ddns-enabled=yes ddns-update-interval=5m
/ip dhcp-client
add comment=defconf disabled=yes interface=ether1
/ip dhcp-server lease
add address=192.168.36.222 client-id=1:0:11:32:be:e1:ca comment=Synology \
    mac-address=00:11:32:BE:E1:CA server=defconf
add address=192.168.36.221 mac-address=D4:5E:EC:39:D0:8D server=defconf
add address=192.168.36.220 mac-address=AC:D5:64:6C:62:A9 server=defconf
add address=192.168.36.200 client-id=1:9c:93:4e:ac:5e:2d comment=Xerox_VRN \
    mac-address=9C:93:4E:AC:5E:2D server=defconf
add address=192.168.36.100 client-id=1:0:e0:4c:68:1:b8 comment=\
    "Toshiba Click2Pro" mac-address=00:E0:4C:68:01:B8 server=defconf
add address=192.168.36.254 client-id=1:fc:5b:39:7d:b3:d8 comment=CiscoAP \
    mac-address=FC:5B:39:7D:B3:D8 server=defconf
add address=192.168.36.100 client-id=1:0:e0:6c:38:69:82 comment=\
    "Toshiba Click2Pro" disabled=yes mac-address=00:E0:6C:38:69:82 server=\
    defconf
/ip dhcp-server network
add address=192.168.36.0/24 comment=defconf dns-server=1.1.1.1,8.8.8.8 \
    gateway=192.168.36.1 netmask=24
/ip dns
set allow-remote-requests=yes servers=94.140.14.14
/ip dns static
add address=192.168.36.1 comment=defconf name=router.lan
/ip firewall address-list
add address=HIDDEN list=WAN-IP
add address=wtfismyip.com list=VPN-destinations
/ip firewall filter
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=accept chain=input comment="allow pptp" dst-port=1723 protocol=tcp
add action=accept chain=input comment="Mikrotik Management" dst-port=8291 \
    protocol=tcp src-address=192.168.89.0/24
add action=accept chain=input dst-port=8291 protocol=tcp src-address=\
    192.168.35.0/24
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment=\
    "defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=drop chain=input comment="defconf: drop all not coming from LAN" \
    in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" \
    ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" \
    ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" \
    connection-state=established,related hw-offload=yes
add action=accept chain=forward comment=\
    "defconf: accept established,related, untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid
add action=drop chain=forward comment=\
    "defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat \
    connection-state=new in-interface-list=WAN
add action=accept chain=input comment="allow IPsec NAT" disabled=yes \
    dst-port=4500 protocol=udp
add action=accept chain=input comment="allow IKE" disabled=yes dst-port=500 \
    protocol=udp
add action=accept chain=input comment="allow l2tp" disabled=yes dst-port=1701 \
    protocol=udp
add action=accept chain=input comment="allow sstp" disabled=yes dst-port=443 \
    protocol=tcp
/ip firewall mangle
add action=change-mss chain=forward new-mss=clamp-to-pmtu passthrough=yes \
    protocol=tcp tcp-flags=syn
add action=mark-connection chain=prerouting connection-mark=no-mark \
    connection-state=new disabled=yes dst-address-list=VPN-destinations \
    in-interface-list=LAN new-connection-mark=via-ipsec passthrough=yes \
    src-address-list=192.168.36.0/24
/ip firewall nat
add action=accept chain=srcnat comment="NAT bypass" dst-address=\
    192.168.35.0/24 src-address=192.168.36.0/24
add action=accept chain=srcnat dst-address=192.168.99.0/24 log=yes \
    src-address=192.168.36.0/24
add action=accept chain=srcnat dst-address=192.168.98.0/24 log=yes \
    src-address=192.168.36.0/24
add action=masquerade chain=srcnat comment="Hairpin NAT" out-interface=\
    pppoe-out1
add action=masquerade chain=srcnat comment="Hairpin NAT" dst-address=\
    192.168.36.0/24 src-address=192.168.36.0/24
add action=masquerade chain=srcnat comment="defconf: masquerade" \
    ipsec-policy=out,none out-interface=pppoe-out1 src-address=\
    192.168.36.0/24
add action=dst-nat chain=dstnat comment="Synology Web Management" \
    dst-address-list=WAN-IP dst-port=5000-5001 protocol=tcp to-addresses=\
    192.168.36.222 to-ports=5000-5001
add action=dst-nat chain=dstnat comment="HTTPS Web Server" dst-address-list=\
    WAN-IP dst-port=443 protocol=tcp to-addresses=192.168.36.222 to-ports=443
add action=dst-nat chain=dstnat comment="Synology Cloud Sync" dst-port=6690 \
    in-interface=pppoe-out1 protocol=tcp to-addresses=192.168.36.222 \
    to-ports=6690
add action=dst-nat chain=dstnat comment="Plex Toshiba" dst-port=32401 \
    in-interface=pppoe-out1 protocol=tcp to-addresses=192.168.36.100 \
    to-ports=32400
add action=masquerade chain=srcnat comment="masq. vpn traffic" src-address=\
    192.168.89.0/24
add action=dst-nat chain=dstnat comment=Plex disabled=yes dst-port=64200 \
    in-interface=pppoe-out1 protocol=tcp to-addresses=192.168.36.222 \
    to-ports=32400
add action=dst-nat chain=dstnat comment="HTTP Web Server" dst-address-list=\
    WAN-IP dst-port=80 protocol=tcp to-addresses=192.168.36.222 to-ports=80
/ip firewall service-port
set sip disabled=yes
/ip ipsec identity
add peer=Athens
/ip ipsec policy
add dst-address=192.168.35.0/24 level=unique peer=Athens src-address=\
    192.168.36.0/24 tunnel=yes
add dst-address=192.168.98.0/24 level=unique peer=Athens src-address=\
    192.168.36.0/24 tunnel=yes
add dst-address=192.168.99.0/24 level=unique peer=Athens src-address=\
    192.168.36.0/24 tunnel=yes
/ip service
set www disabled=yes
/ip upnp interfaces
add interface=bridge type=internal
add interface=ether1 type=external
/ppp secret
add name=vpn
/snmp
set enabled=yes
/system clock
set time-zone-name=Europe/Moscow
/system identity
set name=MikroTik_36
/system scheduler
add interval=1m name="Flush SA" on-event=":if ([/ping 192.168.35.1 interface b\
    ridge count=4]<3) do={\r\
    \n  /ip ipsec installed-sa flush;\r\
    \n  :log info \"IPSEC tunnel is down: Flushing Installed SA !!!\"\r\
    \n} else={\r\
    \n\r\
    \n}" policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon \
    start-date=dec/08/2021 start-time=12:27:38
add disabled=yes interval=1d name=Reboot on-event="/system reboot" policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon \
    start-date=jun/17/2021 start-time=05:00:00
/tool graphing interface
add interface=pppoe-out1
add interface=ether1
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN

On the other side I have double NAT and I don't know if this is a problem also.

I have followed numerous guides and the bottom line is that I have to create a Address List with specific domains I want them getting through IPsec tunnel, and then mark these packets (Mangle).
Everything failed. Maybe my config is wrong also.

Some sites in Address List resolve to the local ISP IP, since the URL is getting redirected to a page which says "blocked". How I can parse this specific DNS query also to the other side?

Thank you

sindy · Mon Jun 06, 2022 5:32 pm

Before a deep dive into your fairly complex configuration, two generic points:

for "vanilla" IPsec, i.e. a "policy based" one, the only way to affect the "routing" by matching an address list is to use src-nat rules matching on that address list. So if you want some devices in country R to connect to some servers via the router in country G, and some devices in country G to connect to other servers via the router in country R, you need to configure IPsec policies 0.0.0.0/0 <-> G.G.G.G and R.R.R.R <-> 0.0.0.0/0, and src-nat rules like action=src-nat to-addresses=R.R.R.R dst-address-list=via-G-WAN in country R and action=src-nat to-addresses=G.G.G.G dst-address-list=via-R-WAN in country G, making the traffic matching the respective address lists "visible" for the corresponding policies.
Therefore, a setup that uses IPsec only to encrypt some other type of tunnel (IPIP, GRE, L2TP - each has its own advantages and disadvantages), and normal "policy routing" (multiple routing tables using these "normal" tunnels as WAN interfaces), may be easier to understand.
the address lists where the address item is a domain name have their limits - since they just resolve particular fqdns to lists of IP numbers, you cannot use wildcards in the domain names. And even worse, since many large companies run multiple services with different fqdns on the same servers, it is not possible to treat such domains differently. As an example, you cannot effectively distinguish between google.com and youtube.com, as both are served from the same local caches.

tharorris · Tue Jun 07, 2022 8:52 am

Thank you for your reply.
I tried and made the changes as you mentioned and now the websites on the address list are failing to load - which is progress in my textbook.

When you say G.G.G.G and R.R.R.R (which is very specific i can say, because letters really depict specific Countries!) I replace with 192.168.35.0/24 and 192.168.36.0/24
Also I add (not replace) the IPsec policy as you said on both Mikrotik devices and the PH2 state is established. But if I move it up to be first, all the remaining tunnels immediately disconnect.

This is the new config.

/interface bridge
add admin-mac=C4:AD:34:D7:15:C2 auto-mac=no comment=defconf name=bridge
/interface ethernet
set [ find default-name=ether1 ] l2mtu=1500
set [ find default-name=ether4 ] l2mtu=1500
set [ find default-name=ether5 ] l2mtu=1500
/interface pppoe-client
add add-default-route=yes disabled=no interface=ether1 name=pppoe-out1 \
    use-peer-dns=yes user=v42049471
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/interface lte apn
set [ find default=yes ] ip-type=ipv4 use-network-apn=no
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip ipsec mode-config
set [ find default=yes ] connection-mark=via-ipsec
/ip ipsec peer
add address=HIDDEN exchange-mode=ike2 local-address=192.168.36.1 \
    name=Athens
/ip ipsec profile
set [ find default=yes ] dh-group=modp2048 enc-algorithm=aes-256 \
    hash-algorithm=sha256 lifetime=8h
/ip ipsec proposal
set [ find default=yes ] auth-algorithms=sha256 enc-algorithms=aes-256-cbc \
    lifetime=8h pfs-group=modp2048
/ip pool
add name=dhcp ranges=192.168.36.10-192.168.36.254
add name=vpn ranges=192.168.89.2-192.168.89.255
/ip dhcp-server
add address-pool=dhcp always-broadcast=yes interface=bridge lease-time=1h \
    name=defconf
/port
set 0 name=serial0
/ppp profile
set *FFFFFFFE dns-server=192.168.36.1 local-address=192.168.89.1 \
    remote-address=vpn
/routing bgp template
set default disabled=no output.network=bgp-networks
/routing ospf instance
add disabled=no name=default-v2
/routing ospf area
add disabled=yes instance=default-v2 name=backbone-v2
/interface bridge port
add bridge=bridge comment=defconf ingress-filtering=no interface=ether2
add bridge=bridge comment=defconf ingress-filtering=no interface=ether3
add bridge=bridge comment=defconf ingress-filtering=no interface=ether4
add bridge=bridge comment=defconf ingress-filtering=no interface=ether5
add bridge=bridge comment=defconf ingress-filtering=no interface=sfp1
/ip neighbor discovery-settings
set discover-interface-list=LAN
/ipv6 settings
set disable-ipv6=yes max-neighbor-entries=8192
/interface detect-internet
set detect-interface-list=all
/interface l2tp-server server
set use-ipsec=yes
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1 list=WAN
add interface=pppoe-out1 list=WAN
/interface ovpn-server server
set auth=sha1,md5
/interface sstp-server server
set default-profile=default-encryption
/ip address
add address=192.168.36.1/24 comment=defconf interface=bridge network=\
    192.168.36.0
/ip cloud
set ddns-enabled=yes ddns-update-interval=5m
/ip dhcp-client
add comment=defconf disabled=yes interface=ether1
/ip dhcp-server lease
add address=192.168.36.222 client-id=1:0:11:32:be:e1:ca comment=Synology \
    mac-address=00:11:32:BE:E1:CA server=defconf
add address=192.168.36.221 mac-address=D4:5E:EC:39:D0:8D server=defconf
add address=192.168.36.220 mac-address=AC:D5:64:6C:62:A9 server=defconf
add address=192.168.36.200 client-id=1:9c:93:4e:ac:5e:2d comment=Xerox_VRN \
    mac-address=9C:93:4E:AC:5E:2D server=defconf
add address=192.168.36.100 client-id=1:0:e0:4c:68:1:b8 comment=\
    "Toshiba Click2Pro" mac-address=00:E0:4C:68:01:B8 server=defconf
add address=192.168.36.254 client-id=1:fc:5b:39:7d:b3:d8 comment=CiscoAP \
    mac-address=FC:5B:39:7D:B3:D8 server=defconf
add address=192.168.36.100 client-id=1:0:e0:6c:38:69:82 comment=\
    "Toshiba Click2Pro" disabled=yes mac-address=00:E0:6C:38:69:82 server=\
    defconf
/ip dhcp-server network
add address=192.168.36.0/24 comment=defconf dns-server=1.1.1.1,8.8.8.8 \
    gateway=192.168.36.1 netmask=24
/ip dns
set allow-remote-requests=yes servers=94.140.14.14
/ip dns static
add address=192.168.36.1 comment=defconf name=router.lan
/ip firewall address-list
add address=HIDDEN list=WAN-IP
add address=wtfismyip.com list=VPN-destinations
add address=facebook.com list=VPN-destinations
/ip firewall filter
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=accept chain=input comment="allow pptp" dst-port=1723 protocol=tcp
add action=accept chain=input comment="Mikrotik Management" dst-port=8291 \
    protocol=tcp src-address=192.168.89.0/24
add action=accept chain=input dst-port=8291 protocol=tcp src-address=\
    192.168.35.0/24
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment=\
    "defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=drop chain=input comment="defconf: drop all not coming from LAN" \
    in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" \
    ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" \
    ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" \
    connection-state=established,related hw-offload=yes
add action=accept chain=forward comment=\
    "defconf: accept established,related, untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid
add action=drop chain=forward comment=\
    "defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat \
    connection-state=new in-interface-list=WAN
add action=accept chain=input comment="allow IPsec NAT" disabled=yes \
    dst-port=4500 protocol=udp
add action=accept chain=input comment="allow IKE" disabled=yes dst-port=500 \
    protocol=udp
add action=accept chain=input comment="allow l2tp" disabled=yes dst-port=1701 \
    protocol=udp
add action=accept chain=input comment="allow sstp" disabled=yes dst-port=443 \
    protocol=tcp
/ip firewall mangle
add action=change-mss chain=forward new-mss=clamp-to-pmtu passthrough=yes \
    protocol=tcp tcp-flags=syn
add action=mark-connection chain=prerouting connection-mark=no-mark \
    connection-state=new disabled=yes dst-address-list=VPN-destinations \
    in-interface-list=LAN new-connection-mark=via-ipsec passthrough=yes \
    src-address-list=192.168.36.0/24
/ip firewall nat
add action=accept chain=srcnat comment="NAT bypass" dst-address=\
    192.168.35.0/24 src-address=192.168.36.0/24
add action=src-nat chain=srcnat comment="NAT bypass" dst-address-list=\
    VPN-destinations to-addresses=192.168.36.0/24
add action=accept chain=srcnat dst-address=192.168.99.0/24 log=yes \
    src-address=192.168.36.0/24
add action=accept chain=srcnat dst-address=192.168.98.0/24 log=yes \
    src-address=192.168.36.0/24
add action=accept chain=srcnat disabled=yes dst-address=192.168.40.0/24 log=\
    yes src-address=192.168.36.0/24
add action=masquerade chain=srcnat comment="Hairpin NAT" out-interface=\
    pppoe-out1
add action=masquerade chain=srcnat comment="Hairpin NAT" dst-address=\
    192.168.36.0/24 src-address=192.168.36.0/24
add action=masquerade chain=srcnat comment="defconf: masquerade" \
    ipsec-policy=out,none out-interface=pppoe-out1 src-address=\
    192.168.36.0/24
add action=dst-nat chain=dstnat comment="Synology Web Management" \
    dst-address-list=WAN-IP dst-port=5000-5001 protocol=tcp to-addresses=\
    192.168.36.222 to-ports=5000-5001
add action=dst-nat chain=dstnat comment="HTTPS Web Server" dst-address-list=\
    WAN-IP dst-port=443 protocol=tcp to-addresses=192.168.36.222 to-ports=443
add action=dst-nat chain=dstnat comment="Synology Cloud Sync" dst-port=6690 \
    in-interface=pppoe-out1 protocol=tcp to-addresses=192.168.36.222 \
    to-ports=6690
add action=dst-nat chain=dstnat comment="Plex Toshiba" dst-port=32401 \
    in-interface=pppoe-out1 protocol=tcp to-addresses=192.168.36.100 \
    to-ports=32400
add action=masquerade chain=srcnat comment="masq. vpn traffic" src-address=\
    192.168.89.0/24
add action=dst-nat chain=dstnat comment=Plex disabled=yes dst-port=64200 \
    in-interface=pppoe-out1 protocol=tcp to-addresses=192.168.36.222 \
    to-ports=32400
add action=dst-nat chain=dstnat comment="HTTP Web Server" dst-address-list=\
    WAN-IP dst-port=80 protocol=tcp to-addresses=192.168.36.222 to-ports=80
/ip firewall service-port
set sip disabled=yes
/ip ipsec identity
add peer=Athens
/ip ipsec policy
add dst-address=192.168.35.0/24 level=unique peer=Athens src-address=\
    192.168.36.0/24 tunnel=yes
add dst-address=192.168.35.0/24 level=unique peer=Athens src-address=\
    0.0.0.0/0 tunnel=yes
add dst-address=192.168.98.0/24 level=unique peer=Athens src-address=\
    192.168.36.0/24 tunnel=yes
add dst-address=192.168.99.0/24 level=unique peer=Athens src-address=\
    192.168.36.0/24 tunnel=yes
/ip service
set www disabled=yes
/ip upnp interfaces
add interface=bridge type=internal
add interface=ether1 type=external
/ppp secret
add name=vpn
/snmp
set enabled=yes
/system clock
set time-zone-name=Europe/Moscow
/system identity
set name=MikroTik_36
/system scheduler
add interval=1m name="Flush SA" on-event=":if ([/ping 192.168.35.1 interface b\
    ridge count=4]<3) do={\r\
    \n  /ip ipsec installed-sa flush;\r\
    \n  :log info \"IPSEC tunnel is down: Flushing Installed SA !!!\"\r\
    \n} else={\r\
    \n\r\
    \n}" policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon \
    start-date=dec/08/2021 start-time=12:27:38
add disabled=yes interval=1d name=Reboot on-event="/system reboot" policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon \
    start-date=jun/17/2021 start-time=05:00:00
/tool graphing interface
add interface=pppoe-out1
add interface=ether1
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN

My config is not complex. It's a PPoE connection, DHCP with some static leases, an IPsec site-to-site VPN, some port forwards and a Hairpin NAT so I can hit some internal devices with their external domain name.
You will definitely find some abnormalities or garbage in the config, since I try different configs - till something work !

Thank you.

sindy · Tue Jun 07, 2022 10:15 am

When you say G.G.G.G and R.R.R.R (which is very specific i can say, because letters really depict specific Countries!) I replace with 192.168.35.0/24 and 192.168.36.0/24

I did not have in mind subnets but indeed individual (/32) addresses, as the purpose is to clearly separate the policies used for "site to site" (LAN subnet to LAN subnet) traffic from policies used for "site to internet via the other country" traffic. Therefore, G.G.G.G and R.R.R.R should be private addresses outside any subnet you use at either site, used solely to make the traffic "visible" to the policies. Using the local subnets doesn't actually break anything, it just makes the overall configuration a bit less clear. There's also another reason, see below.

The added policies I've suggested are just one part of the overall solution; the other part is the action=src-nat rule matching on dst-address-list=VPN-destinations. The way you have configured it, it exempts the traffic towards the VPN destinations from getting masqueraded to the local WAN addresses, which explains why the connections stopped working the normal way; why the connections did not succeed via the remote site's WAN connection is yet to be resolved.

Also I add (not replace) the IPsec policy as you said on both Mikrotik devices and the PH2 state is established. But if I move it up to be first, all the remaining tunnels immediately disconnect.

This sounds strange. By design, the traffic gets matched against the traffic selectors of the policies from the topmost policy to the bottommost one until first match, same like when matching firewall rules, routing rules etc. So a policy x.x.x.0/24 -> 0.0.0.0/0 shadows a policy x.x.x.0/24 -> y.y.y.0/24 if the "wider" one is placed earlier (higher) in the policy list, but if the "wider" policy works properly, the fact that the "narrower" one is shadowed should not break the traffic that matches the "narrower" one provided that both policies establish their SAs between the same pair of peers.

One of the rare cases where Mikrotik "knows better that the users what the user wants to do" is that an IPsec responder marks policies with dst-address=0.0.0.0/0 as Invalid, or just doesn't allow them to become active, under some circumstances. This is a "safety precaution" to prevent the users from locking themselves out from the router by misconfiguring IPsec. However, if this was the reason why the connections to "VPN-destinations" fail, the policy should not be shown as Active.

To rule this out, first change the policies so that the G.G.G.G and R.R.R.R are indeed out of any subnet used at the router (because the danger of locking yourself out is real, and you need to access the router via MAC address to revert such misconfiguration, which may not be easy if the remote site is unattended), and then split the policy into two, one with dst-address=0.0.0.0/1 and the other one with dst-address=128.0.0.0/1 (and correspondingly for src-address at the other peer)

G.G.G.G and R.R.R.R need not be assigned to any interface on any of the routers, they are just used by the src-nat rules and the policies.

tharorris · Tue Jun 07, 2022 12:22 pm

The whole reason I am trying to do this, is that clients on any subnet should be able to reach banned webpages through the other IPsec connected subnet.
192.168.36.0/24 is the R subnet
192.168.35.0/24 is the G subnet
Example: facebook.com is included in the address list.

Nowadays if I want to reach Facebook (or any banned site), I have to open on my device a VPN connection to another site (OpenVPN). But since I have this IPsec, I want to see if it is possible with Mikrotik.
It is ridiculously easy on other firewalls (pfSense, Watchguard, Fortigate), either by NAT rules or SD-Wan policy.
I understand the mentality to do it, I just don't freaking understand how to translate this to Mikrotik.

I think I should:
1) create address list with websites I want to route
2) mark packets (Mangle)
3) Create NAT / Filter rules

I follow some guides regarding NordVPN, "adjusted" to my settings but no dice.
viewtopic.php?t=161967
viewtopic.php?f=23&t=169273

I love Mikrotik devices, they do stuff that other firewalls don't, the power they consume is minimal and I want to learn to program them so as to assist others learn this world.

Another idea is to create a separate OpenVPN connection (interface) pointing to a Watchguard device and route address list through there but I don't want to put extra load since there is a connection already.

Thank you.

sindy · Tue Jun 07, 2022 1:58 pm

The whole reason I am trying to do this, is that clients on any subnet should be able to reach banned webpages through the other IPsec connected subnet.

No need to explain this. I was a bit surprised you want the functionality to be bi-directional, as if some sites were banned in G and you needed to access them via WAN in R, but technically that's not a big deal.

since I have this IPsec, I want to see if it is possible with Mikrotik.

It is. There must be some minor issue somewhere.

I understand the mentality to do it, I just don't freaking understand how to translate this to Mikrotik.

I think I should:
1) create address list with websites I want to route
2) mark packets (Mangle)
3) Create NAT / Filter rules

All correct, although since you use IPsec rather than "normal" routing to route the traffic via the tunnel, you can skip the packet marking phase and use matching against the dst-address-list (normally used to choose packets to get a routing mark) in the src-nat rules.

But IPsec policies need more thinking than normal routes, that's why I have suggested to use some "normal" tunnel and just use IPsec to encrypt it, even using one of the already existing policies. Such as /interface ipip add local-address=192.168.36.1 remote-address=192.168.35.1 in R, and /interface ipip add local-address=192.168.35.1 remote-address=192.168.36.1 in G. Then, you would add routing tables via-R (in G) and via-G (in R), both consisting of a single route default route with gateway=ipip1, and use action=mark-routing dst-address-list=VPN-destinations new-routing-mark=via-X to make the traffic to VPN destinations be routed via the tunnel. For the responses, you have to add routes to 192.168.3x.0/24 via ipip1 to the main routing table, as the policy 192.168.35.0/24 <-> 192.168.36.0/24 will ignore these responses.

When sticking with IPsec policies, the thing is that e.g. a policy 192.168.0.0/24 <-> 0.0.0.0/0 will match also on packets between the router's own address in 192.168.0.0/24 and any host connected to that subnet. So it will prevent hosts in this subnet from managing the router and/or using it as DNS. So you either need to put a policy 192.168.0.0/24 <-> 192.168.0.0/24 action=none above it to shadow the wider one, or you have to use something else than 192.168.0.0/24 for the "wide" policy to avoid that conflict completely. That's why I keep insisting that G.G.G.G and R.R.R.R should be addresses reserved solely for the purpose of sending the traffic for the "remote WAN". /32 addresses are sufficient because at the remote end, all these connections will be src-nat'ed to the single WAN address anyway. And since you have 3 subnets in Athens, the order of policies would be extremely important if you wanted to keep using 192.168.35.0/24 and 192.168.36.0/24 as G.G.G.G and R.R.R.R, such as

src-address=192.168.35.0/24 dst-address=192.168.36.0/24 action=encrypt peer=Moscow
src-address=192.168.98.0/24 dst-address=192.168.36.0/24 action=encrypt peer=Moscow
src-address=192.168.99.0/24 dst-address=192.168.36.0/24 action=encrypt peer=Moscow
src-address=192.168.0.0/16 dst-address=192.168.0.0/16 action=none
src-address=192.168.0.0/16 dst-address=0.0.0.0/0 action=encrypt peer=Moscow
src-address=0.0.0.0/0 dst-address=192.168.0.0/16 action=encrypt peer=Moscow

The first three policies would be the already existing "real" policies, the fourth one would prevent traffic between local subnets from getting "kidnapped" to R, and the fifth and sixth policy would provide the "remote WAN" operation. Instead of the first 4 policies, you'd have to use 9 ones with action=none to prevent all local traffic from being kidnapped to R (35<=>35, 98<=>98, 99<=>99, 35<=>98, 98<=35, 35<=>99, 99<=35, 99<=>98, 98<=99).

Plus there is the thing with the likely need to split 0.0.0.0/0 into halves to work around the safety measure of Mikrotik.

tharorris · Tue Jun 07, 2022 2:57 pm

The thing with 3 subnets ... just forget it and I am sorry that I confused you!
The 192.168.35.0/24 is the one I am using (LAN) and trying to implement changes.

192.168.99.1 is the internet gateway (WAN - as I said, double NAT) and I add it just for the sake to be able to configure ISP equipment remotely through IPsec (port forwarding, restart equipment etc). Because in Greece, SMB internet equipment don't support trunk and DMZ is locked...
192.168.98.1 is a second 4G / voice gateway (WAN#2) in case .99.1 fails. I haven't implement yet a dual WAN failover configuration, but it's the next thing I will focus on, when I find solution for this problem.

Can you please tell me the extra commands I should input, and I will do it one by one to check how it behave?
I already revert back to the original config I sent in the first place.

Thank you.

sindy · Tue Jun 07, 2022 3:12 pm

Which way you want, pure policy-based IPsec or the IPIP tunnel encrypted using an existing policy? In any case, I need the export from the router in Athens as well as it will require a configuration modification too.

tharorris · Tue Jun 07, 2022 5:36 pm

I don't even know the pros / cons of each case. What is the simplest way for easy troubleshoot?

36 site

/interface bridge
add admin-mac=C4:AD:34:D7:15:C2 auto-mac=no comment=defconf name=bridge
/interface ethernet
set [ find default-name=ether1 ] l2mtu=1500
set [ find default-name=ether4 ] l2mtu=1500
set [ find default-name=ether5 ] l2mtu=1500
/interface pppoe-client
add add-default-route=yes disabled=no interface=ether1 name=pppoe-out1 \
    use-peer-dns=yes user=v42049471
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/interface lte apn
set [ find default=yes ] ip-type=ipv4 use-network-apn=no
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip ipsec mode-config
set [ find default=yes ] connection-mark=via-ipsec
/ip ipsec peer
add address=HIDDEN exchange-mode=ike2 local-address=192.168.36.1 \
    name=Athens
/ip ipsec profile
set [ find default=yes ] dh-group=modp2048 enc-algorithm=aes-256 \
    hash-algorithm=sha256 lifetime=8h
/ip ipsec proposal
set [ find default=yes ] auth-algorithms=sha256 enc-algorithms=aes-256-cbc \
    lifetime=8h pfs-group=modp2048
/ip pool
add name=dhcp ranges=192.168.36.10-192.168.36.254
/ip dhcp-server
add address-pool=dhcp always-broadcast=yes interface=bridge lease-time=1h \
    name=defconf
/port
set 0 name=serial0
/routing bgp template
set default disabled=no output.network=bgp-networks
/routing ospf instance
add disabled=no name=default-v2
/routing ospf area
add disabled=yes instance=default-v2 name=backbone-v2
/interface bridge port
add bridge=bridge comment=defconf ingress-filtering=no interface=ether2
add bridge=bridge comment=defconf ingress-filtering=no interface=ether3
add bridge=bridge comment=defconf ingress-filtering=no interface=ether4
add bridge=bridge comment=defconf ingress-filtering=no interface=ether5
add bridge=bridge comment=defconf ingress-filtering=no interface=sfp1
/ip neighbor discovery-settings
set discover-interface-list=LAN
/ipv6 settings
set disable-ipv6=yes max-neighbor-entries=8192
/interface detect-internet
set detect-interface-list=all
/interface l2tp-server server
set use-ipsec=yes
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1 list=WAN
add interface=pppoe-out1 list=WAN
/interface ovpn-server server
set auth=sha1,md5
/interface sstp-server server
set default-profile=default-encryption
/ip address
add address=192.168.36.1/24 comment=defconf interface=bridge network=\
    192.168.36.0
/ip cloud
set ddns-enabled=yes ddns-update-interval=5m
/ip dhcp-client
add comment=defconf disabled=yes interface=ether1
/ip dhcp-server lease
add address=192.168.36.222 client-id=1:0:11:32:be:e1:ca comment=Synology \
    mac-address=00:11:32:BE:E1:CA server=defconf
add address=192.168.36.221 mac-address=D4:5E:EC:39:D0:8D server=defconf
add address=192.168.36.220 mac-address=AC:D5:64:6C:62:A9 server=defconf
add address=192.168.36.200 client-id=1:9c:93:4e:ac:5e:2d comment=Xerox_VRN \
    mac-address=9C:93:4E:AC:5E:2D server=defconf
add address=192.168.36.100 client-id=1:0:e0:4c:68:1:b8 comment=\
    "Toshiba Click2Pro" mac-address=00:E0:4C:68:01:B8 server=defconf
add address=192.168.36.254 client-id=1:fc:5b:39:7d:b3:d8 comment=CiscoAP \
    mac-address=FC:5B:39:7D:B3:D8 server=defconf
add address=192.168.36.100 client-id=1:0:e0:6c:38:69:82 comment=\
    "Toshiba Click2Pro" disabled=yes mac-address=00:E0:6C:38:69:82 server=\
    defconf
/ip dhcp-server network
add address=192.168.36.0/24 comment=defconf dns-server=1.1.1.1,8.8.8.8 \
    gateway=192.168.36.1 netmask=24
/ip dns
set allow-remote-requests=yes servers=94.140.14.14
/ip dns static
add address=192.168.36.1 comment=defconf name=router.lan
/ip firewall address-list
add address=HIDDEN list=WAN-IP
add address=wtfismyip.com list=VPN-destinations
add address=facebook.com list=VPN-destinations
/ip firewall filter
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=accept chain=input comment="allow pptp" dst-port=1723 protocol=tcp
add action=accept chain=input comment="Mikrotik Management" dst-port=8291 \
    protocol=tcp src-address=192.168.89.0/24
add action=accept chain=input dst-port=8291 protocol=tcp src-address=\
    192.168.35.0/24
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment=\
    "defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=drop chain=input comment="defconf: drop all not coming from LAN" \
    in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" \
    ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" \
    ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" \
    connection-state=established,related hw-offload=yes
add action=accept chain=forward comment=\
    "defconf: accept established,related, untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid
add action=drop chain=forward comment=\
    "defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat \
    connection-state=new in-interface-list=WAN
add action=accept chain=input comment="allow IPsec NAT" disabled=yes \
    dst-port=4500 protocol=udp
add action=accept chain=input comment="allow IKE" disabled=yes dst-port=500 \
    protocol=udp
add action=accept chain=input comment="allow l2tp" disabled=yes dst-port=1701 \
    protocol=udp
add action=accept chain=input comment="allow sstp" disabled=yes dst-port=443 \
    protocol=tcp
/ip firewall mangle
add action=change-mss chain=forward new-mss=clamp-to-pmtu passthrough=yes \
    protocol=tcp tcp-flags=syn
add action=mark-connection chain=prerouting connection-mark=no-mark \
    connection-state=new disabled=yes dst-address-list=VPN-destinations \
    in-interface-list=LAN new-connection-mark=via-ipsec passthrough=yes \
    src-address-list=192.168.36.0/24
/ip firewall nat
add action=accept chain=srcnat comment="NAT bypass" dst-address=\
    192.168.35.0/24 src-address=192.168.36.0/24
add action=src-nat chain=srcnat comment="NAT VPN" disabled=yes \
    dst-address-list=VPN-destinations to-addresses=192.168.36.0/24
add action=accept chain=srcnat dst-address=192.168.99.0/24 log=yes \
    src-address=192.168.36.0/24
add action=accept chain=srcnat dst-address=192.168.98.0/24 log=yes \
    src-address=192.168.36.0/24
add action=accept chain=srcnat disabled=yes dst-address=192.168.40.0/24 log=\
    yes src-address=192.168.36.0/24
add action=masquerade chain=srcnat comment="Hairpin NAT" out-interface=\
    pppoe-out1
add action=masquerade chain=srcnat comment="Hairpin NAT" dst-address=\
    192.168.36.0/24 src-address=192.168.36.0/24
add action=masquerade chain=srcnat comment="defconf: masquerade" \
    ipsec-policy=out,none out-interface=pppoe-out1 src-address=\
    192.168.36.0/24
add action=dst-nat chain=dstnat comment="Synology Web Management" \
    dst-address-list=WAN-IP dst-port=5000-5001 protocol=tcp to-addresses=\
    192.168.36.222 to-ports=5000-5001
add action=dst-nat chain=dstnat comment="HTTPS Web Server" dst-address-list=\
    WAN-IP dst-port=443 protocol=tcp to-addresses=192.168.36.222 to-ports=443
add action=dst-nat chain=dstnat comment="Synology Cloud Sync" dst-port=6690 \
    in-interface=pppoe-out1 protocol=tcp to-addresses=192.168.36.222 \
    to-ports=6690
add action=dst-nat chain=dstnat comment="Plex Toshiba" dst-port=32401 \
    in-interface=pppoe-out1 protocol=tcp to-addresses=192.168.36.100 \
    to-ports=32400
add action=masquerade chain=srcnat comment="masq. vpn traffic" src-address=\
    192.168.89.0/24
add action=dst-nat chain=dstnat comment=Plex disabled=yes dst-port=64200 \
    in-interface=pppoe-out1 protocol=tcp to-addresses=192.168.36.222 \
    to-ports=32400
add action=dst-nat chain=dstnat comment="HTTP Web Server" dst-address-list=\
    WAN-IP dst-port=80 protocol=tcp to-addresses=192.168.36.222 to-ports=80
/ip firewall service-port
set sip disabled=yes
/ip ipsec identity
add peer=Athens
/ip ipsec policy
add dst-address=192.168.35.0/24 level=unique peer=Athens src-address=\
    192.168.36.0/24 tunnel=yes
add dst-address=192.168.35.0/24 level=unique peer=Athens src-address=\
    0.0.0.0/0 tunnel=yes
add dst-address=192.168.98.0/24 level=unique peer=Athens src-address=\
    192.168.36.0/24 tunnel=yes
add dst-address=192.168.99.0/24 level=unique peer=Athens src-address=\
    192.168.36.0/24 tunnel=yes
/ip service
set www disabled=yes
/ip upnp interfaces
add interface=bridge type=internal
add interface=ether1 type=external
/ppp profile
set *FFFFFFFE dns-server=192.168.36.1 local-address=192.168.89.1 \
    remote-address=*2
/ppp secret
add name=vpn
/snmp
set enabled=yes
/system clock
set time-zone-name=Europe/Moscow
/system identity
set name=MikroTik_36
/system scheduler
add interval=1m name="Flush SA" on-event=":if ([/ping 192.168.35.1 interface b\
    ridge count=4]<3) do={\r\
    \n  /ip ipsec installed-sa flush;\r\
    \n  :log info \"IPSEC tunnel is down: Flushing Installed SA !!!\"\r\
    \n} else={\r\
    \n\r\
    \n}" policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon \
    start-date=dec/08/2021 start-time=12:27:38
add disabled=yes interval=1d name=Reboot on-event="/system reboot" policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon \
    start-date=jun/17/2021 start-time=05:00:00
/tool graphing interface
add interface=pppoe-out1
add interface=ether1
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN

35 site

/interface bridge
add admin-mac=48:8F:5A:82:D3:17 auto-mac=no comment=defconf name=bridge
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip ipsec peer
add address=HIDDEN exchange-mode=ike2 local-address=192.168.99.50 \
    name=Russia
/ip ipsec profile
set [ find default=yes ] dh-group=modp2048 enc-algorithm=aes-256 \
    hash-algorithm=sha256
/ip ipsec proposal
set [ find default=yes ] auth-algorithms=sha256 enc-algorithms=aes-256-cbc \
    lifetime=8h pfs-group=modp2048
/ip pool
add name=dhcp ranges=192.168.35.200-192.168.35.254
add name=vpn ranges=192.168.89.2-192.168.89.255
/ip dhcp-server
add address-pool=dhcp disabled=no interface=bridge name=defconf
/ppp profile
set *FFFFFFFE local-address=192.168.89.1 remote-address=vpn
/interface bridge port
add bridge=bridge comment=defconf interface=ether2
add bridge=bridge comment=defconf interface=ether3
add bridge=bridge comment=defconf interface=ether1
add bridge=bridge comment=defconf interface=sfp1
/ip neighbor discovery-settings
set discover-interface-list=LAN
/interface detect-internet
set detect-interface-list=all
/interface l2tp-server server
set enabled=yes use-ipsec=yes
/interface list member
add interface=ether1 list=LAN
add interface=ether2 list=LAN
add interface=ether3 list=LAN
add interface=ether4 list=WAN
add interface=ether5 list=WAN
add interface=sfp1 list=LAN
/interface pptp-server server
set enabled=yes
/interface sstp-server server
set default-profile=default-encryption enabled=yes
/ip address
add address=192.168.35.1/24 comment=defconf interface=bridge network=\
    192.168.35.0
/ip cloud
set ddns-enabled=yes ddns-update-interval=10m
/ip dhcp-client
add comment=defconf interface=ether1
/ip dhcp-server lease
add address=192.168.35.4 client-id=1:fc:ec:da:37:42:84 comment=\
    "Ubiquity Access Point" mac-address=FC:EC:DA:37:42:84 server=defconf
add address=192.168.35.5 client-id=1:44:b2:95:31:3a:48 comment=\
    "Camera Entrance" mac-address=44:B2:95:31:3A:48 server=defconf
add address=192.168.35.221 client-id=1:ec:5c:68:c0:62:cb comment=\
    "Sony TV Wireless" mac-address=EC:5C:68:C0:62:CB server=defconf
add address=192.168.35.3 client-id=1:4c:78:97:0:9a:f4 comment=\
    "Home Security Alarm" mac-address=4C:78:97:00:9A:F4 server=defconf
add address=192.168.35.221 client-id=1:cc:98:8b:e6:f3:cb comment=\
    "Sony TV Wired" disabled=yes mac-address=CC:98:8B:E6:F3:CB server=defconf
/ip dhcp-server network
add address=192.168.35.0/24 comment=defconf gateway=192.168.35.1 netmask=24
/ip dns
set allow-remote-requests=yes
/ip dns static
add address=192.168.35.1 comment=defconf name=router.lan
/ip firewall address-list
add address=wtfismyip.com list=VPN-destinations
/ip firewall filter
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=accept chain=input comment="Mikrotik Management" dst-port=8291 \
    protocol=tcp src-address=192.168.40.0/24
add action=accept chain=input dst-port=8291 protocol=tcp src-address=\
    192.168.89.0/24
add action=accept chain=input dst-port=8291 protocol=tcp src-address=\
    192.168.36.0/24
add action=accept chain=input dst-port=8291 protocol=tcp src-address=\
    192.168.35.0/24
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment=\
    "defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=drop chain=input comment="defconf: drop all not coming from LAN" \
    in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" \
    ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" \
    ipsec-policy=out,ipsec
add action=drop chain=forward comment="Block Camera to Internet" src-address=\
    192.168.35.5
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" \
    connection-state=established,related
add action=accept chain=forward comment=\
    "defconf: accept established,related, untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid
add action=drop chain=forward comment=\
    "defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat \
    connection-state=new in-interface-list=WAN
add action=accept chain=input comment="allow IPsec NAT" disabled=yes \
    dst-port=4500 protocol=udp
add action=accept chain=input comment="allow IKE" disabled=yes dst-port=500 \
    protocol=udp
add action=accept chain=input comment="allow pptp" disabled=yes dst-port=1723 \
    protocol=tcp
add action=accept chain=input comment="allow l2tp" disabled=yes dst-port=1701 \
    protocol=udp
add action=accept chain=input comment="allow sstp" disabled=yes dst-port=443 \
    protocol=tcp
/ip firewall nat
add action=accept chain=srcnat comment="NAT bypass" dst-address=\
    192.168.36.0/24 src-address=192.168.35.0/24
add action=src-nat chain=srcnat comment="NAT bypass" dst-address-list=\
    VPN-destinations to-addresses=192.168.35.0/24
add action=accept chain=srcnat dst-address=192.168.36.0/24 src-address=\
    192.168.98.0/24
add action=accept chain=srcnat dst-address=192.168.36.0/24 src-address=\
    192.168.99.0/24
add action=masquerade chain=srcnat comment="Hairpin NAT " disabled=yes \
    out-interface=ether1
add action=masquerade chain=srcnat comment="Hairpin NAT" dst-address=\
    192.168.35.0 src-address=192.168.35.0
add action=accept chain=srcnat disabled=yes dst-address=192.168.40.0/24 \
    src-address=192.168.35.0/24
add action=dst-nat chain=dstnat dst-port=10050-11000 in-interface=ether5 \
    protocol=udp to-addresses=192.168.35.10 to-ports=10050-11000
add action=dst-nat chain=dstnat dst-port=35060-35061 in-interface=ether5 \
    protocol=tcp to-addresses=192.168.35.10 to-ports=35060-35061
add action=masquerade chain=srcnat comment="defconf: masquerade" \
    ipsec-policy=out,none out-interface-list=WAN
add action=masquerade chain=srcnat comment="masq. vpn traffic" src-address=\
    192.168.89.0/24
add action=masquerade chain=srcnat out-interface=ether5 src-address=\
    192.168.35.0/24
/ip firewall service-port
set sip disabled=yes ports=35060,35061
/ip ipsec identity
add peer=Russia
/ip ipsec policy
add dst-address=192.168.36.0/24 level=unique peer=Russia src-address=\
    192.168.35.0/24 tunnel=yes
add dst-address=192.168.36.0/24 level=unique peer=Russia src-address=\
    192.168.98.0/24 tunnel=yes
add dst-address=192.168.36.0/24 level=unique peer=Russia src-address=\
    192.168.99.0/24 tunnel=yes
add dst-address=0.0.0.0/0 level=unique peer=Russia src-address=\
    192.168.35.0/24 tunnel=yes
/ip route
add distance=2 gateway=192.168.98.1
/ip upnp interfaces
add interface=ether1 type=external
add interface=bridge type=internal
/ppp secret
add name=vpn
/system clock
set time-zone-name=Europe/Athens
/system identity
set name=MikroTik_35
/system scheduler
add interval=1m name="Flush SA" on-event=":if ([/ping 192.168.36.1 interface b\
    ridge count=4]<3) do={\r\
    \n  /ip ipsec installed-sa flush;\r\
    \n  :log info \"IPSEC tunnel is down: Flushing Installed SA !!!\"\r\
    \n} else={\r\
    \n\r\
    \n}" policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon \
    start-date=dec/08/2021 start-time=11:26:42
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN

As I said earlier, some sites in address list resolute in the local ISP "banned" page. How it can pass DNS queries from specific sites from the other site?

Thank you.

sindy · Tue Jun 07, 2022 7:01 pm

I don't even know the pros / cons of each case. What is the simplest way for easy troubleshoot?

An IPsec-encrypted tunnel uses headers of both IPsec and the inner tunnel, so less space is left in each packet for the payload. This is not the case if you use transport mode of IPsec and IPIP (=ipencap) as the inner tunnel, but to simplify the start, we'll reuse the existing SAs in tunnel mode as suggested earlier. "Vanilla IPsec" with policy matching has its advantages too but only for simple setups.

As I said earlier, some sites in address list resolute in the local ISP "banned" page. How it can pass DNS queries from specific sites from the other site?

After years, Mikrotik has added a possibility to use regular expression matching on fqdns to choose a DNS server.

I'll send you the scripts for both topics later.

sindy · Tue Jun 07, 2022 8:11 pm

So the bare minimum is as follows.

36:
/ip ipsec policy remove [find src-address=0.0.0.0/0 peer=Athens]
/interface ipip add name=ipip-Athens local-address=192.168.36.1 remote-address=192.168.35.1
/ip route
add dst-address=192.168.35.0/24 gateway=ipip-Athens
add gateway=ipip-Athens routing-mark=via-Athens
/ip firewall filter
set [find action=fasttrack-connection] connection-mark=no-mark
add chain=input place-before=[find action=drop chain=input in-interface-list="!LAN"] protocol=udp dst-port=53 in-interface=via-Athens
/ip firewall mangle
add chain=prerouting src-address=192.168.36.0/24 dst-address-list=VPN-destinations action=mark-routing new-routing-mark=via-Athens
add chain=forward connection-state=!new action=accept
add chain=forward in-interface=ipip-Athens action=mark-connection new-connection-mark=prevent-fasttrack
add chain=forward out-interface=ipip-Athens action=mark-connection new-connection-mark=prevent-fasttrack
/ip dhcp-server network set [find] dns-server=192.168.36.1
/ip dns static
add regexp="google\\.com\$" forward-to=192.168.35.1

35:
/ip ipsec policy remove [find dst-address=0.0.0.0/0 peer=Russia]
/interface ipip add name=ipip-Russia local-address=192.168.35.1 remote-address=192.168.36.1
/ip route
add dst-address=192.168.36.0/24 gateway=ipip-Russia
add gateway=ipip-Russia routing-mark=via-Russia
/ip firewall filter
set [find action=fasttrack-connection] connection-mark=no-mark
add chain=input place-before=[find action=drop chain=input in-interface-list="!LAN"] protocol=udp dst-port=53 in-interface=via-Russia
/ip firewall mangle
add chain=prerouting src-address=192.168.35.0/24 dst-address-list=VPN-destinations action=mark-routing new-routing-mark=via-Russia
add chain=forward connection-state=!new action=accept
add chain=forward in-interface=ipip-Athens action=mark-connection new-connection-mark=prevent-fasttrack
add chain=forward out-interface=ipip-Athens action=mark-connection new-connection-mark=prevent-fasttrack
/ip dhcp-server network set [find] dns-server=192.168.35.1
/ip dns static
add regexp="\\.[rs]u\$" forward-to=192.168.36.1

Try that and let me know the outcome. You have to let the LAN hosts re-request the IP settings via DHCP (or wait until they do that automatically once the lease time is at 1/2) so that their DNS server setting would change to the Mikrotik, so that the query forwarding could work.

tharorris · Wed Jun 08, 2022 11:25 am

Thank you for the config.

On .36 I paste your commands on the Terminal and I got errors. I try to manually input starting with the /ip/route> add gateway=ipip-Athens routing-mark=via-Athens
but there is no "routing-mark" option. Only routing Table.

This is the output of Terminal.

[admin@MikroTik_36] > /ip ipsec policy remove [find src-address=0.0.0.0/0 peer=Ath
ens]
[admin@MikroTik_36] > /interface ipip add name=ipip-Athens local-address=192.168.3
6.1 remote-address=192.168.35.1
[admin@MikroTik_36] > /ip route
[admin@MikroTik_36] /ip/route> add dst-address=192.168.35.0/24 gateway=ipip-Athens

[admin@MikroTik_36] /ip/route> add gateway=ipip-Athens routing-mark=via-Athens
expected end of command (line 1 column 25)
[admin@MikroTik_36] /ip/route> /ip firewall filter
[admin@MikroTik_36] /ip/firewall/filter> set [find action=fasttrack-connection] co
nnection-mark=no-mark
[admin@MikroTik_36] /ip/firewall/filter> add chain=input place-before=[find action
=drop chain=input in-interface-list="!LAN"] protocol=udp dst-port=53 in-interface=
via-Athens
input does not match any value of interface
[admin@MikroTik_36] /ip/firewall/filter> /ip firewall mangle
[admin@MikroTik_36] /ip/firewall/mangle> add chain=prerouting src-address=192.168.
36.0/24 dst-address-list=VPN-destinations action=mark-routing new-routing-mark=via
-Athens
input does not match any value of new-routing-mark
[admin@MikroTik_36] /ip/firewall/mangle> add chain=forward connection-state=!new a
ction=accept
[admin@MikroTik_36] /ip/firewall/mangle> add chain=forward in-interface=ipip-Athen
s action=mark-connection new-connection-mark=prevent-fasttrack
[admin@MikroTik_36] /ip/firewall/mangle> add chain=forward out-interface=ipip-Athe
ns action=mark-connection new-connection-mark=prevent-fasttrack
[admin@MikroTik_36] /ip/firewall/mangle> /ip dhcp-server network set [find] dns-se
rver=192.168.36.1
[admin@MikroTik_36] /ip/firewall/mangle> /ip dns static
[admin@MikroTik_36] /ip/dns/static> add regexp="google\\.com\$" forward-to=192.168
.35.1

Do I need to upgrade firmware to enable this option?

sindy · Wed Jun 08, 2022 11:43 am

Ah, sorry, I have noticed you are running ROS7 at one moment but then forgot about it again. There are a few differences in syntax and logics, I'll send you a "translated" version of the scripts.

sindy · Wed Jun 08, 2022 12:23 pm

Now as reading it on the PC rather than mobile, I can see that most commands succeeded. So I just put here what you have to do to fix the errors for 36:
/routing table add name=via-Athens fib
/ip route add gateway=ipip-Athens routing-table=via-Athens
/ip firewall filter add chain=input place-before=[find action=drop chain=input in-interface-list="!LAN"] protocol=udp dst-port=53 in-interface=
ipip-Athens
/ip firewall mangle add chain=prerouting src-address=192.168.36.0/24 dst-address-list=VPN-destinations action=mark-routing new-routing-mark=via
-Athens

For 35, here's the complete updated version:
/ip ipsec policy remove [find dst-address=0.0.0.0/0 peer=Russia]
/interface ipip add name=ipip-Russia local-address=192.168.35.1 remote-address=192.168.36.1
/routing table add name=via-Athens fib
/ip route
add dst-address=192.168.36.0/24 gateway=ipip-Russia
add gateway=ipip-Russia routing-table=via-Russia
/ip firewall filter
set [find action=fasttrack-connection] connection-mark=no-mark
add chain=input place-before=[find action=drop chain=input in-interface-list="!LAN"] protocol=udp dst-port=53 in-interface=ipip-Russia
/ip firewall mangle
add chain=prerouting src-address=192.168.35.0/24 dst-address-list=VPN-destinations action=mark-routing new-routing-mark=via-Russia
add chain=forward connection-state=!new action=accept
add chain=forward in-interface=ipip-Athens action=mark-connection new-connection-mark=prevent-fasttrack
add chain=forward out-interface=ipip-Athens action=mark-connection new-connection-mark=prevent-fasttrack
/ip dhcp-server network set [find] dns-server=192.168.35.1
/ip dns static
add regexp="\\.[rs]u\$" forward-to=192.168.36.1

tharorris · Wed Jun 08, 2022 12:58 pm

Thank you again!

36 is ok without errors.

On 35 I had these on Terminal.

[admin@MikroTik_35] > /ip ipsec policy remove [find dst-address=0.0.0.0/0 peer=Russia]
[admin@MikroTik_35] > /interface ipip add name=ipip-Russia local-address=192.168.35.1 remote-address=192.168.36.1
[admin@MikroTik_35] > /routing table add name=via-Athens fib
bad command name table (line 1 column 10)
[admin@MikroTik_35] > /ip route
[admin@MikroTik_35] /ip route> add dst-address=192.168.36.0/24 gateway=ipip-Russia
[admin@MikroTik_35] /ip route> add gateway=ipip-Russia routing-table=via-Russia
expected end of command (line 1 column 25)
[admin@MikroTik_35] /ip route> /ip firewall filter
[admin@MikroTik_35] /ip firewall filter> set [find action=fasttrack-connection] connection-mark=no-mark
[admin@MikroTik_35] /ip firewall filter> add chain=input place-before=[find action=drop chain=input in-interface-list="!LAN"] protocol=udp dst-port=53 in-interface=ipip-Russia
[admin@MikroTik_35] /ip firewall filter> /ip firewall mangle
[admin@MikroTik_35] /ip firewall mangle> add chain=prerouting src-address=192.168.35.0/24 dst-address-list=VPN-destinations action=mark-routing new-routing-mark=via-Russia
[admin@MikroTik_35] /ip firewall mangle> add chain=forward connection-state=!new action=accept
[admin@MikroTik_35] /ip firewall mangle> add chain=forward in-interface=ipip-Athens action=mark-connection new-connection-mark=prevent-fasttrack
input does not match any value of interface
[admin@MikroTik_35] /ip firewall mangle> add chain=forward out-interface=ipip-Athens action=mark-connection new-connection-mark=prevent-fasttrack
input does not match any value of interface
[admin@MikroTik_35] /ip firewall mangle> /ip dhcp-server network set [find] dns-server=192.168.35.1
[admin@MikroTik_35] /ip firewall mangle> /ip dns static
[admin@MikroTik_35] /ip dns static> add regexp="\\.[rs]u\$" forward-to=192.168.36.1

And then logged me out!

In IPsec i see PH2 State established but I can't login remotely. I get an error "router does not support secure connection, please enable Legacy Mode if you want to connect anyway".
I did enable Legacy mode, but no luck.

I can ask someone to go by and connect to that site with a laptop, to fix or revert this.

sindy · Wed Jun 08, 2022 1:23 pm

The errors at 35 indicate that it is running ROS 6, so the original script was the right one, except for the IPIP interface name - a copy-paste error.

So here's another try for 35 once you manage to revert it to the previous state:
/ip ipsec policy remove [find dst-address=0.0.0.0/0 peer=Russia]
/interface ipip add name=ipip-Russia local-address=192.168.35.1 remote-address=192.168.36.1
/ip route
add dst-address=192.168.36.0/24 gateway=ipip-Russia
add gateway=ipip-Russia routing-mark=via-Russia
/ip firewall filter
set [find action=fasttrack-connection] connection-mark=no-mark
add chain=input place-before=[find action=drop chain=input in-interface-list="!LAN"] protocol=udp dst-port=53 in-interface=via-Russia
/ip firewall mangle
add chain=prerouting src-address=192.168.35.0/24 dst-address-list=VPN-destinations action=mark-routing new-routing-mark=via-Russia
add chain=forward connection-state=!new action=accept
add chain=forward in-interface=ipip-Russia action=mark-connection new-connection-mark=prevent-fasttrack
add chain=forward out-interface=ipip-Russias action=mark-connection new-connection-mark=prevent-fasttrack
/ip dhcp-server network set [find] dns-server=192.168.35.1
/ip dns static
add regexp="\\.[rs]u\$" forward-to=192.168.36.1

In general, I usually set up an SSTP VPN in parallel to the basic one, so that I had a backdoor for remote management if something goes wrong.

tharorris · Wed Jun 08, 2022 3:33 pm

I understood that maybe I am in ROS6 the moment I logged out and start to think what I did wrong....

Once I get it back working, or I will upgrade it to ROS7 (just to be at the same version) and try the failed commands again or I will put the new commands you send me.

This about the SSTP VPN, is really nice if I thought about it earlier. It's a pity I don't have any Windows / Linux inside to remote login and manage device directly...

By this configuration, which sites go through IPsec and which not? How do I declare what goes through what?

Thank you so much for your time and effort! I will let you know what happens after I apply your config.

sindy · Wed Jun 08, 2022 4:38 pm

This about the SSTP VPN, is really nice if I thought about it earlier. It's a pity I don't have any Windows / Linux inside to remote login and manage device directly...

Well, if you don't have any alternative internet connection at the remote site, it doesn't make a big difference whether a configuration mistake kills the ability of the router to act as SSTP client or the connectivity of the PhC allowing the remote login.

By this configuration, which sites go through IPsec and which not? How do I declare what goes through what?

IPsec policies always win, so whatever matches a traffic selector of an IPsec policy is sent that way, and is dropped if it doesn't arrive via a Security Association linked to that policy.

The rest is routed the normal way, i.e. all packets from 192.168.3x.0/24 to internet destinations listed in the VPN-destinations list go via the IPIP tunnel because that's what the action=mark-routing rules and routing table via-Xyz say; whatever comes from the internet and gets un-src-nated to 192.168.3x.0/24 of the remote site also goes via the IPIP tunnel because that's what the route in routing table main says.

Transport packets of the IPIP tunnel do match the policy 192.168.35.0/24 <-> 192.168.36.0/24 so they are transported encrypted.

tharorris · Thu Jun 09, 2022 12:15 am

Interesting thing. I had a laptop installed in 35 subnet, because after upgrading to ROS7, it seems the configuration is somehow broken or incomplete.
The unit didn't seem to have internet because ether5 doesn't pick up an IP from the ISP equipment - either though the ISP equipment has DHCP enabled and static IP based on Mikrotik MAC address on ether5.

All day I was trying everything I could and after some frustration, in Interface List \ interface, i pressed the button "Detect Internet". I changed some options, hit Apply and then "Interface Statuses". Suddenly there was traffic in ether5, IP\Cloud picked external IP and everything worked as it should.
Curiously, I did a reboot and then it was stuck again - till I do the same process again.

I noticed it creates in IP\Address, an Address List -> Address 192.168.99.50/24, Network 192.168.99.0 and Interface ether5. That wasn't before there.
Also It creates in IP\Routes two entries.
Dst. Address 0.0.0.0/0 Gateway 192.168.99.1 Distance 1, VRF Interface ether5, Routing Table Main
Dst. Address 192.168.99.0/24 Gateway ether5 Distance 0, Routing Table Main

When I try to recreate these by myself, I can't set Distance 0 to second entry. With Distance 1 it just won't work.
I am pulling my hair, I hope I will find solution, even if I will make everything from scratch. Including Dual-WAN Failover configuration.

EDIT: I found this amazing menu called "DHCP Client" and once I saw a dynamic setting, I create one. Also I create an address list and the IP\Route is now normal.

tharorris · Thu Jun 09, 2022 1:09 am

Following your guide for .35 subnet, I paste the code in Terminal and got this result

[admin@MikroTik_35] > /ip ipsec policy remove [find dst-address=0.0.0.0/0 peer=Russia]
[admin@MikroTik_35] > /interface ipip add name=ipip-Russia local-address=192.168.35.1 remote-address=192.168.36.1
[admin@MikroTik_35] > /routing table add name=via-Athens fib
[admin@MikroTik_35] > /ip route
[admin@MikroTik_35] /ip/route> add dst-address=192.168.36.0/24 gateway=ipip-Russia
[admin@MikroTik_35] /ip/route> add gateway=ipip-Russia routing-table=via-Russia
input does not match any value of routing-table
[admin@MikroTik_35] /ip/route> /ip firewall filter
[admin@MikroTik_35] /ip/firewall/filter> set [find action=fasttrack-connection] connection-mark=no-mark
[admin@MikroTik_35] /ip/firewall/filter> add chain=input place-before=[find action=drop chain=input in-interface-list="!LAN"] protocol=udp dst-port=53 in-interface=ipip-Russia
[admin@MikroTik_35] /ip/firewall/filter> /ip firewall mangle
[admin@MikroTik_35] /ip/firewall/mangle> add chain=prerouting src-address=192.168.35.0/24 dst-address-list=VPN-destinations action=mark-routing new-routing-mark=via-Russia
input does not match any value of new-routing-mark
[admin@MikroTik_35] /ip/firewall/mangle> add chain=forward connection-state=!new action=accept
[admin@MikroTik_35] /ip/firewall/mangle> add chain=forward in-interface=ipip-Athens action=mark-connection new-connection-mark=prevent-fasttrack
input does not match any value of interface
[admin@MikroTik_35] /ip/firewall/mangle> add chain=forward out-interface=ipip-Athens action=mark-connection new-connection-mark=prevent-fasttrack
input does not match any value of interface
[admin@MikroTik_35] /ip/firewall/mangle> /ip dhcp-server network set [find] dns-server=192.168.35.1
[admin@MikroTik_35] /ip/firewall/mangle> /ip dns static
[admin@MikroTik_35] /ip/dns/static> add regexp="\\.[rs]u\$" forward-to=192.168.36.1

sindy · Thu Jun 09, 2022 8:04 am

Yes, I have fixed the copy-paste errors in the latest version of my script for V6 but not in the previous one for V7.

So here's a script to fix what has failed on 35:
/routing table
remove [find name=via-Athens]
add name=via-Russia fib
/ip route add gateway=ipip-Russia routing-table=via-Russia
/ip firewall mangle add chain=prerouting src-address=192.168.35.0/24 dst-address-list=VPN-destinations action=mark-routing new-routing-mark=via-Russia
/ip firewall mangle add chain=forward out-interface=ipip-Russia action=mark-connection new-connection-mark=prevent-fasttrack

tharorris · Thu Jun 09, 2022 9:04 am

Thank you, it worked on one website (wtfismyip.com), thought the query was slow, but I got response from Athens instead of 36!

I added Facebook and LinkedIn to address list (36), which are blocked and they are still not working.
Rebooted both units but nothing changed.

The camera inside site 35, is recording at site 36 and after the changes, it's getting disconnected / connected all the time, every 2-5 minutes.
Also from time to time I can't reach Mikrotik 35 with Winbox.

EDIT: I add one website which I can reach from 36, but the images are not visible because the image hosting block this country. After the change I can still reach it but again without images. So it doesn't follow the address list rule.
Also google.com is not working!

sindy · Thu Jun 09, 2022 11:08 am

The thing is that on nowadays' internet, what appears to be a single web page is loaded from tens to hundreds of individual servers on fqdns and IP addresses with no apparent relationship to the original URL (e.g. google uses domains 1e100.net, akamai.net, and several other ones).

So unless you want to spend hours or even days building lists of destination addresses and hostnames to be resolved using the DNS in Athens, while the addresses will change almost on daily basis as the content providers move their stuff from one datacenter to another, maybe it is better to introduce another address list, VPN-users, and add the addresses of LAN hosts that access one of the addresses on the VPN-destinations to this list for a certain period of time (minutes) so that all traffic of these devices would go via the WAN in the other country during that period?

Regarding the interruptions of the camera connection - there were other changes done than just the configuration (upgrade of Athens to 7.x). So to check, I'd suggest to disable the mangle rules assigning the routing-mark, as doing so is enough to effectively deactivate the rest of the "other country WAN" stuff, and see whether that makes the camera start behaving normally again.

tharorris · Thu Jun 09, 2022 11:41 am

That is true about webpages. For the specific one with images, I inspect the code and found around 13 external sites - which I add them to address-list. The website was behaving better but there are at least 20+ websites I want to add to the list.
Let's say I devote a weekend to add them all (crazy right?) and on Monday, a sysadmin decide to change the provider of... for example image host. Then I have to go and update the list.... every time something changes...

I kept backups of new configuration for both Mikrotik devices, and rolled back to pre-VPN rules. I will try to create a new config for both sites, command by command, add Dual WAN failover support to both sites for redundancy (all guides are for ROS6...) and then I will try to use your commands again.

On another firewall appliances, I create a rule that just say "from all external interfaces, first interface to reply, win the election and get routed from ether1 or 2 or 3". - election can be Round-Robin, Failover, hybrid solutions.
For websites like the one with images, we just blacklist the whole website from the Interface that we don't want it to get routed, so it naturally goes to the next one.
This above is for interfaces. For IPsec route, we can do it only with SD-Wan hard rules.

I will not give up on Mikrotik, since with your engagement in this problem, I start to understand differences from ROS6 to ROS7 and the usual problems when something just won't work.

Thank you so much for your time and your help!

sindy · Thu Jun 09, 2022 12:27 pm

On another firewall appliances

This is the key. Mikrotik is not a firewall in the sense the word is used today. It is a router with L3 firewalling capabilities. The address lists resolving complete fqdns to sets of addresses is the maximum it can do above that.

I create a rule that just say "from all external interfaces, first interface to reply, win the election and get routed from ether1 or 2 or 3". - election can be Round-Robin, Failover, hybrid solutions.
For websites like the one with images, we just blacklist the whole website from the Interface that we don't want it to get routed, so it naturally goes to the next one.

To blacklist the whole website completely is easy - if you prevent the browser from downloading the basic html document, it obviously cannot load anything else either. But to ban it from using a particular WAN is equally complicated as forcing it to use a particular WAN if you can only use the weak tools we have discussed until now.

The current application layer firewalls can benefit from the fact that they process the contents of the html documents they forward, so they can dynamically adjust the routing - if a web page refers to www.xyz.tv, they resolve that to an IP address, and when the address that downloaded the basic html document accesses the www.xyz.tv address, they apply the same routing policy they did for its download of the initial document. They can do this because they have the tools to read encrypted traffic, which is essential for malware detection, and at the same time it is a security nightmare - you put an unlimited trust into the vendor of such a firewall.

This kind of processing requires much more CPU power, and thus electric power, than the L3 firewall processing. So not a task for a passively cooled plastic box any more.

tharorris · Thu Jun 09, 2022 9:34 pm

I totally agree with your approach about power and fan less units. However I didn't know that when I bought them and the Greek retail store told me to buy these because it's 4 cores, Gigabit ports and hardware IPsec encryption.
One week ago I read about the ZeroTier support for Mikrotik units and I was amazed! Unfortunately my units are MMIPS and do not support it.
But these Hex S work already two years non stop with little to none hiccups and they are a very nice way to dive into the Mikrotik world.
Another thing which I understood yesterday is that upgrading from ROS6 to ROS7, break the configuration.
For example till now I am searching why I don't receive phone calls from my PBX located in 35 site, while I am inside 36 site. Port forward work when I switch my mobile to 4G but while I am inside 36 subnet (which is connected with ipsec vpn) the PBX client won't connect.

Other than these issues, pretty amazing little boxes.

sindy · Thu Jun 09, 2022 10:22 pm

But these Hex S work already two years non stop with little to none hiccups

and even better, if they eventually stop working, chances are really high that replacing the power adaptor will be enough to make them work for another few years.

and they are a very nice way to dive into the Mikrotik world.

What I like most about the "Mikrotik world" is that both the tiniest and the most powerful devices all have the same functionality, only the throughput differs.

Another thing which I understood yesterday is that upgrading from ROS6 to ROS7, break the configuration.
For example till now I am searching why I don't receive phone calls from my PBX located in 35 site, while I am inside 36 site. Port forward work when I switch my mobile to 4G but while I am inside 36 subnet (which is connected with ipsec vpn) the PBX client won't connect.

Yes, the subtle changes between 6.x and the individual versions of 7.x can cause a lot of surprises if you use more complicated configuratons. But in this particular case (VoIP phone not working any more), your configuration is not that complex to be affected, so I'd assume there is something the phone or the PBX do not like about the LAN interface of the adjacent hEX, not something related to the tunnel.

So I'd suggest to sniff at both routers to find out what gets lost and where. Since you mention a mobile phone and a PBX in a single sentence, I wonder what VoIP phone app do you use and how does it connect to the PBX while using 4G?

tharorris · Fri Jun 10, 2022 1:02 am

The Hex S in Athens site works without power adapter. I have a PoE switch and swapped ether1 for ether5.
Ether5 is primary WAN and ether4 will be the 4G failover. The 4G is connected already, I just need to find how to implement it in ROS7.
As you can see from the configuration, ether1, 2 and 3 are bridged. Ether1 get LAN and power from the PoE switch.
The PBX is a Grandstream UCM and I can connect from outside using TLS certificates. The mobile phone app is Bria Mobile - but you can use pretty much any VoIP app. I use Bria because it support push notification when I have an incoming call - so the app can "wake up" from sleep.
The app try to connect to the DDNS and that's why I think it's a hairpin NAT issue. I believe when I visit the Athens site, it will also not work from inside LAN.
With ROS6 in Athens, it was working perfectly in both sites.

Edit: on Hairpin NAT I replaced In Interface "ether5" to in Interface List "WAN" and after a reboot it worked.
IPsec stuck on reboot so I disabled ether4 for now, till I find how to apply the Dual WAN failover configuration.

tharorris · Fri Jun 10, 2022 11:45 am

I found a guide for Dual-WAN but I don't believe it's failover... I see traffic passing through ether4.
The problem is that if I don't disable ether4, the port forward won't work (outside or on the .36 site). Can you help me with the config please?

/interface bridge
add admin-mac=48:8F:5A:82:D3:17 auto-mac=no comment=defconf name=bridge
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/interface lte apn
set [ find default=yes ] ip-type=ipv4 use-network-apn=no
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip ipsec peer
add address=HIDDEN exchange-mode=ike2 local-address=192.168.99.50 \
    name=Russia
/ip ipsec profile
set [ find default=yes ] dh-group=modp2048 enc-algorithm=aes-256 \
    hash-algorithm=sha256
/ip ipsec proposal
set [ find default=yes ] auth-algorithms=sha256 enc-algorithms=aes-256-cbc \
    lifetime=8h pfs-group=modp2048
/ip pool
add name=dhcp ranges=192.168.35.200-192.168.35.254
add name=vpn ranges=192.168.89.2-192.168.89.255
/ip dhcp-server
add address-pool=dhcp interface=bridge name=defconf
/port
set 0 name=serial0
/ppp profile
set *FFFFFFFE local-address=192.168.89.1 remote-address=vpn
/routing bgp template
set default disabled=no output.network=bgp-networks
/routing ospf instance
add disabled=no name=default-v2
/routing ospf area
add disabled=yes instance=default-v2 name=backbone-v2
/routing table
add disabled=no fib name=to-ISP1
add disabled=no fib name=to-ISP2
/interface bridge port
add bridge=bridge comment=defconf ingress-filtering=no interface=ether2
add bridge=bridge comment=defconf ingress-filtering=no interface=ether3
add bridge=bridge comment=defconf ingress-filtering=no interface=ether1
add bridge=bridge comment=defconf ingress-filtering=no interface=sfp1
/ip neighbor discovery-settings
set discover-interface-list=LAN
/ipv6 settings
set disable-ipv6=yes max-neighbor-entries=8192
/interface detect-internet
set detect-interface-list=all
/interface l2tp-server server
set enabled=yes use-ipsec=yes
/interface list member
add interface=ether1 list=LAN
add interface=ether2 list=LAN
add interface=ether3 list=LAN
add interface=ether4 list=WAN
add interface=ether5 list=WAN
add interface=sfp1 list=LAN
/interface ovpn-server server
set auth=sha1,md5
/interface pptp-server server
# PPTP connections are considered unsafe, it is suggested to use a more modern VPN protocol instead
set enabled=yes
/interface sstp-server server
set default-profile=default-encryption enabled=yes
/ip address
add address=192.168.35.1/24 comment=defconf interface=bridge network=\
    192.168.35.0
add address=192.168.99.50/24 interface=ether5 network=192.168.99.0
add address=192.168.98.50/24 interface=ether4 network=192.168.98.0
/ip cloud
set ddns-enabled=yes ddns-update-interval=5m
/ip dhcp-client
add add-default-route=no interface=ether5
add add-default-route=no interface=ether4
/ip dhcp-server lease
add address=192.168.35.4 client-id=1:fc:ec:da:37:42:84 comment=\
    "Ubiquity Access Point" mac-address=FC:EC:DA:37:42:84 server=defconf
add address=192.168.35.5 client-id=1:44:b2:95:31:3a:48 comment=\
    "Camera Entrance" mac-address=44:B2:95:31:3A:48 server=defconf
add address=192.168.35.221 client-id=1:ec:5c:68:c0:62:cb comment=\
    "Sony TV Wireless" mac-address=EC:5C:68:C0:62:CB server=defconf
add address=192.168.35.3 client-id=1:4c:78:97:0:9a:f4 comment=\
    "Home Security Alarm" mac-address=4C:78:97:00:9A:F4 server=defconf
add address=192.168.35.221 client-id=1:cc:98:8b:e6:f3:cb comment=\
    "Sony TV Wired" disabled=yes mac-address=CC:98:8B:E6:F3:CB server=defconf
/ip dhcp-server network
add address=192.168.35.0/24 comment=defconf gateway=192.168.35.1 netmask=24
/ip dns
set allow-remote-requests=yes servers=1.1.1.1,8.8.8.8
/ip dns static
add address=192.168.35.1 comment=defconf name=router.lan
/ip firewall address-list
add address=wtfismyip.com list=VPN-destinations
/ip firewall filter
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=accept chain=input comment="Mikrotik Management" dst-port=8291 \
    protocol=tcp src-address=192.168.40.0/24
add action=accept chain=input dst-port=8291 protocol=tcp src-address=\
    192.168.89.0/24
add action=accept chain=input dst-port=8291 protocol=tcp src-address=\
    192.168.36.0/24
add action=accept chain=input dst-port=8291 protocol=tcp src-address=\
    192.168.35.0/24
add action=accept chain=forward comment="defconf: accept in ipsec policy" \
    ipsec-policy=in,ipsec
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment=\
    "defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=accept chain=forward comment="defconf: accept out ipsec policy" \
    ipsec-policy=out,ipsec
add action=accept chain=forward comment=\
    "defconf: accept established,related, untracked" connection-state=\
    established,related,untracked
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" \
    connection-state=established,related hw-offload=yes
add action=drop chain=input comment="defconf: drop all not coming from LAN" \
    in-interface-list=!LAN
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=drop chain=forward comment="Block Camera to Internet" src-address=\
    192.168.35.5
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid
add action=drop chain=forward comment=\
    "defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat \
    connection-state=new in-interface-list=WAN
/ip firewall mangle
add action=accept chain=prerouting comment=Accept dst-address=192.168.99.0/24
add action=accept chain=prerouting dst-address=192.168.98.0/24
add action=accept chain=prerouting dst-address=192.168.35.0/24
add action=mark-connection chain=input comment=Input in-interface=ether5 \
    new-connection-mark=ISP1 passthrough=yes
add action=mark-connection chain=input in-interface=ether4 \
    new-connection-mark=ISP2 passthrough=yes
add action=mark-connection chain=prerouting comment=Mark-con in-interface=\
    ether5 new-connection-mark=ISP1 passthrough=yes
add action=mark-connection chain=prerouting in-interface=ether4 \
    new-connection-mark=ISP2 passthrough=yes
add action=mark-connection chain=prerouting comment=PCC dst-address-type=\
    local in-interface=ether5 new-connection-mark=ISP1 passthrough=yes \
    per-connection-classifier=both-addresses:2/0
add action=mark-connection chain=prerouting dst-address-type=local \
    in-interface=ether4 new-connection-mark=ISP2 passthrough=yes \
    per-connection-classifier=both-addresses:2/1
add action=mark-routing chain=output comment=Output connection-mark=ISP1 \
    new-routing-mark=to-ISP1 passthrough=yes
add action=mark-routing chain=output connection-mark=ISP2 new-routing-mark=\
    to-ISP2 passthrough=yes
add action=mark-routing chain=prerouting comment=Mark-Route connection-mark=\
    ISP1 in-interface=bridge new-routing-mark=to-ISP1 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=ISP2 in-interface=\
    bridge new-routing-mark=to-ISP2 passthrough=yes
/ip firewall nat
add action=dst-nat chain=dstnat comment=Port-Forward dst-port=10050-11000 \
    in-interface=ether5 protocol=udp to-addresses=192.168.35.10 to-ports=\
    10050-11000
add action=dst-nat chain=dstnat dst-port=35060-35061 in-interface=ether5 \
    protocol=tcp to-addresses=192.168.35.10 to-ports=35060-35061
add action=src-nat chain=srcnat comment="NAT bypass" disabled=yes \
    to-addresses=192.168.35.0/24
add action=accept chain=srcnat comment="NAT bypass" dst-address=\
    192.168.36.0/24 src-address=192.168.35.0/24
add action=accept chain=srcnat dst-address=192.168.36.0/24 src-address=\
    192.168.98.0/24
add action=accept chain=srcnat dst-address=192.168.36.0/24 src-address=\
    192.168.99.0/24
add action=masquerade chain=srcnat comment="Hairpin NAT " out-interface-list=\
    WAN
add action=masquerade chain=srcnat dst-address=192.168.35.0 src-address=\
    192.168.35.0
add action=masquerade chain=srcnat comment="defconf: masquerade" \
    ipsec-policy=out,none out-interface-list=WAN
add action=masquerade chain=srcnat comment="masq. vpn traffic" src-address=\
    192.168.89.0/24
add action=masquerade chain=srcnat out-interface-list=WAN src-address=\
    192.168.35.0/24
add action=masquerade chain=srcnat
/ip firewall service-port
set sip disabled=yes ports=35060,35061
/ip ipsec identity
add peer=Russia
/ip ipsec policy
add dst-address=192.168.36.0/24 level=unique peer=Russia src-address=\
    192.168.35.0/24 tunnel=yes
add dst-address=192.168.36.0/24 level=unique peer=Russia src-address=\
    192.168.98.0/24 tunnel=yes
add dst-address=192.168.36.0/24 level=unique peer=Russia src-address=\
    192.168.99.0/24 tunnel=yes
/ip route
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.98.1 \
    pref-src=0.0.0.0 routing-table=main scope=30 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.99.1 \
    pref-src=0.0.0.0 routing-table=main scope=30 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.99.1 \
    pref-src=0.0.0.0 routing-table=to-ISP1 scope=30 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.98.1 \
    pref-src=0.0.0.0 routing-table=to-ISP2 scope=30 suppress-hw-offload=no \
    target-scope=10
/ip upnp interfaces
add interface=ether1 type=external
add interface=bridge type=internal
/ppp secret
add name=vpn
/system clock
set time-zone-name=Europe/Athens
/system identity
set name=MikroTik_35
/system scheduler
add interval=1m name="Flush SA" on-event=":if ([/ping 192.168.36.1 interface b\
    ridge count=4]<3) do={\r\
    \n  /ip ipsec installed-sa flush;\r\
    \n  :log info \"IPSEC tunnel is down: Flushing Installed SA !!!\"\r\
    \n} else={\r\
    \n\r\
    \n}" policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon \
    start-date=dec/08/2021 start-time=11:26:42
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN

Thank you.

sindy · Fri Jun 10, 2022 12:48 pm

Yes but later in the afternoon.

sindy · Fri Jun 10, 2022 5:56 pm

Your current setup doesn't monitor transparency of the WAN all the way to the internet, which means that in order that the routing would fail over from ISP 1 to ISP 2, the ISP 1 interface must go physically down so that routes via that interface would become inactive. When this happens, there is no active route left in the table to-ISP1, so the routing uses table main as a backup.

See this on how to use a functionality originally intended for a different purpose for a scriptless monitoring of uplink transparency.
The two rules matching on per-connection-classifier, whose presumable purpose is to distribute the generic LAN->WAN traffic evenly between the two WANs, have currently no effect - as they match on WAN in-interfaces, and as each of them matches on another one, each of them just re-assigns the same connection-mark like one of those two right before them for matching connections. If their intended purpose is indeed to distribute the generic traffic, they have to match on in-interface=bridge instead.
The problem is that if I don't disable ether4, the port forward won't work (outside or on the .36 site).
Here I would assume that it's because when updating the DDNS (the "ip cloud" functionality), RouterOS has sent the update packet via ether4, so the xxxxxxxx.sn.mynetname.net resolves to the public IP behind ether4, so the phone (or more likely the cloud server of Bria) try to register to that address.

To make RouterOS always send the DDNS update via ether5, you have to add a mangle rule to chain output before those translating connection-mark to routing-mark:
chain=prerouting protocol=udp dst-port=15252 action=mark-connection new-connection-mark=ISP1 passthrough=yes
I use Bria because it support push notification when I have an incoming call - so the app can "wake up" from sleep.
...
With ROS6 in Athens, it was working perfectly in both sites.
In order that push notifications about incoming calls worked, the phone app must either register via an external server operated by the app developer that receives the INVITE and sends the push notification via a link to Google/Apple infrastructure before forwarding the INVITE it the phone, or it must provide an https link with some unique identifier in it that the PBX uses, before sending the INVITE, to ask the developer's server to send the push notification to the phone. To date I don't know about any VoIP app using the latter approach any more. So I'd assume no matter whether the phone is connected to WiFi in Athens or in Russia, or to LTE, the registrations come from the same IP of the external server.

The reason why it worked in ROS v6 and doesn't in v7 may be related to different treatment of routes with identical dst-address and distance in the same routing table and with different gateways. In v6, just one of such routes becomes active, and I don't know the rules determining which one it will be; in v7, all of them are active and used in a round-robin manner (ECMP).

tharorris · Mon Jun 13, 2022 2:18 am

Hello again,

1) Getting interface physically down will not help much in case of service getting down. I will try this guide and I will let you know.
What is the difference between script-less and scripted actions? Are scripts considered not upgrade proof with versions changing? Resource allocation? Peace of mind?

2) I changed it to in-interface=bridge and didn't find any difference - I will check it when I will have more time and available computer there.

3) It's not DDNS. The moment I bring down ether4, port-forward starts working and DDNS is not updated yet. Even I force update it, public IP is still the one from ether5.
I want to have ether5 as default internet gateway and DDNS record. When ether5 failes to get internet, I want it to switch to ether4 as failover and get DDNS from there - so all devices connect there. My PCC configuration is just temporary - and wrong!

4) I have continuous PUSH registration, which mean the push servers are always connected to my PBX and they invite the mobile app to accept the call - when a call is taking place. I believe it's not Bria's fault, I put my money on the port forward configuration. How it was possible previously it was working on ROS6 and not in ROS7? I have to find what is wrong. Maybe ether4 when enabled, has priority over ether5 somehow...

sindy · Mon Jun 13, 2022 8:20 am

1) Getting interface physically down will not help much in case of service getting down.

I'm saying the same, I was just trying to explain why you currently had to disable ether4 so that something happened.

What is the difference between script-less and scripted actions? ...

Sometimes indeed scripts need patching to keep working after an upgrade, but it's rare. Aside of many people being scared of scripting, even those who are not often don't want to spend much time learning YAFSL and debugging it. Plus the article by @chupaka is kind of the de facto standard implementation of failover. So it only makes sense to script your own solution if you need to check the uplink transparency more frequenty (or less frequently, for uplinks charged per traffic volume) than 6 times a minute.

2) I changed it to in-interface=bridge and didn't find any difference - I will check it when I will have more time and available computer there.

You can see any effect only for newly established connections initiated from the LAN side. If there is no computer or other device that would establish temporary connections, there is no way how the per-connection-classifier could distribute the connections between the WANs.

3), 4) It's not DDNS. The moment I bring down ether4, port-forward starts working and DDNS is not updated yet... How it was possible previously it was working on ROS6 and not in ROS7? ...

I've missed two points. One is that you haven't disabled fasttracking, which prevents any mangle rules from working for forwarded traffic (i.e. other than the one of the router itself). Once you disable the action=fasttrack-connection rule, newly created connections will not get fasttracked any more, but existing ones will stay fasttracked, so you'll have to use /ip/firewall/connection/remove [find fasttrack] to clean up the connection tracking table. Another one is only important in a specific situation - you set the connection-mark no matter what in PCC rules, which is not correct given that you also set it up as the response pakets arrive via WAN, which is a way to prevent connections that got established while their preferred WAN was not working from breaking once the preferred WAN starts working again. So you should add a connection-mark=no-mark match condition to the PCC rules.
The only reason why it worked in v6 and doesn't in v7 I can imagine is that routing cache was saving it by routing all packets belonging to the same connection the same way like the first one. Routing cache doesn't exist any more in v7 because it has been removed from the underlying linux kernel.

tharorris · Fri Jun 17, 2022 7:57 am

Hello Sindy,

I erased the configuration and created a new one, command by command. Also cleaned up some old rules and now it works as a charm.

Thank you for your time and effort helping me!

(I already kept history of backups in order to not blow it again)

Who is online