As someone think it will be interesting, here is my configuration.
it’s a quite long post, but configuration is complex.
2 hours just to try explaining … I lost count for the hours I spent to make it function!
###############################################
The plan
###############################################
10.1.1.0/24 my local lan network
10.1.2.0/24 my local dmz network
10.2.1.0/24 SITE_2 network
10.3.1.0/24 SITE_3 network
10.4.1.0/24 SITE_4 network
10.5.1.0/24 SITE_5 network
10.6.1.0/16 SITE_6 - main network
10.6.2.0/16 SITE_6 - secondary network
my public addresses:
1.1.1.32/28 my public address class
1.1.1.33 my external router
1.1.1.34 address used for ipsec
… other adresses for services
1.1.1.44 public address of my first routerboard
1.1.1.45 public address of my second routerboard
peers public addresses
2.2.2.2 SITE_2
3.3.3.3 SITE_3
4.4.4.4 SITE_4
5.5.5.5 SITE_5
6.6.6.6 SITE_6
/interface vrrp
# I used a specific vrrp interface only for IPSEC. I gave it priority 255 (so each router thinks always
# to be the the master and starts that interface immediatly when enabled. I use scripts to disable or enable it.
# I use vrid (see script) for synchronizing going master and slave of interesting interfaces depending on one
# of them, that I linked to scripts. ... if they are executed when they should ... (see post ______________)
# anyway, I decided that if WAN becomes master makes two things: increase priority of other interfaces
add interface=LAN name=LAN-V1 vrid=1
add interface=DMZ name=DMZ-V1 vrid=1
add interface=WAN name=WAN-V1 vrid=1 on-backup="/system script run IPSEC-shutdown" on-master="/system script run IPSEC-restart"
add interface=WAN name=WAN-V2-IPSEC vrid=2 priority=255
/ip address
# these change between the two routerboards.
# The second has internally 10.1.1.252 as main address for LAN, 10.1.2.252 for DMZ and 1.1.1.45 on external side
add address=10.1.1.251/24 interface=LAN
add address=10.1.2.251/24 interface=DMZ
add address=1.1.1.44/28 interface=WAN
# these are the same on both routerborads.
add address=10.1.1.2/32 interface=LAN-V1 network=10.1.1.2
add address=10.1.2.2/32 interface=DMZ-V1 network=10.1.2.2
add address=1.1.1.36/32 interface=WAN-V1 network=1.1.1.36
add address=1.1.1.37/32 interface=WAN-V1 network=1.1.1.37
add address=1.1.1.39/32 interface=WAN-V1 network=1.1.1.39
add address=1.1.1.38/32 interface=WAN-V1 network=1.1.1.38
# This is the clue: this interface, specifically for ipsec, is out of standard:
# the /28 mask makes it routeble, and I need it to be so. See on routing: I have the route with lower
# priority with this as "preferred address": it was the only way to have ipsec packest outgoing with the right address
# but when I disable the interface that route became inactive
add address=1.1.1.34/28 comment=IPSEC-ENDPOINT interface=WAN-V2-IPSEC
# Firewall: the most is self-explaining ...
/ip firewall address-list
add address=2.2.2.2 comment=SITE_2 list=IPSEC-peers
add address=3.3.3.3 comment=SITE_3 list=IPSEC-peers
add address=4.4.4.4 comment=SITE_4 list=IPSEC-peers
add address=5.5.5.5 comment=SITE_5 list=IPSEC-peers
add address=6.6.6.6 comment=SITE_6 list=IPSEC-peers
add address=10.2.1.0/24 comment=SITE_2 list=IPSEC-networks
add address=10.3.1.0/24 comment=SITE_3 list=IPSEC-networks
add address=10.4.1.0/24 comment=SITE_4 list=IPSEC-networks
add address=10.5.1.0/24 comment=SITE_5 list=IPSEC-networks
add address=10.6.1.0/24 comment=SITE_6 list=IPSEC-networks
add address=10.6.2.0/24 comment=SITE_6 list=IPSEC-networks
# I had to change mtu only for some of the remote networks ... the ones not behind a cisco
add address=10.2.1.0/24 comment=" SITE_2 " list=IPSEC-mangle-MTU
add address=10.3.1.0/24 comment=" SITE_3" list=IPSEC-mangle-MTU
add address=10.4.1.0/24 comment=" SITE_4" list=IPSEC-mangle-MTU
# I do not post other firewall configuration, but just for better reading:
add address=10.2.1.0/24 comment=SITE_2 list=INT-LAN-SITE_2
add address=10.3.1.0/24 comment=SITE_3 list=INT-LAN-SITE_3
add address=10.4.1.0/24 comment=SITE_4 list=INT-LAN-SITE_4
add address=10.6.1.0/24 comment=SITE_6 list=INT-LAN-SITE_6
add address=10.6.2.0/24 comment=SITE_6 list=INT-LAN-SITE_6
/ip firewall filter
# just remember that action=accept is the default in export compact
add chain=input comment="vrrp accept !" protocol=vrrp
add chain=output comment="vrrp accept !" protocol=vrrp
add chain=input comment="INPUT ipsec peers" dst-port=500 protocol=udp src-address-list=IPSEC-peers
add chain=input comment="INPUT ipsec peers" protocol=ipsec-ah src-address-list=IPSEC-peers
add chain=input comment="INPUT ipsec peers" protocol=ipsec-esp src-address-list=IPSEC-peers
add chain=output comment="OUTPUT ipsec peers" dst-port=500 protocol=udp src-address-list=ipsec-peers
add chain=output comment="OUTPUT ipsec peers" protocol=ipsec-ah src-address-list=ipsec-peers
add chain=output comment="OUTPUT ipsec peers" protocol=ipsec-esp src-address-list=ipsec-peers
/ip firewall mangle
# change MTU on some IPSEC connections
add action=change-mss chain=forward new-mss=1300 protocol=tcp src-address-list=IPSEC-mangle-MTU tcp-flags=syn tcp-mss=!0-1300
add action=change-mss chain=forward dst-address-list=IPSEC-mangle-MTU new-mss=1300 protocol=tcp tcp-flags=syn tcp-mss=!0-1300
/ip firewall nat
add chain=srcnat comment="VRRP - don't nat!" protocol=vrrp
add chain=dstnat comment="VRRP - don't nat!" protocol=vrrp
add chain=srcnat comment="IPSEC - don't nat addresses" dst-address-list=IPSEC-networks src-address-list=IPSEC-networks
add action=masquerade chain=srcnat out-interface=WAN-V2-IPSEC
/ip ipsec proposal
set [ find default=yes ] enc-algorithms=aes-256 lifetime=8h
add auth-algorithms=md5 lifetime=1h name=3DES-MD5
add enc-algorithms=aes-256 lifetime=1h name=AES256-SHA1
/ip ipsec peer
# I needed to be agressive on most of the configurations.
# Cisco for example don't like so much that the peer changes.
# In real life aggressive negotiation jst works... but I plan to insert certificates for better security.
# oh, yes, jou know... a string of the same number is always the better password :-)
add address=2.2.2.2/32 comment=" FRANCE" dpd-interval=5s dpd-maximum-failures=2 exchange-mode=aggressive lifetime=8h nat-traversal=yes secret=22222222
add address=3.3.3.3/32 comment=" SITE_3" dpd-interval=5s dpd-maximum-failures=2 enc-algorithm=aes-256 hash-algorithm=sha1 lifetime=8h secret=33333333
add address=4.4.4.4/32 comment=" SITE_4" dpd-interval=5s dpd-maximum-failures=2 exchange-mode=aggressive lifetime=8h secret=44444444
add address=5.5.5.5/32 comment=" SITE_5" dpd-interval=disable-dpd dpd-maximum-failures=2 lifebytes=4608000 lifetime=8h secret=55555555
add address=6.6.6.6/32 comment=" SITE_6" dpd-interval=5s dpd-maximum-failures=2 exchange-mode=aggressive lifebytes=4608000 lifetime=8h secret=66666666
/ip ipsec policy
add comment="DMZ - SITE_6 2" dst-address=10.6.2.0/24 level=unique priority=4 proposal=AES256-SHA1 sa-dst-address=6.6.6.6 sa-src-address=1.1.1.34 src-address=192.168.250.0/24 tunnel=yes
add comment="DMZ - SITE_6 1" dst-address=10.6.1.0/24 level=unique priority=5 proposal=AES256-SHA1 sa-dst-address=6.6.6.6 sa-src-address=1.1.1.34 src-address=192.168.250.0/24 tunnel=yes
add comment="LAN - SITE_6 2" dst-address=10.6.2.0/24 level=unique proposal=AES256-SHA1 sa-dst-address=6.6.6.6 sa-src-address=1.1.1.34 src-address=10.1.1.0/24 tunnel=yes
add comment="LAN - SITE_6 1" dst-address=10.6.1.0/24 level=unique priority=1 proposal=AES256-SHA1 sa-dst-address=6.6.6.6 sa-src-address=1.1.1.34 src-address=10.1.1.0/24 tunnel=yes
add comment="DMZ - SITE_4" dst-address=10.4.1.0/24 priority=4 proposal=AES256-SHA1 sa-dst-address=217.40.239.148 sa-src-address=1.1.1.34 src-address=192.168.250.0/24 tunnel=yes
add comment="LAN - SITE_4" dst-address=10.4.1.0/24 priority=4 proposal=AES256-SHA1 sa-dst-address=217.40.239.148 sa-src-address=1.1.1.34 src-address=10.1.1.0/24 tunnel=yes
add comment="LAN - SITE_2 " dst-address=10.2.1.0/24 level=unique priority=1 proposal=3DES-MD5 sa-dst-address=2.2.2.2 sa-src-address=1.1.1.34 src-address=10.1.1.0/24 tunnel=yes
add comment="DMZ - SITE_2 " dst-address=10.2.1.0/24 level=unique priority=1 proposal=3DES-MD5 sa-dst-address=2.2.2.2 sa-src-address=1.1.1.34 src-address=192.168.250.0/24 tunnel=yes
add comment="LAN - SITE_3" dst-address=10.3.1.0/24 level=unique priority=1 proposal=AES256-SHA1 sa-dst-address=3.3.3.3 sa-src-address=1.1.1.34 src-address=10.1.1.0/24 tunnel=yes
add comment="DMZ - SITE_3" dst-address=10.3.1.0/24 level=unique priority=1 proposal=AES256-SHA1 sa-dst-address=3.3.3.3 sa-src-address=1.1.1.34 src-address=192.168.250.0/24 tunnel=yes
add comment="LAN - SITE_5" dst-address=10.5.1.0/24 level=unique priority=1 proposal=AES256-SHA1 sa-dst-address=5.5.5.5 sa-src-address=1.1.1.34 src-address=10.1.1.0/24 tunnel=yes
add comment="DMZ - SITE_5" dst-address=10.5.1.0/24 level=unique priority=1 proposal=AES256-SHA1 sa-dst-address=5.5.5.5 sa-src-address=1.1.1.34 src-address=192.168.250.0/24 tunnel=yes
/ip route
# the first one becomes active when the WAN-V2-IPSEC interface is on and takes priority.
# this causes all the outgoing traffic to have this address by default,
# but when I use vrrp I rarely unse "masquerade", I prefer specific srcnat with the right external vrrp address
add distance=1 gateway=1.1.1.33 pref-src=1.1.1.34
add distance=5 gateway=1.1.1.33
/system scheduler
# don't ask me why, but phase 2 key renegotiation with cisco is always a problem.
# Better to have a long lifetime and restart every day when no-one is wotking
# ... but only if this routerboard is the master!
add interval=1d name="Restart ipsec nightly" on-event="/system script run IPSEC-ifmaster-restart" policy=ftp,reboot,read,write,policy,test,winbox,password,sniff,sensitive,api start-date=nov/01/1980 start-time=05:00:00
/system script
#... I'll explain them later
/tool netwatch
# just to confirm everything is working, have a host to ping in every net
add comment="IPSEC SITE_4" host=10.4.1.10 interval=5s timeout=2s
add comment="IPSEC SITE_2 " host=10.2.1.20 interval=5s timeout=2s
add comment="IPSEC SITE_3" host=10.3.1.10 interval=5s timeout=2s
add comment="IPSEC SITE_6 - 2" host=10.6.2.8 interval=5s timeout=2s
add comment="IPSEC SITE_6" host=10.6.1.10 interval=5s timeout=2s
add comment="IPSEC SITE_5" host=10.5.1.140 interval=5s timeout=2s
############################################
The Scripts:
############################################
“IPSEC-restart”
# actually is either "IPSEC-start" and re-start: it disables peers, policies and the VRRP-IPSEC interface
# and then enables them after ten seconds... a time acceptable to have tunnels going on the other router
# I use this variable to define if this router has the role of ipsec master
:global IPSECMaster true
:log info "IPSEC - shutdown before restart"
/ip ipsec peer disable [ /ip ipsec peer find ]
/ip ipsec policy disable [ /ip ipsec policy find ]
/interface disable WAN2-V2-IPSEC
/ip ipsec remote-peers kill-connections
/ip ipsec installed-sa flush
# now I wait 10 seconds: should be a sufficient time for vrrp to change role and
# and trigger IPSEC-Shutdown that puts IPSECMaster to false.
# if this happens next if will abort ipsec starting
# At the same time, if the role remains to this router and IPSAC should start,
# remote peers should have understood (with dpd enabled) that all keys need to be renegotiated
:delay 10
:if ($IPSECMaster) do={
:log info "IPSEC - restarting"
/interface enable WAN2-V2-IPSEC
:delay 5
/ip ipsec peer enable [ /ip ipsec peer find ]
/ip ipsec policy enable [ /ip ipsec policy find ]
/tool netwatch enable [/tool netwatch find where comment~"IPSEC.*"]
} else {
:log info "IPSEC - restart aborted"
}
“IPSEC-shutdown”
# self explaining ...
:global IPSECMaster false
:log info "IPSEC is shutting down"
/interface disable WAN2-V2-IPSEC
/ip ipsec policy disable [ /ip ipsec policy find ]
/ip ipsec peer disable [ /ip ipsec peer find ]
/ip ipsec remote-peers kill-connections
/ip ipsec installed-sa flush sa-type=all
/tool netwatch disable [/tool netwatch find where comment~"IPSEC.*"]
“VRRP-master-ALWAYS”
# i put this router as the only master (255 means don't wait any vrrp packet: you are the master)
:local newpriority 255
:local vrid 1
:log info ("VRRP - setting priority ".$newpriority." for all interfaces with vrid ".$vrid)
/interface vrrp set [/interface vrrp find where priority!=$newpriority and vrid=$vrid] priority=$newpriority
“VRRP-master-normal”
# norlmally, master priority=100 and the backup priority=50
:local newpriority 100
:local vrid 1
:log info ("VRRP - setting priority ".$newpriority." for all interfaces with vrid ".$vrid)
/interface vrrp set [/interface vrrp find where priority!=$newpriority and vrid=$vrid] priority=$newpriority
“VRRP-goMaster”
# this is to make all interfaces with vrid=1 master at the same time
:local newpriority 150
:local vrid 1
:log info ("VRRP - setting priority ".$newpriority." for all interfaces with vrid ".$vrid)
/interface vrrp set [/interface vrrp find where priority!=$newpriority and vrid=$vrid] priority=$newpriority
“VRRP-goSlave”
# norlmally, master priority=100 and the backup priority=50
:local newpriority 50
:local vrid 1
:log info ("VRRP - setting priority ".$newpriority." for all interfaces with vrid ".$vrid)
/interface vrrp set [/interface vrrp find where priority!=$newpriority and vrid=$vrid] priority=$newpriority
/system script run IPSEC-shutdown
“IPSEC-ifmaster-restart”
# self explaining
:global IPSECMaster
:if ($IPSECMaster) do={
:log info "This router seems to be ipsec master ... I try to restart"
/system script run IPSEC-Restart
} else={
:log info "This router seems not to be ipsec master ... I'll do nothing"
}
############################################
What functions and what is problematic
############################################
First of all, on-master and on-backup are problematic, see http://forum.mikrotik.com/t/vrrp-on-backup-script-not-running/60549/1
So I added a schedule to disable vrrp on router reboot on the backup router, else the second can start thinking to be the IPSECmaster if it was before reboot, even if vrrp will go from initialization state to backup
When all is running and master is master and backup is backup I can swap roles manually running IPSEC-gomaster or powering off the master. Swap goes well (10 seconds… you know, for my customer is acceptable).
If the second router is at priority 50 (normal backup priority) and the first starts up with priority 100), the first bring the roles and goes on well.
If I disconnect power cable from master router, or wan cable, swap is ok. (I decided to trigger scripts only on wan, but I can attach to others)
###############################################################
HOPE MY WORK WILL HELP ANYONE (or just be a good read)
NATURALLY … ANY SUGGESTION APPRECIATED, AND IF YOU THINK
THIS IS USEFUL, MAKE ME KNOW !!! (… and give karma…)
###############################################################
