I am trying to set up a couple of CHR to run side by side for a streaming service that we offer. This requires that I have a single WAN and and a single LAN address that I can use. For best availability I need to have one of the routers running as the primary and the other as the secondary so that in the event the primary goes down my traffic will failover to the secondary router. These CHR are located in mirrored VMware datacenters so in the event of an issue at one site the second site can take over. I am not very well versed in Mikrotik so any help would be appreciated.
I guess my initial question is if you have an investment in VMWare… wouldn’t vMotion/etc work better? In that case you’d two identical CHRs, with VMWare controlling which one was active.
VRRP certainly be possible on LAN side pretty easily. And VRRP supports connection tracking sync which keep it hitless. But WAN side gets more tricky, and more specific to what’s going on… And you may want to consider BGP instead of VRRP on WAN side.
I do not understand what a “mirrored datacenter” means. VRRP uses specific MAC addresses so the physical machines running the CHRs have to be interconnected on L2 level so that a single public address could migrate between them using VRRP (there are L3 methods that do not require L2 interconnection but have other complicated requirements). Plus the hypervisors must allow the VMs to use other MAC adddresses than the ones they have been assigned. On Mikrotik, you can configure one VRRP interface to track the state of another one, by means of adjusting the priority of the tracking one when the leading one changes state, using the on-master and on-backup items in its configuration that can hold scripts.
Yeah that’s kinda the central question here.
Based on my now dated VMWare knowledge, but the general idea is that an “application” (i.e. CHR) didn’t NOT have to be aware of redudency — since VMWare managed syncing all machine state and networks between clusters. i.e. VMWare mirrors memory, disk, and network for something like CHR… so in there is only one CHR from RouterOS POV. But this depends on much of VMWare kool-aid you’ve drank… since how “mirrored” things are is a sliding scale and different features sets/licensing.
Sorry, by mirrored datacenters I meant geographically redundant. I have two datacenters in two different physical locations, but each is running the exactly same hardware as the other.
Starting from the ground up I currently have the two CHR up and running and set up VRRP between them, but for some reason the one keep switching from master to remote constantly. For testing right now I have both routers built in the same datacenter. I created a new vlan for the Mikrotik VRRP, and have enable promiscuous mode for that vlan. It seemed to be working fine to start with and then I started to see the flip flop on the master/backup.
I have a problem to wrap my head around a scenario where you’ve got two georedundant datacenters between which a single IP address may freely migrate on an L2 segment (presumably a VxLAN) - in my understanding of networking, it would mean that there is a router in each of the datacenters that is able to advertise the subnet that lives inside that L2 VxLAN to the internet. Is that the case?
Other than that, please post the anonymized configuration exports of both CHRs as they look now, as you refer to switching “from master to remote” whereas the possible states of a VRRP interface are MASTER and BACKUP; what typically happens if there is a misconfiguration is that both machines become masters because they do not receive the VRRP messages from each other.
vMotion is not L3 or Application redundancy. It just helps if the physical host dies.
You can use a vrrp-interface with on-master= and on-backup= scripts to enable/disable other interfaces, tunnels or ip-adresses.
As far as the networking goes all the traffic in the two datacenters are set up in vlans. The datacenters are Cisco UCS/FI stacks being uplinked to our MPLS network through a pair of Cisco Nexus 9k and ASR 9ks for the MPLS ring. We have a video streaming solution we run at one of the datacenter locations on it’s own hardware and that hardware connects to an ASR 9k which brings it into the network and allows my datacenters to get the streams. We bring those streams into the vmware stack on a particular vlan and drop it off to ether 2 on the CHR Mikrotik and then we share that out to our real world customers over the public IP addresses we have set on the Mikrotik via ether 1. I will post the config for the existing Mikrotik below.
I do now have two test CHRs up and running as a VRRP pair with internet access and if I disable the WAN ethernet port on either one my test box stays up and connected to the internet. The next part I need to figure out is how to make this work for HA. Right now both test boxes live in just one of the datacenters so I didn’t have to build out the crossconnects through my MPLS ring. But the end goal is to have 1 CHR in each datacenter.
Hopefully that helps clarify things. I would draw out a diagram of the setup and post that as well, but am unsure of how I would upload that here.
/interface ethernet
set [ find default-name=ether1 ] comment="VLAN 310 Internet"
set [ find default-name=ether2 ] comment="VLAN 116 Video"
/snmp community
set [ find default=yes ] addresses=y.y.65.18/32 name=mikrotik
/system logging action
set 1 disk-file-name=log
/ip neighbor discovery-settings
set discover-interface-list=!dynamic
/ip settings
set max-neighbor-entries=8192
/ip address
add address=x.x.116.1/24 comment="IP Address for Internal LAN" interface=ether2 network=x.x.116.0
add address=y.y.64.165/28 comment="IP Address for VidServer 3" interface=ether1 network=y.y.64.160
add address=y.y.64.162/28 comment="IP Address for VidServer App/Web Portal" interface=ether1 network=y.y.64.160
add address=y.y.64.166/28 comment="IP Address for VidServer 4" interface=ether1 network=y.y.64.160
add address=y.y.64.163/28 comment="IP Address for VidServer1" interface=ether1 network=y.y.64.160
add address=y.y.64.164/28 comment="IP Address for VidServer 2" interface=ether1 network=y.y.64.160
add address=y.y.64.167/28 comment="IP Address for VidServer 5" interface=ether1 network=y.y.64.160
/ip dns
set servers=y.y.64.11,y.y.64.12
/ip firewall filter
add action=accept chain=input comment="Allow Ping" protocol=icmp src-address=y.y.64.15
add action=accept chain=input comment="Allow Ping 2" protocol=icmp src-address=y.y.65.18
add action=accept chain=input comment="Allow ICMP from Other" protocol=icmp src-address=z.z.192.190
add action=accept chain=input comment="Allow ICMP from y.y.64.10" protocol=icmp
add action=accept chain=input comment="Allow port 8000 http access for mgmt" dst-port=8000 protocol=tcp src-address=y.y.65.0/24
add action=accept chain=input comment="Allow SSH access for Mgmt" dst-port=22 protocol=tcp src-address=y.y.65.0/24
add action=accept chain=input comment="Allow Established and Related Connections" connection-state=established,related log=yes
add action=accept chain=input comment=SNMP dst-port=161 protocol=udp src-address=y.y.65.18
add action=accept chain=input comment="VidBox 7022" dst-port=25069 protocol=tcp
add action=accept chain=input comment="VidBox 7022" dst-port=27514 protocol=udp
add action=drop chain=input comment="Drop Everything Else" in-interface=ether1 log=yes log-prefix=DROP
/ip firewall nat
add action=src-nat chain=srcnat comment="VidServer 1 - One to One NAT" out-interface=ether1 src-address=x.x.116.141 to-addresses=y.y.64.163
add action=src-nat chain=srcnat comment="VidServer 2 - One to One NAT" out-interface=ether1 src-address=x.x.116.142 to-addresses=y.y.64.164
add action=src-nat chain=srcnat comment="VidServer 3 - One to One NAT" out-interface=ether1 src-address=x.x.116.143 to-addresses=y.y.64.165
add action=src-nat chain=srcnat comment="VidServer 4 - One to One NAT" out-interface=ether1 src-address=x.x.116.144 to-addresses=y.y.64.166
add action=src-nat chain=srcnat comment="VidServer 5 - One to One NAT" out-interface=ether1 src-address=x.x.116.145 to-addresses=y.y.64.167
add action=dst-nat chain=dstnat comment="VidServer VRRP x.x.116.140 Port 443" dst-address=y.y.64.162 dst-port=443 in-interface=ether1 protocol=tcp to-addresses=x.x.116.140 to-ports=443
add action=dst-nat chain=dstnat comment="VidServer VRRP x.x.116.140 Port 80" dst-address=y.y.64.162 dst-port=80 in-interface=ether1 protocol=tcp to-addresses=x.x.116.140 to-ports=80
add action=dst-nat chain=dstnat comment="VidServer 1 Port 443" dst-address=y.y.64.163 dst-port=443 in-interface=ether1 protocol=tcp to-addresses=x.x.116.141 to-ports=443
add action=dst-nat chain=dstnat comment="VidServer 1 Port 80" dst-address=y.y.64.163 dst-port=80 in-interface=ether1 protocol=tcp to-addresses=x.x.116.141 to-ports=80
add action=dst-nat chain=dstnat comment="VidServer 2 Port 443" dst-address=y.y.64.164 dst-port=443 in-interface=ether1 protocol=tcp to-addresses=x.x.116.142 to-ports=443
add action=dst-nat chain=dstnat comment="VidServer 2 Port 80" dst-address=y.y.64.164 dst-port=80 in-interface=ether1 protocol=tcp to-addresses=x.x.116.142 to-ports=80
add action=dst-nat chain=dstnat comment="VidServer 3 Port 443" dst-address=y.y.64.165 dst-port=443 in-interface=ether1 protocol=tcp to-addresses=x.x.116.143 to-ports=443
add action=dst-nat chain=dstnat comment="VidServer 3 Port 80" dst-address=y.y.64.165 dst-port=80 in-interface=ether1 protocol=tcp to-addresses=x.x.116.143 to-ports=80
add action=dst-nat chain=dstnat comment="VidServer 4 Port 443" dst-address=y.y.64.166 dst-port=443 in-interface=ether1 protocol=tcp to-addresses=x.x.116.144 to-ports=443
add action=dst-nat chain=dstnat comment="VidServer 4 Port 80" dst-address=y.y.64.166 dst-port=80 in-interface=ether1 protocol=tcp to-addresses=x.x.116.144 to-ports=80
add action=dst-nat chain=dstnat comment="VidServer 5 Port 443" dst-address=y.y.64.167 dst-port=443 in-interface=ether1 protocol=tcp to-addresses=x.x.116.145 to-ports=443
add action=dst-nat chain=dstnat comment="VidServer 5 Port 80" dst-address=y.y.64.167 dst-port=80 in-interface=ether1 protocol=tcp to-addresses=x.x.116.145 to-ports=80
add action=dst-nat chain=dstnat comment="VidServer 1 Port TCP 25069" dst-address=y.y.64.163 dst-port=25069 in-interface=ether1 protocol=tcp to-addresses=x.x.116.141 to-ports=25069
add action=dst-nat chain=dstnat comment="VidServer 2 Port TCP 25069" dst-address=y.y.64.164 dst-port=25069 in-interface=ether1 protocol=tcp to-addresses=x.x.116.142 to-ports=25069
add action=dst-nat chain=dstnat comment="VidServer 3 Port TCP 25069" dst-address=y.y.64.165 dst-port=25069 in-interface=ether1 protocol=tcp to-addresses=x.x.116.143 to-ports=25069
add action=dst-nat chain=dstnat comment="VidServer 4 Port TCP 25069" dst-address=y.y.64.166 dst-port=25069 in-interface=ether1 protocol=tcp to-addresses=x.x.116.144 to-ports=25069
add action=dst-nat chain=dstnat comment="VidServer 5 Port TCP 25069" dst-address=y.y.64.167 dst-port=25069 in-interface=ether1 protocol=tcp to-addresses=x.x.116.145 to-ports=25069
add action=dst-nat chain=dstnat comment="VidServer 1 Port UDP 27514" dst-address=y.y.64.163 dst-port=27514 in-interface=ether1 protocol=udp to-addresses=x.x.116.141 to-ports=27514
add action=dst-nat chain=dstnat comment="VidServer 2 Port UDP 27514" dst-address=y.y.64.164 dst-port=27514 in-interface=ether1 protocol=udp to-addresses=x.x.116.142 to-ports=27514
add action=dst-nat chain=dstnat comment="VidServer 3 Port UDP 27514" dst-address=y.y.64.165 dst-port=27514 in-interface=ether1 protocol=udp to-addresses=x.x.116.143 to-ports=27514
add action=dst-nat chain=dstnat comment="VidServer 4 Port UDP 27514" dst-address=y.y.64.166 dst-port=27514 in-interface=ether1 protocol=udp to-addresses=x.x.116.144 to-ports=27514
add action=dst-nat chain=dstnat comment="VidServer 5 Port UDP 27514" dst-address=y.y.64.167 dst-port=27514 in-interface=ether1 protocol=udp to-addresses=x.x.116.145 to-ports=27514
add action=dst-nat chain=dstnat comment="VidServer 5041 authorization" dst-address=y.y.64.162 dst-port=25069 in-interface=ether1 protocol=tcp to-addresses=x.x.116.141 to-ports=25069
add action=dst-nat chain=dstnat comment="Vidserver 5041 syslog" dst-address=y.y.64.162 dst-port=27514 in-interface=ether1 protocol=udp to-addresses=x.x.116.141 to-ports=27514
/ip firewall service-port
set tftp disabled=yes
set h323 disabled=yes
set sip disabled=yes
set pptp disabled=yes
set udplite disabled=yes
set dccp disabled=yes
set sctp disabled=yes
/ip ipsec policy
set 0 disabled=yes
/ip route
add disabled=no dst-address=0.0.0.0/0 gateway=y.y.64.161
/ip service
set telnet disabled=yes
set ftp address=y.y.65.0/24 disabled=yes
set www address=y.y.65.0/24 port=8000
set ssh address=y.y.65.0/24
set www-ssl address=y.y.65.0/24 port=8080
set api disabled=yes
set winbox address=y.y.65.0/24 disabled=yes
set api-ssl disabled=yes
/ip ssh
set allow-none-crypto=yes forwarding-enabled=remote
/routing bfd configuration
add disabled=no interfaces=all min-rx=200us min-tx=200us multiplier=5
/snmp
set enabled=yes trap-generators=interfaces trap-version=2
/system clock
set time-zone-name=America/Denver
/system logging
set 3 action=memory
add topics=firewall
/system note
set show-at-login=no
/system ntp client
set enabled=yes
/system ntp client servers
add address=y.y.65.17
add address=y.y.65.18
/tool graphing interface
add allow-address=y.y.65.18/32 interface=ether1 store-on-disk=no
/tool sniffer
set filter-ip-protocol=0 filter-port=ircu
Just curious, but why not fully utilize the VSM functionality since you already have a bunch of ASR 9Ks? I mean, why use CHRs as edge routers?
Btw, this is how you add an image to a post:
He is the diagram. Not sure what you meant by VSM though.
Thanks, but I have to admit I’m pretty confused by the network diagram as the image doesn’t seem to follow a clear visual logic and it’s hard to make sense of it without additional context.
For example, how does the red-dashed VRRP relate to the four nodes (VRRP1, VRRP2, CHR1, CHR2)? And what role does the blue-dashed CHR play in this layout? I’m also still curious about why use CHR instead of the powerfull ASR9k as CE, which btw already has that role on the right side?
Maybe you can provide an explanatory text and possibly a more logically organized network topology diagram?
Ah, for me a “datacenter” normally means something provided by a 3rd party 
So I figure the two ASRs use BGP to advertise the public subnet where the CHRs live to the internet, which the diagram is silent about. How exactly the L2 is tunneled between the two locations (MPLS, VxLAN, or something else) is not much relevant.
The most important part, i.e. the VRRP configuration, is missing in that export, which kind of denies its purpose.
But it dawned on me earlier today - as you are still a beginner with Mikrotik, could it be that you’ve mistook the RM indication next to the VRRP interface as ReMote whilst it actually indicates Running and Master?
Because in that export, I can see the usual omission - the rules in chain input of ip firewall filter drop the incoming VRRP packets, so the two CHRs do not know about each other and hence both become masters. Which wouldn’t be a big deal if it wasn’t for NAT.
Sorry the existing router I took the export from is the Live router that is currently handling the traffic. The idea is to spin up two new CHR to replace that single and provide an HA type of solution for my current video offering. I am trying to get the two new CHR routers up and running in a test environment to prove the concept will work.
To clarify things maybe a bit more I work for an ISP. We offer an over the top option for video to be streamed to devices on our network. As part of this setup there is a URL that the steaming devices connect back to. This obviously resolves to a single IP that lives on the current CHR. Our goal is to be able to have two new CHR that can either individually (master/backup) or load balance that traffic coming in and in the event of a failure one of them can take over the entire load if needed. So we would prefer and active/active setup, but an active standby setup is agreeable to get going.
As for the transport layer it shouldn’t play a huge part. We bring internet directly to the CHR on a vlan that gets stripped as the VMware host and the same for video so the CHR is never even aware of a vlan being used. So for all intents and purposes they have direct connections to the VidServer and the internet.
Hope that clarifies things some more. Sorry if I am not explaining things very well.
That’s the point - you have experienced the issue on the test pair of CHRs but you have posted an export of something else. So post the configuration of the two test devices together with the output of /interface vrrp print from both in the state you “do not like”, so that we can debug and resolve the actual issue.
The red dashed connections are what I am hoping to try and get working and the blue dashed is what is currently in service. As for using the 9ks they are transparent to the VMs and VidServer they are the L2 transport network.
Alright, got it. As for load balancing (ie vrrp load sharing) and grouping, have you checked if the ROS version has what you need? It might be worth a look, since it doesn’t have all the ‘bells and whistles’ of the Cisco IOS XR equivalent.
The load balancing I can address down the road. My main concern for now is getting the single WAN IP to work with the dual router setup. The uplinks for the CHR are going to be 10Gb so as of right now we have plenty of bandwidth.
The subnet that the WAN IPs live in is a /28 of which 6 IPs are in used. So let say 100.100.64.162-167 in use. 161 is the gateway, and 168-174 are unused right now.
Since it was asked earlier. Here are the two test routers I have set up.
RTR 1
/interface bridge
add name=lanbridge1
/interface ethernet
set [ find default-name=ether1 ] disable-running-check=no
set [ find default-name=ether2 ] disable-running-check=no
set [ find default-name=ether3 ] disable-running-check=no name=ether4
/interface vrrp
add comment=LAN interface=ether1 name=vrrp1 preemption-mode=no priority=254 sync-connection-tracking=yes vrid=49
add comment=WAN disabled=yes interface=ether4 name=vrrp2 preemption-mode=no vrid=59
/ip pool
add name=dhcp_pool0 ranges=192.168.88.2-192.168.88.254
/ip dhcp-server
add address-pool=dhcp_pool0 interface=lanbridge1 name=dhcp1
/port
set 0 name=serial0
set 1 name=serial1
/interface bridge port
add bridge=lanbridge1 interface=ether2
/ip firewall connection tracking
set enabled=yes
/ip address
add address=192.168.1.10/24 interface=ether1 network=192.168.1.0
add address=192.168.1.1 interface=vrrp1 network=192.168.1.1
add address=192.168.88.1/24 interface=lanbridge1 network=192.168.88.0
add address=172.20.120.254/24 comment=Internet interface=ether4 network=172.20.120.0
/ip dhcp-client
add disabled=yes interface=ether1
/ip dhcp-server network
add address=192.168.88.0/24 dns-server=192.168.88.1,8.8.8.8 gateway=192.168.88.1
/ip dns
set servers=8.8.8.8,8.8.4.4
/ip firewall nat
add action=masquerade chain=srcnat comment=Internet
/ip route
add disabled=no dst-address=0.0.0.0/0 gateway=172.20.120.1 routing-table=main suppress-hw-offload=no
/system clock
set time-zone-name=US/Mountain
/system note
set show-at-login=no
RTR 2
/interface bridge
add name=lanbridge1
/interface ethernet
set [ find default-name=ether1 ] disable-running-check=no
set [ find default-name=ether2 ] disable-running-check=no
set [ find default-name=ether3 ] disable-running-check=no
/interface vrrp
add comment=LAN interface=ether1 name=vrrp1 preemption-mode=no sync-connection-tracking=yes vrid=49
add comment=WAN disabled=yes interface=ether3 name=vrrp2 preemption-mode=no priority=200 vrid=59
/ip pool
add name=dhcp_pool0 ranges=192.168.88.2-192.168.88.254
/ip dhcp-server
add address-pool=dhcp_pool0 interface=lanbridge1 name=dhcp1
/port
set 0 name=serial0
set 1 name=serial1
/interface bridge port
add bridge=lanbridge1 interface=ether2
/ip firewall connection tracking
set enabled=yes
/ip address
add address=192.168.88.1/24 interface=lanbridge1 network=192.168.88.0
add address=192.168.1.20/24 interface=ether1 network=192.168.1.0
add address=192.168.1.1 interface=vrrp1 network=192.168.1.1
add address=172.20.120.253/24 comment=Internet interface=ether3 network=172.20.120.0
/ip dhcp-client
add interface=ether1
/ip dhcp-server network
add address=192.168.88.0/24 dns-server=192.168.88.1 gateway=192.168.88.1
/ip dns
set servers=8.8.8.8,8.8.4.4
/ip firewall nat
add action=masquerade chain=srcnat comment=Internet
/ip route
add comment=Internet disabled=no dst-address=0.0.0.0/0 gateway=172.20.120.1 routing-table=main suppress-hw-offload=no
/system clock
set time-zone-name=US/Mountain
/system note
set show-at-login=no
OK, no firewall filter rules at all so VRRP packets from the other router can definitely get in if they make it through the LAN.
With these configurations, what does /interface/vrrp/print where name=vrrp1 show at both test routers?
After discussing the issue internally with some techs, our best guess at this point is that the flip-flop behavior might be caused by a VMware Virtual Network Adapter ‘state change’ which can happen for various reasons like network congestion, resource constraints, virtual switch misconfigurations, bandwidth cap mgmt tools reconfiguring network settings, etc. This state change is signaled to the guest OS, causing ROS to flip VRRP state.
If possible, enable logging in ESXi on both instances and check the ROS logs for VRRP events, then sync the timestamps to see when and why the issues occur (make sure time is synchronized beforehand).
I wouldn’t pay any attention to vrrp2 right now. I just created it this morning and have done nothing with it yet. But my intention is for that to be the WAN vrrp interface and vrrp1 is for the lan side.
RTR 1
Flags: X - DISABLED; F - FAILURE
Columns: NAME, INTERFACE, MAC-ADDRESS, VRID, PRIORITY, INTERVAL, VERSION, V3-PROTOCOL, SYNC-CONNECTION-TRACKING
# NAME INTERFACE MAC-ADDRESS VRID PRIORITY INTERVAL VERSION V3-PROTOCOL SYNC-CONNECTION-TRACKING
;;; LAN
0 F vrrp1 ether1 00:00:5E:00:01:31 49 254 1s 3 ipv4 yes
;;; WAN
1 X vrrp2 ether4 00:00:5E:00:01:3B 59 100 1s 3 ipv4 no
RTR 2
[admin@MikroTik] > interface/vrrp/print
Flags: X - DISABLED; R - RUNNING; M - MASTER
Columns: NAME, INTERFACE, MAC-ADDRESS, VRID, PRIORITY, INTERVAL, VERSION, V3-PROTOCOL, SYNC-CONNECTION-TRACKING
# NAME INTERFACE MAC-ADDRESS VRID PRIORITY INTERVAL VERSION V3-PROTOCOL SYNC-CONNECTION-TRACKING
;;; LAN
0 RM vrrp1 ether1 00:00:5E:00:01:31 49 100 1s 3 ipv4 yes
;;; WAN
1 X vrrp2 ether3 00:00:5E:00:01:3B 59 200 1s 3 ipv4 no