VRRP doesn't work in 2.9.17

RouterOS General
1

My Research in the 2.9.17 Vrrp-Implementation as follows - any ideas or comments are welcome!

IP/Mac/Device List of Lab Equipment:
10.23.45.60  00-d0-bb-d6-4b-00  Cisco Catalyst Switch
10.23.45.61  xx-xx-xx-xx-xx-xx  Vrrp Virtual Address
10.23.45.62  00-20-ed-64-21-6d  RouterOs Vrrp Master
10.23.45.63  00-20-ed-53-6d-e5  RouterOs Vrrp Backup
10.23.45.143 00-01-80-3A-00-C2  Windows XP Diag Pc

All devices are connected to a Cisco Catalyst, the Vrrp Master to Port FastEthernet0/7 and the Backup to FastEthernet0/8.
The Diag PC does a continous ping to all above ip addresses to resolve mac immediately.

As follows there is a dump of ‘debug ethernet-controller address’ on a Cisco Catalyst Switch connected only to the two Vrrp router-os boxes and my Pc. there is only the default vlan 1 active, which is untagged. stp is disabeld on the switch, also portfast is activated which prevents in caching of Mac-addresses on the switch:

At first the Vrrp master comes up alone right after booting:

01:19:59: Add    address 0020.ed64.216d, on port Fa0/7
01:19:59: Add    address 0020.ed64.216d, on port Fa0/7 vlan 1
01:19:59: Add    address 0000.5e00.0101, on port Fa0/7
01:19:59: Add    address 0000.5e00.0101, on port Fa0/7 vlan 1

the first two entries represent the physical Mac of the card, the second the logical Vrrp-Mac address which got registered through Vrrp i think.
thats half correct - only the logical Vrrp ip address should get announced through this logical Vrrp Mac.

on my diag Pc i try as follows (i did a ‘arp -d’ before booting to clean local Arp cache - furthermore i do continous ping to 10.23.45.62 [main ip of Vrrp master], 10.23.45.63 [main ip of Vrrp backup] and 10.23.45.61 [Vrrp ip])

C:\>arp -a
Interface: 10.23.45.143 --- 0x2
  Internetadresse       Physikal. Adresse     Typ
  10.23.45.60           00-d0-bb-d6-4b-00     dynamic
  10.23.45.61           00-00-5e-00-01-01     dynamic
  10.23.45.62           00-00-5e-00-01-01     dynamic
  10.23.45.63           00-00-00-00-00-00     invalid

you can see that both ip addresses are registered under the Vrrp Mac-address and in this situation i don’t get any positive reply from the alone running Vrrp master.
both master ip-addresses are registered under the logical Vrrp Mac address which is only correct for the Vrrp ip address.
furthermore none reponds to any Arp request.

after doing a ‘arp -d 10.23.45.61’ and ‘arp -d 10.23.45.62’ then the pysical Mac-addresses got registered and the master is reachable:

C:\>arp -a
Interface: 10.23.45.143 --- 0x2
  Internetadresse       Physikal. Adresse     Typ
  10.23.45.60           00-d0-bb-d6-4b-00     dynamic
  10.23.45.61           00-20-ed-64-21-6d     dynamic
  10.23.45.62           00-20-ed-64-21-6d     dynamic
  10.23.45.63           00-00-00-00-00-00     invalid

in this scene all is reachable, but technical not correct, because the logical Vrrp address is registered under the physical Mac.

then i boot the Vrrp backup and itselfs registers clean with his physical Mac-address - all three ip’s are reachable

C:\>arp -a
Interface: 10.23.45.143 --- 0x2
  Internetadresse       Physikal. Adresse     Typ
  10.23.45.60           00-d0-bb-d6-4b-00     dynamic
  10.23.45.61           00-20-ed-64-21-6d     dynamic
  10.23.45.62           00-20-ed-64-21-6d     dynamic
  10.23.45.63           00-20-ed-53-6d-e5     dynamic

in this scene still all is reachable, but technical not correct, because the logical Vrrp address is still registered under the physical Mac.

then i disconnect the ethernet cable of the Vrrp master and all things start to get Mac-address crazy - as follows the Cisco dump of the event:

01:46:09: add static addr 0100.0cdd.dddd, port Fa0/8 (109) 00000000 00000001 00000000 00000003
01:46:09: add static addr 0100.0cdd.dddd, port Fa0/7 (109) 00000000 00000001 00000000 00000000
01:46:09: add static addr 0100.5e00.0001, port Fa0/8 (109) 00000000 00000000 00000000 00000003
01:46:09: add static addr 0100.5e00.0001, port Fa0/7 (109) 00000000 00000000 00000000 00000000
01:46:09: add static addr 0100.5e00.0002, port Fa0/8 (109) 00000000 00000000 00000000 00000003
01:46:09: add static addr 0100.5e00.0002, port Fa0/7 (109) 00000000 00000000 00000000 00000000
01:46:09: add static addr ffff.ffff.ffff, port Fa0/8 (109) 00000000 00000400 00000000 00000003
01:46:09: add static addr ffff.ffff.ffff, port Fa0/7 (109) 00000000 00000000 00000000 00000000
01:46:10: %LINK-3-UPDOWN: Interface FastEthernet0/7, changed state to down
01:46:10: Delete address 0020.ed64.216d, on port Fa0/7
01:46:10: Delete address 0020.ed64.216d, on port Fa0/7 vlan 1
01:46:11: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/7, changed state to down
01:46:12: add static addr 0100.0cdd.dddd, port Fa0/8 (109) 00000000 00000001 00000000 00000000
01:46:12: add static addr 0100.5e00.0001, port Fa0/8 (109) 00000000 00000000 00000000 00000000
01:46:12: add static addr 0100.5e00.0002, port Fa0/8 (109) 00000000 00000000 00000000 00000000
01:46:12: add static addr ffff.ffff.ffff, port Fa0/8 (109) 00000000 00000000 00000000 00000000
01:46:12: %LINK-3-UPDOWN: Interface FastEthernet0/8, changed state to down
01:46:12: Delete address 0020.ed53.6de5, on port Fa0/8
01:46:12: Delete address 0020.ed53.6de5, on port Fa0/8 vlan 1
01:46:13: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/8, changed state to down
01:46:14: Add    address 0000.5e00.0101, on port Fa0/8
01:46:14: Add    address 0000.5e00.0101, on port Fa0/8 vlan 1
01:46:14: add static addr 0100.0cdd.dddd, port Fa0/8 (109) 00000000 00000001 00000000 00000003
01:46:14: add static addr 0100.5e00.0001, port Fa0/8 (109) 00000000 00000000 00000000 00000003
01:46:14: add static addr 0100.5e00.0002, port Fa0/8 (109) 00000000 00000000 00000000 00000003
01:46:14: add static addr ffff.ffff.ffff, port Fa0/8 (109) 00000000 00000400 00000000 00000003
01:46:14: %LINK-3-UPDOWN: Interface FastEthernet0/8, changed state to up
01:46:15: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/8, changed state to up

You can see that the Vrrp implementation starts to change the Mac address of the logical Vrrp address to the Vrrp logical Mac-address, furthermore the physical Mac-address of the Vrrp backup got removed
The swapping of the logical Vrrp ip works fine, because of Mac-address changement to the logical Mac-address.
Technical from my point of view this is half correct - the logical ip should change, but not the real Mac of the backup which is still connected to the real ip of the backup.

Now i reconnect the ethernet cable of the master (same Cisco dump as described above):

02:00:38: add static addr 0100.0cdd.dddd, port Fa0/8 (109) 00000000 00000001 00000000 00000043
02:00:38: add static addr 0100.0cdd.dddd, port Fa0/7 (109) 00000000 00000001 00000000 00000083
02:00:38: add static addr 0100.5e00.0001, port Fa0/8 (109) 00000000 00000000 00000000 00000043
02:00:38: add static addr 0100.5e00.0001, port Fa0/7 (109) 00000000 00000000 00000000 00000083
02:00:38: add static addr 0100.5e00.0002, port Fa0/8 (109) 00000000 00000000 00000000 00000043
02:00:38: add static addr 0100.5e00.0002, port Fa0/7 (109) 00000000 00000000 00000000 00000083
02:00:38: add static addr ffff.ffff.ffff, port Fa0/8 (109) 00000000 00000400 00000000 00000043
02:00:38: add static addr ffff.ffff.ffff, port Fa0/7 (109) 00000000 00000400 00000000 00000083
02:00:38: %LINK-3-UPDOWN: Interface FastEthernet0/7, changed state to up
02:00:38: Add    address 0020.ed64.216d, on port Fa0/7 vlan 1
02:00:38: add static addr 0100.0cdd.dddd, port Fa0/7 (109) 00000000 00000001 00000000 00000003
02:00:38: add static addr 0100.0cdd.dddd, port Fa0/8 (109) 00000000 00000001 00000000 00000000
02:00:39: add static addr 0100.5e00.0001, port Fa0/7 (109) 00000000 00000000 00000000 00000003
02:00:39: add static addr 0100.5e00.0001, port Fa0/8 (109) 00000000 00000000 00000000 00000000
02:00:39: add static addr 0100.5e00.0002, port Fa0/7 (109) 00000000 00000000 00000000 00000003
02:00:39: add static addr 0100.5e00.0002, port Fa0/8 (109) 00000000 00000000 00000000 00000000
02:00:39: add static addr ffff.ffff.ffff, port Fa0/7 (109) 00000000 00000400 00000000 00000003
02:00:39: add static addr ffff.ffff.ffff, port Fa0/8 (109) 00000000 00000000 00000000 00000000
02:00:39: %LINK-3-UPDOWN: Interface FastEthernet0/8, changed state to down
02:00:39: Delete address 0000.5e00.0101, on port Fa0/8
02:00:39: Delete address 0000.5e00.0101, on port Fa0/8 vlan 1
02:00:39: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/7, changed state to up
02:00:40: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/8, changed state to down
02:00:40: add static addr 0100.0cdd.dddd, port Fa0/7 (109) 00000000 00000001 00000000 00000083
02:00:40: add static addr 0100.0cdd.dddd, port Fa0/8 (109) 00000000 00000001 00000000 00000043
02:00:40: add static addr 0100.5e00.0001, port Fa0/7 (109) 00000000 00000000 00000000 00000083
02:00:40: add static addr 0100.5e00.0001, port Fa0/8 (109) 00000000 00000000 00000000 00000043
02:00:40: add static addr 0100.5e00.0002, port Fa0/7 (109) 00000000 00000000 00000000 00000083
02:00:40: add static addr 0100.5e00.0002, port Fa0/8 (109) 00000000 00000000 00000000 00000043
02:00:40: add static addr ffff.ffff.ffff, port Fa0/7 (109) 00000000 00000400 00000000 00000083
02:00:40: add static addr ffff.ffff.ffff, port Fa0/8 (109) 00000000 00000400 00000000 00000043
02:00:40: Add    address 0020.ed53.6de5, on port Fa0/8
02:00:40: Add    address 0020.ed53.6de5, on port Fa0/8 vlan 1
02:00:40: %LINK-3-UPDOWN: Interface FastEthernet0/8, changed state to up
02:00:41: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/8, changed state to up

After this both real RouterOs-ip’s are reachable, but not any more the logical Vrrp - on the RouterOs boxes they have been correctly reassigned to master and deactivated on slave.
this reassignement does some Mac-changements again as you can see in the above dump - on my diag Pc the arp -a shows as follows:

C:\>arp -a
Interface: 10.23.45.143 --- 0x2
  Internetadresse       Physikal. Adresse     Typ
  10.23.45.60           00-d0-bb-d6-4b-00     dynamic
  10.23.45.61           00-00-5e-00-01-01     dynamic
  10.23.45.62           00-20-ed-64-21-6d     dynamic
  10.23.45.63           00-20-ed-53-6d-e5     dynamic

and you can see that the .61 has still the virtual Mac but and all other changed correctly to their physical addresses - thats the first time the cache is Vrrp implementation conform - but sorry no response from RouterOs under this Mac!

After cleaning the local PC Arp cache again all changed back to beginning and works till the next failover like described in the begnning:

C:\>arp -a
Interface: 10.23.45.143 --- 0x2
  Internetadresse       Physikal. Adresse     Typ
  10.23.45.60           00-d0-bb-d6-4b-00     dynamic
  10.23.45.61           00-20-ed-64-21-6d     dynamic
  10.23.45.62           00-20-ed-64-21-6d     dynamic
  10.23.45.63           00-20-ed-53-6d-e5     dynamic

Even more interesting is the output of the local mac address on the RouterOs:
on Master while active:

[admin@vrrp-master] > interface ethernet print 
Flags: X - disabled, R - running 
 #    NAME                                   MTU   MAC-ADDRESS       ARP       
 0  R ether1                                 1500  00:00:5E:00:01:01 enabled   
 1 X  ether2                                 1500  00:0E:0C:A0:33:70 enabled

on Backup while Master active:

[admin@vrrp-backup] > /interface ethernet  print 
Flags: X - disabled, R - running 
 #    NAME                                   MTU   MAC-ADDRESS       ARP       
 0  R ether1                                 1500  00:20:ED:53:6D:E5 enabled   
 1 X  ether2                                 1500  00:0E:0C:A0:33:5F enabled

on Backup while Master is down:

[admin@vrrp-backup] > /interface ethernet  print 
Flags: X - disabled, R - running 
 #    NAME                                   MTU   MAC-ADDRESS       ARP       
 0  R ether1                                 1500  00:00:5E:00:01:01 enabled   
 1 X  ether2                                 1500  00:0E:0C:A0:33:5F enabled

From my point of view thats wrong because the real mac of the interface never should change! Maybe the design problem is that RouterOs can handle only on Mac per dedicated interface?

Greets from Austria,
Wolfgang Malecek

2

This is the first I have heard of MT using virtual mac addresses for VRRP. The last time we did an autopsy on MT VRRP we discovered it uses ‘Gratituous ARP’ instead of virtual MAC as per the VRRP RFC.

‘Gratituous ARP’ actually works OK so we didnt care too much. BUT we found our problem.

This has always been an outstanding VRRP bug with MT that we reported 1.5yrs ago and as far as we know has still has not been resolved:-

Usually when you use VRRP you are doing it on 2 identical routers for redundancy, SO the logic is to set one router up perfectly, CLONE the config to a SECOND router and fine-tune VRRP between the two.
EASIER SAID THAN DONE.

BUG:- when you clone an MT’s config, the second MT’s VRRP mac address IS THE SAME AS THE FIRST MT’s REAL mac address !

IF YOU ARE USING VRRP, BUILD THE SECOND ONE FROM SCRATCH AND DONT SHARE CONFIG FILES.

Are you sure you are not seeing this problem ?

OR has something in the VRRP subsystem changed without being in the changelog ? (go figure)
eg is VRRP now using real virtual mac’s instead of gratituous ARP ?
Since what version ?

3

hi airnet,

the problem with mac cloning we alreday have learned in try and error in previous version but is handle-able if you know it.

try any version from 2.9.12 and you’ll see, that vrrp changes the real mac address to a virtual one if vrrp is enabled as an master.

but things get crazy in some version, so it overwrites the real mac with the virtual mac even if no more vrrp is active.
from far away you’ll never get the real mac address back again :frowning:

[w.malecek@x1.lx1] interface ethernet> pri
Flags: X - disabled, R - running 
 #    NAME                                                                                     MTU   MAC-ADDRESS       ARP       
 0  R ;;; s1 fa0/4 mac 00:E0:81:01:A5:D9
      e1-trunk                                                                                 1500  00:E0:81:01:A5:D9 enabled   
 1  R ;;; s1 fa0/10
      e2-trunk-lx1                                                                             1500  00:00:5E:00:01:01 enabled

→ interface #1 was a vrrp interface but still has the virtual mac bound to.

situation gets even more cruel if you mix vrrp on vlan interfaces with and without tagging in combination - nothing then works on any interface.

i think this could be a design problem, because RouterOs is only able to handle one mac per interface.

4

I want to use vrrp on a bgp edge router . We have 2 peers and these routers are all PCI-X based P4 3ghz systems . I did have problems with vrrp in 2.8.x . 4 interfaces on each router wonder if it will work when 2 of the interfaces will have bgp data and handle 400mbps of traffic