I am doing a vxlan setup - 4 CRS510 in a ring connection. Each switch is connected to its neighbor by a routed port (not member of the bridge) on a /31 link net. Each switch have a loopback address. OSFP configured on the loopback and link nets.
Each switch can ping the other swithces loopback interface when sourcing with own loop back interface.
I have created vxlan (sofar 2) interfaces on each switch and created tunnels from own loop back to each other switch loopback. (full mesh)
Vxlan sort of works. from within the vxlan (the vlan mapped to the vni) i can from 1 switch ping a device on the sam vxlan on a neighbor switch. however when i try the same from the next switch in sequesce no reply - however looking at:
- /interface vxlan fdb print - i se that the switch is learning the right mac address from the right vtep
- /interface bridge host print - i see the corect mac addresses
- /ip arp print - the arp table is populated correct.
It must be something stupid i miss - however currently i can’t set it.
Looking through the forum i se that a few other have been writing about something similar, but i cant figure out how the solved it from the post.
I have deployed vxlan on HPE Comware, HPE Aruba-SWOS and HPE Aruba CX - so i like to thing i have the basic knowledge about vxlan, but first time on Mikrotik RouterOS.
I know that EVPN exist, but i wanted a “static” setup (ospf is used to facilitate bfd for fast fail over)
All tunnels seams to work, just only one switch hop away. i have even put in a Aruba SWOS switch - same problem directly connected switches work, but one extra hop away and it fail for forward traffic.
I still se the correct mac-address learned and arp works, but not unicast traffic and dhcp.
Here is my config:
/interface bridge
add name=bridge vlan-filtering=yes
/interface ethernet
set [ find default-name=sfp28-1 ] l2mtu=10218 mtu=9198
set [ find default-name=sfp28-2 ] l2mtu=10218 mtu=9198
set [ find default-name=sfp28-8 ] l2mtu=10218 mtu=9198
/interface vxlan
add bridge=bridge bridge-pvid=501 dont-fragment=enabled local-address=192.0.2.22 name=vxlan-501 vni=501
add bridge=bridge bridge-pvid=502 dont-fragment=enabled local-address=192.0.2.22 name=vxlan-502 vni=502
/interface vlan
add interface=bridge name=vlan-501 vlan-id=501
/routing ospf instance
add disabled=no name=ospf-ipv4 router-id=192.0.2.22
/routing ospf area
add area-id=192.0.2.0 disabled=no instance=ospf-ipv4 name=192.0.2.0
/interface bridge port
add bridge=bridge interface=sfp28-3 pvid=501
add bridge=bridge interface=sfp28-4 pvid=502
/ip neighbor discovery-settings
set discover-interface-list=!dynamic
/interface bridge vlan
add bridge=bridge untagged=sfp28-3 vlan-ids=501
add bridge=bridge untagged=sfp28-4 vlan-ids=502
/interface ethernet switch
set 0 l3-hw-offloading=yes
/interface vxlan vteps
add interface=vxlan-501 remote-ip=192.0.2.11
add interface=vxlan-501 remote-ip=192.0.2.12
add interface=vxlan-501 remote-ip=192.0.2.13
add interface=vxlan-501 remote-ip=192.0.2.21
add interface=vxlan-501 remote-ip=192.0.2.23
add interface=vxlan-502 remote-ip=192.0.2.11
add interface=vxlan-502 remote-ip=192.0.2.12
add interface=vxlan-502 remote-ip=192.0.2.13
add interface=vxlan-502 remote-ip=192.0.2.21
add interface=vxlan-502 remote-ip=192.0.2.23
/ip address
add address=192.0.2.22 interface=lo network=192.0.2.22
add address=10.50.0.22/24 interface=ether1 network=10.50.0.0
add address=192.0.2.41/31 interface=sfp28-1 network=192.0.2.40
add address=192.0.2.43/31 interface=sfp28-2 network=192.0.2.42
add address=192.0.2.46/31 interface=sfp28-8 network=192.0.2.46
add address=10.50.1.22/24 interface=vlan-501 network=10.50.1.0
/ip dhcp-client
add interface=ether1
/ip route
add dst-address=0.0.0.0/0 gateway=10.50.0.1
/routing bfd configuration
add disabled=no interfaces=sfp28-1
add disabled=no interfaces=sfp28-2
/routing ospf interface-template
add area=192.0.2.0 disabled=no interfaces=sfp28-1,sfp28-2,sfp28-8 type=ptp use-bfd=yes
add area=192.0.2.0 disabled=no interfaces=lo passive
Edit 1: MTU would be the goto explanation, However a tiny ping packet does not respond, also if MTU was the cause then the neighbor switch should have the same issue
Edit 2: if i trace vxlan traffic on swith a (192.0.2.11) comming from switch d (192.0.2.22) via switch b (192.0.2.12) i can clearly se activity when i start a ping within the vxlan on switch d to vxlan on swithc a
But when i change the ping size from default to 512 i do not se that reflected on the trace on switch a
[admin@DC-1-A] > /tool sniffer quick ip-address=192.0.2.22 ip-protocol=udp port=4789
Columns: INTERFACE, TIME, NUM, DIR, SRC-MAC, DST-MAC, SRC-ADDRESS, DST-ADDRESS, PROTOCOL, SIZE, CPU
INTERFACE TIME NUM DIR SRC-MAC DST-MAC SRC-ADDRESS DST-ADDRESS PROTOCOL SIZE CPU
sfp28-2 28.724 1 -> 04:F4:1C:B9:92:02 04:F4:1C:B9:91:78 192.0.2.11:40467 192.0.2.22:4789 ip:udp 259 0
sfp28-2 28.725 2 -> 04:F4:1C:B9:92:02 04:F4:1C:B9:91:78 192.0.2.11:47744 192.0.2.22:4789 ip:udp 168 0
sfp28-2 28.725 3 -> 04:F4:1C:B9:92:02 04:F4:1C:B9:91:78 192.0.2.11:50166 192.0.2.22:4789 ip:udp 231 0
sfp28-2 28.726 4 -> 04:F4:1C:B9:92:02 04:F4:1C:B9:91:78 192.0.2.11:51547 192.0.2.22:4789 ip:udp 259 0
sfp28-1 28.726 5 <- 04:F4:1C:B9:8D:8F 04:F4:1C:B9:92:01 192.0.2.22:55792 192.0.2.13:4789 ip:udp 259 0
sfp28-1 28.726 6 <- 04:F4:1C:B9:8D:8F 04:F4:1C:B9:92:01 192.0.2.22:55792 192.0.2.11:4789 ip:udp 259 0
sfp28-1 28.726 7 <- 04:F4:1C:B9:8D:8F 04:F4:1C:B9:92:01 192.0.2.22:48085 192.0.2.13:4789 ip:udp 168 0
sfp28-1 28.726 8 <- 04:F4:1C:B9:8D:8F 04:F4:1C:B9:92:01 192.0.2.22:48085 192.0.2.11:4789 ip:udp 168 0
sfp28-1 28.726 9 <- 04:F4:1C:B9:8D:8F 04:F4:1C:B9:92:01 192.0.2.22:33343 192.0.2.13:4789 ip:udp 231 0
sfp28-1 28.726 10 <- 04:F4:1C:B9:8D:8F 04:F4:1C:B9:92:01 192.0.2.22:33343 192.0.2.11:4789 ip:udp 231 0
sfp28-1 28.726 11 <- 04:F4:1C:B9:8D:8F 04:F4:1C:B9:92:01 192.0.2.22:33343 192.0.2.13:4789 ip:udp 231 0
sfp28-1 28.726 12 <- 04:F4:1C:B9:8D:8F 04:F4:1C:B9:92:01 192.0.2.22:33343 192.0.2.11:4789 ip:udp 231 0
sfp28-8 28.726 13 -> 04:F4:1C:B9:92:08 B0:5A:DA:98:59:20 192.0.2.22:55792 192.0.2.13:4789 ip:udp 259 0
sfp28-8 28.726 14 -> 04:F4:1C:B9:92:08 B0:5A:DA:98:59:20 192.0.2.22:48085 192.0.2.13:4789 ip:udp 168 0
sfp28-8 28.726 15 -> 04:F4:1C:B9:92:08 B0:5A:DA:98:59:20 192.0.2.22:33343 192.0.2.13:4789 ip:udp 231 0
sfp28-8 28.726 16 -> 04:F4:1C:B9:92:08 B0:5A:DA:98:59:20 192.0.2.22:33343 192.0.2.13:4789 ip:udp 231 0
sfp28-2 28.731 17 -> 04:F4:1C:B9:92:02 04:F4:1C:B9:91:78 192.0.2.11:47744 192.0.2.22:4789 ip:udp 168 0
sfp28-2 28.732 18 -> 04:F4:1C:B9:92:02 04:F4:1C:B9:91:78 192.0.2.11:50166 192.0.2.22:4789 ip:udp 231 0
-- [Q quit|D dump|C-z pause]
If i ping switch b with the 512 bytes sized ping packets it clearly shows
[admin@DC-1-B] > tool sniffer quick ip-address=192.0.2.22 ip-protocol=udp port=4789
Columns: INTERFACE, TIME, NUM, DIR, SRC-MAC, DST-MAC, SRC-ADDRESS, DST-ADDRESS, PROTOCOL, SIZE, CPU
INTERFACE TIME NUM DIR SRC-MAC DST-MAC SRC-ADDRESS DST-ADDRESS PROTOCOL SIZE CPU
sfp28-1 15.254 80 <- 04:F4:1C:B9:88:93 04:F4:1C:B9:91:77 192.0.2.22:35339 192.0.2.12:4789 ip:udp 92 0
sfp28-1 15.254 81 -> 04:F4:1C:B9:91:77 04:F4:1C:B9:88:93 192.0.2.12:39610 192.0.2.22:4789 ip:udp 92 0
sfp28-1 15.27 82 <- 04:F4:1C:B9:88:93 04:F4:1C:B9:91:77 192.0.2.22:45426 192.0.2.12:4789 ip:udp 576 0
sfp28-1 15.27 83 -> 04:F4:1C:B9:91:77 04:F4:1C:B9:88:93 192.0.2.12:39927 192.0.2.22:4789 ip:udp 576 0
sfp28-1 15.271 84 -> 04:F4:1C:B9:91:77 04:F4:1C:B9:88:93 192.0.2.12:39610 192.0.2.22:4789 ip:udp 92 0
sfp28-1 15.271 85 <- 04:F4:1C:B9:88:93 04:F4:1C:B9:91:77 192.0.2.22:35339 192.0.2.12:4789 ip:udp 92 0
sfp28-1 16.277 86 <- 04:F4:1C:B9:88:93 04:F4:1C:B9:91:77 192.0.2.22:45426 192.0.2.12:4789 ip:udp 576 0
sfp28-1 16.277 87 -> 04:F4:1C:B9:91:77 04:F4:1C:B9:88:93 192.0.2.12:39927 192.0.2.22:4789 ip:udp 576 0
sfp28-1 17.273 88 <- 04:F4:1C:B9:88:93 04:F4:1C:B9:91:77 192.0.2.22:45426 192.0.2.12:4789 ip:udp 576 0
sfp28-1 17.273 89 -> 04:F4:1C:B9:91:77 04:F4:1C:B9:88:93 192.0.2.12:39927 192.0.2.22:4789 ip:udp 576 0
sfp28-1 18.279 90 <- 04:F4:1C:B9:88:93 04:F4:1C:B9:91:77 192.0.2.22:45426 192.0.2.12:4789 ip:udp 576 0
sfp28-1 18.279 91 -> 04:F4:1C:B9:91:77 04:F4:1C:B9:88:93 192.0.2.12:39927 192.0.2.22:4789 ip:udp 576 0
sfp28-1 19.274 92 <- 04:F4:1C:B9:88:93 04:F4:1C:B9:91:77 192.0.2.22:45426 192.0.2.12:4789 ip:udp 576 0
sfp28-1 19.275 93 -> 04:F4:1C:B9:91:77 04:F4:1C:B9:88:93 192.0.2.12:39927 192.0.2.22:4789 ip:udp 576 0
sfp28-1 20.28 94 <- 04:F4:1C:B9:88:93 04:F4:1C:B9:91:77 192.0.2.22:45426 192.0.2.12:4789 ip:udp 576 0
sfp28-1 20.28 95 -> 04:F4:1C:B9:91:77 04:F4:1C:B9:88:93 192.0.2.12:39927 192.0.2.22:4789 ip:udp 576 0
sfp28-1 21.286 96 <- 04:F4:1C:B9:88:93 04:F4:1C:B9:91:77 192.0.2.22:45426 192.0.2.12:4789 ip:udp 576 0
sfp28-1 21.287 97 -> 04:F4:1C:B9:91:77 04:F4:1C:B9:88:93 192.0.2.12:39927 192.0.2.22:4789 ip:udp 576 0
sfp28-1 22.282 98 <- 04:F4:1C:B9:88:93 04:F4:1C:B9:91:77 192.0.2.22:45426 192.0.2.12:4789 ip:udp 576 0
sfp28-1 22.283 99 -> 04:F4:1C:B9:91:77 04:F4:1C:B9:88:93 192.0.2.12:39927 192.0.2.22:4789 ip:udp 576 0
I have tested the other way round in the ring (via Switch c 192.0.2.21) the larger sized ping packet are not visible here either.
Weather this is caused by hardware offload or not - i do not know.
But even if i get no reply from the switch, both mac address table (interface bridge host) and arp table (ip arp) and Vxlan forwarding table (interface vxlan fdb) are polulated correctly
vlan 501 on switch a
[admin@DC-1-A] > interface vlan print detail
Flags: X - disabled, R - running
0 R name="vlan-501" mtu=1500 l2mtu=1580 mac-address=04:F4:1C:B9:92:03 arp=enabled arp-timeout=auto loop-protect=default loop-protect-status=off loop-protect-send-interval=5s loop-protect-disable-time=5m vlan-id=501
interface=bridge use-service-tag=no mvrp=no
Arp table on switch D
[admin@DC-2-B] > ip arp print
Flags: D - DYNAMIC; C - COMPLETE
Columns: ADDRESS, MAC-ADDRESS, INTERFACE, STATUS
ADDRESS MAC-ADDRESS INTERFACE STATUS
0 DC 10.50.0.1 00:90:0B:97:7F:D4 ether1 reachable
1 DC 192.0.2.40 04:F4:1C:B9:91:77 sfp28-1 reachable
2 DC 192.0.2.42 04:F4:1C:B9:8D:90 sfp28-2 delay
3 DC 10.50.1.21 04:F4:1C:B9:8D:91 vlan-501 stale
4 D 10.50.1.11 04:F4:1C:B9:92:03 vlan-501 failed
5 DC 10.50.1.12 04:F4:1C:B9:91:79 vlan-501 stale
Mac address is correct - and learned from the right vtep
[admin@DC-2-B] > interface bridge host print
Flags: D - DYNAMIC; L - LOCAL; E - EXTERNAL
Columns: MAC-ADDRESS, VID, ON-INTERFACE, BRIDGE, REMOTE-IP
MAC-ADDRESS VID ON-INTERFACE BRIDGE REMOTE-IP
0 DL 04:F4:1C:B9:88:95 bridge bridge
1 DL 2E:A5:71:84:94:E9 vxlan-501 bridge
2 DL E2:B1:51:6A:D2:23 vxlan-502 bridge
3 DL 04:F4:1C:B9:88:95 1 bridge bridge
4 DL 04:F4:1C:B9:88:95 501 bridge bridge
5 D E 04:F4:1C:B9:8D:91 501 vxlan-501 bridge 192.0.2.21
6 D E 04:F4:1C:B9:91:79 501 vxlan-501 bridge 192.0.2.12
7 D E 04:F4:1C:B9:92:03 501 vxlan-501 bridge 192.0.2.11
interface vxlan fdb is also correct
[admin@DC-2-B] > interface vxlan fdb print
0 remote-ip=192.0.2.13 mac-address=B0:5A:DA:98:59:20 interface=vxlan-501
1 remote-ip=192.0.2.12 mac-address=DC:A6:32:20:28:62 interface=vxlan-501
2 remote-ip=192.0.2.21 mac-address=8A:88:01:94:DA:2A interface=vxlan-501
3 remote-ip=192.0.2.21 mac-address=04:F4:1C:B9:8D:91 interface=vxlan-501
4 remote-ip=192.0.2.11 mac-address=04:F4:1C:B9:92:03 interface=vxlan-501
5 remote-ip=192.0.2.21 mac-address=B8:27:EB:48:2F:99 interface=vxlan-501
6 remote-ip=192.0.2.11 mac-address=5E:84:B3:50:8F:A1 interface=vxlan-501