After seeing the notes in 7.24 about Hardware VXLAN now supporting vlans, I thought I'd give it a shot again (After my previous attempts here: Hardware VXLAN Broken recently? )
It works, AND it works using bridged ports, so you don't need to jump through hoops to try to only use a routed, non-bridged interface like the Howto says.
What they MEAN to say is that 'traffic can not leave the same interface it enters, otherwise it leaves untagged'. I kinda think this is a bug in the programming of the switch chip, but whatever.
Here's how I managed to get it to work.
In this example, your VTEP Peers are on VLAN999, and your VTEP traffic is going to be entering, tagged, via sfp1. You want to bring VLAN888 into this switch and have it leave via sfp2 (or any other ports. Just not sfp1)
- Start with a factory-reset CRS317, with two DACs to your main switch, in sfp1 (for ingress) and sfp2 (for normal traffic)
- Fix l2mtu on all the interfaces (
set [ find ] l2mtu=10218) - Create a bridge br0, and add sfp2 (and anything else you want, EXCEPT for sfp1) to it, creating your normal VLANs and IPs so you can access the switch. When you're happy with it, enable VLAN Filtering on it. If you lose access to the switch, factory reset and start again.
- Add VL888 to br0, and set it to be tagged on sfp2 (and any other ports, again, NOT SFP1)
- Enable L3 Hardware Offloading (Switch->Settings->switch1->L3 Hw Offloading). Also double check that the bridge ports have H next to them to confirm they're all being switched in hardware.
- Tricky Bit! Add sfp1 to br0, BUT, you need to remove sfp1 from the spanning tree domain. The easiest way to do this is to set it to be an edge port when you're adding it and ensure that it's set to 'admit only VLAN Tagged'
- Add VL999 to the bridge, and set it to be tagged on br0 and sfp1 ONLY. DO NOT ASSIGN IT TO ANY OTHER PORTS (If you do, the switch chip gets confused and you will have to reboot the switch. Yes seriously)
- Create br0/VL999 and assign the VTEP IP address to it
- If you're using OSPF or BGP, get that sorted out so that you can reach the other VTEP endpoints.
At this point you should be able to ping your other VTEP endpoints. Check your bridge ports to confirm sfp1 is an edge port, and all the rest are Forwarding=yes.
Basically, you have a bridge set up, you can ping your vtep endpoints, and VL888 is ready to go.
Create your VXLAN interface, in this example, 'vxlan888', with the VNI of 10888 (or whatever you want, I just don't like to accidentally mix up a VLAN and VXLAN, so I make sure the numbers are different), and set the bridge to be br0 and PVID being 888.
Add your VTEP Endpoints to the vxlan and it will switch to hardware mode (A 'H' will appear in the first column of the interface)
That's it! That's all it needs.
The most common issue I discovered when working this out is that if the switch chip gets confused it will either send traffic out the egress port (sfp2 in this example) as untagged, or, it won't send BUM traffic to the other endpoints. The only way to resolve this is to reboot the switch. Disabling/re-enabling L3HW does not fix it. It needs a complete reset.
The end result is this! https://mastodon.au/@xrobau/117142965429483736
Remember: If things aren't working as expected, just reboot the switch. If you've done everything right, it's a bug. Reboot!
Good luck!
