My goal is to setup a robust OOB network without using vlans to access the management interfaces of switches and IPMI interfaces of servers.
The OOB network needs to be reachable form the “House” in the topology.
Backbone.sw is a CRS326 running RoS.
Lan.sw and Admin.sw are CSS610 switches.
Here is what the topology looks like, and the problems I’m facing.
My suspicion is that I’m diving deep into L2 loop territory, along with me having some incorrect assumptions (…hopes and dreams) about how traffic flows. What works:
The OOB level of the network, via vlan9 works perfectly from the house, and me connecting directly to the OOB switch to access all interfaces. This part is 100% to my spec.
What doesn’t work:
No regular data flow between Lan.sw and Admin.sw (as part of my uplink chain) nor to the backbone.sw.
What I have tried to mitigate the situation:
I have carefully tried recommended setups on how to configure trunks and access/hybrid ports. I have also tried reducing complexity to bare minimum (ie, VLAN Mode=optional and VLAN Receive=any).
I think I’m facing a fundamental design problem rather than not being able to get the VLAN configuration right.
On setting up the CSS610 with a “management interface”..
The OOB port has “VLAN Mode=disabled” and “VLAN Recieve=any” as it is connected to dumb switch without vlan support.
Port Isolation, where the “OOB port” is unchecked from all other ports. Like this: https://wiki.mikrotik.com/wiki/File:SwOS_Management.png
The CSS610 present the same mac address on every port. Which makes the OOB switch effectively looping Lan.sw and Admin.sw together.
This could be verified by looking at the “Hosts” tab - the learned MAC addresses (between each other, Admin.sw and LAN.sw) were pointing to the OOB-port voiding them from vlan tags, and not my desired link.
To mitigate the above, I added a static host, port → mac mapping, to point to the ports that link admin.sw and Lan.sw together. Which seemingly worked according to the mac address list.
However, the overall problems remain.
Is this topology even remotely feasible? Can it be setup? Any input is appreciated
The fallback of fallbacks would be to just add a managment vlan entry, “Allow from VLAN=vlan9” on the CSS610 switches, and remove OOB from a separate switch entirely.
Granted I’ve become more interested in routing, I would very much prefer having a “as dumb as possible” OOB setup to get my lab out of troubles, heheh!
Your description and drawing don’t match so it’s really hard to know what you are trying to do. You state that you don’t want to use VLANs and then have VLANs running all over the place. Your dashed lines with “no contact” is meaning less. I assume you mean “out of band” for OOB. What are the numbers above the line to house?
I THINK you want management ports that are isolated from the other ports on each device, and a completely separate network for device management. Normally that would require plugging your computer into that dedicated management LAN in order to manage the devices. Logically that is easy enough to accomplish, but then you state it needs to be reachable from the house network.
Also note on your screen captures, hiding the port names, IPs, and MACs of devices on your home LAN is pointless. Unless you are trying to hide details from a hacker on your local LAN (in which case you have bigger problems), all you are doing is making it harder for us to help you. Public addresses MAY be a different story, but we don’t have access to your local LAN so you’re not really improving security.
So, can you clean up your drawing, description, and screen captures so maybe we can figure out what you are trying to accomplish.
Sorry if the drawing is confusing.
I’ve come to learn that it is not always obvious what is meaningsless and not, and thus I attempted to give the fullest picture of the situation.
I will address your concerns to the best of my ability.
They dashed lines are there to showcase my problem - where I expect traffic to flow but it doesn’t.
OOB = Out of band, yes indeed. I mean OOB not in the strict sense of the concept, but loosely, a net where my management interfaces live.
Numbers on the line to the house would be somewhat redundant I suppose (the picture was adapted to this post), it is the vlans that exist on the trunk.
I think I see my culprit in the description.
“My goal is to setup a robust OOB network without using vlans to access the management interfaces of switches and IPMI interfaces of servers.”
The way I’ve accomplish this is to make the “stupid OOB switch” an access switch, with an upstream port that tags all traffic to vlan9.
This means that I can access each and every device, like you say, “by plugging your computer into that dedicated management LAN”, and also allows access from “House”.
Here is hopefully an improved picture.
The problem is that I do not manage to configure properly to avoid this loop:
Should this setup be possible on CSS610 switches?
The problem is that you’re creating loops in your network. RTSP is supposed to detect them and break them (by disabling one of interfaced which form the loop). The complication in your case is that from VLAN perspective there (probably) aren’t any loops. But RSTP doesn’t care about VLANs.
There’s MSTP which does know about VLANs and you should be using it. I don’t have any SwOS drvice so I don’t know if SwOS supports MSTP or not.
Would it help to simply disable RSTP (just on the one management port, or completely if you don’t expect to have any loops)?
Then the management port on the CSS610 might work as expected, of course you must be very careful to not create any loops by accident.
Thank you!
This was a very good step forward.
Checked my VLAN configuration again, and turrned RSTP off for the blocking port.
That turned out well, traffic within each Admin.sw and Lan.sw works as intended.
I can ping between hosts on each switch, on PVID1 and vlan10.
I think the bulk of my OP/problem is fixed by this.
Yet I’m not up and running at all, as there are more components that I apparently have not figured out.
Since there is no routing happening here, but in the “house” in the topology (a pfsense machine in which I’ve got good experience in routing between VLANs).
Currently I seem to be stuck on configuring the CRS326 “Backbone switch”, as it can reach only my default PVID on my devices in the house. Neither it can reach any of the VLAN10s or PVID1 devices on the LAN.sw and Admin.sw
I’m not sure if I’d expand on that problem in this thread, or better creating a new thread as backbone.sw uses RouterOS?
CRS/RouterOS (unlike CSS/SwOS) make it very easy to set up a separate management port - simply remove that port from the bridge, and configure IP address/netmask/default route (static or DHCP) on that one interface. Then you can reconfigure the bridge any way you want, including RSTP/MSTP, without losing management access. I wish all devices with SwOS could also run RouterOS - even using an old, cheap, slow CPU like one in RB931-2nD should be good enough to run RouterOS just for management while the switch chip itself handles the heavy traffic in hardware. Unfortunately, for CSS610 series there is no such choice.
