MikroTik

Hello,

Today we have some switches where we do BGP transit and L3/L2 to our Hypervisors and VM.
Our Hypervisors has local IP instead VM which are used for public and we have two vlans like public and private.
We are about to migrate our DC and separate L2 from L3 finally and we won't mix them so mutch.

Things is to avoid outage as much as we can.
We want to do stack switches where there are connected our Hypervisors and our fiber BGP, which in train to move at Mikrotik Router.

We use those Dell switches for aggregation our Hypervisors and they are connected Active/Active LACP to avoid downtime if one link comes down. Finally this is going to be in stack-switch.

Another Switch is used to connect second ISP who cares for our fiber clients but we don't care BGP, we only care for vlan interface configuration to pass their traffic inside our Network trunk to the Edge. We use only a gigabit port link from second ISP after we trunk to our internal network.

Now I am going the Core Switch to replace with another 2 dell switches in stack and connect my Hypervisors alone. I am confused a little about the second ISP if I am going to connect our fiber clients directly on Mikrotik router or not.

Once I achieve this, I am going to move IP routes to the Mikrotik and vlan as well.

What is the best way to avoid outage, I am very interested to hear from someone experiences in kinda complex things.

Thanks

We work on this type of design frequently.

I would suggest a switch-centric architecture where all of the links terminate in the switch stack and you use LACP to connect the MikroTik routers and hypervisors. Then connect the internet circuits on different switches in the stack for redundancy.

Switch centric design example:

Making switch stack a single point of failure is questionable design. Ill better stick to more disagregated design.

We've been incredibly successful with switch-centric designs over the years and have deployed it on every continent except Antarctica. Switch stacks are not a single point of failure as they form a pair of HA switches that are logically a single switch from sa spanning tree perspective.

Using a switch stack at the core of a small data center network provides a number of benefits.

1) In MikroTik routers, which are CPU and not ASIC based, it allows for the traffic to be spread across multiple lanes into the CPU for a single transit subnet and not just one. This is especially important if you plan to run MPLS as there is no ECMP for LDP in RouterOS and you'll be unable to increase capacity by adding links at layer 3.

2) Operational efficiency - by using VLANs to define traffic paths instead of ports, the physical layer is abstracted and changes become far easier and more templated. New services require a new VLAN interface and subnet under the LACP channel.

3) Upgrades or replacement of routers become very quick and easy and less impacting, by putting all of the config on VLANs, very few changes are required to the config to move to another router. Because the switches terminate all physical connections, replacing or upgrading a router doesn't require disruption of cabling to other routers.

4) Network Function Virtualization (NFV) - If you want to connect a hypervisor and run virtual routers, you need a Layer 3 switch stack as Layer 3 MLAG on disaggregated switches is very unreliable. A layer 3 switch stack provides a single Layer point for NFV routers to connect to and form routing adjacencies. This is a critical element in running CHRs successfully.

5) Increase the number of physical connections available. If you run out of ports on the router, your design will not scale successfully. Switches have far more ports than routers and more services and systems can be connected in this design and aren't dependent on the physical number of ports on a router.

When we design for larger ISP or Enterprise networks, we'll often use a hybrid approach and design a Layer 3 MPLS backbone with a non-mpls switch centric stack connected to the MPLS backbone at Layer3. This provides the best of both worlds as resources in a data center can leverage the switch-centric design to deliver local services and use MPLS transport to reach other data centers.

Hello,

This is what want to achieve in the future. But it looks for the future as per moment we have some different issues even with IP subnetih, bcoz when I joined to this enterprise I found a mess around.

Now my goal is only to avoid outages(downtimes) as mutch as I can to place router in the right manner.

I would like to hear for some engineering skills who did in the past.

We work on this type of design frequently.

I would suggest a switch-centric architecture where all of the links terminate in the switch stack and you use LACP to connect the MikroTik routers and hypervisors. Then connect the internet circuits on different switches in the stack for redundancy.

Switch centric design example:

I've done a lot of Enterprise networks and there are some key things when you're trying to migrate and the network is a mess.

1) If you don't have detailed documentation of how the current network is laid out, take the time to create it.

Layer 1 - Document physical connections to all network equipment
Layer 2 - Document all VLANs in use and the spanning tree root for each VLAN
Layer 3 - Document subnets and routing protocols

2) After documenting, fix any problems you have found that are causing stability issues before you migrate so that you don't bring existing problems into the new network.

3) Build the new network core as a standalone with new hardware and connect a temporary Layer 2 and Layer 3 link to the old network core. Then you can begin migrating hosts and systems over to the new network.

Once you've migrated all the links and end systems, you can turn off the legacy network core.

Thanks for your comments, IPANetEngineer!

It is pleasant to read such informative answer.
When i was writing about switchstack i was thinking about "entry level" stacks.
Here, i presume, you are talking about high grade stacks, that can be maintained without downtimes.

I have only single link to upstream bgp, should I run BGP on Mikrotik router or switch? Whats your best thought...

Regards,

If you run it on a switch, you can ask your upstream provider if you can maintain dual peerings and that way you'll have edge router redundancy if you lose a router or need to upgrade the RouterOS code, it can be done without an outage by failing traffic from one to the other.

It still doesn't help if you have an outage of that fiber upstream, but there's not much you can do about that except ask for another physical handoff from your ISP.

This is quite clear and I am going to start from this point.

Another problem is that there are 2 PUBLIC IP range which are routed from two virtual machine in different vlans.

So I want to use Mikrotik for doing this Routing. This is the most what I have many doubt bcoz they are routed directly in ethernet. I need to place all those routes to Mikrotik and delete those VM machine which makes routes to other VM and particullary to some fiber clients.

I am studying everyday these design and it looks clear to me everyday.
If you have a deep schema or a configuration for this schema it will forward me more deep.

I want to avoid using Routing on VM machine, unless there is a special things incoming in the future and I am thinking for my fiber clients who comes directly to a gigabit and I spare them into vlan switching and out to ISP like (Vlan stack access)