MikroTik

IPANetEngineer

Nice summary of info...thanks!

IPANetEngineer

Ever since I first started using MikroTik 10 years ago, I've been very impressed with how well they handle higher temperatures when a climate controlled environment is not available.

IPANetEngineer

You cannot use subnets that are directly overlapped in different VRFs in RouterOS v6...this is fixed in RouterOSv7

IPANetEngineer

You cannot use Winbox to manage an Arista device as winbox is specific to MikroTik. You can use it as a route reflector if you want though and that will give you some visibility....but is that a good fit for your network architecture? However, you can use programs like Unimus or Ansible to manage al...

IPANetEngineer

Please post the output of the following commands

/routing ospf export

/routing ospf neighbor print detail

/routing ospf as-border-router print detail

IPANetEngineer

mDNS would be a great addition to enable things like chromecast between subnets.

IPANetEngineer

That's why we normally use the ptp OSPF type on RF links and it works very well with rapid convergence.

What radios are you using that don't support multicast?

IPANetEngineer

Have you tried changing the interface queues for both ports from hardware-only to ethernet-default ? You need buffering when mixing interface speeds on any router and i'm not sure what the default buffer capabilities are for the CCR2004. I would at least try a few different queue types for the inter...

IPANetEngineer

If you can plug into it locally, use mac-telnet with winbox which will bypass all L3 and login to the router

IPANetEngineer

same problem as well

IPANetEngineer

It seems like this would be a natural extension of the DUDE. I would love to see NETCONF capabilities to standardize with the rest of the networking industry.

IPANetEngineer

I'm curious, what is the intent behind not wanting DR/BDR?

You might consider eBGP for RF links. It converges pretty fast and doesn't require multicast

IPANetEngineer

What's the lowest throughput (on the 1G) you've seen the packet loss?

IPANetEngineer

Depending on the type of wireless device you're connecting to, you need to verify the station type under your wireless interface settings. station-bridge is ideal if the other router is a MikroTik. Otherwise you can try station-pseudobridge, but you'll have some limits outside of IPv4 traffic. If it...

IPANetEngineer

If i'm understanding what you're trying to do, all you really need is to bridge the wireless interface on the hAP lite to the physical ports the security devices are plugged into. That will extend the RFC1918 network in the home router through to those devices. 1. under 'bridge' add a new bridge usi...

IPANetEngineer

We have mainly packet loss pinging from the remote end to ccr2004 gigabit interface, the problem probably occurs when the upstream traffic from the 10G->1G fullfill the ethernet..

So does the packet loss only occur when the 1G interface is full?

IPANetEngineer

Yes, you'll need to enable default-originate (if installed) on all peerings in this type of design. I would use BGP communities to set either weight or localpref I did a presentation a few years ago on this type of design at the US MUM in Denver. Here is an overview of the way we used communities. e...

IPANetEngineer

This is great news...have been waiting for this feature for a while.

IPANetEngineer

Thanks for the feedback...i'll check the hypervisor and see if it's creating a bottleneck somewhere.

IPANetEngineer

Did some work on testing the L3 performance last week in 7.1beta2 and published it today.

https://stubarea51.net/2020/10/12/mikro ... e-testing/

IPANetEngineer

I would build a highly available BRAS for PPPoE and use x86 +CHR. We wrote an article on how to design it

https://stubarea51.net/2018/04/23/pppoe ... atorsbras/

IPANetEngineer

Either an RB4011 or a CCR1009 will handle this task easily.

IPANetEngineer

While you can create a bridge and tunnel as others have said, honestly ZeroTier is the easiest way to do this and has the best performance Just install it on each computer and they'll be connected with great performance and none of the typical issues that come with L2 extensions. https://www.zerotie...

IPANetEngineer

Just curious, what application are these servers hosting? It seems like an application load balancer would be a better tool than ECMP in routing since you're expecting TCP sessions from a specific source to stay with the server they hit.

This is the exact problem a load balancer solves :-)

IPANetEngineer

Check the security settings on the ESXI VSWITCH and make sure they are as permissive as possible.

IPANetEngineer

I may have found an answer to this in the v7 docs. I am going to test this with an empty route filter and see if it only advertises interfaces in OSPF that are configured. EDIT: 10/10/2020 - This was the issue - verified it in the original lab https://help.mikrotik.com/docs/display/ROS/ROSv7+Basic+R...

IPANetEngineer

We need more details and examples of your configuration and DHCP lease output to be helpful

IPANetEngineer

Mikrotik default MPLS mtu is 1530, if I understood correctly, then it must be raised if the end customer needs qinq, for example.

yes if you want QinQ then you'll need at least 1534 to support two tags.

IP ArchiTechs has engineers in the EU (+2 time zone) as well as North and South America.

IPANetEngineer

If you're going to keep the traffic at Layer 2, then you'll be able to achieve 10Gbps between hosts that are connected to the switches. In RouterOS v7, L3 at wirespeed is possible since routing is offloaded into hardware. https://help.mikrotik.com/docs/display/ROS/CRS3xx+series+switches#heading-L3Ha...

IPANetEngineer

Can you share more about the network topology and configuration? The CCR1072 will support 15G of traffic

IPANetEngineer

A few thoughts here. 1) You don't want separate instances...that would require redistribution and isn't needed 2) You want a router that isn't a border to act as a route reflector often these are core routers - this can even be out of path like a CHR 3) You need to either set next-hop-self on each b...

IPANetEngineer

When you say teaming with ESXi 6.0 do you mean active/passive teaming or LACP?

IPANetEngineer

That's not the same. The laptop will most likely get the address using SLAAC, or possibly using DHCP but asking for a single address, not for a prefix. You can actually use Linux to test for prefix delegation. This is the syntax for debian based flavors. dhclient -d -6 -P <interface_name> --prefix-...

IPANetEngineer

Id love to understand why (in which situations) Id use PtMP? PtMP was very commonly implemented about 10 to 15 years ago in Frame Relay and ATM networks that were not broadcast capable but could otherwise have more than two hosts on a Layer 2 segment. It is sometimes used in wireless networks with ...

IPANetEngineer

Yes of course, it will be implemented.

Is VPNv6 planned for implementation as well?

IPANetEngineer

why the dhcp give same address to active user ppp ??? i check everything But I couldn't solve the problem
SERVER : 1100AHX2 (6.47.4)
I use Radius

What does the radius log say for those two connections?

IPANetEngineer

I tried with a simpler config and got the same result. This appears to be a bug and not configuration related.

IPANetEngineer

Have you tested that prefix delegation is working with another non-mikrotik device or a laptop?

IPANetEngineer

I agree that HW accelerated security devices at a low price point is a *huge* gap in the market. One feature that would really push the adoption of this is a stateful failover feature between two CRS switches so that it doesn't have to be a customized script + vrrp.

IPANetEngineer

This is typically caused by a duplication of a MAC address in two sessions. How many PPPoE clients do you have and what type of router are you using?

IPANetEngineer

If the /29 isn't routed to you and your ISP will be the GW for the /29, then you really have two options

1) Bridge all the ports together and use L2 to connect hosts together
2) Use Proxy ARP and assign a /32 public to each endpoint (as a loopback) with a route pointing to the /32 on your router.

IPANetEngineer

According to the bridge hw offload guide, that model doesn't support Bridge VLAN filtering in HW offload. Have you tried the same config under the switch menu?

https://wiki.mikrotik.com/wiki/Manual:I ... Offloading

IPANetEngineer

I really hope so, 100G on a CCR or CRS would be amazing!!

We have use cases in the data center and in service providers for 100G ports this would be perfect for.

IPANetEngineer

Are you expecting traffic to grow beyond 1G? If so, i'd go with the CCR2004. Also, if you have the budget, the cost of the CCR2004 is only a few hundred dollars more, so unless you're on a tight budget, the CCR2004 is a better choice. That said, the 4011 will do 2G to 3G of Internet traffic (1500 MT...

IPANetEngineer

I'm doing some lab work with IPv6 routing and ROS7.1beta2 CHR. I'm trying to understand why OSPFv3 is advertising a specific prefix as I didn't explicitly configure it to be advertised and it doesn't show up as an external route indicating redistribution. Here is the network topology https://stubare...

IPANetEngineer

Have you created IRR entries for your aggregate ranges? That will often cause a prefix to be dropped from advertisement to peers.

You can either do this with your RIR (aka ARIN, RIPE, etc) or an independent IRR like https://www.radb.net/

IPANetEngineer

It sounds like your prefixes either aren't being advertised or accepted/announced by your upstream.

What is the output of the following command?

routing bgp advertisements print peer=peer_name

IPANetEngineer

The first thing i'd try is to disable all FW, NAT and MANGLE rules, even if you're sure they aren't interfering and test to see if TCP passes through to that host.

IPANetEngineer

Yes the 4011 will beat the CPU of any CRS switch for routing. Using a router to act as the gateway for LAN subnets and terminate the WAN as well as a switch for port density is a very reliable setup that has been used for over 20 years. It's nice to be able to combine the functions together and the ...

IPANetEngineer

i know that MPLS is sensitive about MTU. Any recommendations / basic rules? I typically recommend that most ISPs running MPLS start with the settings below. This is from a presentation I gave at the MikroTik US MUM https://mum.mikrotik.com/presentations/US16/presentation_3327_1462279781.pdf https:/...

IPANetEngineer

Start with basic reachability checks...can you ping the GW using ICMP or ARP?

Are you peering on a directly connected subnet and not the loopbacks?

IPANetEngineer

MikroTik will only load balance (ECMP) with iBGP when peering via loopbacks and using an IGP (OSPF or RIP) or static routes.

https://wiki.mikrotik.com/wiki/Manual:B ... _with_iBGP

IPANetEngineer

Normally, you don't want spanning tree running on the bridges that connect VPLS together. Trying to converge RSTP across a large network creates a lot of problems. I would eliminate loops and turn spanning tree off. If this isn't possible, I would at least use MSTP instead of RSTP as it calculates t...

IPANetEngineer

I really hope so. BGP EVPN and VxLAN would be an amazing combination for MikroTik.

We could also use Type 5 routes for VRF signalling and have a replacement for MPLS in certain use cases.

IPANetEngineer

Here is an article I did with a basic overview of DHCPv6-PD config on MikroTik

https://stubarea51.net/2018/09/14/wisp- ... your-wisp/

IPANetEngineer

True that I consider Cisco today more really as a software company, where 5-10 years ago "hardware" was more the focus with monolithic software designs. Agree on the licensing too, you almost need a phd to understand that (same with Microsoft etc) and pricing. Like you say, sooo much equipment out ...

IPANetEngineer

So yes ... they pack a lot of performance. Should jolly well hope so for £3,500!! Do Mikrotik do a 48 port switch? I can find MikroTik CRS328-24P so would need two for £750. Serious question, what extra does the Cisco Catalyst 9300 bring to the table? For context (because i'll probably be called a ...

IPANetEngineer

If you can bond the links, it's much simpler and you should use LACP.

BGP Load balancing is for use cases when bonding is not possible.

IPANetEngineer

From an architecture perspective as it relates to PPPoE you need something that can load balance and scale laterally. This is true whether it's MikroTik or Cisco/Juniper. This will allow you to add resources as needed based on load and performance observations. Most of the time we use CHR to termina...

IPANetEngineer

This is an unfair comparison. The CCR is a fix-chassis toy compared to the ASR1006-X which was (waaaaaaaaaay)³ too powerful for such simple requirements stated...so whoever "spec'ed" this setup overdid it...A LOT. You can blast 10 interfaces with 10Gbits/sec each concurrently and it will still work...

IPANetEngineer

I think, that Mikrotik is overly complicated in a lot of things. For example Layer-2 configuration: - subinterfaces plus bridges - vlan filtering within the bridge - vlan-handling on the switch itself This is very annoying. No other vender forces you to learn three ways to do simple VLAN stuff. Rea...

IPANetEngineer

Since we run the largest MikroTik consulting firm in the world, I have some thoughts on this :) We have used MikroTik in very large environments for enterprise, data center and ISPs around the world. Like any other piece of network equipment, you have to understand how to design and deploy it. One o...

IPANetEngineer

Not needed as they are inherently secure. That doesn't make any sense....UTM is not just Internet facing, it's designed to detect internal and external threats across an organization's infrastructure. Can you share some details of how blockchain and torrent would prevent malware from replicating on...

IPANetEngineer

UTM could be done for free if all routers had Torrent and BlockChain

How does this solve the problem of analyzing and mitigating threats at L4 - L7?

IPANetEngineer

I just noticed you're talking about IPv6. In IPv4, this works However woth Ipv6, recursive routing does not work in RouterOS 6.x so iBGP does not work...the routes will advertise but are not reachable. You'll have to use OSPFv3 past the BGP Border router to advertise routing reachability. This has b...

IPANetEngineer

1. MikroTik is already working on stacking, I've talked with them at length about the need for this at the MUMs. The last I heard, MikroTik was using a standards based protocol to implement a redundant switching control plane but I don't remember which one. A decent guess would be either SPB (https:...

IPANetEngineer

can anyone please help me to configuring wireguard

i setup peer and wg interface
but traffic cant go throw the tunnel

There is a good tutorial here from Rick Frey

https://rickfreyconsulting.com/wireguard/

IPANetEngineer

This feature is available in Router OS v7.1 beta1 so you can test it, but it's not considered ready for production

https://help.mikrotik.com/docs/display/ ... col+Status

"Routing filter match community/large community lists"

IPANetEngineer

Here is one method of traffic engineering that works very well with MikroTik. From a presentation I did at the MIkroTik US MUM in 2017

https://mum.mikrotik.com/presentations/ ... 062656.pdf

IPANetEngineer

With BFD on for OSPF, you should be able to fail over in a few seconds even without fast reroute.

IPANetEngineer

Have you tried BFD?

IPANetEngineer

Just got my RB4011, and I can confirm that SFP+ works very well with passive DAC. Of course I didn't use the Mikrotik cable. ;) iperf shows around 800 MB/s against my NAS, so I'd say it actually works. Also, I didn't see any flapping. I would be careful until they officially come back with word tha...

IPANetEngineer

Can you share which prefixes you do want to advertise....it looks like you're trying to write a rule to block all IPv6 prefixes

IPANetEngineer

Hi Leandro, There are some _major_ issues with MPLS in RouterOS v6. The major ones being the lack of Fast Re-Route, RSVP paths not failing over(or back), NLRI not being updated for L3VPN's and stale labels causing traffic to disappear. It is usable for basic stuff like VPLS, but it is all done 100%...

IPANetEngineer

MikroTik is using the slow LACP PDU rate of 30s, what rate is the Cisco LACP channel using? If it's set for 1s, i'd reconfigure to 30 and retest.

IPANetEngineer

I'm setting up a couple new CCR2004 with 3 redundant paths between them on the bench using a OSPF, BGP, MPLS, VPLS configuration. When I drop out the link carrying traffic, it takes 20 seconds for traffic to re-establish on the backup link. Is this normal convergence time, or possibly a setting I a...

IPANetEngineer

/31 network between two mikrotik devices - here is how you do it First thing - you do not actually use a /31 in the IP address ( this is important ) Here is how you use a /31 network of only two IP address to get two Mikrotik devices to talk to each other. I am going to demonstrate 192.168.168.0/31...

IPANetEngineer

What are your throughput requirements?

IPANetEngineer

Have you disabled hyperthreading in the BIOS of the host?

IPANetEngineer

I'm not sure if you're running a WISP, but this MUM presentation I did will probably be helpful. Essentially you need to set MTU in 3 places: L2MTU on the interface MTU (which is Layer 3) on the interface or VLAN - This is 1500 by default which is fine for VPLS but if you want to test larger packets...

IPANetEngineer

v7beta does not have this limit.

So if I understand this answer correctly, VRFs are limited to system resources available? Which should mean that several thousand is realistically possible

IPANetEngineer

What is the output of


routing bgp advertisements print peer="peer_name"

IPANetEngineer

an iBGP peering ensures that both border routers have a full view of the routes of the other router and also the interior of the network (if you're iBGP based on the inside) In most cases, an iBGP peering is desired between two routers in the same ASN that take in full tables. Otherwise, traffic cou...

IPANetEngineer

OpenFlow is still very much in use! ISPs, Internet Exchanges and Data Centers all have use cases for OpenFlow. There is a fantastic open source controller called Faucet that is easy to get up and running. https://faucet.nz/ We have been working on a project to use OpenFlow to manage traffic engineer...

IPANetEngineer

I'm waiting on my CCR2004 to get here. Then i'm going to benchmark it with our iperf3 and BGP full table performance lab that we've used for MUM Presentations in the past.

We maxed out the CCR1072 when it first came out with 80Gbps of traffic, so we should be able to make the CCR1004 fall over

IPANetEngineer

It's also worth pointing out, that many IP transit providers (not all) will allow you to announce a prefix length greater than a /24 (i.e /25 through /32) over multiple connections to the same provider & will aggregate the prefix to /24 or less when announcing it to peers in the DFZ.

IPANetEngineer

I'm curious to know the answer to this as well.

Does the long term image load and run properly?

IPANetEngineer

However for WISP networks, hands down EIGRP. And since it's quite simple i'd wager its a lot easier to program and implement than IS-IS As consultants, we design and build a *lot* of WISP networks globally and I can probably count on one hand the number of times i've seen EIGRP used in a WISP in th...

IPANetEngineer

I don't disagree that /31 would be useful but i'd rather see the time spent on improving IPv6 support. The number of IPv6 networks we've been doing consulting work on in the last 12 months has skyrocketed. Once dual stack is in place, IPv4 public requirements don't disappear but are certainly dimini...

IPANetEngineer

when Router C receive the route it is not complete and it will not choose as best path, this is the issue

If a route is not valid or active, it often means the next hop is unreachable. Check that first.

IPANetEngineer

Are you going to put all of the 1Gbps links into the same data center?

IPANetEngineer

This is normal for most BGP routes and does not indicate a problem. This is typically due to a route being redistributed into BGP. Here are the origin types from the BGP-4 RFC (EGP is no longer used so it's listed but not important anymore) https://tools.ietf.org/html/rfc4271 a) ORIGIN (Type Code 1)...

IPANetEngineer

Does you upstream provider support BGP communities? If so, I'd use those instead of prepending which doesn't work as well anymore.

Can you post the config and routing table for each router?

IPANetEngineer

Yes, please add this to the list. More and more people are asking for it.

IPANetEngineer

Can you post configs for all 4 routers in the A,B,C,D chain and the output of the following on A and D?

mpls ldp neighbor print detail

mpls forwarding-table print detail

interface vpls print detail

IPANetEngineer

Ok - next question is re-adjacency – with OSPF can take up to 5mins will iBGP be the same

How many routes and routers do you have and what OSPF network types are you using - boradcast, point-to-point, etc?

IPANetEngineer

Is stubbing and range/summary mandatory for take advantage of the benefits? It definitely helps. The topology information is what contributes to CPU overhead and thus convergence time. When you're summarizing, you're only sending routes and not the detailed area topology info. At what level (# of r...

IPANetEngineer

You can definitely run an additional area and connect it at more than one point. This is why the backbone area exists and when there are redundant paths, they will be used. Areas will definitely help with scaling OSPF when applied to areas of the network that aren't core transit. CPU should be a bit...

IPANetEngineer

Yes, OSPF (or any IGP) is needed to provide loopback reachability for an iBGP AS so issues in OSPF that cause routes to flap can affect BGP and cause the peerings go down.

If the network is well built and designed, this shouldn't happen very often.

IPANetEngineer

You can't separate them. OSPF no matter how much you want to try and pretend differently, it has certain design restrictions that are very restrictive for service providers. Those restrictions make sense in many enterprise environments because of the vastly different topologies and real world condi...

IPANetEngineer

But we have no other protocol to use, we have OSPF and thats it. So rather than picking OSPF to 'solve problems' we have to figure out ways to solve the problems of an OSPF design. Problems like summarization at key points, but spanning multiple areas. I've had to use multiple OSPF area 0 instances...

IPANetEngineer

I have not tried to replicate this exact issue, but I did publish an article a few months ago about Juniper to MIkroTik MPLS with VPNv4 which works well. You can at least use it to verify MPLS between MikroTIk and Juniper. https://stubarea51.net/2020/01/22/juniper-to-mikrotik-mpls-and-vpnv4-interop/

IPANetEngineer

OSPF works fine for corporate/enterprise IS-IS works far better for 'service provider' environments EIGRP works in both This is **NOT** the way to look at routing protocols. Routing protocols solve problems. We have to stop looking at them as enterprise vs. service provider. OSPF and ISIS are link ...

IPANetEngineer

I would much rather have IS-IS or OSPF segment routing for WISP environments. Nobody wants to run MPLS on EIGRP when you have SR available

EIGRP while released to open standards, wasn't completely opened up so most of the best features are missing anyway.

IPANetEngineer

This is more of a general WISP post than anything but there are some specific MikroTik references in here. i've been wanting to do this for a long time but it needed some depth and attention so it's been on the back burner for a while. I finally got a basic comparison written of all the major WISP r...

IPANetEngineer

We've run into this with a few clients that run the 4011 and have used the following as a workaround while we work on a ticket with MikroTik. It seems to be more stable in the few we have tested.

Set the CPU frequency to 1200mhz

system routerboard settings set cpu-frequency=1200

IPANetEngineer

We've run into this with a few clients that run the 4011 and have used the following as a workaround while we work on a ticket with MikroTik. It seems to be more stable in the few we have tested.

Set the CPU frequency to 1200mhz

system routerboard settings set cpu-frequency=1200

IPANetEngineer

We use Unimus for our Data Center and also for our clients. It can backup, notify about config changes and execute automated commands on multiple devices. It supports MikroTik and lots of other vendors as well. It is not expensive and is an excellent tool. https://unimus.net/ https://unimus.net/imag...

IPANetEngineer

Having VxLAN support in HW on the CRS326-24S+2Q+RM would be just amazing.

IPANetEngineer

A couple of questions:

Can you ping the subnets from within your ASN? Also can you post your routing table?

IPANetEngineer

I completely get that it's not trivial. I spend a lot of time consulting in disaggregated networks where we deploy IP Infusion or Cumulus Linux on ONIE switches and sometimes we have to contend with issues in getting the FIB pushed down into the ASIC. My point is that it's possible in software and t...

IPANetEngineer

There is an important point the OP is asking about that is not being addressed. The marvell prestera ASIC that is in the CRS317 and CRS309 is capable of routing in hardware (not the CPU). The question he is asking is if MIkroTik is considering enabling that functionality (which already exists in the...

IPANetEngineer

Confederation is really designed to allow eBGP style policy enforcement within an iBGP AS and break up smaller sub-ASNs within one logical ASN. I've done migration and merging of Public ASNs a numer of times and what you really want to look at is running different instances of BGP on routers that ne...

IPANetEngineer

Now that VxLAN is supported in v7, EVPN would be fantastic to have as well to use as a control plane for VxLAN and for multihoming.

IPANetEngineer

Can you post the full config and active routing tables for the MikroTik and PF Sense device?

IPANetEngineer

So the MikroTik router is receiving the ICMP traffic from the host behind the PF Sense firewall and sending it back. Have you performed a capture on the PF Sense FW to see if it receives the traffic? It seems like your issue is in PF Sense based on the data you sent. The MikroTik is routing correctly.

IPANetEngineer

If you look correctly, he need to route 192.168.0.0/16, as 192.168.5.1 is a part of this network

You're correct, I missed that it's a /16 mask, but I believe the route will still need to be on PF Sense.

IPANetEngineer

Hello, You'll need to add a route : ip route add dst-address=192.168.0.0/16 gateway=10.100.0.60 distance=1 He shouldn't need a route on the MikroTik as both of these will be directly connected networks. More than likely (making an assumption from the drawing), the default GW for the 10.0.0.0/8 netw...

IPANetEngineer

Netwatch would probably be the easiest tool.

You can ping a VPN endpoint and then take an action when it is up or down...like enabling a firewall rule to block all traffic out a certain port to the Internet or changing the routes.

https://wiki.mikrotik.com/wiki/Manual:Tools/Netwatch

IPANetEngineer

If you are going to be moving a lot of traffic between ports at different speeds , you may be exceeding the buffer (which is required for moving traffic between ports of different speeds.

IPANetEngineer

In the lab, I just lowered the VxLAN MTU to 1400 because it was quick and easy. But in production, I would raise the IP MTU of the routed network to something larger like 9000 so that 1500 byte frames can pass without issue.

IPANetEngineer

The CCRs will be too slow until RouterOSv7 is out as a stable option with BGP. The example below is 17 minutes of convergence on CCRs. Here are some examples of convergence time for CCR vs. CHR using Free Range Routing as the RR. The full presentation from MUM US 2019 is here: PDF https://mum.mikrot...

IPANetEngineer

You're welcome! I'm not sure if an IPv6 underlay is supported, but I may try that later today

EDIT: Looks like the VTEPs and VxLAN interface group multicast address are IPv4 only for now.

IPANetEngineer

Here is a blog post I did with a VxLAN lab in EVE-NG between 3 routers and 3 linux servers:

The blog and configurations are available here:
https://stubarea51.net/2020/02/15/mikro ... ook-vxlan/

IPANetEngineer

Here is an example working lab in EVE-NG between 3 routers and 3 linux servers from an article I wrote:

Configs are in the link here:
https://stubarea51.net/2020/02/15/mikro ... ook-vxlan/

IPANetEngineer

Agreed. If we see EVPN added to BGP to support VxLAN in the future, MikroTik will see a massive increase in sales from people who need inexpensive devices to act as a VTEP.

IPANetEngineer

This meas TCP port 179 for BGP is not answering on the remote end which can be one of several things

1) The source of your peering
2) Firewall rules
3) Issue with the router you are peering with.

IPANetEngineer

Can you post your config? What type of AirFibers are you using?

IPANetEngineer

This article I wrote will probably be very helpful since you're familiar with Cisco. It translates cisco switching configs for VLANs to the MikroTik equivalent.

https://stubarea51.net/2019/02/06/cisco ... and-vlans/

IPANetEngineer

Yes, you can do this. Here is an example witjh two Loopback/bridge interfaces

[admin@lab1.iparchitechs.com] > ip route vrf add routing-mark=vrf-1 interfaces=lo1,lo2

IPANetEngineer

Typically when everything else is identical, the oldest route wins

IPANetEngineer

If you have 200 sites, then OSPF is not the right protocol anyway.

I would look at BGP and filtering to solve this problem

IPANetEngineer

I really prefer BGP for what you're trying to do as I really dislike filtering in OSPF. That said, you could filter the routes for each HQ on the ospf-out chain at the Branch. That should prevent the HQs from transiting the branch in an outage but let the branch still talk to each HQ. Probably somet...

IPANetEngineer

I agree, i'd like to see buffer numbers on all the switching chips to be able to plan. The Marvell Prestera chipset this switch is based on (98DX3236) is used in a few other platforms and the listed buffer space is 12MB for the 24 port model. https://eltex-co.com/upload/iblock/9af/MES_2324P_2348P_da...

IPANetEngineer

This is happening because you have them in separate instances. The BGP best path algorithm is only used for routes within the same instance. Otherwise MikroTik's general route selection algorithm is used between instances. From the wiki: "Best path algorithm compares routes received by a single BGP ...

IPANetEngineer

Cool....thanks for all the hard work!

IPANetEngineer

I would look at this platform on either side if you want to solve the window problem.

http://wanos.co/wan-optimization/how-wa ... ion-works/

IPANetEngineer

Agreed, this is great work.

IPANetEngineer

I think a routed Data Center Interconnect (DCI) is better than Layer 2, so yes I would route between them and then use VPLS if you must extend VLANs.

But, to be clear, this will not solve the TCP window problem.

IPANetEngineer

Here is an article we did on high availability for PPPoE servers. It has example configs in it. Hope it helps!

https://stubarea51.net/2018/04/23/pppoe ... atorsbras/

IPANetEngineer

I would recommend using OSPF for transit subnets/loopbacks and iBGP (peering to an RR with loopbacks) for customer subnets. Add MPLS if you want to deliver L2VPN or L3VPN services. Here is a topology overview: https://stubarea51.net/wp-content/uploads/2020/02/WISP-Routing-Arch.png Take a look at thi...

IPANetEngineer

But how can it help if we cannot change the window size on the tunneled clients connections ? The only way I've ever been able to deal with this is by having a WAN optimization box between the DCs like Riverbed or WanOS. We had a DC migration between DCs with 50ms latency and had to use a Riverbed ...

IPANetEngineer

iBGP is a little easier. OSPF can support ECMP and since iBGP uses a loopback to peer - advertised in OSPF as a recursive next hop, it works well.

The only caveat is if you also run MPLS - the LDP implementation that MikroTik uses is not ECMP capable.

IPANetEngineer

Are the data centers in production? How much traffic is normally on the link while you are testing? The CRS326 most likely has 12 MB of packet buffer memory (based on other Marvell Prestera DX3236 board specs) while the CRS 317 packet buffer memory is unknown. As primarily a 10 gig switch, i'd assum...

IPANetEngineer

The only way this works without using routing marks (which i'd avoid) is if you have an eBGP multi-hop peering.

https://wiki.mikrotik.com/wiki/Manual:B ... interfaces

IPANetEngineer

I would much rather see stacking. MLAG is fine for Layer 2, but it's a nightmare for Layer 3. Stacking works well for both. Considering a lot of the CRS3XX chipsets have L3 HW offload that has yet to be taken advantage of, it would be nice to be able to form an LACP chanel across two or more switche...

IPANetEngineer

This is the way.

Indeed

It would be nice to be able to run either fq_codel or cake in RouterOS for better shaping options. Please consider adding this MikroTik.

IPANetEngineer

I am anxious to test this as well. There are a number of things in BGP that v7 is supposed to fix/improve.

Hopefully we'll see a new RouterOS v7 in February with BGP added

IPANetEngineer

LISP is a protocol that Cisco helped to develop and it never really saw widespread use.

https://orhanergun.net/is-lisp-locator- ... ocol-dead/

What's the specific use case you have in mind to implement in MikroTIk?

IPANetEngineer

One idea I had was to change everything to internal addressing and set up MPLS then have a VPLS instance to each base station and run a central PPPoE concentrator in our datacenter. The problems with this setup is loss of the data-center or the concentrator would cause a 100% outage everywhere. It ...

IPANetEngineer

I sure hope so! MIkroTIk WIFI6 would be amazing!

IPANetEngineer

I would consider the CGN NAT range for this as it's not publicly routable 100.64.0.0/10 We use it extensively in our designs for PTP addressing where public addresses would otherwise be used. It works very well. You can then use a public loopback to NAT the traffic to. Here is an example: https://st...

IPANetEngineer

Intel server cards like the DA520 require Intel chipped optical SFPs or coaxial DACs.

If you are using generic SFPs (either optical or DAC), they must be chipped for Intel on the NIC side.

The CRS326 will use a generic SFP without issue.

IPANetEngineer

Yes, it is a great platform. We have used it to test the CHR for each of these presentations. CHR as a BGP Border Router https://mum.mikrotik.com/presentations/EU18/presentation_5188_1524562405.pdf CHR as an MPLS router https://mum.mikrotik.com/presentations/EU19/presentation_6291_1554448059.pdf CHR...

IPANetEngineer

What error is the PPPoE client showing in the log for not connecting?

IPANetEngineer

It's a performance issue. Hyperthreading is helpful for certain types of applications, but not for virtualized routing. There is a good explanation by @TomjNorthIdaho in the linked thread https://forum.mikrotik.com/viewtopic.php?t=138549 Why disable hyper-threading ? Hyper-Threading is a CPU trick t...

IPANetEngineer

I prefer 9.9.9.9 / 2620:fe::fe

It has malware protection and is very transparent about not storing or tracking user data.

https://www.quad9.net/policy/

IPANetEngineer

But the privacy/restriction problem will only move from the ISP resolver to the DoH resolver chosen. Whether that is an improvement, depends on the local situation. That's a great point, but at least the user has the choice of which DNS resolver to trust and it's obscured to the transit providers.

IPANetEngineer

I might be a minority here, but all this DNS over https/TLS,etc, in my opinion, has nothing to do with user's privacy at all, but it has everything to do with making ad blocking and corporate filtering obsolete. I would tend to disagree. The case you mention is only one possible application of DNS ...

IPANetEngineer

Have you done a packet capture of your video to ensure that all ports/protocols and src/dst have been properly identified to mark?

What does your config look like?

IPANetEngineer

The Maximum MTU on that model is 10218 bytes for ports 1-24 and the sfp+ ports ref: https://wiki.mikrotik.com/wiki/Manual:Maximum_Transmission_Unit_on_RouterBoards The following commands should raise the MTU for what you need. select the interface numbers to apply interface ethernet set l2mtu=10218 ...

IPANetEngineer

Are you using the same version on both sides?

IPANetEngineer

I believe in version 7, MikroTik is using the native VRF capability of the Linux Kernel, so it will be interesting to see if this is resolved once BGP is released in the beta

IPANetEngineer

Just a warning on using two /32 addresses as a workaround for the /31 problem.

We've had a number of problems with OSPF stability using this approach if you're planning on using OSPF for a dynamic routing protocol.

IPANetEngineer

What does your current config look like?

IPANetEngineer

Can you also post the routing tables for each node and the IPs of the VPLS endpoints?

IPANetEngineer

I wouldn't redistribute into OSPF for reachability. It would be better to build iBGP in your AS and then build a full table route reflector so that you can deploy a PE router to peer with customers. Here is an example and the presentation I did last year: https://mum.mikrotik.com/presentations/US19/...

IPANetEngineer

Currently, it is not possible, but because of a lot of customer requests for it, we are strongly considering to implement it! :]

This is the single most important switching feature our customers for MikroTik consulting are asking for.

IPANetEngineer

So it may be helpful to clarify that you have two options to connect the server to the MikroTik switch 1) You had originally asked about a DAC cable which is a coaxial copper cable with SFPs molded and permanently attached to the ends. You can either purchase one from Lenovo, one that is compatible ...

IPANetEngineer

Here is a sample config that may be helpful for a subscriber router from an article I did on IPv6 for ISPs It sounds like you are almost there. Make sure you have assigned a subnet from the IPv6 prefix that's delegated to the router. It needs to be on the interface/bridge that yourwired/wireless net...

IPANetEngineer

I especially enjoyed reading about the network in Serbia to detect hail. Good stuff!

IPANetEngineer

It should be the real one...VRRP only answers ARP for the common MAC address and passes the traffic to the real interface. IIRC outbound traffic should originate from the physical interface.

IPANetEngineer

Is there any reason you wouldn't use DHCP relay instead of bridging a Layer 2 tunnel segment?

IPANetEngineer

Still, SNMP monitoring of BGP parameters (peer up, number of prefixes received, number of active routes to that peer) would be welcome, e.g. for graphing.

I agree, I've been dying to get SNMP monitoring of BGP peers for a long time.

IPANetEngineer

A few things to look at

1) Cost can be set on both sides of the link, so make sure you've set the cost on the correct side
2) Cost does not cross areas, if one of the links is in a different area, that could also contribute to the problem

Can you post the MikroTik and Cisco configs?

IPANetEngineer

I’ve just received an answer from MikroTik: Thank you for the report. There is a known problem with enabled connection tracking and fragmented packets. If OSPF packet is being fragmented, then connection tracking passes it to OSPF twice, causing an error. Currently as a workaround I can suggest to ...

IPANetEngineer

It looks like you're applying this on routes learned inbound. Have you tried applying a similar policy to the routes as they are advertised out to the RR clients?

IPANetEngineer

If the routes are received by RIP from a RIP neighbor, they are not "redistributed" but rather learned routes in RIP. You can however filter the routes with a prefix list by interface. First create the prefix lists with the routes you want to permit/deny in and then another prefix list for the outbo...

IPANetEngineer

A few questions:

1) What version or versions of RouterOS are involved?
2) Are the RR clients all MikroTIk/RouterOS or are there other NOS?

IPANetEngineer

This is a pretty simple solution. You can purchase DAC cables that are flashed on one end for Lenovo and are generic at the other end (since MikroTik accepts almost all types) https://www.fs.com/c/10g-sfp-dac-1114 We've had a lot of success with FS.com when connecting hypervisors, servers, storage, ...

IPANetEngineer

It's only a matter of time before RPKI becomes a standard for most IP transit peerings. This is a feature we desperately need to keep using MikroTik routers to peer into the DFZ.

BGP FlowSpec is also important for DDoS mitigation.

IPANetEngineer

You need to use iBGP throughout your AS - redistributing in and out of OSPF for this kind of use case is something i'd avoid. Here are config examples from a presentation I did: https://mum.mikrotik.com//presentations/US13/kevin.pdf Ideally you want a full table Route Reflector to take full tables a...

IPANetEngineer

A few questions

1. How many PPPoE sessions do you have?
2. What is the total throughput?
3. Did you recently add more clients?
4. What do the CPU/Memory resources look like?

IPANetEngineer

OSPF is not an ideal routing protocol for hub/spoke tunnels, issues with one spoke can affect all the others - it's not scalable. You'll find similar guidance in Cisco as well. BGP is really the way to go as it has much better filtering options and you can assign a different AS for each spoke site t...

IPANetEngineer

Glad you found the transit fabric helpful!

One of the largest WISPs in the US who is a client of ours (2000+ towers) uses it in certain areas of their network to increase capacity between towers.

You should file a bug report with MikroTik since you found a repeatable error so that work isn't lost

IPANetEngineer

Kicking off the new year of blog posting with an article on MPLS and VPNv4 interop between Juniper and MIkroTik. Here is the article link and overview of the design with the configs from EVE-NG. https://stubarea51.net/2020/01/22/juniper-to-mikrotik-mpls-and-vpnv4-interop/ https://stubarea51.net/wp-c...

IPANetEngineer

I've used quite a few x520s in Hypervisors and they definitely need Intel-chipped DAC or SFPs. That's most likely your issue.

IPANetEngineer

Awesome, now all we need is stacking on the 40G ports. MikroTik was working on it when I talked to them in Vienna last year at MUM Europe. I hope we see that as a feature this year. It would be nice to be able to build LACP channels across multiple switches.

IPANetEngineer

This is what you're after....

https://wiki.mikrotik.com/wiki/Manual:R ... ting_rules

IPANetEngineer

add address=2001:44b8:2159:cf01::1 interface=ether2 add address=2001:44b8:2159:cf00::1 advertise=no interface=ether1 add address=2001:44b8:2159:cfff::1 advertise=no interface=pppoe-out1 Are these all supposed to be /128? I don't see a mask assigned. I wrote a brief primer for designing and deployin...

IPANetEngineer

https://apps.apple.com/us/app/openvpn-c ... d590379981

IPANetEngineer

If you want more rapid and reliable failover than Layer 2 MSTP, a routed network with OSPF or BGP would be more resilient and could failover in milliseconds using BFD.

IPANetEngineer

Take a look at OpenVPN on RouterOS. You can put it into the ethernet mode and hand out IPv4 and IPv6.

https://wiki.mikrotik.com/wiki/OpenVPN#Bridge_mode

There is a client for Android and Apple.

IPANetEngineer

edit: forgot the most important bit, thanks IPANetEngineer for chiming in! No prob....the other thing you might want to consider for mgmt if it only allows outbound connections is an SSTP outbound tunnel (which traverses NAT with no issue) to a MIkroTik CHR in Digital Ocean or AWS. That way you can...

IPANetEngineer

There is a good thread on it here.

viewtopic.php?t=136465

IPANetEngineer

This is a pretty broad topic. What features do you need and what kind of network/locations will this serve?

IPANetEngineer

The behavior you described sounds a lot like an MTU problem. Have you tried pinging from a user computer with the DF bit set to see if you can get 1500 bytes through?

IPANetEngineer

Well shucks. That adds about 23 grand to the project. I guess I'll need to rethink the whole thing. Would I be able to pay you to go over this design in a little more detail with me? Charles No problem, shoot an e-mail to consulting@iparchitechs.com with your contact info. We've designed and built ...

IPANetEngineer

Instead of using the interface list on the NAT statement, try creating two NAT rules with explicit out interface statements - one for the fiber WAN and one for the LTE interface

I've done a similar setup numerous times without issues switching from fiber to LTE on the private LAN side.

IPANetEngineer

VRF Lite in MIkroTik would be using the routing marks as a standalone without MPLS/VPNv4. This is supported.

Are you trying to use routing protocols in a VRF as well?

Search found 1262 matches