MikroTik

Larsa

I would like to get some feedback from the developers.

Since this is a user forum, I believe you have a better chance of getting a response if you direct your question to: support@mikrotik.com.

Larsa

Not necessarily. It ultimately depends on how well the driver is developed specifically for each solution. With a single NIC used solely by one guest OS, the difference is probably not even measurable with modern drivers. The major difference is that a NIC using PCI passthrough (VMware DirectPath) b...

Larsa

Cron job :D You underestimate Atlassian. It's such a complicated mess. Well, Jira/Confluence might be perceived as 'messy' in the same way as ROS might be for novices. 😉 These products are complex toolkits capable of doing almost anything but requires solid knowledge and experience to set up effect...

Larsa

@mugeno - if you've already paid for it and obtained the public IP address, this guide serves as a good starting point: " Microsoft - Associate a public IP address to a virtual machine ". Here is some other good stuff about Azure networking: https://learn.microsoft.com/en-us/azure/virtual-...

Larsa

Now I get it. I completely missed the part that CMC wasn't configured with OSPF.

Larsa

Checkout "originate-default" in "help.mikrotik.com/docs/display/ROS/OSPF". It can also be combined with routing filters.

Larsa

SD-WAN has been around for over a decade and is now more or less a de facto standard so calling it 'hype' feels somewhat exaggerated. A general guideline is to consider implementing SD-WAN when your network exceeds 10 links. Anyhow, regarding this particular case it's important to consider future ne...

Larsa

@trivex, no offense intended, but a great place to start your research before buying any networking gear is always the manufacturer's own website. MikroTik has organized all its products into categories like switches, routers, and more: mikrotik.com/products.

Larsa

By "THIRD PARTY," I presume you mean third-party "cloud services." Most SD-WAN solutions offer both cloud-based services and on-premises support. If you prefer, Mikrotik ZeroTier includes an on-premises controller that makes you independent of third-party cloud services. However,...

Larsa

What's new in 7.15beta9 (2024-Mar-27 21:55): *) console - added "sanitize-names" property under "/console/settings" menu (option for replacing reserved characters with underscores for files, disabled by default); Thank you! The opt-in method is preferred when introducing breakin...

Larsa

We initially started using WireGuard but as we scaled up it became unmanageable (a real pain in the neck to be honest) to administer so we've completely transitioned to ZeroTier for OOB administration. Also, the overhead for path search traffic is negligible, even in 4G. ZeroTier is extremely easy t...

Larsa

Check out the Pro Custodibus blogs about WireGuard which are absolutely outstanding in my opinion. For example, start with "Primary WireGuard Toplogies"

Happy Easter!

Larsa

The issue is not really a configuration issue as much as a question on how the VPN protocol works, and if this can be explained. Check out the Pro Custodibus blogs about WireGuard which are absolutely outstanding in my opinion. For example, have a look at " Primary WireGuard Toplogies " I...

Larsa

Couldn't agree more. There is clearly something flawed when all sorts of workarounds pop up in the flow..

Larsa

:return [[:parse ":global $1 ; :return [\$$1 $2]"]] Yeah, that's a good one-liner. Here's another neat trick if you want to call system scripts with arguments. This also works with "[/file get /dirname/scriptname contents]" if you prefer to store your scripts in a different loca...

Larsa

Regarding 7.15beta8 (2024-Mar-21 09:12) and inconsistent rules for valid characters in filenames. Check viewtopic.php?p=1065213#p1065213

Larsa

The arbitrary acceptance and rejection of certain characters in filenames cause unnecessary support system disruptions. There is still a bug in 7.15beta8 (2024-Mar-21 09:12) that prevents our backup and version control systems from working properly when filenames contains spaces due to script incom...

Larsa

Just a friendly reminder: Never ever expose TFTP or similar services directly to the internet. Doing so poses serious security risks, otherwise you don't have to worry about CVE-2024-2169.

Larsa

What's new in 7.15beta8 (2024-Mar-21 09:12): *) wireguard - added option to mark peer as responder only (CLI only); *) route - rework of route attributes; Regrettably, I haven't spent as much time on testing as I planed, but wonder if this might possibly solve the issue with the handshake response ...

Larsa

In short:

1. If you're running CHR/x64, use IPsec. This platform can scale up practically infinitely.
2. If you're running a Mikrotik with AES hardware acceleration, use IPsec. Check throughput limitation using the 512-byte column on the product page.
3. In all other cases, use WireGuard.

Larsa

Some suggestions: Set up your own TailScale address pool , use IPv6, or switch to ZeroTier. RB5009 has built-in support for ZeroTier which allows you to pick any or multiple private subnets and also set individual static addresses on any device. There is no problem running ZeroTier and Tailscale in ...

Larsa

I hadn't looked at the ZT changes in a bit – the config has gotten grow a lot. I just don't see how RouterOS could keep up in a reasonable time frame. Yeah, it feels like I've been waiting far too long for both Multipath and Trusted Path for ROS. And yes, JSON support would be awesome! Another thin...

Larsa

Yeah, looks like we need to start collecting some dough to sort this out once and for all! ;-) The ZeroTier client library itself is very small and accessible using a single API. Configuration is managed using parameters that are either retrieved from a configuration file or controlled directly via ...

Larsa

Hello @nocivo! If you want to explore similar solutions to figure out how they work, you can search for mikrotik Cloudflare script on github.

Larsa

There are some highly important factors I think you should consider before making any decisions: Encryption and throughput bottlenecks: WireGuard encryption (ChaCha20) is software-based and lacks hardware acceleration support (on any platform) unlike IPsec. Consequently, the total throughput is cons...

Larsa

Well, I would also call those options hidden since they all are a part of the current ZeroTier version included with RouterOS which simply lacks the ability to configure them. Adding AES hardware acceleration would also be a major enhancement as well as an upgrade to v1.12. This version prevents pat...

Larsa

I'd start by hiring the Canadian Lama, he's probably dead cheap but still a rascal at finding bugs and possible workarounds!

Larsa

There are some GPO hacks using scripting that might be used as a basline but I'd never use them as a replacement for SD-WAN. You still have to support end users or the branch office with manual administration when things go south. If you prefer not to depend on a third-party web server provider for ...

Larsa

I strongly advise against using WireGuard in this case. Manually administering 150 WireGuard connections will likely be a counterproductive solution. It will probably result in complex manual administrational (nightmare) tasks with the risk of long lead times and ultimately lead to increased costs f...

Larsa

It's true that OpenVPN is often configured in a "client/server" style especially for remote access use cases. However, the same applies to WireGuard. Both of these tunnel protocols, along with IPsec and SSTP, have the flexibility to act as "initiators" or passive "responders...

Larsa

I'm sorry, but I have terrible allergies to such things so I've never dared to try!

Btw, @DarkNate, can you please explain what a "client/server" tunnel is to a dummy like me?

Larsa

Okay, I thought your question was: 'My question remains valid: why do you need spaces? Or is it just a personal decision?' (Or did I miss something??)

Larsa

@t0mm13b: *) console - replace reserved characters to backup and certificate export file names with underscores;

Yes @t0mm13b, you've nailed the core issue of this thread!
--

@infabo: I think it was stated pretty clear in the previous post. Is there anything I need to clarify?

Larsa

SUP-147326 - "v7.15beta brakes file naming and script compatibility"

Larsa

@infabo The real question to be asked is: why do you need them? @infabo: If you had read the thread from the beginning, you wouldn't have needed to ask that question. @t0mm13b: The core issues are compatibility and why Mikrotik's proposed changes would break existing scripts and support systems. De...

Larsa

I'd prefer if we focus on OP's issue of how to best preserve script compatibility when it comes to potential limitations in file naming. In my opinion, at an absolute minimum, "spaces" and printable 7-bit ASCII characters that are compatible across common file systems (Windows, Linux, macO...

Larsa

The major issue at stake here is script compatibility when using spaces (and similar common characters) in filenames, not control characters or UTF-8/16.

Larsa

The technical stuff you write about might very well be true, and I truly agree regarding the poor choices that MT is about to make in this case. As I wrote in an another comment: To maintain script compatibility as much as possible, I believe it would be easier to focus on allowed characters rather ...

Larsa

Problem is: where do you define the bounds. Characters like / : \ can also cause trouble. People have used date/time as part of a filename and ran into "inexplicable problems". At least that does not happen anymore. To maintain script compatibility as much as possible, I believe it would ...

Larsa

@jaclaz, regarding the second link, it seems less focused on the actual problem regarding script compatibility issues caused by spaces in filenames and more like 'whataboutism' disguised as academic debate. I mean, this has a serous impact for both the OP and others who rely scripts that handle spac...

Larsa

Well, no! ;-) Windows defaults to UTF-16 as its internal representation but has strong support for working with UTF-8 in addition to the legacy CP-1252 and similar encodings. For example, Notepad uses either ANSI or UTF-8. The rest of the world defaults to UTF-8. However, none are limited to legacy ...

Larsa

All major operating systems like Windows, macOS, Linux, z/OS, Android and iOS utilize UTF-8. What other OS might have the compatibility issue you are referring to?

Larsa

That's beside the point. You should NEVER EVER break script compatibility unless absolutely necessary. And the potential identity issue you're describing is merely a side effect of the change that breaks script compatibility, not the root cause! I do have a certain understanding they want to avoid c...

Larsa

MikroTik has once AGAIN managed to break script compatibility by prohibiting something as common as spaces(!) in file names. I have zero understanding of this as it affects our current solutions for version control and backup which now must be modified and tested on all nodes before we can even cons...

Larsa

This is yet another piece of evidence and major reason one should try to avoid RoS scripting in production at all costs as Mikrotik might break compatibility without notice at any time. Since this isn't the first time (and probably not the last) that Mikrotik breaks script compatibility, I think it'...

Larsa

OT - Yeah, BPF has evolved from a pure filtering mechanism into a highly versatile virtual machine (VM) or "sandbox" within the kernel. Just as Wasm, source code is compiled in user-space to bytecode and executed using JIT within the VM. eBPF is incredibly flexible and might work wonders i...

Larsa

I've made the same mistake plenty of times. My first thought that always pops up is there might be an issue with the NIC before I finally realize I forgot to activate the license, i.e. CHR is running in 'free license mode'. I think MikroTik should introduce some kind of warning when running in 'free...

Larsa

Neither DPDK nor eBPF/XDP is in any way related to SR-IOV, which is a standard hardware-level technology for I/O virtualization offering bare-metal throughput. Additionally, ROS uses Linux kernel netfilter/nftables, not Berkeley Packet Filter or DPDK which are a bunch of user-land network drivers an...

Larsa

Hi! Since this is mainly a user forum, you have better chance of getting a relevant answer directly from Mikrotik by contacting support@mikrotik.com.

Larsa

@Anav - I'm biding my time by exploring possible alternatives since I have no need for quick fixes. Meanwhile, I do appreciate and rely on your tireless effort to make life easier for the users in this forum! 😘 @Amm0: You read my mind! I was thinking of testing that along with some variations of nat...

Larsa

Considering the recent fiasco where the change of date format broke script compatibility we want to minimize script use in production environments whenever possible. And the sad thing is, the date format could have been easily fixed without breaking script compatibility. This 'small' oversight makes...

Larsa

@wfburton/Amm0, I have a similar idea that doesn't involve separate routing tables.

Larsa

Yep, that sounds about right! The whole exercise has currently resulted in two different issues: Q1. Why are WireGuard handshake responses sent through default gateway rather than the originating interface? My initial research indicates this is a known issue with some proposed fixes already sent ups...

Larsa

You'll probably have a greater chance of getting assistance in connecting VyOS with ROS if you open a separate thread for it.

Larsa

WireGuard, like IPsec, doesn't appear as a service like FTP, they have separate configuration menus. Btw, what are you trying to say using the VyOS commands?

Larsa

Yup, it's the starting point itself that creates the initial hurdle in a multi-WAN environment. I'm trying to identify how different configurations behave, for example by using different subnets on the WAN interfaces. One test I've performed is with ether1 as the default gateway and five WAN interfa...

Larsa

@anav: RoS is acrtually following correctly its Operating System code on how to route traffic. I'm sorry, but there is no such thing! The Linux network engine is configured and controlled dynamically entirely by ROS. That's how Linux-based routers operates. It does whatever you tell it to do. If yo...

Larsa

Haha, but of course! My personal take on this is that all built-in services should behave the same when it comes to routing and connection tracking. I see no obvious reason why they shouldn't.

Larsa

I'm pretty sure the standard response would be it's a feature, not a bug! :-) But it is the kernel that actually stores, manages, and executes the routing rules using nftables, it's just the configuration hassle that occurs in userland, i.e. ROS. The connection tracker is tightly coupled to the nfta...

Larsa

As the new 60Pro AC implements 802.11ay it should support AES-GCM or WPA3.

Larsa

One wouldn't need specialized DHCP scripts if Mikrotik fixed its connection tracker to use the incoming interface address as the outgoing source address. I'll try to create a simple diagram and some packet traces that illustrate the whole thing, but considering your previous response you seem to hav...

Larsa

@Anav, unfortunately you're still missing the point but Ammo seems to grasp it. In short, ROS connection tracker mishandles WireGuard handshakes. It forces response packets through the default gateway, breaking the protocol if the initial handshake came from a different interface. See Example 2 for ...

Larsa

The OP asked what type of security is used which unfortunately isn't stated in the product description. Presumably, the wireless encryption is performed with some kind of AES-GCM/WPA3, but to be sure drop an email to sales@mikrotik.com. EDIT: feel free to ask the Mikrotik sales team to update the pr...

Larsa

thanks for the answer. How did you go about configuring routing policies for multiple vans? I have set incoming connection marking and routing marking for the appropriate WAN link, but it does not work for wireguard because during the handshake, the peer that responds to the query sends traffic thr...

Larsa

Well, NO! but let me get back to you with a full trace FYI. I dare you to set up your own lab environment with just two WAN interfaces and test it yourself. You don't have to bother using dynamic IP addresses. The task you are to perform is to connect a WireGuard client with a fully functioning conn...

Larsa

@wfburton, please create a seperate thread if you are not intressed in this specific topic. @Anav, all that dst-nat, prerouting, and connection marking stuff you posted about is completely irrelevant when it comes to the handshake dilemma. Are you sure you understand where the issue occurs according...

Larsa

I'm sorry, but I don't understand what you mean by "user/group policy" and "User333 belongs to vpn333 group connect to wan333" ?? How does this in any way relate to the asymmetric routing issues that I described earlier in example 2?

Larsa

I can use the script, but I consider it a dirty work, why Mikrotik simply don't let us to assing a default gateway from dynamic connection to a routing table? This is also a mystery. I completely agree! And I truly hope Mikrotik implements a simpler solution like /routing/rule src-interface =xxxx o...

Larsa

I guess I dont understand your point then, wish I could help but its beyond my knowledge scope. It isn't that complicated. Here's a brief illustration of how the issue with WireGuard differs from a built-in service like FTP that works as expected. Let's use a couple of examples to show the handshak...

Larsa

I've done it myself so there should be no problem at all using OSPF and optional BFD for fast failover.

Another option is to use ZeroTier which automatically utilizes all available links and also enables easy access from mobile devices, home offices, etc.

Larsa

Thanks for the response but that wasn't a particularly good suggestion for a cleaner policy routing to address the issue with multiple WAN addresses. As I've mentioned several times now: 1) you are not able to make use of mangling during the handshake process until it is completed. 2) To complete th...

Larsa

Let me rephrase that for both of you!

WireGuard is an encrypted tunnel protocol that can be used in all types of topologies, including client/server, spoke/hub, mesh, and much more. @mozerd, great articles btw!

Larsa

G'day Anav, my sincere apologies if this is a bit to complex for you! :-) I meant precisely what I wrote: a conceptual question regarding issues with the internal WireGuard handshake process in a multi-WAN environment with no specific scenario in mind. One challenge with the WireGuard initial handsh...

Larsa

I have a conceptual question regarding WireGuard in a multi-WAN environment using dynamic addresses. Problem: in ROS, when a passive WireGuard peer receives its initial handshake (i.e., when connection-state = new), the state machine doesn't keep track of either the destination address or the inboun...

Larsa

@hargen: I can confirm that it works, but one has to wait for 20 attempts before receiving the message "Handshake for peer did not complete after 20 attempts, giving up," and then it goes silent. If you re-enable "Keep alive" it starts all over again. Well spotted in finding the ...

Larsa

You are welcome, have a nice weekend!

Larsa

Yeah, Cake is only implemented in v7.

Larsa

Yeah, good suggestion. If the wg-interface used for OSPF isn't listed in the LAN device list, you'll need to specify that port explicitly. This also affects the forward chain for routing.

Larsa

We are working on a similar case but we need to verify that it's not caused by a misconfiguration due to some old static routes or an actual bug. Please feel free to report back any feedback from Mikrotik.

Larsa

I'm sorry, but that simply isn't true! Are you taking advice from ChatGPT?

Larsa

Is 7.14 removing some NIC drivers?

Similar issues have been reported regarding other virtual environments. Check forum.mikrotik.com/viewtopic.php?t=205097 for possible workarounds

Larsa

Nice picture, but unfortunately it's pretty difficult to say anything else since it lacks info about networks and interface addresses. Let's begin with router 2 and 3. Btw, is this a single or multi-area topology?

Larsa

Thanks for all your comments and hoping that MikroTik will upgrade the ZT package to a higher version soon. I hope so too, but the current version of ZeroTier in ROS actually supports features like Multi-Path, Low Bandwidth, Trusted Path, as well as hardware AES acceleration. However, none of these...

Larsa

Regarding "wireguard, debug: Sending handshake initiation to peer (0.0.0.0:0)" on passive peers. This is just pure speculation and I might be completely wrong; but after some troubleshooting it seems that MNDP might trigger passive WireGuard peers to attempt to establish a connection despi...

Larsa

@strods, what about "Sending handshake initiation to peer (0.0.0.0:0)" from passive peers? Btw, IMO flooding standard "info" with misleading error messages sends wrong signals. > @Znevna: You have packets flying towards those peers. Stop the packets, the flooding will stop. Or hi...

Larsa

Something fishy is going on with passive WireGuard peers since it seems they are all trying to establish an active connection to the destination address 0.0.0.0, port 0. The WireGuard debug log is flooded with entries like: " wireguard, debug: WG-xxxx: ... Sending handshake initiation to peer (...

Larsa

I'm getting endless messages 'Handshake for peer did not complete after 5 seconds, retrying (try 2)' in log. I've upgraded a couple of lab routers and I'm getting the exact same status flooding from all passive WireGuard peers, ie those defined without endpoint addresses. This applies to both IPv6 ...

Larsa

Unfortunately, there is no built-in automatic "discovery" functionality in OSPF. All included networks/subnets must be explicitly defined somewhere. For example, if a router is connecting two areas (i.e. acting as an OSPF Area Border Router) both networks must be defined for their respecti...

Larsa

I'm not exactly sure what you mean by "OSPF for routing networks behind the router," but you have to define all networks that should be routed using OSPF. Adjacent ones don't propagate automatically. Here are a couple of short and concise step-by-step labs that might cover what you need: &...

Larsa

Ampere Computing LLC, with brands like Ampere Altra and Ampere One, is a family of processors with different design objectives where some models are optimized for networking.

Larsa

It's working alright, though you'll need to be more specific about your intentions regarding subnetting and NATting, for example if you plan to use ULA or specify a prefix, etc. Additionally, including a brief overview of your network topology might help members of this forum better understand your ...

Larsa

Great, well done! Regarding the lte1 ipv4 address you are correct as the 100.75.30.120 ip is a CGNAT address. Enable IPv6 to obtain a public ip address (IPv6 GUA).

Larsa

Okay, you have at least a registered LTE connection which is a good start. Run these commands so we might see where it cracks. Btw, feel free to mask out any public ip address if not NATed. /interface/lte/monitor lte1 once without-paging /ip/address/print /ip/route/print proplist=dst-addres,gateway,...

Larsa

Unfortunately I don't have the opportunity to participate but a colleague of mine will be there.

EDIT:
Forgot to mention that he will be wearing a bat hat if you want to pass along a message to me!

Larsa

IPv6 subnetting works just like IPv4, meaning you divide the /64 prefix into smaller parts, each of which has to use its own DHCPv6 server for the respective subnet. There are plenty of resources online. For more detailed information, Google "subnet IPv6 /64 prefix" and "MikroTik NAT6...

Larsa

Will there be a native version for macOS as well?

OT - Btw, please add support to
- detach child windows from the MDI parent area
- move the "Windows" menu (the one with all active windows) to the title bar, or make the location configurable.

Larsa

As previously mentioned, T-Mobile assigns a /64 prefix as standard and it might be pretty hard to explain the different subnet options if you're not familiar with IPv6. As a personal side note, the initial intent with IPv6 was to provide everyone with enough subnet space (prefixes) and host addresse...

Larsa

Nowadays, most MNOs typically assign a /64 prefix to mobile devices and the same applies to T-Mobile. For details regarding T-Mobile, Google " T-Mobile IPv6 /64 Prefix " or call T-Mobile tech support . If you want/need subnetting using a stationary broadband router, here are some options: ...

Larsa

AFAIK, you cannot alter the setting of Rekey-Timeout as it is most likely hardcoded to 5 seconds. Check the constants used for the timer state system in paragraph 6.1 of the paper "https://www.wireguard.com/papers/wireguard.pdf. 6.1 The following constants are used for the timer state system: S...

Larsa

Okay, then I suppose they're using some other type of traffic pacing control required by fq-codel. A potential transition to standard BQL would likely simplify code management in the long run.

Larsa

AFAIK, device drivers also need to support BQL. Since it's just a matter of pretty basic counters, it shouldn't be too complicated to implement. However, considering that BQL has been around for about 10-12 years, are you absolutely sure they haven't implemented it already or using some equivalent p...

Larsa

Google is your best friend and it’s free!

help.mikrotik.com/docs/display/ROS/Packet+Sniffer

tojaj.com/packet-capture-from-mikrotik-to-wireshark/

Larsa

You might have misunderstood or somehow missed what I wrote but the session is "controlled" by PCF. When a dedicated flow is initiated from the network its initial set of flow control parameters are retrieved from the MNO's "operations center" (OSS/BSS) which manages the contract...

Larsa

Guaranteed bit-rate might not be the only consideration for CCTV/VMS systems but I get your point. Capabilities like for example guaranteed latency, bandwidth/bitrate, QoS, and reliability within PDUs are controlled by the 5G Core Network through the "Policy Control Function" (PCF). The NM...

Larsa

@mdadmin, what are you trying to imply and what source(s) are you relying on? Check https://multirbl.valli.org/dnsbl-lookup ... 4.120.html

Larsa

@sbert, do you have a specific issue you're trying to address using dedicated flow? Dedicated flow relies on the capabilities of the user equipment and the services provided by the MNO. Some advanced 5G devices may support it but it's not a common feature on consumer devices equipped with chips like...

Larsa

Replaced with a more detailed post down below.

Larsa

It's pretty hard to say anything at all without knowing how, when, which DDNS provider, RouterOS 7 version, network topology, etc..

Larsa

@l2sverige, check for any traffic on the wg interface using Winbox Tools -> Packet Sniffer. If not, there might be a mismatch in the wg peer configuration, either with the keys or the allowed addresses.

Larsa

Haha! Well, I think my solution is WAY better since it's just a single firewall rule which restricts any source to the destination. Remember KISS ;-D ;-D

Larsa

@Ammo, thanks for very interesting info! Personally I love Parallels Desktop but for various reasons we are exploring alternative solutions. UTM/VMF might be an option when it becomes stable enough. Will definitely look into it further..

Larsa

@l2sverige - as suggestion create a new WireGuard interface, for example "WG-restricted", and place all connections (peers) that need to be restricted to 10.0.0.10-10.0.0.12 on that interface. Don't add "WG-restricted" to the LAN interface list, instead use: "/ip/firewall/fi...

Larsa

There is no "vanilla" except for the actual tunnel protocol. The resolver and wg peer setup process is implementation-specific and you can make it work using standard configuration settings on a regular Linux machine. However, in this case I am looking for a solution for MikroTik boxes whe...

Larsa

We are working with some customers where the regional NMO will soon phase out all public IPv4 addresses to be replaced by CGNAT. The NMO has implemented IPv6, though only dynamic /64 prefixes are available. To address this potential issue, we would like to prioritize IPv6 connectivity on all affecte...

Larsa

An end is being put to the misuse of CVE reports from individuals and companies outside the Linux kernel community. Hopefully, this will lead to fewer inaccurate CVE reports. [2024-02-17] phoronix.com - Linux 6.8-rc5 Released With Documented Process For CVE Security Vulnerabilities https://github.co...

Larsa

No problems running V7 on Azure. I recommend using Bicep to streamline your CHR installations for easier deployment on Azure.

Larsa

FIXED!!!

Thank you for the feedback and great to hear you’ve managed to locate the root cause. Even though it might be challenging when things don't work as expected, you usually learn a whole lot during the troubleshooting process.

Have a nice weekend!

Larsa

To begin with I think your English is almost perfect, so there are absolutely no problems understanding what you mean. Back to business: The standard system log in RouterOS for Wireguard lacks logging at the packet level so you need to use WinBox "Packet Sniffer" to trace the Wireguard ing...

Larsa

When testing your Linux WireGuard Config following link provides you with excellent clues I absolutely love the format of the Pro Custodibus blogs ! A brilliantly elaborate pedagogy using images in combination with a well-thought-out flow of explanatory text is among the best resources you can find...

Larsa

I had a quick glance at the configuration, though only for WireGuard and the firewall. Everything seems to be in order, and considering that the mobile devices are working, there probably isn't any issue with your RB2011. Thus, unfortunately you'll have to continue troubleshooting with your Linux bo...

Larsa

Here is a link to Anav's user guide "Wireguard Success For The Beginner" which might come in handy..

Larsa

@resca: Have you checked the handshake status of the Wireguard peer using WinBox? When everything is okay, the handshake timer will increment up to two minutes and then start over again. If the handshake is okay, you might have other problems like routing or a firewall blocking the payload traffic.

Larsa

Yeah, the Magic Keyboard is a winner!

Larsa

That usually works fine with "normal" keyboards. However, on a MacBook, pressing shift + grave accent + space might, under certain conditions, produce "±". That's why using opt+n might be a better choice. MacBook keyboards equipped with a hat key (^) can use it in combination wit...

Larsa

I believe that packet-level tracing provides an excellent starting point to ensure that packets reach their destination without obstacles along the way. However, it's up to you to choose the tools that best fit your situation. A tip to improve your chances of getting help in this user forum is to at...

Larsa

Well thanks, but Opt+n followed by spacebar is still the standard procedure for producing a plain 'tilde'..

Larsa

On a US Mac keyboard, use Opt+N and then press spacebar to generate "~".

Larsa

To check how the raw Wireguard packets might appear on the Mikrotik, use Winbox by going to "Tools -> Packet Sniffer". Select the WAN interface and port 13231. Click on [Apply], [Start], and finally the [Packets] button to open the window where the tracing is displayed. Remember to press t...

Larsa

@resca; since you only have one peer on the Mikrotik, ensure there isn't already an active session on it. If the handshake of the peer is under two minutes there is likely an active tunnel.

Larsa

I forgot to mention that the Wireguard endpoint in the Mikrotik also needs to match the network addresses of the received packets. Even if the Linux box is using the correct keys, ROS will simply discard the packets if the "allowed addresses" do not match the Linux address. You can enable ...

Larsa

@Axo123, test if DHCP is working properly by running WinBox "Tools -> Packet Sniffer" on the WAN interface. When the router sends out a DHCPREQUEST, you should receive a DHCPOFFER with an IP address. As a side note, some sites impose restrictions where you are allowed to use only one IP ad...

Larsa

@resca, to trace traffic on the Linux box, use for example, "tcpdump -i name-of-wg-interface". On the MikroTik, use Winbox "Tools -> Packet Sniffer" and select the wg-interface to trace packets in real-time. If you don't receive any traffic on the Linux box, you might have a fire...

Larsa

I have exactly the same behavior with WinBox on macOS/Wine (both version 8/9) when "Open in New Window" is checked. I started debugging Wine in a development environment but never managed to identify the root-couse. The problem closely resembles old MS Windows issues that occurred when a p...

Larsa

2. Zerotier: Allows one to stitch together all your subnets as if they were on the same subnet, L2 connection. Great for multicasting etc but harder to separate out users from each other as its one happy LAN. Note; Relies on zerotier servers (third party). That's not entirely true. ZeroTier default...

Larsa

There's nothing inherently wrong with PPTP any more than with GRE or even older tunnels like IPIP from the mid-1980s, but they all require encryption to secure the connection. The primary reason why PPTP is considered insecure on ROS is that Mikrotik didn't bother to implement stronger encryption me...

Larsa

No problem using SR-IOV on KVM provided the NIC and drivers support it. We have some 10-year-old legacy servers (HP DL380 G5/G6 IIRC) in our testlab to play with and they run just fine using SR-IOV. Regarding the new licensing model and considering all the frustrating comments where many feel comple...

Larsa

Another interesting platform is Nutanix Acropolis Hypervisor (AHV) which is based on the open-source KVM hypervisor and includes standard features such as live migration and VM-centric snapshots. Nutanix has tools to migrate ESXi to their platform. Read more about it in the article "All About H...

Larsa

Thank you, but it looks like a rather old post from Sep '23. I can't find any statement from ZeroTier regarding a new license model and it seems more like two customers have complained about incorrect license quotes.

Larsa

Btw, here is the new Broadcom VMware licensing model for those unlucky ones who lack the original perpetual licenses. "Foundation" is needed to enable SR-IOV, DirektPath (PCI Passthroug) etc. A one-year subscription is about 40% more expensive.

VMware lic.jpg

Larsa

@gotsprings, I noticed the Reddit discussion speculating about a possible new licensing model but haven't seen any official statement regarding this. Do you know where to find it?

Larsa

OpenNebula, Proxmox VE, KVM, Xen, XCP-ng, Virt-Manager, oVirt ... and others all utilize more or less the same fundamental Linux kernel capabilities. However, they differ in their integration methods for installation/configuration, admin GUI, Docker support, tools for operations, monitoring, online ...

Larsa

Just as a reference, here is the official Oxidized RoS plugin that supports the new v7 date header with the courtesy of Brian Candler (candlerb)
https://github.com/ytti/oxidized/blob/master/lib/oxidized/model/routeros.rb

Larsa

Certainly, license costs might have a decisive significance but the original question was primarily about performance and which platforms are available with SR-IOV. However, my comment was aimed more at a general recommendation considering Proxmox VE or proprietary solutions like vSphere/Hyper-V. Wh...

Larsa

Proxmox VE can definitely be a performant open-source solution if you are willing to invest time in how to configure PCIe Passthrough and SR-IOV , analyze and fix potential issues yourself. However, if you need data center features such as hight end performance, central administration and monitoring...

Larsa

SR-IOV is supported in almost all virtual hosts, including ESXi. It's up to the NIC device driver to implement the capabilities. I'd start by checking the SR-IOV capabilities of your NICs and drivers with the manufacturers. Similar to specialized variants such as DirectPath, DirectIO etc which only ...

Larsa

Hi, there are plenty of guides online. Here are some examples: - Azure VPN Gateway and Mikrotik IPSEC/IKE Configuration - MikroTik site-to-site IPsec VPN connection to Azure Resource Manager based gateway - Azure VPN [SOLVED] Youtube: - Easy IPSEC Site-To-Site VPN Guide, MikroTik ROSv7 Microsoft: - ...

Larsa

Thank you, what a pleasant surprise and especially that the event is held in cozy Prague. Cheers!

Larsa

CVE-2023-6200 Detail - " AWAITING ANALYSIS " This vulnerability is currently awaiting analysis. The remote attack is potentially possible in the local network only Ongoing analysis is still being conducted regarding when, how, etc. It's not possible at this time to point out which platfor...

Larsa

CPU PCI-E lanes can't handle/sustain that speed - other factors will be problem too ( example: LATENCY ). The ASR9K/NCS series can do that kind of job. ASR9x and similar models nowadays act more like "regular" linux blade servers with Cisco Linux (IOS XR). Blade cards mainly utilize stand...

Larsa

As I mentioned in another comment, in terms of CHR performance using today's modern drivers supporting DirectIO/DirectPath/SR-IOV, it's as fast as bare metal and the overhead of the supervisor is barely measurable. A properly configured virtual system can easly push many hundreds of gigabits without...

Larsa

\interface lte1 allow-roaming yes <- something like that in terminal and I've not set myself as SU ... did I make it good or not ? You had it almost right, it should be: " /interface/lte/set lte1 allow-roaming=yes " When using web admin, you should find the check box under: WebFig -> Inte...

Larsa

Since it's just the top line that is set with the actual export date, you can simply skip that using a rewrite rule in Oxidized.

# 2024-01-25 09:35:49 by RouterOS 7.12.1
# software id = KAVV-XYZQ
# . . .
# . . .

Larsa

You are most welcome! : -) Btw, fixed broken link "Getting started - First Time Configuration"

Larsa

Mikrosoft docs: - RouterOS Documentation - Getting started - First Time Configuration - wAP ac kit-series documentation Some usefull user articles: - Beginner Basics - New User Config - Firewall Setup - Other useful user articles Edit: fixed broken link "Getting started - First Time Configurati...

Larsa

why use CHR ?? is mikrotik v7 RoS runs great in bare metal... OT - In my opinion, one of the major advantages of CHR is that the platform becomes hardware-agnostic and also enables it to move or upgrade "live" including network sessions to new hw without any downtime (aka Hyper-v/vSphere ...

Larsa

According to Mikrotik's product page for hAP lite TC , it supports the latest version ROS v7.13.2 if you want to try upgrading. There's some information on their website and various posts in the forum on how to upgrade. Remember to first make a backup if you would like to go back to v6. https://help...

Larsa

Okay, why is that? As far as i know, WireGuard is supported on all platforms using ROS v7.

Larsa

Okay, that’s probably the main reason for the bottleneck.

If possible, give WireGuard a try as it tends to be a bit more lenient when it comes to software encryption.

Larsa

Check if your router model supports hardware acceleration for AES (IPSec). If not, encryption will be performed using software and the maximum throughput
will be limited to the CPU power.

https://help.mikrotik.com/docs/display/ ... celeration

Larsa

Unfortunately still software encryption on all platforms. Hopefully it will be addressed in future releases of ROS.

Larsa

Try web.archive.org ..

Larsa

Until now, the only outcome of this discussion was a few guideline posts being removed. I might be wrong, but I think @Anav foremost wants MikroTik to engage much more in practical matters regarding the forum on order to create better conditions to eliminate structural problems that unnecessarily o...

Larsa

Since WireGuard utilizes ChaCha20, which is pure software encryption, the bottleneck is almost always the CPU power. When the CPU hits 100% on either endpoint, that's the maximum throughput you will get.

Larsa

”Try to using cpu affinity for dedicated CHR cpu and SR-IOV to bypass esxi kernel for using dedicated CHR NIC.”

Yeah, that should be pinned somewhere as best practice.

Larsa

A suggestion is to start by focusing on the network interface which is generally the most crucial component whether is used as "bare metal" or as a virtual Network Interface Card (vNIC) in CHR. A well-developed driver is also a prerequisite and can be a showstopper determining whether the ...

Larsa

An LTS/SLTS kernel should for obvious reasons be a better choice, AFAIK v5.6.3 is not a such version.

Larsa

That link is for podcasters only. This is for us normal people ;-)

https://open.spotify.com/show/7sq8IetuZCDDKEvuLX3SL2

Larsa

If we're talking about AES, just search "linux arm aes instructions" on Google eg

https://www.linaro.org/blog/accelerated ... ux-kernel/

Larsa

As I mentioned earlier, it’s feasible with SD-WAN in general using open-source or paid solutions. There is no magic with the VPS; it's simply another node employing the SD-WAN protocol that might be used as default gateway to internet for the SD-WAN network. SD-WAN is by design fault-tolerant and ut...

Larsa

@Gotsprings, I don’t get what you mean by "load-balanced single IP." Zerotier, Tailscale, and most other SD-WAN solutions can utilize multipath and internal load balancing with an "exit node" to a public IP on internet. It’s just a matter of configuration.

Larsa

Confirmed, WireGuard is blocked by ISP.

Unlikely, but try changing the port number. Btw, this is OT thus please create a new thread to continue troubleshooting.

Larsa

If Mikrotik marketing was more aggressive, you could call RouterOS tunnels+mangle+scripts as a "software-defined WAN" too .

Yup, so it is!

Regarding 'black box' solutions like B.L, Gartner also expressed concern about the lack of technical details.

Larsa

@Amm0, BigLeaf is just a regular SD-WAN solution with options like public internet access branded “cloud routing”.

Larsa

Well, I beg to differ. I believe that there is absolutely no exaggeration in striving to achieve simple configuration and administration of VPN links for network management. On the contrary, SD-WAN like ZerTier is way much easier to manage compared to manually configured static links like WireGuard,...

Larsa

Just a side note before jumping on the SD-WAN train with solutions like BigLeaf, first make sure your objectives are in order before making any decisions. Fwiw, ZeroTier is already integrated into ROS v.7. As for remote management, I’d choose ZeroTier anytime over WireGuard but the latter might serv...

Larsa

The Altra series (30/64/80C) isn't exactly cheap but would probably fit pretty well as a natural successor to the CCR1036/72..

Larsa

Well, perhaps for some special case using a single tunnel.

A beefy CPU typically includes an even more beefy AES hw acceleration. If performance is a concern use IPsec otherwise invest in a Wireguard server farm. ;- )

Larsa

Bottom line to achive maximum throughput:
- Use IPsec when hardware acceleration is available at both ends.
- In other cases, use Wireguard.

Larsa

Thanks. A brief description of the network topology would be helpful for getting an idea of how everything is connected and which site that is problematic, for example: <hex local sub-net x.x.x.x ipsec> Wan xx <internet> Wan xx <windows SITE1 ipsec local subnet x.x.x.x.> <hex local sub-net x.x.x.x i...

Larsa

I If you provide a config export, it would greatly help to analyse the problem.

Did you setup policies to match the correct subnets and src-nat to allow IPsec to intercept egress packets? How is the remote site configured and do you have control over it?

Larsa

@msatter: Yes, the previous v6 kernel global routing cache (that was prone to pollution attacks) has been removed and replaced with a more efficient (faster) multi-layer cache in the v7 kernel. However, in some specific scenarios it might consume more CPU resources which could be noticeable on older...

Larsa

Once again, that's a myth and misconception spread on this forum. The current V7 kernel utilizes a more modern network stack that divides the cache into distinct layers, achieving greater efficiency where it's most needed. Some relevant reading on the subject: Routing Decisions in the Linux Kernel -...

Larsa

If the installation is intended for commercial use in an IXP as OP stated in the first post, that network diagram might become pretty complex and might not be suitable to share in this forum.

The crucial question is on which side of the IXP, or perhaps on both sides (i.e., CO=L2 or ISP=L3).

Larsa

Just BGP-EVPN for use with VXLAN. - S6730-H24X6C is an L2 switch with some limited L3 functionality. - CCR1072-1G-8S+ is a high-end L3 router . Which of the two suits best depends entirely on your business case. Hence my previous question: Is the objective of the solution to act as an interconnectio...

Larsa

Thanks, but did you notice my previous questions?

Larsa

Are you sure you have the correct model number? The S6730 series are essentially L2 switches with some L3 functionality. Is the target solution supposed to act as an interconnection point (IXP) for a Communication Service Provider using some form of L2 MPLS or pure L3 on the ISP side?

Larsa

If this thread is only about best practices for using Packet Sniffer on IPsec traffic, then this answer is OT. Plain IPsec usually doesn't pose any significant issues. If your IPsec peer is active (ie established SA for each ip), it typically involves routing problems like forgetting to set 'src-nat...

Larsa

Here is some additional info regarding PCI DirectPath:

- vSphere - Enable Passthrough for a Network Device on a Host
- vSphere VMDirectPath I/O and Dynamic DirectPath I/O: Requirements for Platforms and Devices (2142307)

Larsa

Add the wireguard interface to the interface-list LAN (Interfaces->tab “interface list”)

Larsa

Well, you either troubleshoot or downgrade. If you still want to troubleshoot the root cause, bring a laptop back home and try different key sizes that match the CCR hardware offload. Start using pure IPsec. Then test it again at work. Once everything works as expected, you can start adding other tu...

Larsa

Okay, so the network topology looks like this:

Work PC (IPsec initiator) -> Office default gateway (Fortigate) -> Home (CCR IPsec responder), where the Fortigate just acts as a regular internet gateway, correct?

Larsa

Whether the VPN tunnel is persistent or not is completely irrelevant. However, I still don't get it regarding the "PC". Please be more precise and describe what the network topology looks like in more detail and where the problem occurs: 1. PC (IPsec 'road warrior') -> Home 2. PC (IPsec 'r...

Larsa

Well, your description is very unprecise as to exactly what problem you are referring to. If it concerns a road warrior connection with, for example a smartphone using IPsec, there is nothing strange about 10-20 Mbit. Regarding the Office (Fortigate model ???) to Home (CCR1009) connection, I'd say t...

Larsa

Just a suggestion, start by conducting regular IPsec tests in both directions to verify that encryption hardware acceleration is functioning properly. Then, dig into stuff like MTU and other tunnel settings.

Btw, what encryption methods are supported in hardware by the Fortigate?

Larsa

It's completely impossible to say since you forgot to mention the small detail about what you're running at home and what encryption method is being used...

Larsa

@Samiojtm1, never ever open services like ssh for public access from internet. Instead use VPN to access your local network. Here are som usefull user guides courtesy of @Anav: - Beginner Basics - The DEFACTO DEFAULT FIREWALL Setup - New User Pathway To Config Success - Wireguard Success For The Beg...

Larsa

How protect then our microtik's ip from ddos

MikroTik and other manufacturers of regular routers typically don't have built-in ability to stop DDoS attacks. To address this issue you need to utilize external services like Cloudflare, Google Cloud Armor and similar solutions.

Larsa

Please check the product page for CPU model and IPSec performance specs. Then check the IPsec specs to determine which encryption method should be used at both ends in order to optimize hardware acceleration.

Larsa

Yeah, unfortunately. We can only hope that MT will enable hardware acceleration anytime soon for AES (where available in hw that is) which was introduced already in ZeroTier 1.6.

Larsa

WireGuard’s ChaCha20 encryption lacks hardware acceleration on all platforms (not just on Mikrotik) unlike IPsec which is the preferred way if you are looking for fast VPN throughput. Check the product IPsec specs . When vector extensions such as AVX get wider and faster like on upcoming high end AR...

Search found 1190 matches