RouterOS blatantly ignores pref-src. Can this really be a bug?

[lurker888]

Actually this is not a bug, but simply an effect of how wireguard and linux routing works. By the way as far as I can tell, Mikrotik uses the stock linux implementation - and yes, the same thing happens on straight linux and yes, this is a frequently asked question on linux forums as well.

To break it down:

• wg sends out the packet and does not specify a source address (this is in line with the philosophy around wg - so I wouldn’t hold my breath until this is changed)

I completely disagree with the statement it’s not a bug. Saying it’s a big is just an excuse for not fixing bad code. Yes, wireguard uses UDP and yes, UDP is stateless.

However, when the road warrier first sends a request to 192.0.2.210:51820, it creates a “virtual state” inside wireguard. It is the user space applications’ duty to then fill out the source address (see https://blog.cloudflare.com/everything-you-ever-wanted-to-know-about-udp-sockets-but-were-afraid-to-ask-part-1/#sourcing-packets-from-a-wildcard-socket). I can’t think of any meaningful reason why this shouldn’t happen. There is no reason on earth, the response packet should ever come from a different source address than it was connected to. There is no reason to leave this decision to the operating system.

I agree with the Cloudflare sentiment, and as someone who writes network services professionally, neglecting this irritates me to no end. (Btw. you can feel the writer’s frustration dripping between the lines )

The problem is that wg itself is connection-less, i.e. there is no “new” connection, just another handshake.

Let’s assume that wg did it the way you suggest. I am using my phone and in communication with a wg “server”. I’m using mobile data. I walk into my favorite coffee shop. My phone connects to their wifi. My phone sends out wg packets with the src address appropriate for my previously used LTE connection and not with the address acquired over wifi. Connection fails.

Clearly the packet should go out on the default route and pick up its src address from the pref address there.

This is what I tried to explain. The two decisions:

• don’t differentiate between peers as “server” and “client”
• support roaming IPs
  don’t allow for setting the src address.

It’s not lazy coding, but a result of a philosophy regarding the protocol. One which obviously quite a few people don’t agree with… but not exactly a bug.

• the routing decision is made and as the packet has no source address, pref addr is used - however your packet at this point has no routing mark attached, and quite correctly, the main table is consulted

Hmm… Are you sure that’s true? If the main table would have been consulted, the response packet wouldn’t even find its way out, because the default route for the main table is vlan2 (see my posted config) and it be MASQ’ed and go out there.

I am fairly certain that the packet does use the right table (“default_myas”) and right route because of this in my posted config:

/routing rule
add action=lookup-only-in-table comment="Connections must never go through other links" disabled=no routing-mark=default_isp_only table=default_isp
add action=lookup comment="main table" disabled=no table=main
add action=lookup disabled=no src-address=192.0.2.0/24 table=default_myas

Yet still, pref-src of that default route in table default_myas is ignored.

Again, if that route would not be used, not even the packet with source .177 would make its way out.

If you disagree, can you elaborate?

I do disagree. Refer to the packet flow compiled by the nice Mikrotik guys at: https://help.mikrotik.com/docs/spaces/ROS/pages/328227/Packet+Flow+in+RouterOS

• you packet ingresses the diagram titled “routing” at K
• routing decision is made - please bear in mind that you packet has no src address, and so your third rule does not mach your packet - main is selected
• src addr is set based on pref src in main
• packet goes to output
• (if snat happens, it happens here - but now this is of no interest for us)
• at the end of output “routing adjustment” occurs - here the src address is already set and your third rule now matches: the packet is rerouted according to the *myas routing table; however in the routing adjustment phase, pref addr - as I wrote - is not consulted

Hence your packet goes out on the correct interface with the src address not set as you expected.

Please feel free to say so if you disagree on how/why this is happening…

• later, even if you apply a routing mark to the packet, in the routing adjustment phase of the packet flow, pref src is disregarded (because that’s just how it works - at this point the packet has a source address and regardless of how it was acquired, it’s assumed to be correct)

This behavior leads to a number of sometimes surprising effects. And yes, regardless of Mikrotik or non-Mikrotik, dnat is the universally accepted solution.

But I’m not even applying any route marks. I just use “ip rule” that selects table “default_myas” whenever source address is 192.0.2.0/24. See my config.

Explained above. In the “routing decision” phase your rule cannot and does not match the packet, only in the “routing adjustment” does it pick up the route it eventually takes.

For other protocols this does not manifest so blatantly, because there is a clear request-response nature to them (e.g. ping: echo request - echo response; dns: again, request - response; and of course for all things TCP) and in these cases the source ip is set by the originator of the packet. Wg on the other hand does not differentiate meaningfully between client and server, and also supports roaming (changing ip addresses and ports) of each peer, and in line with this philosophy maintains that fw/routing should take care of the situation you are in. Thus every major contributor (as far as I can tell) maintains that dnat is not some workaround but an appropriate solution; of course everyone is entitled to their opinion. shrug

Understand now what your argument is. Yet I still don’t see what it would break if the response packet of a new (=unseen) connection comes from the right IP address.

Anyway, I am now OK to use DNAT but as I wrote, still no working setup for me in sight

EDIT:
By the way, adding proper VRF support to wg would solve this. Although this was proposed, it never made it into the kernel (at least the last time I was aware of such an effort…)

100%

The lack of VRF isn’t only a pain for this example but for my complete setup: I establish wireguard tunnels for uplinks over different ISPs with dynamic IP. Right now I have to use dirty hacks with route marks (see my posted config). But the correct solution would be to put the respective wireguard connections into the VRFs of the respective ISPs … and the resulting wg interfaces into my VRF “myas”.

It’s a major lacking feature

I have written Mikrotik and they answered it’s on their list and they will implement this some time, but of course, who knows how long it would take.

Let me be a bit clearer on this: someone actually wrote the code and it was not included. I know not what the future holds but if I were in a position to suggest what Mikrotik should do (which I am clearly not in,) I would probably wait until it’s settled in the kernel.

I think VRF support would be more elegant, however - and more practically - if the underlying behavior is clearly understood, dnat is in 99% of cases totally fine.

Thank you for your unusually thoughtful reply. Hope this clears things up at least somewhat.

EDIT:
Messed up snat/dnat naming. Corrected.