So, 7.24.1 is out, and it has no effect on the problem my RB951Ui-2HnD "MikroTik3B" is having.
And, something really weird: in Winbox, under WireGuard, viewing the Status tab of the WireGuard interface, the "Last Link Up Time" field FLUCTUATES between "{today} 19:21:16" and "{today} 19:21:15" (which was approximately the time of the most recent reboot of the router). Testing: rebooting again now; Last Link Up Time field now again shows the most recent reboot time, though is no longer fluctuating between seconds.
(I've updated the support ticket with MikroTik, including a new supout.rif)
Blow it apart with a fresh install and start over would likely be my next step. Best case, it won't reproduce, and worst case you are right back where you are now. Other than the downtime, not really seeing a downside from here. (Well, better still would be get a hold of another unit and build it from scratch, and swap it in to test . . .).
Yeah, reinstall it. Something is probably corrupted beyond repair.
This is only strange if you don't think about it First of all, wireguard interfaces are always up, so that they come up at reboot is no surprise. The "one second back-and-forth" thing is just an artifact of having elapsed time being subtracted from the current time and rounding. Many interface stats (and last-handshake for wg) are like this.
I'm not surprised about the WireGuard interface coming up at boot time. I'm always surprised when a "last status" field can fluctuate backwards in time. Why should the WinBox display round differently on a one-second granularity display field? (Okay, don't answer that, I already know ).
So, I got it working. It was bad WireGuard configuration. That it wasted so much of my time (and yours! thank you all for your generosity!) is the result of RouterOS having inadequate /system/logging relating to WireGuard, hiding WireGuard handshake request packets from /ip/firewall/filter ... log=yes, and hiding WireGuard handshake request packets from /tool/sniffer. If any of those things had been complete enough, this would have made itself obvious from the start.
More specifically, it was an incorrect WireGuard peer public key specification (made a bit worse by the way RouterOS generates a public key for a WireGuard peer if a private key is present; in WinBox if you overwrite the public key but you leave the private key present, it discards what you've put into the public key field, and re-writes it with the - wrong - public key generated from the private key; so even when I saw and "fixed" that, as soon as I hit Apply/OK it threw away the corrected configuration data I have given it. It took a while until I noticed that fact).
But, still, this should generate a /system/logging/topics=wireguard,debug event, as well as /ip/firewall/filter/chain=output dst-port=${whatever} log=yes messages. And it doesn't.
Although there is probably something else amiss. I think the handshake request wasn't sent for some reason, because RouterOS does log failed handshake attempts (and as info, not just debug), and the packets aren't "hidden" in any way from the firewall or the sniffer. In fact these packets are routinely manipulated by the firewall.
Yes, at least one of /system/logging, firewall logging, and packet sniffing, should have seen some evidence either that it was trying and failing, or that something about the configuration was causing it to fail to try. Something else is definitely wrong. We'll see whether MikroTik support responds any further; I've provided them all of this information.
If I understand you correctly, you specified the private key of the peer, and therefore the router generates the public key from the private one and overwrites the public key. And (just for good measure) the generated public key is wrong as well. Is this the situation?
In case it is, then the whole thing becomes clearer: probably the wireguard module does some internal validation at some stage of generating the handshake, and when that fails, it doesn't send anything. It's completely conceivable that such an error is not logged.
More or less. I'm not sure exactly under what confusion I suffered at the time but, yes, I specific a private key, in the private key field of the WireGuard peer configuration, and that causes a public key to be generated from the private key. As long as that/something remains in the private key field on that WireGuard peer configuration page in WinBox, manually replacing the public key field will be ignored, and the original public key as generated from the private key will be maintained. (I think I was working under the mis-impression that WireGuard peer authentication was mutual with two keypairs, rather than one-way of only the responder being able to prove its identity by signing with its private key, and the petitioner validating that the signature matches its expected public key; with mutuality being assured by the PSK, not by two complete keypairs. I still find the overall design decisions of WireGuard to be a bit mysterious. But then, I'm old enough to remember with IPsec was new...).
And, clearly, some error is going un-logged, which wasted everyone's time.
It is exactly mutual. The local side uses its private key (given in the interface private key field) to sign the handshake request, which is verified by the responder (against the public key given in their peer configuration), who then signs their response by their private key (stored in their interface configuration), which the local side verified against the the public key stored in its peer configuration.
So both sides validate each other, but both sides only need to know their owb private key and the other's public key.
Hm. So every peer(client) signs its connection requests to any peer(responder) using that peer(client)'s main WireGuard interface's private key. I had expected there to be a unique pair of keypairs for every peer-peer/client-server WireGuard pair, but I suppose there's no practical security risk in doing it the way it's done. Other than that it takes a while to get my old brain around it.
thanks again.
(Nothing more yet from MikroTik about the lack of transparency in the failures in this case).
Yeah. Wireguard's use of public key crypto usually takes people a bit of time to digest.
Wireguard basically assumes that a device has a single unique identity per interface. It's not that unusual: e.g. an SSH server verifies itself with a single host key to every client (user), verified by the public key stored in the known_hosts file.
When client certificates are used in TLS (what is now commonly referred to as mTLS), again, the same sort of thing happens.
So it's not like wg uses crypto in some surprising or different way, it's just that in the other situations one is not directly exposed to the mechanism. (It's "someone else's job.")
WireGuard's use of public key cryptography still seems a bit different to me.
Taking your example of SSH:
Each SSH server has its one persistent keypair, and all clients of that server received that server's public key (and, on the first time they connect to that server, are asked to validate that public key's fingerprint out-of-band, or just blindly accept the risk that THAT connection had a MITM attack). Client connections habitually don't use public key cryptography, though they can, in which case the individual user's keypair is used to authenticate the client to the server - specifically, to authenticate the client user to the user database of the system on which the SSH server is running.
WireGuard, a bit differently, doesn't have users, and sort-of doesn't even have "client" and "server", being that all WireGuard connections are peer-based; rather than "users", it has endpoints (to which they is not generally a to-a-specific-human association). That the client-peer uses its WireGuard interface ""server's"" keypair to authenticate itself to its server-peer seems like a different association of keypairs/authenticable-identities to "users" than the SSH model.
I'm not saying that it's 'wrong' or that its 'insecure', just that it still seems different, and I'm still unable to convince my started-in-crypto-in-the-early-1980s brain that it's quite .... 'right'.
(Also, still no further feedback from MikroTik Support on the inadequate ROS WireGuard debuggability...)
A lot of this comes down to the peer concept as a mutual association of equals. Wireguard was specifically not designed as "yet another vpn," and the traditional server-client-user concepts translate poorly - if this is interesting to you, the original white paper is a good read.
It's much more useful to compare wireguard with mtls as used in service meshes, where similarly its endpoints that are connected and there is no fixed server-client or initiator-responder. In most situations certificates were customarily deployed with both tls_server and tls_client key usage attributes.
Now there's a whole kerfuffle about this: the browser forum decided to outlaw these sort of certs, and insist that each endpoint should have a separate cert as a server and as a client, and should select which to use based on which side initiated the exchange.
I think in the situations where the two ends of the connection are truly symmetric in their roles, the wg (and former mtls dual-use cert) arrangement makes way more sense. But considering the decision of the browser forum, you're not alone in finding it strange
MikroTik support's reply was "We're continually working to improve it". No specific comment on the inconsistency/ lack of meaningful error messaging....
To add to this topic, I also found that the endpoints didn't even attempt to connect to the other side and handshake if persistent keepalives is disabled, which is the default.
With keepalive, I believe that there should be handshakes, even without traffic.
If neither side is set to keepalive, and no traffic needs to traverse the tunnel, I think we don't expect to see any traffic at all relating to the tunnel.
First of all, wireguard interfaces are always up, so that they come up at reboot is no surprise. The "one second back-and-forth" thing is just an artifact of having elapsed time being subtracted from the current time and rounding. Many interface stats (and last-handshake for wg) are like this.
).
but, yes, I specific a private key, in the private key field of the WireGuard peer configuration, and that causes a public key to be generated from the private key. As long as that/something remains in the private key field on that WireGuard peer configuration page in WinBox, manually replacing the public key field will be ignored, and the original public key as generated from the private key will be maintained. (I think I was working under the mis-impression that WireGuard peer authentication was mutual with two keypairs, rather than one-way of only the responder being able to prove its identity by signing with its private key, and the petitioner validating that the signature matches its expected public key; with mutuality being assured by the PSK, not by two complete keypairs. I still find the overall design decisions of WireGuard to be a bit mysterious. But then, I'm old enough to remember with IPsec was new...).