Hello!
I have a Wireguard both IPv4/v6 setup between CHR VPS (ROS 7.15) and hap ax3 (ROS 7.14.1) connected through PPPoE (but not behind NAT). Now I have a task to hide WG service packets, so I decided to raise SSTP tunnel. Then I discovered through Wireshark, that IPv4 handshake is 176 bytes, keepalive is 60, and 196/80 for IPv6 respectively. So I created mangle rules on hap ax3 for those packet size. After everything is set up, I have noticed, that for IPv6 everything works ok, but for IPv4 - after some time, WG connection seems to timeout, and Wireshark shows that handshakes are now sent by CHR instead, despite the fact that on CHR the peer is marked as Responder (so I suggest it shouldn’t send handshakes at all). Packets in mangle rule are also not counting at this point. I’ve also noticed, that despite persistent-keepalive is set to 30s, it instead coupled to 2min right after handshake packet.
Your widespan action=masquerade rule is most likely causing the issues - the initial transport packet is a “handshake” one so you send it via SSTP, but as the masquerade rule is not selective, the reply-dst-address of the Wireguard connection becomes the one attached to the SSTP interface - have a look at the output of /ip/firewall/connection/print detail where dst-address~“:46379”. Once the negotiation is finished and transport packets carrying the actual payload start to flow, they get src-nated to that (probably private) address as well since the NAT rules are only consulted while handling the first packet of each connection, and the rest of the packets belonging to that connection inherit the same treatment. So depending on how meticulous or careless your ISP is, the packets with the “wrong” source address you send directly via WAN may or may not make it to the responder peer, but the responder peer in any case auto-learns the address of the initiator peer from them so it sends its own packets to that address. Which either means all of them go via the SSTP or that they get lost as the routing at the responder peer’s ISP cannot deliver them to the initiator peer.
The wireguard packets are sent by the router itself, so their source address is chosen in the first stage of routing, so it should be the one attached to the WAN. Leaving aside that you are setting a private address using the action=src-nat rule, which means the whole thing would still fail even if the rule acted because the source address of the packets being sent via SSTP must be the public one the remote peer sees in the packets that went through WAN, the fact that the rule does not count means that the packet that has created the tracked connection in the firewall did not get the routing mark. This may be caused by the fact that the lifetime of a tracked UDP connection in the firewall is 3 minutes by default so a connection created a long time ago has survived all your reconfiguration attempts.
So disable the WG peer on the hAP ax³ and ideally also on the responder, list the connections matching dst-address~“:46379” on the hAP ax³, remove them if found, open a command line window on the hAP ax³, make it as wide as your screen allows, run /tool sniffer quick port=46379 in it, and re-enable the peer(s). You should see what is actually going on. If you don’t get it from the packet list, copy-paste it as text, replace the public addresses, and post the result.
You can currently choose from two “skins” on the forum, one of them shows the text between [code] and [/code] properly.
Now after restart it works as expected (for a while, as usual). I’ve tried the /tool sniffer thing, but there’s a ton of packets passing into it and they just fly, so I think using shark could be equal tool.
/tool sniffer
set filter-interface=all filter-port=46379 filter-stream=yes streaming-enabled=yes streaming-server=_my_laptop_
What happens after connection is just started and everything works:
(hap ax3) → (CHR): Handshake Initiation
(CHR) → (hap ax3): Handshake response
(hap ax3) → (CHR): Keepalive
But after a while, something happens, the tunnel itself may even stay alive and keep transferring all traffic, and the direction of packets is reversed, so:
(CHR) → (hap ax3): Handshake Initiation
(hap ax3) → (CHR): Handshake response
(CHR) → (hap ax3): Keepalive
All IPs shown in wireshark are static white for CHR and PPPoE external for hap ax3. I’ve noticed, that when I first tried those mangle rules, shark sometimes showed SSTP-related 172.20 src and dst IPs, but now it isn’t. This “something” is maybe some ISP action I cannot reproduce myself, so it may take hours to reappear again, but it already happened twice after wireguard interface restart.
Wireshark would be interesting if you were interested in the contents of the packets, but since they are encrypted anyway, there is not much point.
After stopping the sniffer, you can use /tool sniffer packer print to show the buffer (for some minutes, then it is automatically cleared), which is sufficient to see whether the concept works and whether the addresses are correct.
I would not be so sure as for the roles of the packets, so maybe the swapped order of packets of the “interesting” sizes means nothing.
If “PPPoE external” means a private one, then it is still critical that the src-nat rule handling the initial packet passing through the SSTP tunnel would assign to the connection the public address that the CHR sees the traffic to come from. And if that address changes during the connection, the whole thing is sentenced to fail because the initiator side will keep the tracked connection in the firewall alive, so once the external public address changes, the CHR will start receiving the “normal” WG packets from the new one but those coming via SSTP from the “old” one, so it will get confused. But such a change of the external public address should make the SSTP connection fail and restart, so it should be possible to detect that event and take measures.
In Wireshark I do filter by wg-specific packets (wg.type == 1 || wg.type == 2 || wg.keepalive) so I can detach them clearly from the data packets which are also sniffed by udp port.
PPPoE external means the address that I can see in /ip address print on PPPoE interface and on any IP detection site if I visit it from behind the router (this is the same because I’m not behind ISP NAT).
OK, I never dug that deep, given my prejudice against WG
OK, so you do have a public IP on the PPPoE - it was not clear as you have expressly mentioned a public one in case CHR but haven’t in case of PPPoE/hAP ax³, so I figured it was a private one. So ok, one more thing you don’t need to bother about.
Yep, maybe I expressed myself wrong. It’s dynamic, so may change after reboot and so on, but it’s public. That’s why the srcnat rule’s action is masquerade but not src-nat
Understood, but as the initial packet passes (or at least should pass) through the SSTP, you have to use src-nat and modify the rule using the on-up script of the PPPoE client.
Once that is fixed, why it should be important which peer is the first one to send the keepalive?
When the direction of packets is reversed, the tunnel still functions as normal. I’m doing SSTP to hide the fact of using wireguard, so even if it’s working, it’s not exactly what I want to achieve
I do understand the purpose, I would just expect that you send the distinctive types of packets via SSTP in both directions, hence the reversed order would not obstruct the purpose…?
Maybe this would be the only way, but ideally this should be one-way only, because if I attach just one additional client to vps, the complexity of mangling reverse packets grows significantly, as I should create a mangle rule per-user, create an sstp ppp profile for each and so on.
I’m curious if it’s somehow broken implementation of WG in 7.14, since I expect the responder to not send handshakes (not a WG expert, so could be wrong), and there’s no tick for peer to be a responder or not in 7.14 (this is only set on 7.15 CHR now in peer settings)
Same here, but I can easily imagine the whole “responder” thing just means that if the session towards the peer is currently not established, an arrival of a payload packet that would normally trigger its establishing is ignored, but while a session exists, the “responder” setting plays no role.
Well, I would use a single mangle rule to force use of a “via-SSTP” routing table and use the on-up and on.down scripts of the ppp profile row common for all users to add & remove routes to that table, sourcing the remote addresses from the caller-id parameter (http://forum.mikrotik.com/t/ppp-on-up-on-down-variables/94303/1) and setting the gateway to the tunnel interface name.
Handling the case where the WAN IP of the “client” side is private is more complicated as the client has to learn its outermost public address first. In a country that takes the effort to actively prevent use of VPNs I don’t expect all the interested users to enjoy the luxury of a public address.
This sounds complicated for me, I’m imagining now that a client initiates connection through sstp and has 172.20.0.x ip range. It also has wg peer set in 172.16.1.x ip range. How would I map those things together, so maybe a handshake for 172.16.1.1 wireguard peer is sent over 172.20.0.2 sstp peer and so on?
would change of language help?