I have a peculiar situation. We’re forced to use separate interfaces on our router on the same subnet, because of ISP limitations. We use ether1-3 with 3 separate addresses, but a single default GW for the router, also on the same subnet. I’ve set up Wireguard with a known working config, and see that the session initiation packets come in, say, ether3 on the address configured for that interface, but the response goes out on ether2 (or vice versa). Obviously, the connection fails.
How can I fix this? Srcnat doesn’t work for router-originated packets. I tried to mark packets with route mark and use that for routing, but it doesn’t work. The packets get tagged but traffic isn’t steered out where I expect it to (I might be making mistakes here, but let’s forget it for now, as I seek for any advice). Bind address can’t be specified for WG.
Yes, I know it’s not recommended as the outbound interface gets selected randomly, but there must be a way to fix this for this particular purpose…
It can be treated as default config with 3 interfaces (ether1-3) configured for a specific IP in the uplink public subnet and a default WG config with a single client. I traced the traffic via packet dump and noticed that following the incoming init WG packet, it’s not replied to on the same interface/IP. That’s the gist of the problem. It would be very hard to weed out the private parts of the live config, so I’ll just make a POC on a virtual instance. If I can reproduce it there, I can post that full config. If not, I’ll start working on presenting the live site’s router config, thanks.
I was able to reproduce the problem via POC in a CHR. Full config export and packet trace below. I wanted to attach the pcap, but it won’t let me (maybe I should use some specific extension).
EDIT: 10.11.12.101 is my windows client on the LAN, 192.168.13.0/24 is a network I want access to via WG and 10.11.120.0/24 is the WG network. 10.11.120.1 is assigned to the server and 10.11.120.2 is assigned to the client.
Config:
# 2023-08-19 12:32:32 by RouterOS 7.11
# software id = x
#
/interface ethernet
set [ find default-name=ether1 ] disable-running-check=no
set [ find default-name=ether2 ] disable-running-check=no
set [ find default-name=ether3 ] disable-running-check=no
/interface wireguard
add listen-port=51820 mtu=1420 name=wg1
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/interface wireguard peers
add allowed-address=10.11.120.2/32 interface=wg1 public-key=\
"2UuA5Vp+61D+qwG4ciJsq+i03WNMg3MKHrGJ3QVZ2m8="
/ip address
add address=10.11.12.155/24 interface=ether1 network=10.11.12.0
add address=10.11.12.156/24 interface=ether2 network=10.11.12.0
add address=192.168.13.1/24 interface=ether3 network=192.168.13.0
add address=10.11.120.1/24 interface=wg1 network=10.11.120.0
/ip dns
set allow-remote-requests=yes servers=10.11.12.1
/ip route
add gateway=10.11.12.1
/system clock
set time-zone-name=Europe/Berlin
/system identity
set name=mt7test
/system logging
add topics=wireguard
/system note
set show-at-login=no
/tool sniffer
set file-limit=10000KiB file-name=dump.pcap filter-ip-protocol=udp \
filter-port=51820
I’ve noted everything in OP and provided some additional info. This is what I need to work with, any suggestions are welcome that might help in this situation.
Wise choice. It didn’t seem you had the intent to help anyway.
For anyone willing to help, to re-iterate: we had arrived at this solution after testing out various other ones. Not related to WG, but before that. This is not ideal, but again, sadly, this is what I have now. I have no control over it. I need to add the IPs separately to the interfaces, period. The rest is not relevant. If we look at the POC, that in a way, mimics the live site, it’s even less relevant. MikroTik makes great routers with very flexible config options, so there must be some kind of solution… I think. Funny thing is that other services, like accessing the router via Winbox or SSH, do work. They’re on TCP, maybe that behaves differently WRT routing and/or conntrack?
You were the first who asked for more info - I greatly appreciate your input and I provided as much as I can. Patrick7 stated what’s already obvious: it’s not a good practice, I’ve already said in my first post that I know that. I also gave every piece of info I could. It is a concrete situation that I would like to ask for help with, not something else I cannot achieve. Yet I’m accused of not giving more information. I simply can’t as this is all I have, this is what I work with. These things have been ignored. I remember you helping me with an earlier issue (or multiple ones), and I still appreciate that.
I admit, saying that he had no intent to help, could be a bit much, I apologize for that.
So please, if you have suggestions, I’d appreciate hearing them. I need help with this. If it’s impossible to do, that’s also a definitive answer.
Thanks. I’ll try to be concise. We have multiple uplinks for the same public subnet on their switch, but ISP doesn’t support bonding of any kind. Customer wants to keep all 4x1G of BW. Connecting the uplinks on an MT bridge triggers ISP’s port guards and we’re blacked out. No matter what we set, (r)stp, force edge port, etc, tried basically all possibilities. Foreign traffic flows through our router, port guard kicks in, etc, etc. We called it quits on trying to solve this and chose what does work. It’s IPs added to the separate interfaces so we can keep all BW to our servers. Forwarding is fine, it’s the issue with router originated traffic. This is where we’re now. We’d rather not bother with this ISP any more. But for now, we need to solve this. We have workarounds for the situation but if possible, at least I’d keep the VPN on the router. So if you think the info above is required to better understand the problem, sorry for not providing it earlier. But as it’s pretty much cemented at this point, I saw no reason to add this background info. Now I think I’ve told everything.
To address the above points: I’ll look into VRF, thanks. What I want to achieve is simple: in this setup, WG needs to work. Ideally, I also want to be able to manage the uplink IPs without (too many) quirks. Yes, I know. This sucks and I’m peeing upwind. Sometimes happens.
Yeah, naturally, this is what I’ve started with, this would be the “normal” way of things. The issue is that the upstream switch downs our links because guards trigger. Either on BPDU packets or foreign traffic originating from our interfaces or something else (also see here). Their own suggestion was to add the IPs to separate interfaces. It works, but with… quirks.
BPDU can be disabled. A distinct separate bridge should see no other Layer 2 traffic where a shared bridge may allow other Layer 2 and if so consider switch port rules that allows only specific MAC set
A bridge consisting of ports, bridged on the other side as well, with disabled STP will:
create a loop
It will not:
balance traffic to use 4 Gbps… Oh sorry it will, but only for broadcast packets filling up all ports to their limit
Sometimes there is no real, usable and good technical solution for a problem.
Or the solution would be to pay more money for getting the service on an 10G link. Or convince the ISP of supporting bonding.
Nobody knows what this “limitation” is.