Tunneling VLAN traffic over Wireguard

verbylab · Fri Jul 02, 2021 3:43 am

Hello,

First of all, I'd like to say thanks for all of the great resources in this forum. I have been reading loads and learning a lot, but I am now stuck on my little VLAN <-> Wireguard setup.

Setup and goals:

MikroTIk hAP ac2 that connects a switch to the internet and provides IP services and routing (RoaS) (I followed viewtopic.php?f=23&t=143620 for the base setup: VLAN, DHCP, PPPoE)
One trunk port carries untagged and tagged traffic between the switch and MikroTik router (native LAN and one VLAN) ✅
Router assigns IP addresses for the LAN and VLAN ✅
Untagged traffic (native network) should flow over the PPPoE connection ✅
VLAN traffic should be tunneled through the Wireguard connection (I am using the v7 beta) ❌

After having established the base, I followed viewtopic.php?f=1&t=173952 for the Wireguard setup. The Wireguard connection stands and I can ping the peer, but I am struggling to get VLAN traffic routed through the Wireguard interface. I think I am missing something in the routing and firewall section.

Here is my config (/export compact):

# Interfaces
/interface bridge
add admin-mac=XX:XX:XX:XX:XX:XX auto-mac=no comment=defconf name=bridge vlan-filtering=yes

/interface pppoe-client
add add-default-route=yes disabled=no interface=ether1 keepalive-timeout=disabled name=pppoe-out1 \
    password=***** user=*****

/interface wireguard
add listen-port=13231 mtu=1420 name=wireguard-1 private-key=\
    "XXXXXXXXXXXXXXXXXXX"

# Edit: Error 1 was located here. WebFig v7.1 beta6 doesn't allow to specify allowed-address in the UI (bug)
/interface wireguard peers
add comment=peer1 endpoint-address=HOST_IP endpoint-port=51820 interface=wireguard-1 \
    persistent-keepalive=30s public-key="XXXXXXXXX" allowed-address=0.0.0.0/0

/interface vlan
add interface=bridge name=vlan20 vlan-id=20

/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
add name=VLAN

/interface bridge port
add bridge=bridge comment=defconf ingress-filtering=yes interface=ether2
add bridge=bridge comment=defconf interface=ether3
add bridge=bridge comment=defconf interface=ether4

/interface bridge vlan
add bridge=bridge tagged=bridge,ether2,ether3,ether4 vlan-ids=20

/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1 list=WAN
add interface=ether5 list=LAN
add interface=pppoe-out1 list=WAN
add interface=vlan20 list=VLAN
add interface=vlan20 list=LAN
add interface=wireguard-1 list=VLAN


# Adresses
/ip address
add address=192.168.1.1/24 interface=bridge network=192.168.1.0
add address=192.168.20.1/24 interface=vlan20 network=192.168.20.0
add address=192.168.88.1/24 interface=ether5 network=192.168.88.0
add address=10.0.0.5/8 interface=wireguard-1 network=10.0.0.0


# DHCP
/ip dhcp-client
add comment=defconf disabled=no interface=ether1

/ip pool
add name=dhcp-pool-native ranges=192.168.1.100-192.168.1.254
add name=dhcp-pool-vlan20 ranges=192.168.20.100-192.168.20.254
add name=dhcp-pool-mgmt ranges=192.168.88.100-192.168.88.254

/ip dhcp-server
add address-pool=dhcp-pool-native disabled=no interface=bridge name=dhcp-native
add address-pool=dhcp-pool-vlan20 disabled=no interface=vlan20 name=dhcp-vlan20
add address-pool=dhcp-pool-mgmt disabled=no interface=ether5 name=dhcp-mgmt

/ip dhcp-server network
add address=192.168.1.0/24 dns-server=1.1.1.1 gateway=192.168.1.1
add address=192.168.20.0/24 dns-server=1.1.1.1 gateway=192.168.20.1
add address=192.168.88.0/24 dns-server=1.1.1.1 gateway=192.168.88.1

# DNS
/ip dns
set allow-remote-requests=yes servers=1.1.1.1

/ip dns static
add address=192.168.88.1 comment=defconf name=router.lan


# Firewall
/ip firewall filter
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment="defconf: accept to local loopback (for CAPsMAN)" dst-address=\
    127.0.0.1
add action=accept chain=input comment="" in-interface-list=VLAN
add action=drop chain=input comment="defconf: drop all not coming from LAN" in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=\
    established,related hw-offload=yes
add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf: drop all from WAN not DSTNATed" connection-nat-state=\
    !dstnat connection-state=new in-interface-list=WAN

# NAT
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" ipsec-policy=out,none out-interface-list=\
    WAN
add action=masquerade chain=srcnat out-interface=wireguard-1

# Routing
/ip vrf
add list=VLAN name=vrf_vlan20

/ip route
add dst-address=0.0.0.0/0 gateway=wireguard-1@vrf_vlan20 routing-table=vrf_vlan20

Besides the overall setup, I was wondering what the advantage of a VRF table is compared to a simple firewall mangle rule (routing mark) and the creation of a standard routing table and route referring to that routing table. Both seem to use the same underlying concept, but I am not sure how they differ apart from some automatic routes being set up.

Also, I believe the hAP ac2 has a switch chip, but I configured all VLAN settings in the Bridge section of the device. Would it be wise to optimise this setup to take some load off the router's main CPU? Or is the setup above considered good practice?

Thanks a lot!

mducharme · Fri Jul 02, 2021 4:43 am

You are missing allowed-addresses it looks like, and possibly other things are wrong.

Have a look at this thread, it may be helpful: viewtopic.php?f=23&p=865133

anav · Fri Jul 02, 2021 4:55 am

Should have raised the topic in the beta forum if using beta firmware.

pcunite · Fri Jul 02, 2021 6:10 am

I will soon be looking into a solution to enable remote staff to use physical telephony devices (VoIP phones) alongside their personal laptops running behind their home internet service plans. Allowing them to VPN into the corporate network using Wireguard, running on a MikroTik, is the goal.

My thinking is that corporate would install a WireGuard enabled MikroTik server behind whatever their main router is and then the home user would be shipped a preconfigured WireGuard enabled hAP. The home user would plug the hAP into a free port on their home router, then plug the phone into the hAP. With the laptop they would have the option of plugging in or using the wifi corporate SSID from the hAP. There will also be soft-phones on PCs and apps on mobile devices.

Hopefully this will be a drop in solution and everything just works. Naturally, there will be a port forward on the corporate side to the WireGuard server. Now, the fun part will be how to keep the phones a part of the VoIP VLAN and the laptops on the data VLAN.

verbylab · Fri Jul 02, 2021 12:40 pm

Should have raised the topic in the beta forum if using beta firmware.

I could have, but the beta forum has a tagline of "Please report all issues with RouterOS v7beta" and until I read @mducharme's tip on the allowed-addresses glitch in the UI, nothing indicated that my problem is caused by ROSv7. And having fixed this one issue, my solution is still incomplete. Once I have studied and trialled & errored my way through to a working solution, I plan to summarise all (unreported) bugs related to Wireguard and post it in the beta forum, so that RouterOS developers can work with solid evidence rather than helping a new MikroTik customer learn the basics.

verbylab · Fri Jul 02, 2021 1:24 pm

You are missing allowed-addresses it looks like, and possibly other things are wrong.

Have a look at this thread, it may be helpful: viewtopic.php?f=23&p=865133

Thanks for this hint! I've worked with Wireguard before and when I first tried to set the Allowed Addresses field via WebFig, it turned 0.0.0.0/0 into to blank field, and I thought that's just how MikroTik presents an "allow all" rule. But it really does seems like a ROS v7beta6 bug.

I was able to set the value via CLI instead:

/interface wireguard peers set <ID> allowed-addresses=0.0.0.0/0

The result is that I can now ping the internet but no other traffic seems to pass through (FWIW, I also can't access WebFig from the VLAN). But I seem to be one step closer and will go through my routes and firewall rules again.

verbylab · Fri Jul 02, 2021 1:57 pm

I will soon be looking into a solution to enable remote staff to use physical telephony devices (VoIP phones) alongside their personal laptops running behind their home internet service plans. Allowing them to VPN into the corporate network using Wireguard, running on a MikroTik, is the goal.

My thinking is that corporate would install a WireGuard enabled MikroTik server behind whatever their main router is and then the home user would be shipped a preconfigured WireGuard enabled hAP. The home user would plug the hAP into a free port on their home router, then plug the phone into the hAP. With the laptop they would have the option of plugging in or using the wifi corporate SSID from the hAP. There will also be soft-phones on PCs and apps on mobile devices.

Hopefully this will be a drop in solution and everything just works. Naturally, there will be a port forward on the corporate side to the WireGuard server. Now, the fun part will be how to keep the phones a part of the VoIP VLAN and the laptops on the data VLAN.

Nice! For the corporate side, you could simply install Wireguard on any Linux instance and port-forward to it instead of having an extra MikroTik device (unless you want or need to of course). I have done this before and it's been very stable and reliable for my usage pattern (mind you, less than 5 devices connected on average).

For the hAPs in your scenario, Wireguard is only supported in the latest beta versions at the moment, and I would be conservative in my hopes around "everything just works". In terms of VLANs, your plans sound very much like what I am trying to achieve, just on a larger scale and with less opportunities to reconfigure devices (I guess you'd have to remote-desktop into people's home networks to update gateway settings if need be). Hopefully there will come a day soon when I have a working setup together, and I will of course be happy to share it here for others to learn and re-use.

pcunite · Fri Jul 02, 2021 4:27 pm

Nice! For the corporate side, you could simply install Wireguard on any Linux instance and port-forward to it instead of having an extra MikroTik device (unless you want or need to of course). I have done this before and it's been very stable and reliable for my usage pattern (mind you, less than 5 devices connected on average).

I'm still learning, but I have been playing with a Raspberry Pi as a Wireguard server. The trouble is, I don't yet know how to identify an incoming Wireguard peer as belonging to a particular VLAN or not. Traffic leaving the Wireguard server all has the same ip address. I was thinking that I needed a Linux box with multiple ports (aka a MikroTik hEX) and thus I would have a physical Ethernet port to represent each VLAN (each port on the hEX would be connected to different Access ports on a switch).

Is there a way, in Ubuntu 20.04, to VLAN tag traffic (based on ip address I guess) coming from the wg0 interface? I would need different VLAN tags based on which peer is sending traffic.

anav · Fri Jul 02, 2021 6:45 pm

HI pcunite, I too contemplated using the raspberri pi for WG but I think your throughput will suffer if using that device??

pcunite · Fri Jul 02, 2021 7:45 pm

Hi pcunite, I too contemplated using the raspberry pi for WG but I think your throughput will suffer if using that device??

The Raspberry Pi might not have enough horsepower, I don't know yet. I was using it as a test. Could build a Ubuntu system if necessary. Would also consider using MikroTik hardware if it tests well. Interesting reading here about Linux based tunneling.

verbylab · Fri Jul 02, 2021 8:16 pm

Nice! For the corporate side, you could simply install Wireguard on any Linux instance and port-forward to it instead of having an extra MikroTik device (unless you want or need to of course). I have done this before and it's been very stable and reliable for my usage pattern (mind you, less than 5 devices connected on average).

I'm still learning, but I have been playing with a Raspberry Pi as a Wireguard server. The trouble is, I don't yet know how to identify an incoming Wireguard peer as belonging to a particular VLAN or not. Traffic leaving the Wireguard server all has the same ip address. I was thinking that I needed a Linux box with multiple ports (aka a MikroTik hEX) and thus I would have a physical Ethernet port to represent each VLAN (each port on the hEX would be connected to different Access ports on a switch).

Is there a way, in Ubuntu 20.04, to VLAN tag traffic (based on ip address I guess) coming from the wg0 interface? I would need different VLAN tags based on which peer is sending traffic.

Depends on your needs. For enterprise(ish) use cases I would personally not bank on a Raspberry Pi, even though mine have never let me down for small-scale projects. But even with a Pi, you can install VLAN support on it and let all traffic flow over one physical interface. But hard to be more precise without detailed network topology.

Either way, it might be worth creating a separate topic in this forum or in a general Linux/Wireguard forum like https://www.reddit.com/r/WireGuard. I'd also love to learn more about the VLAN filtering and routing requirements you mentioned, but feel like this discussion is starting to override the original intent of the thread.

tdw · Fri Jul 02, 2021 9:39 pm

AFAIK Wireguard is a layer 3 VPN so there is no concept of VLANs - it will route packets between different subnets at each end and firewall rules can be used to restrict which subnets can communicate with each other. If you really need to extend the layer 2 domain then VxLAN, GRETAP or in the Mikrotik case EoIP would work.

pcunite · Sat Jul 03, 2021 7:26 pm

AFAIK Wireguard is a layer 3 VPN so there is no concept of VLANs - it will route packets between different subnets at each end and firewall rules can be used to restrict which subnets can communicate with each other. If you really need to extend the layer 2 domain then VxLAN, GRETAP or in the Mikrotik case EoIP would work.

Correct, but I still want to use WireGuard. Just need a way to map VLANs as I don't really need the tag itself on the frame to get propagated. I understand VxLAN support is on its way. If I could have two WG instances on the client, or two WG instances at the server, that could do it.

verbylab · Sun Jul 04, 2021 1:59 am

AFAIK Wireguard is a layer 3 VPN so there is no concept of VLANs - it will route packets between different subnets at each end and firewall rules can be used to restrict which subnets can communicate with each other. If you really need to extend the layer 2 domain then VxLAN, GRETAP or in the Mikrotik case EoIP would work.

Regarding my initial post, I do not intend to extend the concept of VLANs beyond the Wireguard interface. My problem is simply that I haven't been able to route internet traffic from my local VLAN to the Wireguard interface and back.

anav · Sun Jul 04, 2021 3:58 am

You need to provide a route from the vlan to the wireguard interface
use a routing table entry and a route rule reference the table, lookup only in, and the source address(es) that are appropriate

verbylab · Sun Jul 04, 2021 2:50 pm

Yes, it should be rather simple.

I am currently trying to solve this using VRF, since it nicely separates the VLANs by default.

Adding the VRF for my VLAN worked, but adding the default route for it fails to establish an internet connection for VLAN clients:

add comment="Default route for VLAN" dst-address=0.0.0.0/0 routing-table=*3 gateway
=pppoe-out1 distance=1
# I don't quite understand the *3 reference, but that what showed up for the rules that were automatically created after I set up the VRF, so I just copied it.
# Edit: Turns out that creating a VRF via CLI only creates an entry in /ip/vrf but not /routing/table. When I used WebFig to create the VRF, I get both. Then the *3 is replaced with the actual name of the routing table. Strange.

I think I have to research in detail how VRFs work on ROS v7 since it seems to be a bit different to older versions.

If that fails, I have to delete the VRF, create a simple second routing table, and add a routing lookup rule as you suggested.

verbylab · Tue Jul 06, 2021 3:52 am

I almost got this to work now. I eventually created a VRF (which automatically created a routing table), then I created a default route in that routing table. No extra mangle rule our routing rule - nice and simple.

Then the pings (ICMP) started to work. Traceroute shows that it's tunneled through the VPN.

However, I cannot open a website on the client. So I thought it must be the firewall, until I realized that opening an IP address works.

Looks like something isn't right with the DNS setup. Some sites which the browser had cached before work, until I empty the cache and try it again.

verbylab · Wed Jul 07, 2021 2:51 am

I found a solution to my problem in viewtopic.php?t=150377. Was a really strange thing to debug, cause some sites worked and some others didn't.

/ip firewall mangle
add action=change-mss chain=forward new-mss=clamp-to-pmtu passthrough=yes protocol=tcp tcp-flags=syn

I have no idea why the MSS only caused an issue with some sites, so I want to learn more about it, but at least the setup is functional now.

Cablenut9 · Wed Jul 07, 2021 3:01 am

Clamping MSS also makes things load faster because there's less fragmentation, so adding that rule is always a good thing.

Tunneling VLAN traffic over Wireguard

Tunneling VLAN traffic over Wireguard

Re: Tunneling VLAN traffic over Wireguard

Re: Tunneling VLAN traffic over Wireguard

Re: Tunneling VLAN traffic over Wireguard

Re: Tunneling VLAN traffic over Wireguard

Re: Tunneling VLAN traffic over Wireguard

Re: Tunneling VLAN traffic over Wireguard

Re: Tunneling VLAN traffic over Wireguard

Re: Tunneling VLAN traffic over Wireguard

Re: Tunneling VLAN traffic over Wireguard

Re: Tunneling VLAN traffic over Wireguard

Re: Tunneling VLAN traffic over Wireguard

Re: Tunneling VLAN traffic over Wireguard

Re: Tunneling VLAN traffic over Wireguard

Re: Tunneling VLAN traffic over Wireguard

Re: Tunneling VLAN traffic over Wireguard

Re: Tunneling VLAN traffic over Wireguard

Re: Tunneling VLAN traffic over Wireguard

Re: Tunneling VLAN traffic over Wireguard

Who is online