Need help with inter-VLAN communication and IPSec

teisen · Tue Jan 11, 2022 9:20 pm

Hello,

i am setting up my home network and need some help to understand firewall / routing please.
Last week i successfully setup my RB5009 with VLANS following the guide from pcunite.
I added an aruba switch to the trunk port ether8 and it's working fine. I can reach the WAN and also communicate between VLANS, which i want to be able to.

My use case is that if i connect a computer to the BLUE VLAn it should route normally to WAN and when i connect a computer to the GREEN VLAN it should go through a IPsec to NordVPN (to access streaming content of another country).

So i followed the guide at https://support.nordvpn.com/Connectivit ... ordVPN.htm and that part worked great as well.

But when I have the VPN active my clients cannot communicate anymore to the other VLANS. A ping from a client with IP 192.168.2.27 to 192.168.1.3 fails with a response from a host in the VPN Network (185.196.x.x address: Reply from 185.196.xx.x: Destination host unreachable.) I am toggling the VPN on/off by activating/deactivating the address list 'over-vpn' btw.

What is the best way to route any traffic intended for any of my own VLANS to go to them and not over the IPsec tunnel?

Here is my config:

# jan/11/2022 14:15:26 by RouterOS 7.2rc1
# software id = C5MD-RX8K
#
# model = RB5009UG+S+
/interface bridge
add name=BR1 protocol-mode=none vlan-filtering=yes
/interface vlan
add interface=BR1 name=BASE_VLAN vlan-id=99
add interface=BR1 name=BLUE_VLAN vlan-id=10
add interface=BR1 name=GREEN_VLAN vlan-id=20
/interface list
add name=WAN
add name=VLAN
add name=BASE
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip dhcp-server option
add code=43 name=unifi value=0x0104C0A80103
/ip ipsec mode-config
add name=NordVPN responder=no src-address-list=over-vpn
/ip ipsec policy group
add name=NordVPN
/ip ipsec profile
add name=NordVPN
/ip ipsec peer
add address=xxxxx.nordvpn.com exchange-mode=ike2 name=NordVPN profile=\
    NordVPN
/ip ipsec proposal
add name=NordVPN pfs-group=none
/ip pool
add name=BLUE_POOL ranges=192.168.1.20-192.168.1.254
add name=GREEN_POOL ranges=192.168.2.20-192.168.2.254
add name=BASE_POOL ranges=192.168.88.20-192.168.88.254
/ip dhcp-server
add address-pool=BLUE_POOL interface=BLUE_VLAN name=BLUE_DHCP
add address-pool=GREEN_POOL interface=GREEN_VLAN name=GREEN_DHCP
add address-pool=BASE_POOL interface=BASE_VLAN name=BASE_DHCP
/interface bridge port
add bridge=BR1 frame-types=admit-only-untagged-and-priority-tagged interface=\
    ether2 pvid=99
add bridge=BR1 frame-types=admit-only-untagged-and-priority-tagged interface=\
    ether3 pvid=10
add bridge=BR1 frame-types=admit-only-untagged-and-priority-tagged interface=\
    ether4 pvid=20
add bridge=BR1 frame-types=admit-only-vlan-tagged interface=ether5
add bridge=BR1 frame-types=admit-only-vlan-tagged interface=ether6
add bridge=BR1 frame-types=admit-only-vlan-tagged interface=ether7
add bridge=BR1 frame-types=admit-only-vlan-tagged interface=ether8
add bridge=BR1 frame-types=admit-only-vlan-tagged interface=sfp-sfpplus1
/ip neighbor discovery-settings
set discover-interface-list=all
/interface bridge vlan
add bridge=BR1 tagged=BR1,ether3,ether5,ether6,ether7,ether8,sfp-sfpplus1 \
    vlan-ids=10
add bridge=BR1 tagged=ether4,ether5,ether6,ether7,ether8,sfp-sfpplus1,BR1 \
    vlan-ids=20
add bridge=BR1 tagged=BR1,ether2,ether5,ether6,ether7,ether8,sfp-sfpplus1 \
    vlan-ids=99
/interface detect-internet
set detect-interface-list=all
/interface list member
add interface=ether1 list=WAN
add interface=BASE_VLAN list=VLAN
add interface=BLUE_VLAN list=VLAN
add interface=GREEN_VLAN list=VLAN
add interface=BASE_VLAN list=BASE
/ip address
add address=192.168.88.1/24 interface=BASE_VLAN network=192.168.88.0
add address=192.168.1.1/24 interface=BLUE_VLAN network=192.168.1.0
add address=192.168.2.1/24 interface=GREEN_VLAN network=192.168.2.0
/ip dhcp-client
add comment=defconf interface=ether1
/ip dhcp-server lease
add address=192.168.1.8 client-id=1:0:a0:98:63:9a:97 mac-address=\
    00:A0:98:63:9A:97 server=BLUE_DHCP
add address=192.168.1.10 client-id=1:2:25:90:9c:d6:9b mac-address=\
    02:25:90:9C:D6:9B server=BLUE_DHCP
add address=192.168.1.3 client-id=1:2:25:90:87:0:b5 mac-address=\
    02:25:90:87:00:B5 server=BLUE_DHCP
add address=192.168.1.9 client-id=1:2:25:90:40:ed:c3 mac-address=\
    02:25:90:40:ED:C3 server=BLUE_DHCP
add address=192.168.1.6 client-id=1:2:25:90:14:fa:a mac-address=\
    02:25:90:14:FA:0A server=BLUE_DHCP
add address=192.168.1.5 client-id=1:2:25:90:ba:b5:82 mac-address=\
    02:25:90:BA:B5:82 server=BLUE_DHCP
/ip dhcp-server network
add address=192.168.1.0/24 dns-server=192.168.1.1 gateway=192.168.1.1
add address=192.168.2.0/24 dns-server=192.168.2.1 gateway=192.168.2.1
add address=192.168.88.0/24 dns-server=192.168.88.1 gateway=192.168.88.1
/ip dns
set allow-remote-requests=yes servers=1.1.1.1,1.0.0.1 use-doh-server=\
    https://cloudflare-dns.com/dns-query verify-doh-cert=yes
/ip dns static
add address=192.168.88.1 name=router.lan
add address=192.168.88.2 name=switch.lan
add address=192.168.1.4 name=freenas.lan
add address=192.168.1.5 name=nextcloud.lan
add address=192.168.1.7 name=unifi.lan
add address=192.168.1.9 name=zoneminder.lan
/ip firewall address-list
add address=192.168.2.0/24 list=over-vpn
/ip firewall filter
add action=accept chain=input comment="Allow Estab & Related" \
    connection-state=established,related
add action=accept chain=input comment="Allow VLAN" in-interface-list=VLAN
add action=accept chain=input comment="Allow Base_Vlan Full Access" \
    in-interface=BASE_VLAN
add action=drop chain=input comment=Drop
add action=accept chain=forward comment="Allow Estab & Related" \
    connection-state=established,related
add action=accept chain=forward comment="VLAN Internet Access only" \
    connection-state=new disabled=yes in-interface-list=VLAN \
    out-interface-list=WAN
add action=accept chain=forward comment=\
    "VLAN Internet Access and VLAN access" connection-state=new \
    in-interface-list=VLAN out-interface-list=all
add action=drop chain=forward comment=Drop
/ip firewall nat
add action=masquerade chain=srcnat comment="Default masquerade" \
    out-interface-list=WAN
/ip ipsec identity
add auth-method=eap certificate="" eap-methods=eap-mschapv2 generate-policy=\
    port-strict mode-config=NordVPN peer=NordVPN policy-template-group=\
    NordVPN username=xxxxx
/ip ipsec policy
add dst-address=0.0.0.0/0 group=NordVPN proposal=NordVPN src-address=\
    0.0.0.0/0 template=yes
/system clock
set time-zone-name=America/New_York
/system identity
set name=RouterSwitchAP
/system routerboard settings
set cpu-frequency=auto
/tool mac-server
set allowed-interface-list=VLAN
/tool mac-server mac-winbox
set allowed-interface-list=VLAN

Thanks!

anav · Tue Jan 11, 2022 10:46 pm

Your explanation is not clear.

First, yes its relatively painless to send clients on one subnet out the vpn tunnel vice out the normal internet (same router).
As to vlans talking to one another, the purpose of vlans is to prevent that and the purpose of firewall rules to prevent that at layer 2.

Again its not clear what is not happening that you wish to happen

anav · Tue Jan 11, 2022 10:57 pm

You didnt quite follow pcunite correctly.

Check it out:
add bridge=BR1 frame-types=admit-only-untagged-and-priority-tagged interface=\
ether2 pvid=99
add bridge=BR1 tagged=BR1,ether2,ether5,ether6,ether7,ether8,sfp-sfpplus1 \
vlan-ids=99

add bridge=BR1 frame-types=admit-only-untagged-and-priority-tagged interface=\
ether3 pvid=10
add bridge=BR1 tagged=BR1,ether3,ether5,ether6,ether7,ether8,sfp-sfpplus1 \
vlan-ids=10

Suggesting you have two choices.....
Remove mention of ether2 or ether3 in the rules above, and the router will create the untagging automatically/dynamically due to the pvid configs, OR
you could add the untagging manually, which allows you to visually crosscheck the config to ensure it makes sense...........

SO either:
/interface bridge vlan
add bridge=BR1 tagged=BR1,ether5,ether6,ether7,ether8,sfp-sfpplus1 \
vlan-ids=10
add bridge=BR1 tagged=ether4,ether5,ether6,ether7,ether8,sfp-sfpplus1,BR1 \
vlan-ids=20
add bridge=BR1 tagged=BR1,ether5,ether6,ether7,ether8,sfp-sfpplus1 \
vlan-ids=99
OR
/interface bridge vlan
add bridge=BR1 tagged=BR1ether5,ether6,ether7,ether8,sfp-sfpplus1 \
untagged=ether3 vlan-ids=10
add bridge=BR1 tagged=ether4,ether5,ether6,ether7,ether8,sfp-sfpplus1,BR1 \
vlan-ids=20
add bridge=BR1 tagged=BR1,ether5,ether6,ether7,ether8,sfp-sfpplus1 \
untagged=ether2 vlan-ids=99

The difference is if you print a config the manual method shows your untagging clearly and to me is less confusing but everyone is different.

anav · Tue Jan 11, 2022 10:58 pm

Okay I found this..........
/ip firewall address-list
add address=192.168.2.0/24 list=over-vpn

But where is 192.168.2.0/24 defined anywhere???

anav · Tue Jan 11, 2022 11:00 pm

What is the difference between these two rules........... think about it....
add action=accept chain=input comment="Allow VLAN" in-interface-list=VLAN
add action=accept chain=input comment="Allow Base_Vlan Full Access" \
in-interface=BASE_VLAN

Hint: Will any packets from the LAN side ever reach the second rule??

anav · Tue Jan 11, 2022 11:03 pm

What are the purpose of these two rules??? What is the difference between the two??
add action=accept chain=forward comment="VLAN Internet Access only" \
connection-state=new disabled=yes in-interface-list=VLAN \
out-interface-list=WAN
add action=accept chain=forward comment=\
"VLAN Internet Access and VLAN access" connection-state=new \
in-interface-list=VLAN out-interface-list=all

Hint: The second rule does all the first rule will do....................

With this and the previous post one could conclude.
a. you dont really know what you want to allow or disallow
b. you dont really understand firewall rules on the router.

An observation is that your vlans are not a very well thought out concept if you allow full access all over the place..............why have separate vlans then.......

anav · Tue Jan 11, 2022 11:04 pm

This can be set to NONE, (is not a secure form of access)
/tool mac-server
set allowed-interface-list=VLAN

teisen · Tue Jan 11, 2022 11:05 pm

Okay I found this..........
/ip firewall address-list
add address=192.168.2.0/24 list=over-vpn

But where is 192.168.2.0/24 defined anywhere???

It's the IP network of the GREEN_VLAN. I guess the firewall route is sending all traffic from the VLAN to the IPSEC. Maybe this is where the trouble is. How else can i tell the firewall to only route traffic destined for the internet to the IPsec and not all traffic from that network, therefore still allowing communication to the other VLANS?

/ip address
add address=192.168.88.1/24 interface=BASE_VLAN network=192.168.88.0
add address=192.168.1.1/24 interface=BLUE_VLAN network=192.168.1.0
add address=192.168.2.1/24 interface=GREEN_VLAN network=192.168.2.0

anav · Tue Jan 11, 2022 11:10 pm

Finally, in terms of sending out traffic to the tunnel specifically will require some IP routes.
Typically
dst-address=0.0.0.0/0 gwy=WANIP gateway table=main is the default route created in IP DHCP Client which is probably what you have since I see no IP route rule in the config.

What we want to do is create another such rule but only for the source subnet desired.
dst-addres=0.0.0.0/0 gwy=NordVPN table=usetunnel

Where a route rule is in place that states: Source address=192.168.2.0/24, Action=lookup-only-in-table, table=usetunnel
Where one creates the table but only through the CLI as this part is not available yet in version 7.1.1

If you want the users to revert to the regular interface (main local router) if the VPN tunnel is not working then simply use ACTION=lookup-in-table.

teisen · Tue Jan 11, 2022 11:10 pm

With this and the previous post one could conclude.
a. you dont really know what you want to allow or disallow
b. you dont really understand firewall rules on the router.

An observation is that your vlans are not a very well thought out concept if you allow full access all over the place..............why have separate vlans then.......

Yeah... I do not really understand the configuration it, I admit. My original thinking was that it would be easiest to segment my network into two VLANS, so that i can plug all devices that should exit in another country into one VLAN and all normal devices into the other. Maybe i should build the rules not based of VLANS but something else?

Use case 1: Chromecast is on GREEN_VLAN so it can stream TV content, same with a device that would cast to the chromecast
Use case 2: Plex receiver should be able to stream TV content from other country (GREEN_VLAN) but also be able to connect to a local server on the BLUE_VLAN to stream from there.

anav · Tue Jan 11, 2022 11:11 pm

Okay my bad, I see now that is the subnet you want over the tunnel, I lost the forest for the trees there.....
However the rest needs to get fixed first.

anav · Tue Jan 11, 2022 11:16 pm

Even your use cases are confusing and not clear.

Where is the plex server located,
Which users are connecting to it?
Its not clear whats going on,

Draw some diagrams......perhaps to explain the flows........

Sob · Wed Jan 12, 2022 12:55 am

I don't remember exact details, but this VPN creates some dynamic firewall rules, right? Srcnat rule, I think? If so, is it possible to put another before it, with local subnets as source and destination, and action=accept? If not (in case it's added at the top), it should be possible to switch mode config to use connection marks instead of address list, and then you could use mangle rules to mark what should use VPN and what shouldn't.

urchino · Wed Jan 12, 2022 11:14 am

Is there some other benefit of having separate VLANs? You seem to be trying to overcome the segmentation they're designed to implement, and the main reason for having them. Why not get rid of the VLANs and just use firewall address-list membership to route traffic over the VPN? This is how I have my network configured and no devices have any trouble with local network access whilst they're connected to the internet through the VPN. I used this guide for a simpler configuration: viewtopic.php?f=23&t=169273.

teisen · Wed Jan 12, 2022 4:58 pm

Even your use cases are confusing and not clear.

Where is the plex server located,
Which users are connecting to it?
Its not clear whats going on,

Draw some diagrams......perhaps to explain the flows........

So, i am not sure if using VLAN is the correct strategy, was thinking that it would be a good opportunity to learn more about VLANs. Please see below image for the main usecase, normal computers, servers etc are all in BLUE_VLAN and connect to WAN normally. Devices that i want to exit in foreign content to be able to stream I put in GREEN_VLAN. That all works fine, except once the Firewall Address list is enabled (and therefore green is routed to the VPN) i cannot connect from GREEN to BLUE anymore.

teisen · Wed Jan 12, 2022 5:06 pm

I don't remember exact details, but this VPN creates some dynamic firewall rules, right? Srcnat rule, I think? If so, is it possible to put another before it, with local subnets as source and destination, and action=accept? If not (in case it's added at the top), it should be possible to switch mode config to use connection marks instead of address list, and then you could use mangle rules to mark what should use VPN and what shouldn't.

Yes, the config suggested uses an address list (in my case the addresses from the GREEN_VLAN of 192.168.2.0/24) and then uses that in Chain srcnat.
Src. Address List: over-vpn
Dst. Address List: !over-vpn

I do not know how routing is to be setup differently, or at all LOL. Is there a place i can read more about how to mark something that is originating in green_vlan and not going to either green_VLAn or any of my internal addresses and sending only that traffic to IPSEC?

anav · Wed Jan 12, 2022 5:46 pm

Okay in summary.

Every device in the Green VLAN should get internet via the Wireguard/VPN tunnel.
Every device in the Green VLAN needs access ONLY to the plex server in the blue vlan.

Its not clear if you want the plex receiver to be able to connect to the internet via the VPN or from the main router (normal internet)

+++++++++
As to the dilemma it appears to be that you cannot have at the same time
a. green clients going out the vpn to the internet
b. green clients accessing the plex server (another vlan).

I think the way to solve this might be through IP routes as I think that is the root of the problem...

The issue is typically one states
dst-address=0.0.0.0/0 gwy=vpn For GREEN VLAN
as the way access all possible internet addresses which unfortunately also includes addresses on the local vlans.

Best case would be a routing hierarchy
dst-address=PLex Server gwy=vpn distance=5 FOR Green VLAN
dst-address=0.0.0.0/0 gwy=vpn distance=10 FOR Green VLAN

In this way, the router will always select the lower distance routing for traffic going to the plex server even though the second rule overlaps the same numbers..
IN fact, I believe that you dont even have to differentiate between distance as the router will see that the plex server IP as a more defined/refined (aka higher value destination) and solve that routing first between the two rules.

Is that what you are looking for?

Sob · Thu Jan 13, 2022 3:24 am

The problem is that dynamic srcnat rule added by vpn takes all packets from sources listed in over-vpn address list and changes source address. And that new source address matches ipsec policy, so regardless of routes, ipsec takes those packets.

The fix should be if you remove src-address-list=over-vpn from mode-config, replace it with connection-mark=to-vpn and then do something like this:

/ip firewall mangle
add chain=prerouting src-address-list=over-vpn connection-state=new dst-adress=192.168.0.0/16 action=accept
add chain=prerouting src-address-list=over-vpn connection-state=new action=mark-connection new-connection-mark=to-vpn

teisen · Thu Jan 13, 2022 8:49 pm

Sob,

you are right, i removed the src-address-list=over-vpn from mode-config and then replaced it with connection-mark=NordVPN and then

/ip ipsec mode-config
add connection-mark=NordVPN name=NordVPN responder=no

/ip firewall address-list
add address=192.168.2.0/24 list=over-vpn
add address=192.168.1.0/24 list=local-vlans
add address=192.168.2.0/24 list=local-vlans
add address=192.168.88.0/24 list=local-vlans

/ip firewall mangle
add action=mark-connection chain=forward connection-state=new \
    dst-address-list=!local-vlans new-connection-mark=NordVPN passthrough=no \
    src-address-list=over-vpn

I need to try our your 192.168.0.0/16 - which might be easier to manage.

Sob · Thu Jan 13, 2022 9:47 pm

It's basically the same thing. You have list of subnets, I used one larger subnet that covers all. I used two rules, but one with dst-adress=!192.168.0.0/16 would be ok too.

Need help with inter-VLAN communication and IPSec [SOLVED]

Need help with inter-VLAN communication and IPSec

Re: Need help with inter-VLAN communication and IPSec

Re: Need help with inter-VLAN communication and IPSec

Re: Need help with inter-VLAN communication and IPSec

Re: Need help with inter-VLAN communication and IPSec

Re: Need help with inter-VLAN communication and IPSec

Re: Need help with inter-VLAN communication and IPSec

Re: Need help with inter-VLAN communication and IPSec

Re: Need help with inter-VLAN communication and IPSec

Re: Need help with inter-VLAN communication and IPSec

Re: Need help with inter-VLAN communication and IPSec

Re: Need help with inter-VLAN communication and IPSec

Re: Need help with inter-VLAN communication and IPSec

Re: Need help with inter-VLAN communication and IPSec

Re: Need help with inter-VLAN communication and IPSec

Re: Need help with inter-VLAN communication and IPSec

Re: Need help with inter-VLAN communication and IPSec

Re: Need help with inter-VLAN communication and IPSec [SOLVED]

Re: Need help with inter-VLAN communication and IPSec

Re: Need help with inter-VLAN communication and IPSec

Who is online