Need help with Vlan routing

tatianaroma · Sun Mar 28, 2021 10:43 am

Hi guys,

I come from an Ubiquiti ER-4 but I needed a router capable of multi-gig routing. And so I bought the CCR2004-1G-12S+2XS.
I have a fairly simple home installation, with:
- my router connected to switch via a trunk port (sfp-sfpplus12) and then this same switch is connected to another switch via another trunk port.
- a 'main' vlan with all my trusted devices and full access to everything
- a 'iot' vlan where I put all my trash devices. In this vlan they can reach each other but are unable to reach the main vlan unless a connection was already established.

It used to work pretty well with my ER-4 but when I tried to adapt these set of rules to my CCR2004 I struggle:
- traffic within each vlan seems OK
- when I try to reach the 'iot' vlan from my 'main' the connection takes a lot of time to establish. Around 6-8 seconds and then I runs alright with no packet loss I think. I can see this behavior with iperf, once the connection is established, I get full wire speed.

Can you please take a look into my conf and see if something is misconfigured. I been banging my head for days now, I am starting to loose my sleep and thinking of bringing back my ER-4 :(

[andrep@RT-POLARIS] > ip address print
Flags: X - disabled, I - invalid, D - dynamic 
 #   ADDRESS            NETWORK         INTERFACE                                                                                                                                                                                            
 0   ;;; defconf
     192.168.88.1/24    192.168.88.0    ether1                                                                                                                                                                                               
 1   192.168.10.1/24    192.168.10.0    vlan10_home                                                                                                                                                                                          
 2   192.168.11.1/24    192.168.11.0    vlan11_iot                                                                                                                                                                                           
 3   192.168.12.1/24    192.168.12.0    vlan12_guest                                                                                                                                                                                         
 4   192.168.13.1/24    192.168.13.0    vlan13_iot-offline                                                                                                                                                                                   
 5 D xxx.xxx.xx3.183/24  xxx.xxx.xx3.0    sfp-sfpplus1

*******************************************************************

 [andrep@RT-POLARIS] > /interface print     
Flags: D - dynamic, X - disabled, R - running, S - slave 
 #     NAME                                TYPE       ACTUAL-MTU L2MTU  MAX-L2MTU MAC-ADDRESS      
 0     ether1                              ether            1500  1600       9586 08:55:31:A5:9C:27
 1  R  sfp-sfpplus1                        ether            1500  1592       9578 B4:FB:E4:CB:06:B2
 2     sfp-sfpplus2                        ether            1500  1592       9578 08:55:31:A5:9C:29
 3     sfp-sfpplus3                        ether            1500  1592       9578 08:55:31:A5:9C:2A
 4     sfp-sfpplus4                        ether            1500  1592       9578 08:55:31:A5:9C:2B
 5     sfp-sfpplus5                        ether            1500  1592       9578 08:55:31:A5:9C:2C
 6     sfp-sfpplus6                        ether            1500  1592       9578 08:55:31:A5:9C:2D
 7     sfp-sfpplus7                        ether            1500  1592       9578 08:55:31:A5:9C:2E
 8     sfp-sfpplus8                        ether            1500  1592       9578 08:55:31:A5:9C:2F
 9     sfp-sfpplus9                        ether            1500  1592       9578 08:55:31:A5:9C:30
10     sfp-sfpplus10                       ether            1500  1592       9578 08:55:31:A5:9C:31
11     sfp-sfpplus11                       ether            1500  1592       9578 08:55:31:A5:9C:32
12  R  sfp-sfpplus12                       ether            1500  1592       9578 08:55:31:A5:9C:33
13     sfp28-1                             ether            1500  1592       9578 08:55:31:A5:9C:34
14     sfp28-2                             ether            1500  1592       9578 08:55:31:A5:9C:35
15  R  vlan10_home                         vlan             1500  1588            08:55:31:A5:9C:33
16  R  vlan11_iot                          vlan             1500  1588            08:55:31:A5:9C:33
17  R  vlan12_guest                        vlan             1500  1588            08:55:31:A5:9C:33
18  R  vlan13_iot-offline                  vlan             1500  1588            08:55:31:A5:9C:33

*******************************************************************

 [andrep@RT-POLARIS] > /interface vlan print
Flags: X - disabled, R - running 
 #   NAME                            MTU ARP                 VLAN-ID INTERFACE                                                                                              
 0 R vlan10_home                   1500 enabled              10 sfp-sfpplus12                                                                                          
 1 R vlan11_iot                    1500 enabled              11 sfp-sfpplus12                                                                                          
 2 R vlan12_guest                  1500 enabled              12 sfp-sfpplus12                                                                                          
 3 R vlan13_iot-offline            1500 enabled              13 sfp-sfpplus12

*******************************************************************

[andrep@RT-POLARIS] > /ip route print
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit 
 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 0 ADS  0.0.0.0/0                          185.66.103.254            1
 1 ADC  185.66.103.0/24    xxx.xxx...3.183  sfp-sfpplus1              0
 2 ADC  192.168.10.0/24    192.168.10.1    vlan10_home               0
 3 ADC  192.168.11.0/24    192.168.11.1    vlan11_iot                0
 4 ADC  192.168.12.0/24    192.168.12.1    vlan12_guest              0
 5 ADC  192.168.13.0/24    192.168.13.1    vlan13_iot-offline        0
 6  DC  192.168.88.0/24    192.168.88.1    ether1                  255
 7 A S  239.255.255.0/24                   vlan11_iot                1

*******************************************************************

[andrep@RT-POLARIS] > /ip firewall filter export
# mar/28/2021 09:23:45 by RouterOS 6.48.1
# software id = TPHC-LDWX
#
# model = CCR2004-1G-12S+2XS
# serial number = 
/ip firewall filter
add action=accept chain=input comment="accept established,related" connection-state=established,related
add action=drop chain=input comment="drop invalid state" connection-state=invalid
add action=accept chain=input comment="accept KNET ICMP probes " limit=50/1m,1:packet log=yes log-prefix="[knet icmp probes]" protocol=icmp src-address-list=KNET_ICMP-PROBES
add action=drop chain=input comment="block everything else from WAN" in-interface-list=wan_ipv4
add action=jump chain=input comment="input IOT rules" in-interface=vlan11_iot jump-target=input_IOT
add action=jump chain=input comment="input GUEST rules" in-interface=vlan12_guest jump-target=input_RESTRICTED
add action=jump chain=input comment="input IOT OFFLINE rules" in-interface=vlan13_iot-offline jump-target=input_RESTRICTED
add action=accept chain=input_RESTRICTED comment="Accept DHCP (udp)" dst-port=67 protocol=udp src-port=68
add action=drop chain=input_RESTRICTED comment="drop everything else"
add action=accept chain=input_RESTRICTED comment="Accept DHCP (tcp)" dst-port=67 port="" protocol=tcp src-port=68
add action=accept chain=input_IOT comment="Accept DNS (tcp)" port=53 protocol=tcp
add action=accept chain=input_IOT comment="Accept DNS (udp)" port=53 protocol=udp
add action=accept chain=input_IOT comment="Accept DHCP (tcp)" dst-port=67 port="" protocol=tcp src-port=68
add action=accept chain=input_IOT comment="Accept DHCP (udp)" dst-port=67 protocol=udp src-port=68
add action=drop chain=input_IOT comment="drop everything else"
add action=fasttrack-connection chain=forward comment="fast-track for established,related" connection-state=established,related
add action=accept chain=forward comment="accept established,related" connection-state=established,related
add action=drop chain=forward comment="drop invalid state" connection-state=invalid
add action=drop chain=forward comment="drop access to clients behind NAT form WAN" connection-nat-state=!dstnat connection-state=new in-interface-list=wan_ipv4
add action=jump chain=forward comment="forward IOT rules" in-interface=vlan11_iot jump-target=forward_IOT
add action=jump chain=forward comment="forward GUEST rules" in-interface=vlan12_guest jump-target=forward_GUEST
add action=jump chain=forward comment="forward IOT OFFLINE rules" in-interface=vlan13_iot-offline jump-target=forward_FULL-ISOLATION
add action=accept chain=forward_IOT comment="accept all within IOT" out-interface=vlan11_iot
add action=accept chain=forward_IOT comment="accept WAN outbound traffic (tcp)" out-interface-list=wan_ipv4 port="" protocol=tcp
add action=accept chain=forward_IOT comment="accept WAN outbound traffic (udp)" out-interface-list=wan_ipv4 port="" protocol=udp
add action=accept chain=forward_IOT comment="accept multicast" disabled=yes dst-address-type=multicast out-interface=vlan10_home
add action=reject chain=forward_IOT comment="drop everything else" reject-with=icmp-network-unreachable
add action=accept chain=forward_GUEST comment="accept DNS (tcp)" dst-address-list=GUEST_DNS port=53 protocol=udp
add action=accept chain=forward_GUEST comment="accept DNS (udp)" dst-address-list=GUEST_DNS port=53 protocol=tcp
add action=accept chain=forward_GUEST comment="accept omada captive portal" dst-address=192.168.10.21 port=8088 protocol=tcp
add action=accept chain=forward_GUEST comment="accept HTTP HTTPS (tcp)" out-interface-list=wan_ipv4 port=80,8080,443,8443 protocol=tcp
add action=accept chain=forward_GUEST comment="accept HTTP HTTPS (udp)" out-interface-list=wan_ipv4 port=80,8080,443,8443 protocol=udp
add action=drop chain=forward_GUEST comment="drop everything else"
add action=drop chain=forward_FULL-ISOLATION comment="drop everything else"

*******************************************************************

[andrep@RT-POLARIS] > /ip firewall nat export      
# mar/28/2021 09:37:07 by RouterOS 6.48.1
# software id = TPHC-LDWX
#
# model = CCR2004-1G-12S+2XS
# serial number = 
/ip firewall nat
add action=masquerade chain=srcnat comment="masquarade for WAN" out-interface-list=wan_ipv4
add action=dst-nat chain=dstnat comment="DNS interception for IOT (udp)" dst-address=!192.168.11.1 in-interface=vlan11_iot port=53 protocol=udp to-addresses=192.168.11.1
add action=dst-nat chain=dstnat comment="DNS interception for IOT (tcp)" dst-address=!192.168.11.1 in-interface=vlan11_iot port=53 protocol=tcp to-addresses=192.168.11.1
add action=dst-nat chain=dstnat comment="DNS interception for GUEST (udp)" dst-address-list=!GUEST_DNS in-interface=vlan12_guest port=53 protocol=udp to-addresses=1.0.0.3
add action=dst-nat chain=dstnat comment="DNS interception for GUEST (tcp)" dst-address-list=!GUEST_DNS in-interface=vlan12_guest port=53 protocol=tcp to-addresses=1.1.1.3
add action=dst-nat chain=dstnat comment="SSH to pireserver" in-interface-list=wan_ipv4 port=2287 protocol=tcp to-addresses=192.168.10.10 to-ports=22
add action=dst-nat chain=dstnat comment="PLEX to pireserver" in-interface-list=wan_ipv4 port=23004 protocol=tcp to-addresses=192.168.11.10 to-ports=32400
add action=dst-nat chain=dstnat comment="OPENVPN (udp)" in-interface-list=wan_ipv4 port=123 protocol=udp to-addresses=192.168.10.10 to-ports=1194
add action=dst-nat chain=dstnat comment="OPENVPN (tcp)" in-interface-list=wan_ipv4 port=443 protocol=tcp to-addresses=192.168.10.10 to-ports=1194
add action=dst-nat chain=dstnat comment="HASSIO (tcp)" in-interface-list=wan_ipv4 port=8123 protocol=tcp to-addresses=192.168.10.21 to-ports=8123

tatianaroma · Sun Mar 28, 2021 12:28 pm

Quick update:

I narrowed the problem to one firewall rule:

add action=drop chain=forward comment="drop invalid state" connection-state=invalid

but why? I don't understand...

anav · Mon Mar 29, 2021 3:47 pm

Dont think multicasting or openvpn work very well on MT routers..
Besides that your firewall config is full of un-needed fluff

If you want to focus on vlans this is your best guide.
viewtopic.php?f=23&t=143620

tatianaroma · Mon Mar 29, 2021 9:07 pm

Dont think multicasting or openvpn work very well on MT routers..
Besides that your firewall config is full of un-needed fluff

If you want to focus on vlans this is your best guide.
viewtopic.php?f=23&t=143620

Thanks for your advice. I tried my best to simplify my firewall rules, while maintaining a structure that I find easily readable. I know I could still cut down some rules but I really want to maintain the custom chains :)

My openvpn server is not my MT router. And I plain removed the multicast package.

Anyway, here are my un-fluffed rules:

[andrep@RT-POLARIS] > /ip firewall export 
# mar/29/2021 20:01:16 by RouterOS 6.48.1
# software id = TPHC-LDWX
#
# model = CCR2004-1G-12S+2XS
# serial number = 
/ip firewall address-list
add address=178.250.208.0/23 list=KNET_ICMP-PROBES
add address=92.118.98.0/24 list=KNET_ICMP-PROBES
add address=192.168.0.0/16 list=LAN_RF-1918
add address=172.16.0.0/12 list=LAN_RF-1918
add address=10.0.0.0/8 list=LAN_RF-1918
add address=1.1.1.3 list=GUEST_DNS
add address=1.0.0.3 list=GUEST_DNS
add address=208.67.222.123 list=GUEST_DNS
add address=208.67.220.123 list=GUEST_DNS
/ip firewall filter
add action=accept chain=input comment="accept established,related" connection-state=established,related
add action=drop chain=input comment="drop invalid state" connection-state=invalid
add action=accept chain=input comment="accept KNET ICMP probes " limit=50/1m,1:packet log=yes log-prefix="[knet icmp probes]" protocol=icmp src-address-list=\
    KNET_ICMP-PROBES
add action=jump chain=input comment="input HOME chain" in-interface=vlan10_home jump-target=input_HOME
add action=jump chain=input comment="input IOT chain" in-interface=vlan11_iot jump-target=input_IOT
add action=jump chain=input comment="input GUEST chain" in-interface=vlan12_guest jump-target=input_RESTRICTED
add action=jump chain=input comment="input IOT OFFLINE chain" in-interface=vlan13_iot-offline jump-target=input_RESTRICTED
add action=drop chain=input comment="default policy DROP"
add action=accept chain=input_HOME comment="default policy ACCEPT"
add action=accept chain=input_IOT comment="Accept DNS (udp)" port=53 protocol=udp
add action=accept chain=input_IOT comment="Accept DNS (tcp)" port=53 protocol=tcp
add action=accept chain=input_IOT comment="Accept DHCP (udp)" dst-port=67 protocol=udp src-port=68
add action=accept chain=input_IOT comment="Accept DHCP (tcp)" dst-port=67 port="" protocol=tcp src-port=68
add action=accept chain=input_RESTRICTED comment="Accept DHCP (udp)" dst-port=67 protocol=udp src-port=68
add action=accept chain=input_RESTRICTED comment="Accept DHCP (tcp)" dst-port=67 port="" protocol=tcp src-port=68
add action=fasttrack-connection chain=forward comment="fast-track for established,related" connection-state=established,related
add action=accept chain=forward comment="accept established,related" connection-state=established,related
add action=drop chain=forward comment="drop invalid state" connection-state=invalid disabled=yes
add action=jump chain=forward comment="forward HOME chain" in-interface=vlan10_home jump-target=forward_HOME
add action=jump chain=forward comment="forward IOT chain" in-interface=vlan11_iot jump-target=forward_IOT
add action=jump chain=forward comment="forward GUEST chain" in-interface=vlan12_guest jump-target=forward_GUEST
add action=accept chain=forward comment="accept traffic for port forwarding (dnat)" connection-nat-state=dstnat connection-state=new in-interface-list
add action=drop chain=forward comment="default policy DROP"
add action=accept chain=forward_HOME comment="default policy ACCEPT"
add action=accept chain=forward_IOT comment="accept all within IOT" out-interface=vlan11_iot
add action=accept chain=forward_IOT comment="accept WAN outbound traffic (tcp)" out-interface-list=wan-tun_ipv4 port="" protocol=tcp
add action=accept chain=forward_IOT comment="accept WAN outbound traffic (udp)" out-interface-list=wan-tun_ipv4 port="" protocol=udp
add action=accept chain=forward_GUEST comment="accept DNS (udp)" dst-address-list=GUEST_DNS port=53 protocol=tcp
add action=accept chain=forward_GUEST comment="accept DNS (tcp)" dst-address-list=GUEST_DNS port=53 protocol=udp
add action=accept chain=forward_GUEST comment="accept omada captive portal" dst-address=192.168.10.21 port=8088 protocol=tcp
add action=accept chain=forward_GUEST comment="accept HTTP HTTPS (tcp)" out-interface-list=wan-tun_ipv4 port=80,8080,443,8443 protocol=tcp
add action=accept chain=forward_GUEST comment="accept HTTP HTTPS (udp)" out-interface-list=wan-tun_ipv4 port=80,8080,443,8443 protocol=udp
/ip firewall nat
add action=masquerade chain=srcnat comment="masquarade for WAN" out-interface-list=wan-tun_ipv4
add action=dst-nat chain=dstnat comment="DNS interception for IOT (udp)" dst-address=!192.168.11.1 in-interface=vlan11_iot port=53 protocol=udp to-add
    192.168.11.1
add action=dst-nat chain=dstnat comment="DNS interception for IOT (tcp)" dst-address=!192.168.11.1 in-interface=vlan11_iot port=53 protocol=tcp to-add
    192.168.11.1
add action=dst-nat chain=dstnat comment="DNS interception for GUEST (udp)" dst-address-list=!GUEST_DNS in-interface=vlan12_guest port=53 protocol=udp 
    1.0.0.3
add action=dst-nat chain=dstnat comment="DNS interception for GUEST (tcp)" dst-address-list=!GUEST_DNS in-interface=vlan12_guest port=53 protocol=tcp 
    1.1.1.3
add action=dst-nat chain=dstnat comment="SSH to pireserver" in-interface-list=wan-tun_ipv4 port=2287 protocol=tcp to-addresses=192.168.10.10 to-ports=
add action=dst-nat chain=dstnat comment="PLEX to pireserver" in-interface-list=wan-tun_ipv4 port=23004 protocol=tcp to-addresses=192.168.11.10 to-port
add action=dst-nat chain=dstnat comment="OPENVPN (udp)" in-interface-list=wan-tun_ipv4 port=123 protocol=udp to-addresses=192.168.10.10 to-ports=1194
add action=dst-nat chain=dstnat comment="OPENVPN (tcp)" in-interface-list=wan-tun_ipv4 port=443 protocol=tcp to-addresses=192.168.10.10 to-ports=1194
add action=dst-nat chain=dstnat comment="HASSIO (tcp)" in-interface-list=wan-tun_ipv4 port=8123 protocol=tcp to-addresses=192.168.10.21 to-ports=8123

I found yesterday that it was not a problem routing VLANS (I read the thread you suggested, thanks) but more a problem related to this rule (when enabled, of course):

add action=drop chain=forward comment="drop invalid state" connection-state=invalid disabled=yes

Do you know why inter-VLAN traffic is sometimes interpreted as 'INVALID'?

anav · Mon Mar 29, 2021 10:33 pm

Like I said, let me be more blunt, the FW rules are a farce and make it too difficult to even read.
If you are interested in vlans that work and not cutesy fw rules let me know otherwise someone else can chime in.

tatianaroma · Tue Mar 30, 2021 1:04 pm

Like I said, let me be more blunt, the FW rules are a farce and make it too difficult to even read.
If you are interested in vlans that work and not cutesy fw rules let me know otherwise someone else can chime in.

Your remarks are not very constructive. I read the entire VLAN thread you suggested and I can say with confidence that my firewall rules are not very different from those of @pcunite.
I agree that those I first posted were a mess but I am talking of those from my second post.

Defaut DROP for input / forward and ALLOW necessary traffic.

But of course I am interested in making my VLANs better (they work btw if I don't drop the invalid connections which I will log later to better understand what is going on).

anav · Tue Mar 30, 2021 2:18 pm

The drop invalid rules in both the forward and input chain do not cause issues unless you have configured something else in error.

Also you need to post your entire config not just the firewall rules to determine what is going on....................

As for firewall rules you would be best to get rid of all that you have in place and use the defaults to narrow down the problem. After you have some starter rules in place AND A WORKING CONFIG, then we can talk about what else is required for firewall rules that seems to have driven you to add extra rules.........
................

/ip firewall filter
add action=accept chain=input comment=\
"defconf: accept established,related,untracked" connection-state=\
established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment=\
"defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1 (if not using capsman this rule can be removed)
add action=drop chain=input comment="defconf: drop all not coming from LAN" \
disabled=yes in-interface-list=!LAN
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
add action=accept chain=forward comment="defconf: accept in ipsec policy" \
ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" \
ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" \
connection-state=established,related
add action=accept chain=forward comment=\
"defconf: accept established,related, untracked" connection-state=\
established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
connection-state=invalid
add action=drop chain=forward comment=\
"defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat \
connection-state=new in-interface-list=WAN

If you want a bit better security modify the above to........

/ip firewall filter
add action=accept chain=input comment=\
"defconf: accept established,related,untracked" connection-state=\
established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
invalid
add action=accept chain=input dst-address=127.0.0.1 comment="defconf: accept to local loopback (for CAPsMAN)" {remove if not using capsman}
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment="Allow ADMIN to Router" in-interface-list=allowedsubnets\     {allowed subnets could be the LAN or another interface list you created}
src-address-list=adminaccess  {adminaccess is a firewall address list of static LANIPs of admin devices (desktop, laptop, ipad, smartphone etc)}
add action=accept chain=input comment="Allow LAN DNS queries - TCP" \
connection-state=new dst-port=53 in-interface-list=LAN protocol=tcp
add action=accept chain=input comment="Allow LAN DNS queries-UDP" \
connection-state=new dst-port=53 in-interface-list=LAN protocol=udp
add action=drop chain=input comment="Drop anything else! # ONLY ENABLE WHEN ADMIN RULE ABOVE IN PLACE
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
add action=accept chain=forward comment="defconf: accept in ipsec policy" \
ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" \
ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" \
connection-state=established,related
add action=accept chain=forward comment=\
"defconf: accept established,related, untracked" connection-state=\
established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
connection-state=invalid
add action=accept chain=forward comment="ENABLE Internet traffic" \
in-interface-list=LAN out-interface-list=WAN
add action=accept chain=forward comment="Allow Port Forwarding" \ {you can disable this rule until you need it}
connection-nat-state=dstnat connection-state=new in-interface-list=WAN
add action=drop chain=forward comment="DROP ALL other FORWARD traffic"
/ip firewall nat
add action=masquerade chain=srcnat out-interface-list=WAN

tatianaroma · Tue Mar 30, 2021 2:53 pm

thanks for the input.
I will try it tonight or tomorrow.

Need help with Vlan routing

Need help with Vlan routing

Re: Need help with Vlan routing

Re: Need help with Vlan routing

Re: Need help with Vlan routing

Re: Need help with Vlan routing

Re: Need help with Vlan routing

Re: Need help with Vlan routing

Re: Need help with Vlan routing

Who is online