Routing between VLANs stopped working after PCC load balancing.

TheLordOfTheShells · Wed Jul 18, 2018 11:41 am

Hi everybody,
First I'm sorry for there are so much topic of PPC Load Balancing, I have searched some topics but It doesn't solve my own problem.
I'm setting a network with 2 WAN using PPPOE, It worked well until I create new Vlans (First I just had one LAN). The problem is that I can't access through Vlans after PPC Load Balancing. Below is my script, is there anyone here who has had the same problems?.
I would appreciate hearing your opinion on this case.
Thanks

Code: Select all


# jul/18/2018 15:24:01 by RouterOS 6.42.4

# software id = 5RKA-6A72

#

# model = CRS326-24G-2S+

# serial number = 763C07B33D5A

/interface bridge

add fast-forward=no name=LAN

add fast-forward=no name=Trunk vlan-filtering=yes

/interface pppoe-client

add disabled=no interface=ether1 name=pppoe-out1 user=user1

add disabled=no interface=ether2 name=pppoe-out2 user=user2

/interface vlan

add interface=Trunk name=vlan10 vlan-id=10

add interface=Trunk name=vlan20 vlan-id=20

add interface=Trunk name=vlan30 vlan-id=30

add interface=Trunk name=vlan40 vlan-id=40

/ip pool

add name=dhcp_pool0 ranges=192.168.1.2-192.168.1.254

add name=dhcp_pool1 ranges=10.10.10.2-10.10.10.254

add name=dhcp_pool2 ranges=10.10.30.2-10.10.30.254

add name=dhcp_pool3 ranges=10.10.20.2-10.10.20.254

add name=dhcp_pool4 ranges=10.10.40.2-10.10.40.254

/ip dhcp-server

add address-pool=dhcp_pool0 disabled=no interface=LAN name=dhcp1

add address-pool=dhcp_pool1 disabled=no interface=vlan10 name=dhcp2

add address-pool=dhcp_pool2 disabled=no interface=vlan30 name=dhcp3

add address-pool=dhcp_pool3 disabled=no interface=vlan20 name=dhcp4

add address-pool=dhcp_pool4 disabled=no interface=vlan40 name=dhcp5

/interface bridge port

add bridge=LAN interface=ether3

add bridge=Trunk interface=ether4 pvid=10

add bridge=Trunk interface=ether6 pvid=20

add bridge=Trunk interface=ether8 pvid=30

add bridge=Trunk interface=ether10 pvid=40

/interface bridge vlan

add bridge=Trunk tagged=ether6,ether8,ether10,Trunk untagged=ether4 vlan-ids=10

add bridge=Trunk tagged=ether4,ether8,ether10,Trunk untagged=ether6 vlan-ids=20

add bridge=Trunk tagged=ether4,ether6,ether10,Trunk untagged=ether8 vlan-ids=30

add bridge=Trunk tagged=ether4,ether6,ether8,Trunk untagged=ether10 vlan-ids=40

/ip address

add address=192.168.1.1/24 interface=LAN network=192.168.1.0

add address=10.10.10.1/24 interface=vlan10 network=10.10.10.0

add address=10.10.20.1/24 interface=vlan20 network=10.10.20.0

add address=10.10.30.1/24 interface=vlan30 network=10.10.30.0

add address=10.10.40.1/24 interface=vlan40 network=10.10.40.0

/ip dhcp-server network

add address=10.10.10.0/24 gateway=10.10.10.1

add address=10.10.20.0/24 gateway=10.10.20.1

add address=10.10.30.0/24 gateway=10.10.30.1

add address=10.10.40.0/24 gateway=10.10.40.1

add address=192.168.1.0/24 gateway=192.168.1.1

/ip dns

set servers=8.8.8.8

/ip firewall mangle

add action=mark-connection chain=input in-interface=pppoe-out1 \

    new-connection-mark=WAN1_conn passthrough=yes

add action=mark-connection chain=input in-interface=pppoe-out2 \

    new-connection-mark=WAN2_conn passthrough=yes

add action=mark-routing chain=output connection-mark=WAN1_conn \

    new-routing-mark=to_WAN1 passthrough=yes

add action=mark-routing chain=output connection-mark=WAN2_conn \

    new-routing-mark=to_WAN2 passthrough=yes

add action=mark-connection chain=prerouting dst-address-type=!local \

    in-interface=LAN new-connection-mark=WAN1_conn passthrough=yes \

    per-connection-classifier=both-addresses:2/0

add action=mark-connection chain=prerouting dst-address-type=!local \

    in-interface=LAN new-connection-mark=WAN2_conn passthrough=yes \

    per-connection-classifier=both-addresses:2/1

add action=mark-connection chain=prerouting dst-address-type=!local \

    in-interface=vlan10 new-connection-mark=WAN1_conn passthrough=yes \

    per-connection-classifier=both-addresses:2/0

add action=mark-connection chain=prerouting dst-address-type=!local \

    in-interface=vlan10 new-connection-mark=WAN2_conn passthrough=yes \

    per-connection-classifier=both-addresses:2/1

add action=mark-connection chain=prerouting dst-address-type=!local \

    in-interface=vlan20 new-connection-mark=WAN1_conn passthrough=yes \

    per-connection-classifier=both-addresses:2/0

add action=mark-connection chain=prerouting dst-address-type=!local \

    in-interface=vlan20 new-connection-mark=WAN2_conn passthrough=yes \

    per-connection-classifier=both-addresses:2/1

add action=mark-connection chain=prerouting dst-address-type=!local \

    in-interface=vlan30 new-connection-mark=WAN1_conn passthrough=yes \

    per-connection-classifier=both-addresses:2/0

add action=mark-connection chain=prerouting dst-address-type=!local \

    in-interface=vlan30 new-connection-mark=WAN2_conn passthrough=yes \

    per-connection-classifier=both-addresses:2/1

add action=mark-connection chain=prerouting dst-address-type=!local \

    in-interface=vlan40 new-connection-mark=WAN1_conn passthrough=yes \

    per-connection-classifier=both-addresses:2/0

add action=mark-connection chain=prerouting dst-address-type=!local \

    in-interface=vlan40 new-connection-mark=WAN2_conn passthrough=yes \

    per-connection-classifier=both-addresses:2/1

add action=mark-routing chain=prerouting connection-mark=WAN1_conn \

    in-interface=LAN new-routing-mark=to_WAN1 passthrough=yes

add action=mark-routing chain=prerouting connection-mark=WAN2_conn \

    in-interface=LAN new-routing-mark=to_WAN2 passthrough=yes

add action=mark-routing chain=prerouting connection-mark=WAN1_conn \

    in-interface=vlan10 new-routing-mark=to_WAN1 passthrough=yes

add action=mark-routing chain=prerouting connection-mark=WAN2_conn \

    in-interface=vlan10 new-routing-mark=to_WAN2 passthrough=yes

add action=mark-routing chain=prerouting connection-mark=WAN1_conn \

    in-interface=vlan20 new-routing-mark=to_WAN1 passthrough=yes

add action=mark-routing chain=prerouting connection-mark=WAN2_conn 

    in-interface=vlan20 new-routing-mark=to_WAN2 passthrough=yes

add action=mark-routing chain=prerouting connection-mark=WAN1_conn 

    in-interface=vlan30 new-routing-mark=to_WAN1 passthrough=yes

add action=mark-routing chain=prerouting connection-mark=WAN2_conn 

    in-interface=vlan30 new-routing-mark=to_WAN2 passthrough=yes

add action=mark-routing chain=prerouting connection-mark=WAN1_conn 

    in-interface=vlan40 new-routing-mark=to_WAN1 passthrough=yes

add action=mark-routing chain=prerouting connection-mark=WAN2_conn 

    in-interface=vlan40 new-routing-mark=to_WAN2 passthrough=yes

/ip firewall nat

add action=masquerade chain=srcnat out-interface=pppoe-out1

add action=masquerade chain=srcnat out-interface=pppoe-out2

/ip route

add check-gateway=ping distance=1 gateway=pppoe-out1 routing-mark=t

add check-gateway=ping distance=1 gateway=pppoe-out2 routing-mark=t

add check-gateway=ping distance=1 gateway=pppoe-out1

add check-gateway=ping distance=2 gateway=pppoe-out2

sindy · Wed Jul 18, 2018 12:30 pm

Let's translate the L2 term "VLANs" to an L3 term "connected subnets", i.e. IP subnets in which the Mikrotik has its own IP addresses.

A connected subnet also needs a route; it is added dynamically (i.e. automatically once the IP address/mask configuration is added), with distance=0, but only to the default routing table (called "main").

You seem to be another victim of wishful thinking, assuming that dst-address-type=local matches to any destination address from any connected subnet, while in reality it only matches Mikrotik's own addresses. So by using dst-address-type=!local in the mangle rules assigning connection marks, you do not prevent packets towards local subnets from being connection-marked, so these packets consequently get routing marks, and the route for them is chosen among those bearing the same routing mark.

Since a matching route with that routing mark exists, these packets are routed out via WAN 1 or WAN 2 rather than being sent to the connected LAN subnet because the route for that subnet only exists in routing table "main".

So while the topics title, "vlans do not work" suggests that VLAN tagging stopped working, actually routing between connected subnets (which in your case happen to occupy VLANs) stopped working.

But you want a solution, right? One possibility is to create an address list of all the connected subnets:
/ip firewall address-list
add list=connected-subnets address=192.168.1.1/24
add list=connected-subnets address=10.10.10.1/24
add list=connected-subnets address=10.10.20.1/24
add list=connected-subnets address=10.10.30.1/24
add list=connected-subnets address=10.10.40.1/24

and use dst-address-list=!connected-subnets instead of dst-address-type=!local in the action=mark-connection rules.

Another (simpler to configure but possibly slightly more CPU-intensive) way is to use

/ip route rule
add action=lookup table=main dst-address=192.168.1.1/24
add action=lookup table=main dst-address=10.10.10.1/24
add action=lookup table=main dst-address=10.10.20.1/24
add action=lookup table=main dst-address=10.10.30.1/24
add action=lookup table=main dst-address=10.10.40.1/24

to negate the effect of the routing-mark (route rules are evaluated just before route lookup and override or translate an eventually existing routing-mark assignment)

Unrelated to your issue, just to "structured-oriented firewalling": whichever of the two ways above you choose, I would recommend you to use an interface list:
/interface list
add name=all-lans
/interface list member
add list=all-lans interface=LAN
add list=all-lans interface=vlan10
add list=all-lans interface=vlan20
add list=all-lans interface=vlan30
add list=all-lans interface=vlan40

and replace each 5 identical mangle rules differing only in in-interface value by a single rule using in-interface-list=all-lans instead.

TheLordOfTheShells · Wed Jul 18, 2018 1:59 pm

Let's translate the L2 term "VLANs" to an L3 term "connected subnets", i.e. IP subnets in which the Mikrotik has its own IP addresses.

A connected subnet also needs a route; it is added dynamically (i.e. automatically once the IP address/mask configuration is added), with distance=0, but only to the default routing table (called "main").

You seem to be another victim of wishful thinking, assuming that dst-address-type=local matches to any destination address from any connected subnet, while in reality it only matches Mikrotik's own addresses. So by using dst-address-type=!local in the mangle rules assigning connection marks, you do not prevent packets towards local subnets from being connection-marked, so these packets consequently get routing marks, and the route for them is chosen among those bearing the same routing mark.

Since a matching route with that routing mark exists, these packets are routed out via WAN 1 or WAN 2 rather than being sent to the connected LAN subnet because the route for that subnet only exists in routing table "main".

So while the topics title, "vlans do not work" suggests that VLAN tagging stopped working, actually routing between connected subnets (which in your case happen to occupy VLANs) stopped working.

But you want a solution, right? One possibility is to create an address list of all the connected subnets:
/ip firewall address-list
add list=connected-subnets address=192.168.1.1/24
add list=connected-subnets address=10.10.10.1/24
add list=connected-subnets address=10.10.20.1/24
add list=connected-subnets address=10.10.30.1/24
add list=connected-subnets address=10.10.40.1/24

and use dst-address-list=!connected-subnets instead of dst-address-type=!local in the action=mark-connection rules.

Another (simpler to configure but possibly slightly more CPU-intensive) way is to use

/ip route rule
add action=lookup table=main dst-address=192.168.1.1/24
add action=lookup table=main dst-address=10.10.10.1/24
add action=lookup table=main dst-address=10.10.20.1/24
add action=lookup table=main dst-address=10.10.30.1/24
add action=lookup table=main dst-address=10.10.40.1/24

to negate the effect of the routing-mark (route rules are evaluated just before route lookup and override or translate an eventually existing routing-mark assignment)

Unrelated to your issue, just to "structured-oriented firewalling": whichever of the two ways above you choose, I would recommend you to use an interface list:
/interface list
add name=all-lans
/interface list member
add list=all-lans interface=LAN
add list=all-lans interface=vlan10
add list=all-lans interface=vlan20
add list=all-lans interface=vlan30
add list=all-lans interface=vlan40

and replace each 5 identical mangle rules differing only in in-interface value by a single rule using in-interface-list=all-lans instead.

Thank you so much for your kindly support.
P/s: I have also changed the topic title to fit the content.

sindy · Wed Jul 18, 2018 3:57 pm

Nope

Now the title is definitely wrong. "Routing between VLANs stopped working ..." would be a correct one.

TheLordOfTheShells · Wed Jul 18, 2018 5:49 pm

Nope Now the title is definitely wrong. "Routing between VLANs stopped working ..." would be a correct one.

Hope everything will be fine now

P/s: Thank you once again.

TheLordOfTheShells · Mon Jul 30, 2018 1:32 pm

Hi Sindy.
Sorry to bother you again.
But after apply those configuration I feel like there is some problem with forwarding port. Inside the local i want to NAT some port to out side internet, but it seem not working.
Hope you will check my configuration again and give me some opinions.
Thanks

sindy · Mon Jul 30, 2018 2:07 pm

Can you be more precise? In the only configuration posted above, there is nothing what would resemble a port forwarding setup, so what have you already tried and what didn't work?

TheLordOfTheShells · Mon Jul 30, 2018 4:27 pm

Can you be more precise? In the only configuration posted above, there is nothing what would resemble a port forwarding setup, so what have you already tried and what didn't work?

Hi Sindy.
Here is the mangle code which i have follow your guild before.

Code: Select all

/ip firewall mangle

add action=mark-connection chain=input in-interface=pppoe-out1 \

    new-connection-mark=WAN1_conn passthrough=yes

add action=mark-connection chain=input in-interface=pppoe-out2 \

    new-connection-mark=WAN2_conn passthrough=yes

add action=mark-connection chain=input in-interface=ether3 new-connection-mark=\

    WAN3_conn passthrough=yes

add action=mark-routing chain=output connection-mark=WAN1_conn \

    new-routing-mark=to_WAN1 passthrough=yes

add action=mark-routing chain=output connection-mark=WAN2_conn \

    new-routing-mark=to_WAN2 passthrough=yes

add action=mark-routing chain=output connection-mark=WAN3_conn \

    new-routing-mark=to_WAN3 passthrough=yes

add action=mark-connection chain=prerouting dst-address-list=!connected-subnets \

    in-interface-list=all-lans new-connection-mark=WAN1_conn passthrough=yes \

    per-connection-classifier=both-addresses-and-ports:3/0

add action=mark-connection chain=prerouting dst-address-list=!connected-subnets \

    dst-address-type="" in-interface-list=all-lans new-connection-mark=\

    WAN2_conn passthrough=yes per-connection-classifier=\

    both-addresses-and-ports:3/1

add action=mark-connection chain=prerouting dst-address-list=!connected-subnets \

    dst-address-type="" in-interface-list=all-lans new-connection-mark=\

    WAN3_conn passthrough=yes per-connection-classifier=\

    both-addresses-and-ports:3/2

add action=mark-routing chain=prerouting connection-mark=WAN1_conn \

    in-interface-list=all-lans new-routing-mark=to_WAN1 passthrough=yes

add action=mark-routing chain=prerouting connection-mark=WAN2_conn \

    in-interface-list=all-lans new-routing-mark=to_WAN2 passthrough=yes

add action=mark-routing chain=prerouting connection-mark=WAN3_conn \

    in-interface-list=all-lans new-routing-mark=to_WAN3 passthrough=yes

/ip firewall nat

add action=masquerade chain=srcnat out-interface=ether3

add action=masquerade chain=srcnat out-interface=pppoe-out1

add action=masquerade chain=srcnat out-interface=pppoe-out2

add action=dst-nat chain=dstnat dst-port=8080 in-interface=pppoe-out1 protocol=\

    tcp to-addresses=10.1.1.5 to-ports=80

/ip route

add check-gateway=ping distance=1 gateway=pppoe-out1 routing-mark=to_WAN1

add check-gateway=ping distance=1 gateway=pppoe-out2 routing-mark=to_WAN2

add check-gateway=ping distance=1 gateway=45.xxx.xxx.xxx routing-mark=to_WAN3

add distance=1 gateway=45.xxx.xxx.xxx

add distance=2 gateway=pppoe-out1

add distance=3 gateway=pppoe-out2

But when i nat port 80 and check it in ping.eu the result show that is closed.
Do you have any idea about that.
Thanks

sindy · Mon Jul 30, 2018 4:36 pm

Yes, the idea is that while you've properly added the action=dst-nat rule to /ip fireewall nat chain=dstnat, you haven't permitted the packets from WAN(s) to LAN to be forwarded in the /ip firewall filter chain=forward. Not knowing how your rules look like, I just suggest you that there is a nice matcher, connection-nat-state, and if you put action=accept chain=forward in-interface-list=WAN connection-nat-state=dstnat to an appropriate place, you will let through anything what you dst-nat in the nat table.

TheLordOfTheShells · Tue Jul 31, 2018 6:13 am

Yes, the idea is that while you've properly added the action=dst-nat rule to /ip fireewall nat chain=dstnat, you haven't permitted the packets from WAN(s) to LAN to be forwarded in the /ip firewall filter chain=forward. Not knowing how your rules look like, I just suggest you that there is a nice matcher, connection-nat-state, and if you put action=accept chain=forward in-interface-list=WAN connection-nat-state=dstnat to an appropriate place, you will let through anything what you dst-nat in the nat table.

It's my mistake, the service was not running yet When I check port opening.
I also found that my server can not connect from inside Lan through Ip public, After some google I found the reason https://wiki.mikrotik.com/wiki/Hairpin_NAT.
But after change the NAT rule i still can not connect to my server inside LAN.
Here is the NAT code

Code: Select all

/ip firewall nat

add action=masquerade chain=srcnat dst-address=192.168.15.77 dst-address-type="" dst-port=8080 out-interface-list=all-lans protocol=tcp src-address-list=connected-subnets

add action=dst-nat chain=dstnat dst-address=14.xxx.xxx.xxx dst-address-type=local dst-port=8080 protocol=tcp to-addresses=192.168.15.77 to-ports=80

Thank you very much for your cooperation and time

sindy · Tue Jul 31, 2018 12:18 pm

The srcnat chain (where the masquerade and src-nat rules live) is parsed as one of the last ones in the chain - this picture has it all. So in the masquerade rule you have posted above, just replace dst-port=8080 (which is only true until the dst-nat is made) by dst-port=80 (which is the value the packet has when it reaches the masquerade rule).

Routing between VLANs stopped working after PCC load balancing. [SOLVED]

Routing between VLANs stopped working after PCC load balancing.

Re: VLANs doesn't work after PCC load balancing. [SOLVED]

Re: VLANs doesn't work after PCC load balancing.

Re: VLAN tagging stopped working after PCC load balancing.

Re: VLAN tagging stopped working after PCC load balancing.

Re: Routing between VLANs stopped working after PCC load balancing.

Re: Routing between VLANs stopped working after PCC load balancing.

Re: Routing between VLANs stopped working after PCC load balancing.

Re: Routing between VLANs stopped working after PCC load balancing.

Re: Routing between VLANs stopped working after PCC load balancing.

Re: Routing between VLANs stopped working after PCC load balancing.

Who is online