Routing between VLANs stopped working after PCC load balancing.

RouterOS General
1

Hi everybody,
First I’m sorry for there are so much topic of PPC Load Balancing, I have searched some topics but It doesn’t solve my own problem.
I’m setting a network with 2 WAN using PPPOE, It worked well until I create new Vlans (First I just had one LAN). The problem is that I can’t access through Vlans after PPC Load Balancing. Below is my script, is there anyone here who has had the same problems?.
I would appreciate hearing your opinion on this case.
Thanks

jul/18/2018 15:24:01 by RouterOS 6.42.4

software id = 5RKA-6A72

model = CRS326-24G-2S+

serial number = 763C07B33D5A

/interface bridge
add fast-forward=no name=LAN
add fast-forward=no name=Trunk vlan-filtering=yes
/interface pppoe-client
add disabled=no interface=ether1 name=pppoe-out1 user=user1
add disabled=no interface=ether2 name=pppoe-out2 user=user2
/interface vlan
add interface=Trunk name=vlan10 vlan-id=10
add interface=Trunk name=vlan20 vlan-id=20
add interface=Trunk name=vlan30 vlan-id=30
add interface=Trunk name=vlan40 vlan-id=40
/ip pool
add name=dhcp_pool0 ranges=192.168.1.2-192.168.1.254
add name=dhcp_pool1 ranges=10.10.10.2-10.10.10.254
add name=dhcp_pool2 ranges=10.10.30.2-10.10.30.254
add name=dhcp_pool3 ranges=10.10.20.2-10.10.20.254
add name=dhcp_pool4 ranges=10.10.40.2-10.10.40.254
/ip dhcp-server
add address-pool=dhcp_pool0 disabled=no interface=LAN name=dhcp1
add address-pool=dhcp_pool1 disabled=no interface=vlan10 name=dhcp2
add address-pool=dhcp_pool2 disabled=no interface=vlan30 name=dhcp3
add address-pool=dhcp_pool3 disabled=no interface=vlan20 name=dhcp4
add address-pool=dhcp_pool4 disabled=no interface=vlan40 name=dhcp5
/interface bridge port
add bridge=LAN interface=ether3
add bridge=Trunk interface=ether4 pvid=10
add bridge=Trunk interface=ether6 pvid=20
add bridge=Trunk interface=ether8 pvid=30
add bridge=Trunk interface=ether10 pvid=40
/interface bridge vlan
add bridge=Trunk tagged=ether6,ether8,ether10,Trunk untagged=ether4 vlan-ids=10
add bridge=Trunk tagged=ether4,ether8,ether10,Trunk untagged=ether6 vlan-ids=20
add bridge=Trunk tagged=ether4,ether6,ether10,Trunk untagged=ether8 vlan-ids=30
add bridge=Trunk tagged=ether4,ether6,ether8,Trunk untagged=ether10 vlan-ids=40
/ip address
add address=192.168.1.1/24 interface=LAN network=192.168.1.0
add address=10.10.10.1/24 interface=vlan10 network=10.10.10.0
add address=10.10.20.1/24 interface=vlan20 network=10.10.20.0
add address=10.10.30.1/24 interface=vlan30 network=10.10.30.0
add address=10.10.40.1/24 interface=vlan40 network=10.10.40.0
/ip dhcp-server network
add address=10.10.10.0/24 gateway=10.10.10.1
add address=10.10.20.0/24 gateway=10.10.20.1
add address=10.10.30.0/24 gateway=10.10.30.1
add address=10.10.40.0/24 gateway=10.10.40.1
add address=192.168.1.0/24 gateway=192.168.1.1
/ip dns
set servers=8.8.8.8
/ip firewall mangle
add action=mark-connection chain=input in-interface=pppoe-out1
new-connection-mark=WAN1_conn passthrough=yes
add action=mark-connection chain=input in-interface=pppoe-out2
new-connection-mark=WAN2_conn passthrough=yes
add action=mark-routing chain=output connection-mark=WAN1_conn
new-routing-mark=to_WAN1 passthrough=yes
add action=mark-routing chain=output connection-mark=WAN2_conn
new-routing-mark=to_WAN2 passthrough=yes
add action=mark-connection chain=prerouting dst-address-type=!local
in-interface=LAN new-connection-mark=WAN1_conn passthrough=yes
per-connection-classifier=both-addresses:2/0
add action=mark-connection chain=prerouting dst-address-type=!local
in-interface=LAN new-connection-mark=WAN2_conn passthrough=yes
per-connection-classifier=both-addresses:2/1
add action=mark-connection chain=prerouting dst-address-type=!local
in-interface=vlan10 new-connection-mark=WAN1_conn passthrough=yes
per-connection-classifier=both-addresses:2/0
add action=mark-connection chain=prerouting dst-address-type=!local
in-interface=vlan10 new-connection-mark=WAN2_conn passthrough=yes
per-connection-classifier=both-addresses:2/1
add action=mark-connection chain=prerouting dst-address-type=!local
in-interface=vlan20 new-connection-mark=WAN1_conn passthrough=yes
per-connection-classifier=both-addresses:2/0
add action=mark-connection chain=prerouting dst-address-type=!local
in-interface=vlan20 new-connection-mark=WAN2_conn passthrough=yes
per-connection-classifier=both-addresses:2/1
add action=mark-connection chain=prerouting dst-address-type=!local
in-interface=vlan30 new-connection-mark=WAN1_conn passthrough=yes
per-connection-classifier=both-addresses:2/0
add action=mark-connection chain=prerouting dst-address-type=!local
in-interface=vlan30 new-connection-mark=WAN2_conn passthrough=yes
per-connection-classifier=both-addresses:2/1
add action=mark-connection chain=prerouting dst-address-type=!local
in-interface=vlan40 new-connection-mark=WAN1_conn passthrough=yes
per-connection-classifier=both-addresses:2/0
add action=mark-connection chain=prerouting dst-address-type=!local
in-interface=vlan40 new-connection-mark=WAN2_conn passthrough=yes
per-connection-classifier=both-addresses:2/1
add action=mark-routing chain=prerouting connection-mark=WAN1_conn
in-interface=LAN new-routing-mark=to_WAN1 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN2_conn
in-interface=LAN new-routing-mark=to_WAN2 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN1_conn
in-interface=vlan10 new-routing-mark=to_WAN1 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN2_conn
in-interface=vlan10 new-routing-mark=to_WAN2 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN1_conn
in-interface=vlan20 new-routing-mark=to_WAN1 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN2_conn
in-interface=vlan20 new-routing-mark=to_WAN2 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN1_conn
in-interface=vlan30 new-routing-mark=to_WAN1 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN2_conn
in-interface=vlan30 new-routing-mark=to_WAN2 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN1_conn
in-interface=vlan40 new-routing-mark=to_WAN1 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN2_conn
in-interface=vlan40 new-routing-mark=to_WAN2 passthrough=yes
/ip firewall nat
add action=masquerade chain=srcnat out-interface=pppoe-out1
add action=masquerade chain=srcnat out-interface=pppoe-out2
/ip route
add check-gateway=ping distance=1 gateway=pppoe-out1 routing-mark=t
add check-gateway=ping distance=1 gateway=pppoe-out2 routing-mark=t
add check-gateway=ping distance=1 gateway=pppoe-out1
add check-gateway=ping distance=2 gateway=pppoe-out2

2

Let’s translate the L2 term “VLANs” to an L3 term “connected subnets”, i.e. IP subnets in which the Mikrotik has its own IP addresses.

A connected subnet also needs a route; it is added dynamically (i.e. automatically once the IP address/mask configuration is added), with distance=0, but only to the default routing table (called “main”).

You seem to be another victim of wishful thinking, assuming that dst-address-type=local matches to any destination address from any connected subnet, while in reality it only matches Mikrotik’s own addresses. So by using dst-address-type=!local in the mangle rules assigning connection marks, you do not prevent packets towards local subnets from being connection-marked, so these packets consequently get routing marks, and the route for them is chosen among those bearing the same routing mark.

Since a matching route with that routing mark exists, these packets are routed out via WAN 1 or WAN 2 rather than being sent to the connected LAN subnet because the route for that subnet only exists in routing table “main”.

So while the topics title, “vlans do not work” suggests that VLAN tagging stopped working, actually routing between connected subnets (which in your case happen to occupy VLANs) stopped working.

But you want a solution, right? One possibility is to create an address list of all the connected subnets:
/ip firewall address-list
add list=connected-subnets address=192.168.1.1/24
add list=connected-subnets address=10.10.10.1/24
add list=connected-subnets address=10.10.20.1/24
add list=connected-subnets address=10.10.30.1/24
add list=connected-subnets address=10.10.40.1/24
and use dst-address-list=!connected-subnets instead of dst-address-type=!local in the action=mark-connection rules.

Another (simpler to configure but possibly slightly more CPU-intensive) way is to use

/ip route rule
add action=lookup table=main dst-address=192.168.1.1/24
add action=lookup table=main dst-address=10.10.10.1/24
add action=lookup table=main dst-address=10.10.20.1/24
add action=lookup table=main dst-address=10.10.30.1/24
add action=lookup table=main dst-address=10.10.40.1/24
to negate the effect of the routing-mark (route rules are evaluated just before route lookup and override or translate an eventually existing routing-mark assignment)

Unrelated to your issue, just to “structured-oriented firewalling”: whichever of the two ways above you choose, I would recommend you to use an interface list:
/interface list
add name=all-lans
/interface list member
add list=all-lans interface=LAN
add list=all-lans interface=vlan10
add list=all-lans interface=vlan20
add list=all-lans interface=vlan30
add list=all-lans interface=vlan40
and replace each 5 identical mangle rules differing only in in-interface value by a single rule using in-interface-list=all-lans instead.

3

Thank you so much for your kindly support.
P/s: I have also changed the topic title to fit the content.

4

Nope :slight_smile: Now the title is definitely wrong. “Routing between VLANs stopped working …” would be a correct one.

5

Hope everything will be fine now :laughing:
P/s: Thank you once again.

6

Hi Sindy.
Sorry to bother you again.
But after apply those configuration I feel like there is some problem with forwarding port. Inside the local i want to NAT some port to out side internet, but it seem not working.
Hope you will check my configuration again and give me some opinions.
Thanks

7

Can you be more precise? In the only configuration posted above, there is nothing what would resemble a port forwarding setup, so what have you already tried and what didn’t work?

8

Hi Sindy.
Here is the mangle code which i have follow your guild before.
/ip firewall mangle
add action=mark-connection chain=input in-interface=pppoe-out1
new-connection-mark=WAN1_conn passthrough=yes
add action=mark-connection chain=input in-interface=pppoe-out2
new-connection-mark=WAN2_conn passthrough=yes
add action=mark-connection chain=input in-interface=ether3 new-connection-mark=
WAN3_conn passthrough=yes
add action=mark-routing chain=output connection-mark=WAN1_conn
new-routing-mark=to_WAN1 passthrough=yes
add action=mark-routing chain=output connection-mark=WAN2_conn
new-routing-mark=to_WAN2 passthrough=yes
add action=mark-routing chain=output connection-mark=WAN3_conn
new-routing-mark=to_WAN3 passthrough=yes
add action=mark-connection chain=prerouting dst-address-list=!connected-subnets
in-interface-list=all-lans new-connection-mark=WAN1_conn passthrough=yes
per-connection-classifier=both-addresses-and-ports:3/0
add action=mark-connection chain=prerouting dst-address-list=!connected-subnets
dst-address-type=“” in-interface-list=all-lans new-connection-mark=
WAN2_conn passthrough=yes per-connection-classifier=
both-addresses-and-ports:3/1
add action=mark-connection chain=prerouting dst-address-list=!connected-subnets
dst-address-type=“” in-interface-list=all-lans new-connection-mark=
WAN3_conn passthrough=yes per-connection-classifier=
both-addresses-and-ports:3/2
add action=mark-routing chain=prerouting connection-mark=WAN1_conn
in-interface-list=all-lans new-routing-mark=to_WAN1 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN2_conn
in-interface-list=all-lans new-routing-mark=to_WAN2 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN3_conn
in-interface-list=all-lans new-routing-mark=to_WAN3 passthrough=yes
/ip firewall nat
add action=masquerade chain=srcnat out-interface=ether3
add action=masquerade chain=srcnat out-interface=pppoe-out1
add action=masquerade chain=srcnat out-interface=pppoe-out2
add action=dst-nat chain=dstnat dst-port=8080 in-interface=pppoe-out1 protocol=
tcp to-addresses=10.1.1.5 to-ports=80
/ip route
add check-gateway=ping distance=1 gateway=pppoe-out1 routing-mark=to_WAN1
add check-gateway=ping distance=1 gateway=pppoe-out2 routing-mark=to_WAN2
add check-gateway=ping distance=1 gateway=45.xxx.xxx.xxx routing-mark=to_WAN3
add distance=1 gateway=45.xxx.xxx.xxx
add distance=2 gateway=pppoe-out1
add distance=3 gateway=pppoe-out2But when i nat port 80 and check it in ping.eu the result show that is closed.
Do you have any idea about that.
Thanks

9

Yes, the idea is that while you’ve properly added the action=dst-nat rule to /ip fireewall nat chain=dstnat, you haven’t permitted the packets from WAN(s) to LAN to be forwarded in the /ip firewall filter chain=forward. Not knowing how your rules look like, I just suggest you that there is a nice matcher, connection-nat-state, and if you put action=accept chain=forward in-interface-list=WAN connection-nat-state=dstnat to an appropriate place, you will let through anything what you dst-nat in the nat table.

10

It’s my mistake, the service was not running yet When I check port opening.
I also found that my server can not connect from inside Lan through Ip public, After some google I found the reason https://wiki.mikrotik.com/wiki/Hairpin_NAT.
But after change the NAT rule i still can not connect to my server inside LAN.
Here is the NAT code
/ip firewall nat
add action=masquerade chain=srcnat dst-address=192.168.15.77 dst-address-type=“” dst-port=8080 out-interface-list=all-lans protocol=tcp src-address-list=connected-subnets
add action=dst-nat chain=dstnat dst-address=14.xxx.xxx.xxx dst-address-type=local dst-port=8080 protocol=tcp to-addresses=192.168.15.77 to-ports=80Thank you very much for your cooperation and time

11

The srcnat chain (where the masquerade and src-nat rules live) is parsed as one of the last ones in the chain - this picture has it all. So in the masquerade rule you have posted above, just replace dst-port=8080 (which is only true until the dst-nat is made) by dst-port=80 (which is the value the packet has when it reaches the masquerade rule).

12

Thank you, I had the same issue with !local, corrected with your suggestion.

13

Similarly could you have not used that firewall adddress list as a first rule in the mangle chain

/ip firewall mangle
add action=accept chain=prerouting in-interface-list=LAN dst-address-list=connected-subnets

Which says let any traffic between vlans be executed before any mangling!

THEN the mangle rules you had including
!dst-address=local which was really stating dont mark traffic from LAN to the router itself. I made the same mistake thinking it meant subnet to subnet traffic but really it only means, to local interfaces, not local users.

14

I’m unsure if you replied to me, but I would assume so.

/ip firewall mangle
add action=accept chain=prerouting in-interface-list=LAN dst-address-list=connected-subnets

This does the trick, however, I do not want inter VLAN traffic, unless the parts I specified for, e.g. access to a printer. I know my config is not included in this topic, and user requirements were not specified. But I tried your suggestion, just for testing, and it does work.

!dst-address=local which was really stating dont mark traffic from LAN to the router itself. I made the same mistake thinking it meant subnet to subnet traffic but really it only means, to local interfaces, not local users.

Here I lost you, you refer to dst-address-type=!local maybe?

15

Good to know, thanks for the feedback.
However you are not quite right YOU DO NEED THAT RULE TO ensure any local traffic CAN reach other subnets prior to mangling for load balancing etc. AKA The POSSIBILITY is created.
What is ALLOWED to happen is determined by your firewall rules.

So, Then you use firewall to create what traffic is permitted and the last rule drops any other traffic cold. Simple, clean!

add action=accept chain=forward comment=“common printer” in-inteface-list=LAN dst-address=IPaddress of Printer.
…
…
add action=drop chain=forward comment=“drop all else”

Now you could modify the rules to narrow it down if desired.
/ip firewall mangle
add action=accept chain=prerouting in-interface-list=LAN dst-address-list=servers_on_LAN

+++++++++++++++++

As for the second part, just wanted to state that you can then use the original rules found in most LB which on per connection class basis identify
dst-address-type=!local meaning dont mark traffic going to the router itself (or router local interfaces ).