I'm having problems with load balance since upgrade to ver 4, I downgraded but lost config (no backups!) so I had to recreate it, but still having the same problem:
Sometimes webpages would not open, just gives 'cannot display...' or just a white screen in IE, sometimes webpage download but it seems 'corrupt', i.e. does not load completely, frames out of place, etc.
I'm not sure if it's a MTU/MSS issue or with my LoadBalance config.
I load balance 6 lines, the first 4 (WAN1-WAN4)is static IP ADSL routers, the last two(ADSL5 & ADSL6) is dynamic IP pppoe-clients. My ISP says that MTU is 1492 but I left default pppoe-client setting as 1480.
I tried to change the WAN1-4 MTU to 1480, 1440, 1454, etc. but still same problem. I tried to create change-MSS rules for WAN1-4 so it's the same rules as the automatic created for pppoe, but no change. After a test with www.speedguide.net, hich gave erratic results, I added rule below to change to lowest value that speedguide gave:
add action=change-mss chain=forward new-mss=1414 protocol=tcp tcp-flags=syn
With this rule it works much better, but still not like it was/should be. If I disable this rule, it is much worse but even with it I still get timeouts and have to refresh to get a page loaded.

Any advice on MTU/MSS issues regarding my mixed-adsl setup or possible problems in my config below would be greatly appreciated.
Thanks in advance
Ekkas


/ip address
add address=10.1.1.4/24 interface=TRUE
add address=000.36.9.146/29 interface=WAN1
add address=000.212.64.187/29 interface=WAN2
add address=000.34.167.131/29 interface=WAN3
add address=000.212.66.179/29 interface=WAN4

/ip firewall mangle
add action=change-mss chain=forward new-mss=1414 protocol=tcp tcp-flags=syn
#rule to send out to mail server
add action=mark-connection chain=prerouting in-interface=TRUE connection-state=new dst-address=000.36.9.144/29 new-connection-mark=Cwan1 passthrough=yes comment="MAIL SERVER"
add action=mark-routing chain=prerouting in-interface=TRUE connection-mark=Cwan1 new-routing-mark=Rwan1 passthrough=no comment="MAIL ROUTE"

#mark all https and a dst-address-list traffic to go out 1 WAN
add action=mark-connection chain=prerouting connection-state=new dst-port=443,2222,8443,5192 in-interface=TRUE new-connection-mark=Cwan6 passthrough=yes protocol=tcp comment="CMark https+other ports"
add action=mark-connection chain=prerouting connection-state=new dst-address-list=LFOUR in-interface=TRUE new-connection-mark=Cwan6 passthrough=yes comment="CMark addr.list"
add action=mark-routing chain=prerouting connection-mark=Cwan6 in-interface=TRUE new-routing-mark=Rwan6 passthrough=no comment="RM443 *WAN5"

#mark all incoming from TRUE with Nth
add action=mark-connection chain=prerouting comment=CM1 connection-state=new in-interface=TRUE new-connection-mark=Cwan1 nth=6,1 passthrough=yes
add action=mark-routing chain=prerouting comment=RM1 connection-mark=Cwan1 in-interface=TRUE new-routing-mark=Rwan1 passthrough=no
add action=mark-connection chain=prerouting comment=CM2 connection-state=new in-interface=TRUE new-connection-mark=Cwan2 nth=5,1 passthrough=yes
add action=mark-routing chain=prerouting comment=RM2 connection-mark=Cwan2 in-interface=TRUE new-routing-mark=Rwan2 passthrough=no
add action=mark-connection chain=prerouting comment=CM3 connection-state=new in-interface=TRUE new-connection-mark=Cwan3 nth=4,1 passthrough=yes
add action=mark-routing chain=prerouting comment=RM3 connection-mark=Cwan3 in-interface=TRUE new-routing-mark=Rwan3 passthrough=no
add action=mark-connection chain=prerouting comment=CM4 connection-state=new in-interface=TRUE new-connection-mark=Cwan4 nth=3,1 passthrough=yes
add action=mark-routing chain=prerouting comment=RM4 connection-mark=Cwan4 in-interface=TRUE new-routing-mark=Rwan4 passthrough=no
add action=mark-connection chain=prerouting comment=CM5 connection-state=new in-interface=TRUE new-connection-mark=Cwan5 nth=2,1 passthrough=yes
add action=mark-routing chain=prerouting comment=RM5 connection-mark=Cwan5 in-interface=TRUE new-routing-mark=Rwan5 passthrough=no
add action=mark-connection chain=prerouting comment=CM6 connection-state=new in-interface=TRUE new-connection-mark=Cwan6 passthrough=yes
add action=mark-routing chain=prerouting comment=RM6 connection-mark=Cwan6 in-interface=TRUE new-routing-mark=Rwan6 passthrough=no

/ip route
add comment=WAN1 dst-address=0.0.0.0/0 gateway=000.36.9.145 routing-mark=Rwan1
add comment=WAN2 dst-address=0.0.0.0/0 gateway=000.212.64.185 routing-mark=Rwan2
add comment=WAN3 dst-address=0.0.0.0/0 gateway=000.34.167.129 routing-mark=Rwan3
add comment=WAN4 dst-address=0.0.0.0/0 gateway=000.212.66.177 routing-mark=Rwan4
add comment=WAN5 dst-address=0.0.0.0/0 gateway=ADSL5 routing-mark=Rwan5
add comment=WAN6 dst-address=0.0.0.0/0 gateway=ADSL6 routing-mark=Rwan6
add comment=ALL dst-address=0.0.0.0/0 gateway=000.34.167.129 distance=2

#static route to network (TRUE interface)
add dst-address=10.0.0.0/8 gateway=10.1.1.254 comment="Wireless network"
add dst-address=192.168.168.0/24 gateway=10.1.1.254 comment="LAN"

/ip firewall nat
add action=src-nat chain=srcnat comment=WAN1 connection-mark=Cwan1 to-addresses=000.36.9.146
add action=src-nat chain=srcnat comment=WAN2 connection-mark=Cwan2 to-addresses=000.212.64.187
add action=src-nat chain=srcnat comment=WAN3 connection-mark=Cwan3 to-addresses=000.34.167.131
add action=src-nat chain=srcnat comment=WAN4 connection-mark=Cwan4 to-addresses=000.212.66.179
add action=masquerade chain=srcnat comment=ADSL5 out-interface=ADSL5
add action=masquerade chain=srcnat comment=ADSL6 out-interface=ADSL6


/ip address
add address=10.1.1.4/24 interface=TRUE
add address=000.36.9.146/29 interface=WAN1
add address=000.212.64.187/29 interface=WAN2
add address=000.34.167.131/29 interface=WAN3
add address=000.212.66.179/29 interface=WAN4

/ip firewall mangle
add action=change-mss chain=forward new-mss=1414 protocol=tcp tcp-flags=syn
#rule to send out to mail server
add action=mark-connection chain=prerouting in-interface=TRUE connection-state=new dst-address=000.36.9.144/29 new-connection-mark=Cwan1 passthrough=yes comment="MAIL SERVER"
add action=mark-routing chain=prerouting in-interface=TRUE connection-mark=Cwan1 new-routing-mark=Rwan1 passthrough=no comment="MAIL ROUTE"

#mark all https and a dst-address-list traffic to go out 1 WAN
add action=mark-connection chain=prerouting connection-state=new dst-port=443,2222,8443,5192 in-interface=TRUE new-connection-mark=Cwan6 passthrough=yes protocol=tcp comment="CMark https+other ports"
add action=mark-connection chain=prerouting connection-state=new dst-address-list=LFOUR in-interface=TRUE new-connection-mark=Cwan6 passthrough=yes comment="CMark addr.list"
add action=mark-routing chain=prerouting connection-mark=Cwan6 in-interface=TRUE new-routing-mark=Rwan6 passthrough=no comment="RM443 *WAN5"

#mark all incoming from TRUE with Nth
add action=mark-connection chain=prerouting comment=CM1 connection-state=new in-interface=TRUE new-connection-mark=Cwan1 nth=6,1 passthrough=yes
add action=mark-routing chain=prerouting comment=RM1 connection-mark=Cwan1 in-interface=TRUE new-routing-mark=Rwan1 passthrough=no
add action=mark-connection chain=prerouting comment=CM2 connection-state=new in-interface=TRUE new-connection-mark=Cwan2 nth=5,1 passthrough=yes
add action=mark-routing chain=prerouting comment=RM2 connection-mark=Cwan2 in-interface=TRUE new-routing-mark=Rwan2 passthrough=no
add action=mark-connection chain=prerouting comment=CM3 connection-state=new in-interface=TRUE new-connection-mark=Cwan3 nth=4,1 passthrough=yes
add action=mark-routing chain=prerouting comment=RM3 connection-mark=Cwan3 in-interface=TRUE new-routing-mark=Rwan3 passthrough=no
add action=mark-connection chain=prerouting comment=CM4 connection-state=new in-interface=TRUE new-connection-mark=Cwan4 nth=3,1 passthrough=yes
add action=mark-routing chain=prerouting comment=RM4 connection-mark=Cwan4 in-interface=TRUE new-routing-mark=Rwan4 passthrough=no
add action=mark-connection chain=prerouting comment=CM5 connection-state=new in-interface=TRUE new-connection-mark=Cwan5 nth=2,1 passthrough=yes
add action=mark-routing chain=prerouting comment=RM5 connection-mark=Cwan5 in-interface=TRUE new-routing-mark=Rwan5 passthrough=no
add action=mark-connection chain=prerouting comment=CM6 connection-state=new in-interface=TRUE new-connection-mark=Cwan6 passthrough=yes
add action=mark-routing chain=prerouting comment=RM6 connection-mark=Cwan6 in-interface=TRUE new-routing-mark=Rwan6 passthrough=no

/ip route
add comment=WAN1 dst-address=0.0.0.0/0 gateway=000.36.9.145 routing-mark=Rwan1
add comment=WAN2 dst-address=0.0.0.0/0 gateway=000.212.64.185 routing-mark=Rwan2
add comment=WAN3 dst-address=0.0.0.0/0 gateway=000.34.167.129 routing-mark=Rwan3
add comment=WAN4 dst-address=0.0.0.0/0 gateway=000.212.66.177 routing-mark=Rwan4
add comment=WAN5 dst-address=0.0.0.0/0 gateway=ADSL5 routing-mark=Rwan5
add comment=WAN6 dst-address=0.0.0.0/0 gateway=ADSL6 routing-mark=Rwan6
add comment=ALL dst-address=0.0.0.0/0 gateway=000.34.167.129 distance=2

#static route to network (TRUE interface)
add dst-address=10.0.0.0/8 gateway=10.1.1.254 comment="Wireless network"
add dst-address=192.168.168.0/24 gateway=10.1.1.254 comment="LAN"

/ip firewall nat
add action=src-nat chain=srcnat comment=WAN1 connection-mark=Cwan1 to-addresses=000.36.9.146
add action=src-nat chain=srcnat comment=WAN2 connection-mark=Cwan2 to-addresses=000.212.64.187
add action=src-nat chain=srcnat comment=WAN3 connection-mark=Cwan3 to-addresses=000.34.167.131
add action=src-nat chain=srcnat comment=WAN4 connection-mark=Cwan4 to-addresses=000.212.66.179
add action=masquerade chain=srcnat comment=ADSL5 out-interface=ADSL5
add action=masquerade chain=srcnat comment=ADSL6 out-interface=ADSL6

Hey Ekkas,

I'm now trying to do Local vs International traffic splitting using route markings, but its causing quite a bit of trouble on the return path, since the LAN routes aren't in the Local/International routing tables. I've then added a routing policy for all the LAN destinations to explicitly use the main routing table. I figured that out after enable logging on the mangle entries, which indicated that the ICMP packets were sent & received, but never routed back to my LAN devices.

Originally the traffic splitting script added like 600+ routes, which can take a while to add/remove/disable/enable, which is why I want to replace that with address lists instead. If anyone has a better solution, I would be grateful!

Anyhow, I think you've made a mistake with the following line:

add action=mark-connection chain=prerouting comment=CM6 connection-state=new in-interface=TRUE new-connection-mark=Cwan6 passthrough=yes

, since you left out the "new-connection-mark=Cwan6 nth=1,1" -> resulting in: In ROS v4.x, the nth="Every,Counter,Packet" unlike with v3.x which is "Every,Packet"

Then you may have to do the same as I did, by explicitly using the main routing table for your LAN routes,

add dst-address=10.0.0.0/8 gateway=10.1.1.254 comment="Wireless network"
add dst-address=192.168.168.0/24 gateway=10.1.1.254 comment="LAN"

, like follow:

I'm not sure why it would cause problems if you specify in-interface=LAN on the marking, as only outgoing packets will be marked. Once the connection is marked, return packets should be routed to your LAN via default route. Or do you perhaps forward public IPs to your LAN? (i.o.w not using NAT). I now have action=accept rules before the marking start for packets destined to LAN as to prevent them from marking and thus using default routes.

Then you may have to do the same as I did, by explicitly using the main routing table for your LAN routes,

Just by having default routes, it works ok for me.

Anyhow, I think you've made a mistake with the following line:

The logic was that all the rest of the packets will be marked with CM6 as it was the last in the list of markings, but counters get reset once it reach an 'every' packet rule, so I did add the '1,1', however it seems to work ok even without it.

I'm not sure if I understand your problem correctly, or were you just trying to point out possible errors in my config?

Ekkas

I was just trying to find possible errors, since the one was maybe related to my problem.

* EDIT:
Ai, I can't believe I was so stupid to exclude the in-interface with the route marking mangle rule, causing the return packets to be marked too!

Btw, your initial post has your config posted twice.
Also, does all your WAN connections have the same latency/bandwidth, since you're assigning the same "Every" property to them all?

I have 3X1Mbps and 3X4Mbps lines, but assign same priority to all, if a client connection hits a 4Mbps, good for him. 2 of my 4Mbps lines are paid per-GB (not uncapped) so I see no need to assign extra traffic to the 4Mbps lines(and pay a lot more), as here in SA, during office hours, I rarely get 4Mbps in any case. The new IS 4Mbps uncapped lines seems nice, as it's not port shaped, just port-prioritised, so I'll probably change all my lines to 4Mbps soon.

Ai, I can't believe I was so stupid to exclude the in-interface with the route marking mangle rule, causing the return packets to be marked too!

AHA!

Just one more note, if you use prerouting chain and you use NAT, then your LAN IPs wouldn't show yet (in rules) as dst-nat is only done after prerouting mangle, if you need to mark incoming packets based on LAN IPs, then rather put the rules in forward chain.

Ekkas