Load Balancing problem and openvpn client , server

samuellsk · June 18, 2015, 8:32am

Hellou,

I m not used to asking for help. Usually i google the answer myself but not today. Today my google-fu (art of using google) failed me horribly. So heres the usual setup:

store:
1 local network (duuuh)
2 isp providers
1 openvpn-client

not on mikrotik and nowhere near:
central:
another local network,
ovpn-server

This setup with load balancer is working fine on router os 6.18 , after the upgrade on newer version it does not , i was not paying attention much to the problem as i was thinking it would be fixed in future releases…

So we have “central” here is local network , servers… our shops all over the country are connectiong through the ovpn-client to our server, and local network.

As i said, everything was working and is (i m still on 6.18).

Now what is working and not working if i update:
working - traffic from central to store mikrotik local address itself (but not store local network, computers, cam recordes…)
not working - traffic from central to store local network (computers, cam recordes…)

so if mikrotik in store has 10.21.25.1 local ip , than everything else 2-254 is not reachable from central

firewall mangle , and routes

/ip firewall mangle
add chain=prerouting dst-address=192.168.1.0/24  action=accept in-interface=ether2
add chain=prerouting dst-address=192.168.0.0/24  action=accept in-interface=ether2
add chain=prerouting in-interface=ether1 connection-mark=no-mark action=mark-connection new-connection-mark=ether1_conn
add chain=prerouting in-interface=ether3 connection-mark=no-mark action=mark-connection new-connection-mark=ether3_conn
add chain=prerouting  in-interface=ether2 connection-mark=no-mark dst-address-type=!local per-connection-classifier=both-addresses:2/0 action=mark-connection new-connection-mark=ether1_conn 
add chain=prerouting  in-interface=ether2 connection-mark=no-mark dst-address-type=!local per-connection-classifier=both-addresses:2/1 action=mark-connection new-connection-mark=ether3_conn
add chain=prerouting connection-mark=ether1_conn in-interface=ether2 action=mark-routing new-routing-mark=to_ether1
add chain=prerouting connection-mark=ether3_conn in-interface=ether2 action=mark-routing new-routing-mark=to_ether3
add chain=output connection-mark=ether1_conn action=mark-routing new-routing-mark=to_ether1     
add chain=output connection-mark=ether3_conn action=mark-routing new-routing-mark=to_ether3

/ip route
add dst-address=0.0.0.0/0 gateway=192.168.1.1 routing-mark=to_ether1 check-gateway=ping
add dst-address=0.0.0.0/0 gateway=192.168.0.1 routing-mark=to_ether3 check-gateway=ping
add dst-address=0.0.0.0/0 gateway=192.168.1.1 distance=1 check-gateway=ping
add dst-address=0.0.0.0/0 gateway=192.168.0.1 distance=2 check-gateway=ping

Can anyone have idea why it stoped working ? Just in case, i tried burn it whole down a build it from scratch using someone elses “load balancers” even the one from that pdf presentation http://mum.mikrotik.com/presentations/US12/tomas.pdf

Yes, yes, routes are good, and as i said, its working on 6.18
2015-06-18 10_25_52-harlock [Running] - Oracle VM VirtualBox _ 2.png
What i m i missing ?

Thank you.

pukkita · June 18, 2015, 9:48am

have you checked in “central” that a route through 10.21.25.1 is created to 10.21.25.0/24?

If it was previously working maybe you were doing nat? (i.e. the mikrotik natted everything coming from central as coming from 10.21.25.1 to store hosts)

samuellsk · June 18, 2015, 10:07am

I know that you are trying to help and i do appreciate that at least someone answered, but this is not a new instalation, nor anything has changed on server aka “central” side. This is working for months (if I use as an example older store, than its working for years, upgrade after upgrade until we hit the 6.19, than i simply done downgrade). And don’t really see the point of checking something on that side, if I upgrade to 6.29 , everything stops, do force downgrade (i m using dude for this), than on 6.18 everything is working as it should.

I know that this is not the answer you was hoping for (neither was I), but you have to understand that checking serv is a dead end.

pukkita · June 18, 2015, 10:22am

Have you checked when upgrading, on the IP > firewall rules,mangle,nat tabs that nothing is marked in red?

There were several changes that made previous allowable settings to become invalid after 6.18; some specific nat settings being one of them, maybe that’s what’s happening when you upgrade.

Or maybe OpenVPN client broke or settings were changed and you need to polish them.

samuellsk · June 18, 2015, 10:32am

There is only one not importat thing in red, which is rule to accept from “other” ovpn-client , its red, becouse that interface is not active (disabled at the moment), it will become active if ovpn-server is going down, so the mikrotik will switch to another one (disabling ovpn-client, and enabling the right one, to connect to another ovpn-server) - so as i said its not important, its a cosmetic “error”

Nat , there are only masquerade for ether1 and ether3 , and 80 port redirection. Than there are few rules for specific domains which are not ours. If client wants something from those domains, they need to ask they dns servers… nothing much of important what could contribute to this problem, and nothing in red.

I suspect the problem is rooted in marking packets, or somewhere around that. If i upgrade to 6.29 , and disable needed rulse for mangle - aka “load balancing” everything is normal , except that load balancer is not working.

edit 1:
at this configuration it works

mangle ( “load balancer only part shown”):

/ip firewall mangle
add action=mark-connection chain=input in-interface=ether1 new-connection-mark=ether1_conn passthrough=no
add action=mark-connection chain=input in-interface=ether3 new-connection-mark=ether3_conn passthrough=no
add action=mark-routing chain=output connection-mark=ether1_conn new-routing-mark=to_ether1 passthrough=no
add action=mark-routing chain=output connection-mark=ether3_conn new-routing-mark=to_ether3 passthrough=no

routes have one diff too (0.0.0.0/0 part only shown).

0 A S dst-address=0.0.0.0/0 gateway=192.168.1.1
gateway-status=192.168.1.1 reachable via ether1 check-gateway=ping
distance=3 scope=30 target-scope=10 routing-mark=to_ether1

1 A S dst-address=0.0.0.0/0 gateway=192.168.0.1
gateway-status=192.168.0.1 reachable via ether3 check-gateway=ping
distance=3 scope=30 target-scope=10 routing-mark=to_ether3

2 A S dst-address=0.0.0.0/0 gateway=192.168.0.1,192.168.1.1
gateway-status=192.168.0.1 reachable via ether3,192.168.1.1 reachable
via ether1
check-gateway=ping distance=2 scope=30 target-scope=10

0.0.0.0/0 without mark have now 2 gateways

edit 2:
I have to go to store and check if i disconnect “isp cable” from “isp device” - so that mikrotik will still have its “link up” without change, if its going to work as it should. Will report later.

pukkita · June 18, 2015, 2:20pm

I’m a little lost, but

/ip firewall mangle
add chain=prerouting dst-address=192.168.1.0/24  action=accept in-interface=ether2
add chain=prerouting dst-address=192.168.0.0/24  action=accept in-interface=ether2
[...]

Makes connections from those networks not to be balanced. Is that your original setup? Or did adding that to a working balancing setup fix the problem with central?

FWIW, Changelog for 6.19:

*) firewall - packet defragmenting will only happen with connection tracking enabled;
*) firewall - optimized option matching order with-in a rule;
*) firewall - rules that require CONNTRACK to work will now have Invalid flag
when CONNTRACK is disabled;
*) firewall - rules that require use-ip-firewall to work will now have invalid flag
when use-ip-firewall is disabled;
*) firewall - rules that have interface with “Slave” flag specified as in-/out-interface
will now have Invalid flag;
*) firewall - rules that have interface without “Slave” flag specified as in-/out-bridge-port
will now have Invalid flag;
*) firewall - rules with Invalid flags will now be auto-commented to explain why;

Have you “torched” the LAN/WAN connections to see what arrives when pinging from central to hosts? If you suspect mangle is hosing central → LAN connectivity checking firewall connections for connection marks / routing marks is unvaluable.

samuellsk · June 19, 2015, 12:08pm

/ip firewall mangle
add chain=prerouting dst-address=192.168.1.0/24  action=accept in-interface=ether2
add chain=prerouting dst-address=192.168.0.0/24  action=accept in-interface=ether2
[...]

Is in original “mangle for load balancing” , and as a part of the rest, it works without a problem at routeros version 6.18.

To changelog: Yes i m reading those , did not seen there anything significant. As its stated in changelog, it would be commented out and why (what is the problem with rule).

No i have not tried torch at those moments.

Setup after yeasterday tests is in fully working order (as i wrote before). For anyone interested , first post is of course from version 6.18 , and now the 6.29.1:

ether1 gw 192.168.1.1
ether3 gw 192.168.0.1
ether2 is local network

mangle rules for loadbalancer:

/ip firewall mangle
add action=mark-connection chain=input in-interface=ether1 new-connection-mark=ether1_conn passthrough=no
add action=mark-connection chain=input in-interface=ether3 new-connection-mark=ether3_conn passthrough=no
add action=mark-routing chain=output connection-mark=ether1_conn new-routing-mark=to_ether1 passthrough=no
add action=mark-routing chain=output connection-mark=ether3_conn new-routing-mark=to_ether3 passthrough=no

routes to 0.0.0.0:

 0 A S  dst-address=0.0.0.0/0 gateway=192.168.1.1 gateway-status=192.168.1.1 reachable via  ether1 
        check-gateway=ping distance=3 scope=30 target-scope=10 routing-mark=to_ether1 

 1 A S  dst-address=0.0.0.0/0 gateway=192.168.0.1 gateway-status=192.168.0.1 reachable via  ether3 
        check-gateway=ping distance=3 scope=30 target-scope=10 routing-mark=to_ether3 

 2 A S  ;;; gateway
        dst-address=0.0.0.0/0 gateway=192.168.0.1,192.168.1.1 
        gateway-status=192.168.0.1 reachable via  ether3,192.168.1.1 reachable via  ether1 check-gateway=ping 
        distance=3 scope=30 target-scope=10

And this is how it all works on 6.29.1 version

pukkita · June 19, 2015, 3:41pm

It is actually load balancing (by ECMP) spreading connections amongst the two WANs with that setup?

Do internal hosts browse https sites fine?

samuellsk · June 20, 2015, 6:12pm

I have googled that ECMP setup , i had no idea that it existed but as i was comparing it to mine, its the same thing. Yes https sites works fine, should they not ?