Load balancing - One static IP sometimes is unreachable

eXtremer · July 7, 2011, 8:35am

Hi all.

I’m using 2 providers with RB750G (load balancing + routing failover), since I’ve changed one provide a week ago I’m having sporadic problems reaching one of the static IP, I’m pinging the IP everything is all right after 5 minutes can’t reach it any more, then I can reach it again after some time, really can’t understand what’s going on, this problem happens only with the old provider, the new one is reachable all the time don’t have any issues with it, I’m 100% sure there is nothing wrong with the provider, something is not right with the config. or I don’t know…
The problem is with: 77.89.XXX.142, I must say there was no changes in configuration, just added the new (188.23X.XX.115) provider, that’s it.
Need some help, thank you.

[admin@MikroTik] > /ip route print detail        
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme,
B - blackhole, U - unreachable, P - prohibit
0 A S  dst-address=0.0.0.0/0 gateway=188.23X.XX.1 gateway-status=188.23X.XX.1 reachable MTC check-gateway=ping distance=1 scope=30 target-scope=10
        routing-mark=MTC

1 A S  dst-address=0.0.0.0/0 gateway=77.89.XXX.141 gateway-status=77.89.XXX.141 reachable Orange check-gateway=ping distance=1 scope=30 target-scope=10
        routing-mark=Orange

2 A S  dst-address=0.0.0.0/0 gateway=77.89.XXX.141,188.23X.XX.1 gateway-status=77.89.XXX.141 reachable WAN2,188.23X.XX.1 reachable MTC 
        check-gateway=ping distance=1 scope=30 target-scope=10 

3 A S  dst-address=77.89.XXX.0/24 gateway=77.89.XXX.141 gateway-status=77.89.XXX.141 reachable Orange check-gateway=ping distance=1 scope=30
        target-scope=10

4 ADC  dst-address=77.89.XXX.140/30 pref-src=77.89.XXX.142 gateway=Orange gateway-status=Orange reachable distance=0 scope=10

5 ADC  dst-address=188.23X.XX.0/24 pref-src=188.23X.XX.105 gateway=MTC gateway-status=MTC reachable distance=0 scope=10

6 A S  dst-address=188.237.77.105/32 gateway=188.23X.XX.1 gateway-status=188.23X.XX.1 reachable MTC check-gateway=ping distance=1 scope=30
        target-scope=10

7 ADC  dst-address=192.168.0.0/24 pref-src=192.168.0.1 gateway=LAN1 gateway-status=LAN1 reachable distance=0 scope=10
[admin@MikroTik] >

[admin@MikroTik] > /ip route nexthop print detail
0 address=77.89.XXX.141 gw-state=reachable scope=10 check-gateway=icmp gw-check-ok=yes

1 address=188.23X.XX.1 gw-state=reachable scope=10 check-gateway=icmp gw-check-ok=yes
[admin@MikroTik] >

Juwei · July 7, 2011, 10:54am

Can you please post the mangle (or full) configuration for the both lines?
Maybe the answer paket is not sending thru the line where the packen got received?

When used two lines with PCC i initially got the same problem, fixed this after adding some rules to ensure the paket is leaving the correct line.

eXtremer · July 7, 2011, 11:13am

There are no mangle rules, I’m using ECMP not PCC.

Juwei · July 7, 2011, 11:21am

So how do you tell the router to send the answer (outgoing) thru the same line as they came in?

Juwei · July 7, 2011, 11:24am

Please see this link:
http://wiki.mikrotik.com/wiki/ECMP_load_balancing_with_masquerade

Connections to the router itself

With all multi-gateway situations there is a usual problem to reach router from public network via one, other or both gateways. Explanations is very simple - Outgoing packets uses same routing decision as packets that are going trough the router. So reply to a packet that was received via wlan1 might be send out and masqueraded via wlan2.

To avoid that we need to policy routing those connections.

eXtremer · July 7, 2011, 12:16pm

Added those lines..

/ ip firewall mangle
add chain=input in-interface=wlan1 action=mark-connection new-connection-mark=wlan1_conn
add chain=input in-interface=wlan2 action=mark-connection new-connection-mark=wlan2_conn
add chain=output connection-mark=wlan1_conn action=mark-routing new-routing-mark=to_wlan1
add chain=output connection-mark=wlan1_conn action=mark-routing new-routing-mark=to_wlan2

Till the day I changed one provider everything was working just fine without this rules.
Will see if this will solve the issue…
10x.

eXtremer · July 10, 2011, 6:47am

Nothing changed, still have this issue…
I’ve made a rule to check every 20 seconds if port 25 is unreachable (I have a mail server behind mikrotik) it’s not such a big probelm because mail are keep on coming through the other IP but this situation needs to be solved, can’t understand why it’s happening like this…

Needs some advice, thank you.

Juwei · July 10, 2011, 8:11am

Please post your config.

eXtremer · July 10, 2011, 8:47am

Once again I must say that I’m having issues only with Orange interface (77.89.XXX.142), with the second Provider (MTC) everything is all right, allways reachable.
Another thing, after adding the Mangle rules, the IP now is always reachable (ping), but when trying to telnet IP:PORT, sometimes there is a reply, sometimes not, and what’s interesting I can always login with Winbox with IP that has the problem.
Thank you…

[admin@MikroTik] > ip export
# jul/10/2011 11:22:48 by RouterOS 4.11
#
/ip hotspot profile
set default dns-name="" hotspot-address=0.0.0.0 html-directory=hotspot http-cookie-lifetime=3d http-proxy=0.0.0.0:0 login-by=cookie,http-chap name=\
    default rate-limit="" smtp-server=0.0.0.0 split-user-domain=no use-radius=no

/ip hotspot user profile
set default idle-timeout=none keepalive-timeout=2m name=default shared-users=1 status-autorefresh=1m transparent-proxy=no

/ip ipsec proposal
set default auth-algorithms=sha1 comment="" disabled=no enc-algorithms=3des lifetime=30m name=default pfs-group=modp1024

/ip pool
add name=dhcp_pool1 ranges=192.168.0.3-192.168.0.199

/ip dhcp-server
add add-arp=yes address-pool=dhcp_pool1 authoritative=after-2sec-delay bootp-support=static disabled=no interface=LAN1 lease-time=12h name=dhcp1

/ip accounting
set account-local-traffic=no enabled=yes threshold=256

/ip accounting web-access
set accessible-via-web=yes address=0.0.0.0/0

/ip address
add address=192.168.0.1/24 broadcast=192.168.0.255 comment="" disabled=no interface=LAN1 network=192.168.0.0
add address=77.89.XXX.142/30 broadcast=77.89.XXX.143 comment="" disabled=no interface=Orange network=77.89.XXX.140

/ip dhcp-client
add add-default-route=yes comment="" default-route-distance=0 disabled=no interface=MTC use-peer-dns=yes use-peer-ntp=yes

/ip dhcp-server config
set store-leases-disk=5m

/ip dhcp-server network
add address=192.168.0.0/24 comment="" gateway=192.168.0.1 netmask=24

/ip dns
set allow-remote-requests=yes cache-max-ttl=1w cache-size=2048KiB max-udp-packet-size=512 servers=212.56.19X.XX,212.56.19X.XX


/ip firewall connection tracking
set enabled=yes generic-timeout=10m icmp-timeout=10s tcp-close-timeout=10s tcp-close-wait-timeout=10s tcp-established-timeout=1d tcp-fin-wait-timeout=\
    10s tcp-last-ack-timeout=10s tcp-syn-received-timeout=5s tcp-syn-sent-timeout=5s tcp-syncookie=no tcp-time-wait-timeout=10s udp-stream-timeout=3m \
    udp-timeout=10s

/ip firewall filter
nreachable
add action=drop chain=input comment="" disabled=no dst-port=8080 in-interface=Orange protocol=tcp src-address=0.0.0.0/0
add action=drop chain=input comment="" disabled=no dst-port=8080 in-interface=MTC protocol=tcp src-address=0.0.0.0/0
add action=accept chain=input comment="" disabled=no protocol=icmp
add action=accept chain=input comment="" disabled=no dst-port=8282 protocol=tcp src-address=192.168.0.0/24
add action=accept chain=input comment="" disabled=no dst-port=8282 protocol=tcp src-address=92.115.XXX.XXX
add action=accept chain=input comment="Added by webbox" connection-state=established disabled=no in-interface=Orange
add action=accept chain=input comment="Added by webbox" connection-state=established disabled=no in-interface=MTC
add action=accept chain=input comment="Added by webbox" connection-state=related disabled=no in-interface=Orange
add action=accept chain=input comment="Added by webbox" connection-state=related disabled=no in-interface=MTC
add action=drop chain=input comment="Added by webbox" disabled=no in-interface=Orange
add action=drop chain=input comment="Added by webbox" disabled=no in-interface=MTC
add action=jump chain=forward comment="Added by webbox" disabled=no in-interface=Orange jump-target=customer
add action=jump chain=forward comment="Added by webbox" disabled=no in-interface=MTC jump-target=customer
add action=accept chain=customer comment="Added by webbox" connection-state=established disabled=no
add action=accept chain=customer comment="Added by webbox" connection-state=related disabled=no
add action=drop chain=input comment="" disabled=no dst-port=8282 protocol=tcp

/ip firewall mangle
add action=mark-connection chain=input comment="" disabled=no in-interface=Orange new-connection-mark=Orange passthrough=yes
add action=mark-connection chain=input comment="" disabled=no in-interface=MTC new-connection-mark=MTC passthrough=yes
add action=mark-routing chain=output comment="" connection-mark=Orange disabled=no new-routing-mark=Orange passthrough=yes
add action=mark-routing chain=output comment="" connection-mark=MTC disabled=no new-routing-mark=MTC passthrough=yes

/ip firewall nat
add action=dst-nat chain=dstnat comment="" disabled=no dst-port=2222 protocol=tcp to-addresses=192.168.0.187 to-ports=2222
add action=dst-nat chain=dstnat comment="" disabled=no dst-port=1111 protocol=tcp to-addresses=192.168.0.250 to-ports=1111
add action=dst-nat chain=dstnat comment="" disabled=no dst-port=20633 protocol=udp to-addresses=192.168.0.18 to-ports=20633
add action=dst-nat chain=dstnat comment="" disabled=no dst-port=20633 protocol=tcp to-addresses=192.168.0.18 to-ports=20633
add action=dst-nat chain=dstnat comment="" disabled=no dst-address-type=local dst-port=25 protocol=tcp to-addresses=192.168.0.6 to-ports=25
add action=dst-nat chain=dstnat comment="" disabled=no dst-address-type=local dst-port=143 protocol=tcp to-addresses=192.168.0.6 to-ports=143
add action=dst-nat chain=dstnat comment="" disabled=no dst-address-type=local dst-port=465 protocol=tcp to-addresses=192.168.0.6 to-ports=465
add action=dst-nat chain=dstnat comment="" disabled=no dst-address-type=local dst-port=993 protocol=tcp to-addresses=192.168.0.6 to-ports=993
add action=dst-nat chain=dstnat comment="" disabled=no dst-address-type=local dst-port=995 protocol=tcp to-addresses=192.168.0.6 to-ports=995
add action=masquerade chain=srcnat comment="Added by webbox" disabled=no out-interface=Orange
add action=masquerade chain=srcnat comment="" disabled=no out-interface=MTC
add action=dst-nat chain=dstnat comment="" disabled=no dst-address=188.237.XX.105 dst-port=443 protocol=tcp to-addresses=192.168.0.2 to-ports=443
add action=dst-nat chain=dstnat comment="" disabled=no dst-address=77.89.XXX.142 dst-port=443 protocol=tcp to-addresses=192.168.0.2 to-ports=443
add action=dst-nat chain=dstnat comment="" disabled=no dst-port=81 protocol=tcp to-addresses=192.168.0.2 to-ports=81
add action=dst-nat chain=dstnat comment="" disabled=no dst-address=188.237.XX.105 dst-port=80 protocol=tcp to-addresses=192.168.0.2 to-ports=81
add action=dst-nat chain=dstnat comment="" disabled=no dst-address=77.89.XXX.142 dst-port=80 protocol=tcp to-addresses=192.168.0.2 to-ports=81
add action=dst-nat chain=dstnat comment="" disabled=no dst-port=80 protocol=tcp to-addresses=192.168.0.1 to-ports=8080
add action=masquerade chain=srcnat comment="" disabled=no dst-address=192.168.0.6 out-interface=LAN1 src-address=192.168.0.0/24

/ip firewall service-port
set ftp disabled=yes ports=21
set tftp disabled=yes ports=69
set irc disabled=yes ports=6667
set h323 disabled=yes
set sip disabled=yes ports=5060,5061
set pptp disabled=yes

/ip hotspot service-port
set ftp disabled=no ports=21

/ip neighbor discovery
set MTC discover=no
set Orange discover=no
set LAN1 discover=no
set LAN2 discover=no
set LAN3 discover=no
/
ip proxy
set always-from-cache=no cache-administrator=Administrator cache-hit-dscp=4 cache-on-disk=no enabled=yes max-cache-size=none max-client-connections=600 \
    max-fresh-time=3d max-server-connections=600 parent-proxy=0.0.0.0 parent-proxy-port=0 port=8080 serialize-connections=no src-address=0.0.0.0

/ip route
add check-gateway=ping comment="" disabled=no distance=1 dst-address=0.0.0.0/0 gateway=188.237.XX.1 routing-mark=MTC scope=30 target-scope=10
add check-gateway=ping comment="" disabled=no distance=1 dst-address=0.0.0.0/0 gateway=77.89.XXX.141 routing-mark=Orange scope=30 target-scope=10
add check-gateway=ping comment="" disabled=no distance=1 dst-address=0.0.0.0/0 gateway=77.89.XXX.141,188.237.XX.1 scope=30 target-scope=10
add check-gateway=ping comment="" disabled=no distance=1 dst-address=77.89.XXXX.0/24 gateway=77.89.XXX.141 scope=30 target-scope=10
add check-gateway=ping comment="" disabled=no distance=1 dst-address=188.237.XX.105/32 gateway=188.237.XX.1 scope=30 target-scope=10

/ip service
set telnet address=0.0.0.0/0 disabled=yes port=23
set ftp address=192.168.0.0/24 disabled=yes port=21
set www address=192.168.0.0/24 disabled=no port=3030
set ssh address=0.0.0.0/0 disabled=yes port=22
set www-ssl address=0.0.0.0/0 certificate=none disabled=yes port=443
set api address=0.0.0.0/0 disabled=yes port=8728
set winbox address=0.0.0.0/0 disabled=no port=8282

/ip socks
set connection-idle-timeout=2m enabled=no max-connections=200 port=1080

/ip traffic-flow
set active-flow-timeout=30m cache-entries=4k enabled=no inactive-flow-timeout=15s interfaces=Orange

/ip upnp
set allow-disable-external-interface=no enabled=yes show-dummy-rule=yes

/ip upnp interfaces
add disabled=yes interface=MTC type=internal
add disabled=yes interface=Orange type=external
add disabled=yes interface=LAN1 type=internal
add disabled=yes interface=LAN2 type=internal
add disabled=yes interface=LAN3 type=internal
add disabled=yes type=external
[admin@MikroTik] >

[admin@MikroTik] > /ip route print detail        
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme,
B - blackhole, U - unreachable, P - prohibit
0 A S  dst-address=0.0.0.0/0 gateway=188.23X.XX.1 gateway-status=188.23X.XX.1 reachable MTC check-gateway=ping distance=1 scope=30 target-scope=10
        routing-mark=MTC

1 A S  dst-address=0.0.0.0/0 gateway=77.89.XXX.141 gateway-status=77.89.XXX.141 reachable Orange check-gateway=ping distance=1 scope=30 target-scope=10
        routing-mark=Orange

2 A S  dst-address=0.0.0.0/0 gateway=77.89.XXX.141,188.23X.XX.1 gateway-status=77.89.XXX.141 reachable WAN2,188.23X.XX.1 reachable MTC 
        check-gateway=ping distance=1 scope=30 target-scope=10 

3 A S  dst-address=77.89.XXX.0/24 gateway=77.89.XXX.141 gateway-status=77.89.XXX.141 reachable Orange check-gateway=ping distance=1 scope=30
        target-scope=10

4 ADC  dst-address=77.89.XXX.140/30 pref-src=77.89.XXX.142 gateway=Orange gateway-status=Orange reachable distance=0 scope=10

5 ADC  dst-address=188.23X.XX.0/24 pref-src=188.23X.XX.105 gateway=MTC gateway-status=MTC reachable distance=0 scope=10

6 A S  dst-address=188.237.77.105/32 gateway=188.23X.XX.1 gateway-status=188.23X.XX.1 reachable MTC check-gateway=ping distance=1 scope=30
        target-scope=10

7 ADC  dst-address=192.168.0.0/24 pref-src=192.168.0.1 gateway=LAN1 gateway-status=LAN1 reachable distance=0 scope=10
[admin@MikroTik] >

[admin@MikroTik] > /ip route nexthop print detail
0 address=77.89.XXX.141 gw-state=reachable scope=10 check-gateway=icmp gw-check-ok=yes

1 address=188.23X.XX.1 gw-state=reachable scope=10 check-gateway=icmp gw-check-ok=yes
[admin@MikroTik] >

eXtremer · July 10, 2011, 9:09am

Just checked, NO connection to port 25, but I can connect to the router with Winbox - same IP.

Juwei · July 10, 2011, 9:10am

Try adding this mangle rules:

/ip firewall mangle
add action=mark-connection chain=prerouting comment="" disabled=no dst-address-type=!local in-interface=Orange new-connection-mark=Orange passthrough=yes
add action=mark-connection chain=prerouting comment="" disabled=no dst-address-type=!local in-interface=MTC new-connection-mark=MTC passthrough=yes
add action=mark-routing chain=prerouting comment="" connection-mark=Orange disabled=no in-interface=LAN1 new-routing-mark=Orange passthrough=yes
add action=mark-routing chain=prerouting comment="" connection-mark=MTC disabled=no in-interface=LAN1 new-routing-mark=MTC passthrough=yes

Juwei · July 10, 2011, 9:20am

Hmm, forgot to add “connection-mark=no-mark” to the incoming mangle (so it doesn’t mark replying packets from outgoing connections).
Also “dst-address-type=!local” should not be necessary for this prerouting. This ist just an example, try this and test if this is working for you
Correct:

/ip firewall mangle
add action=mark-connection chain=prerouting connection-mark=no-mark comment="" disabled=no in-interface=Orange new-connection-mark=Orange passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark comment="" disabled=no in-interface=MTC new-connection-mark=MTC passthrough=yes
add action=mark-routing chain=prerouting comment="" connection-mark=Orange disabled=no in-interface=LAN1 new-routing-mark=Orange passthrough=yes
add action=mark-routing chain=prerouting comment="" connection-mark=MTC disabled=no in-interface=LAN1 new-routing-mark=MTC passthrough=yes

eXtremer · July 10, 2011, 11:28am

Juwei:

Hmm, forgot to add “connection-mark=no-mark” to the incoming mangle (so it doesn’t mark replying packets from outgoing connections).
Also “dst-address-type=!local” should not be necessary for this prerouting. This ist just an example, try this and test if this is working for you
Correct:
/ip firewall mangle
add action=mark-connection chain=prerouting connection-mark=no-mark comment="" disabled=no in-interface=Orange new-connection-mark=Orange passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark comment="" disabled=no in-interface=MTC new-connection-mark=MTC passthrough=yes
add action=mark-routing chain=prerouting comment="" connection-mark=Orange disabled=no in-interface=LAN1 new-routing-mark=Orange passthrough=yes
add action=mark-routing chain=prerouting comment="" connection-mark=MTC disabled=no in-interface=LAN1 new-routing-mark=MTC passthrough=yes

Added those lines, till now all good, I will enable the “25 port alert” too see if the port is always reachable.
Thank you Juwei and have a nice day.

Another thing, how to check an IP for example 8.8.8.8 and not the gateway for a route and another ? becase sometimes the gateway is reachable but there is no internet and the routing failover is not working as it should.
Thank you in advance.

peterd · July 11, 2011, 7:48am

Hi,

Check the post below, should help you.

http://forum.mikrotik.com/t/if-one-route-stopped/48310/1

PJD

eXtremer · July 11, 2011, 7:56am

Thanks, but I don’t want script, I know there is a easier workaround, in my case both IP’s are static, no modems or whatsoever.

sophal · July 11, 2011, 8:59am

you try to change the 1st upstream don’t change distance=1, 2nd upstream change distance=2

example:
/ip route
add check-gateway=ping comment=“1st upstream” disabled=no distance=1 dst-address=0.0.0.0/0 gateway=x.x.x.x/x scope=30 target-scope=10

add check-gateway=ping comment=“2nd upstream” disabled=no distance=2 dst-address=0.0.0.0/0 gateway=x.x.x.x/x scope=30 target-scope=10

eXtremer · July 11, 2011, 9:12am

Sorry guys, let me explain once again, it’s not about the distance…So, it says that the gateway is reachable, although I don’t have internet because my Provider doesn’t block the Port, I can navigate to local websites (intranet) but I don’t have access to external websites (internet), so I need not to ping my gateway but an external IP, for example 8.8.8.8 so if it s not reachable that route will not be used but only the good one.

peterd · July 11, 2011, 11:05am

My script does exactly that, the ips on the top 192.168.2.1 and 3.1 could be replaced with your static or something like 8.8.8.8, thats on Internet then just adapt the routes so its checking via each line, and disables the one with no internet access.

Simple.

Btw.

I don’t think there is another way of doing it, If someone knows please share it.

Regards,

PJD

eXtremer · July 11, 2011, 11:58am

What else should be change in the script in my case ? What should I put instead of “ether1_traffic” ? Something else ?

:global ping1 [/ping 8.8.8.8 count=3 routing-table=ether1_traffic]
:global ping2 [/ping 8.8.8.8 count=3 routing-table=ether2_traffic]
:global gw1 [/ip route get number=0  gateway-status]
:global gw2 [/ip route get number=1  gateway-status]

:if (($ping1=0) && ($ping2=3) && ($gw1="77.89.XXX.141 reachable ether1")) do={/ip route disable numbers=[find gateway=77.89.XXX.141];  [:log info  "disabling route 141"];}
:if (($ping1=3) && ($ping2=3) && ($gw1!="77.89.XXX.141 reachable ether1")) do={/ip route enable numbers=[find gateway=77.89.XXX.141]; [:log info  "enabling route 141"];}

:if (($ping1=3) && ($ping2=0) && ($gw2="188.237.XX.1 reachable ether2")) do={/ip route disable numbers=[find gateway=188.237.XX.1];  [:log info  "disabling route 1"];}
:if (($ping1=3) && ($ping2=3) && ($gw2!="188.237.XX.1 reachable ether2")) do={/ip route enable numbers=[find gateway=188.237.XX.1]; [:log info  "enabling route 1"];}

:if (($ping1=0) && ($ping2=0)) do={[/ip route enable numbers=[find gateway=77.89.XXX.141]]; [/ip route enable numbers=[find gateway=188.237.XX.1]]; [:log info  "fail safe enabling 141 and 1"];}

peterd · July 11, 2011, 12:17pm

Hi,

the routing table bit you can change to interface=ether1 and 2 accordingly that should work, cause you want to ping out on a certain interface to check if it has internet conectivity.

:global ping1 [/ping 8.8.8.8 count=3 interface=ether1]
:global ping2 [/ping 8.8.8.8 count=3 interface=ether2]

Although I haven’t tested it myself, please report on progress.

PJD