Community discussions

MUM Europe 2020
 
leon84
Member Candidate
Member Candidate
Topic Author
Posts: 184
Joined: Wed Dec 02, 2009 12:15 pm

RBM33G with 2 LTE module (how to manage failover)

Tue Jul 10, 2018 2:12 pm

Dear all,
I'm using RBM33G with 2 Huawei 909s-120 LTE module. I would to configure a failover internet connection on lte. Could you help me?
I tried this one https://wiki.mikrotik.com/wiki/Advanced ... _Scripting but I didn't help me because with LTE interface I don't have a default gateway specified as ip address but only with name of the interface. In my case lte1 and lte2.
I tried also with static route and different distance but It run correctly only if the lte interface goes down. In many case the interface remain up but I don't have internet connectivity and then the failover doesn't run.
Have you some suggestion?
Thanks in advance.
 
User avatar
krafg
Member
Member
Posts: 336
Joined: Sun Jun 28, 2015 7:36 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Wed Jul 11, 2018 10:42 pm

Play with route distances. 1 for main LTE module and 2 for backup LTE module.

Regards.
If you want to know my network, you can find it here.

Image
 
sindy
Forum Guru
Forum Guru
Posts: 4218
Joined: Mon Dec 04, 2017 9:19 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Wed Jul 11, 2018 11:17 pm

To what I know you cannot do that without scripting. You can use various setups to force pinging of some reference IP addresses on the internet via lte1 and other reference IP addresses via lte2 so that you knew that not only the link is up but you can actually get somewhere, but for gateways other than IP addresses, it is not possible to translate this information into route availability without scripting.

The script might look like this.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
leon84
Member Candidate
Member Candidate
Topic Author
Posts: 184
Joined: Wed Dec 02, 2009 12:15 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Thu Jul 12, 2018 9:55 am

Dear all,
I realized a little script to manage failover but on lte interface is more complicated.
I have a configure 2 routes and some firewall rule to allow ping to 8.8.8.8 only for LTE1 interface and 8.8.4.4 for LTE2 interface.
Then I have 2 default routes, one through LTE1 (with distance 1 because for me is primary link) and another one through LTE2 (with distance of 2).
With a script and the scheduler I check ping to 8.8.8.8 and 8.8.4.4 every 1 minute and if one of them fails I disable relative default route. For example if 8.8.8.8 ping fails I disable default route on LTE1 interface.
Where is the problem? I have to contact from remote my RBM33G and I'm not using SIM that allow bidirectional traffic on their APN. Then my routerboard can generate trafffic to INTERNET but it is not contactable from internet. In the past I used specific APN (more expensive for application machine 2 machine) but now I implemented a VPN with SSTP protocol.
When I switch from LTE1 to LTE2 the sstp client on my rotuerboard disconnect and immediatly create a new connection. Because I use RBM33G for transmit real data stream this disconnections can create some problem. Often disconnection of LTE1 interface can be very short and then the connection swith on LTE2 interface and after few seconds (when LTE1 become up) another one on LTE1. This create 2 sstp client disconnection. In case of using only one sim (in this case) I would have had only one disconnection :-).
Have you some idea?
 
sindy
Forum Guru
Forum Guru
Posts: 4218
Joined: Mon Dec 04, 2017 9:19 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Thu Jul 12, 2018 1:53 pm

Since years the problem of short outages of the primary path and the associated transient effects is solved by a "WTR timer" (Wait to Restore) which is used to delay the swicthover back to the primary path after it becomes OK. This is normally used to avoid falling back on false OKs, but could help a bit to you in terms that the total time of the outage would be shorter.

However, my approach would be to use one SSTP tunnel via each LTE interface and use a failover between these two tunnels for the critical data. It could then detect failures much faster than once in a minute.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
leon84
Member Candidate
Member Candidate
Topic Author
Posts: 184
Joined: Wed Dec 02, 2009 12:15 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Thu Jul 12, 2018 2:00 pm

Since years the problem of short outages of the primary path and the associated transient effects is solved by a "WTR timer" (Wait to Restore) which is used to delay the swicthover back to the primary path after it becomes OK. This is normally used to avoid falling back on false OKs, but could help a bit to you in terms that the total time of the outage would be shorter.

However, my approach would be to use one SSTP tunnel via each LTE interface and use a failover between these two tunnels for the critical data. It could then detect failures much faster than once in a minute.
Thanks a lot for your suggestion! A question: if I implement 2 sstp tunnel I have 2 different ip address client side. right?
 
sindy
Forum Guru
Forum Guru
Posts: 4218
Joined: Mon Dec 04, 2017 9:19 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Thu Jul 12, 2018 4:02 pm

Yes, you need to create two SSTP client interfaces, each will have another user and thus get another IP addresses. Only if each SSTP client would connect to a different SSTP server, there would be a danger that both interfaces would get the same IP address assigned.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
leon84
Member Candidate
Member Candidate
Topic Author
Posts: 184
Joined: Wed Dec 02, 2009 12:15 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Thu Jul 12, 2018 4:19 pm

Actually I have a service that acquire realtime data from remote RBM33G through a unique SSTP Tunnel. This service is connected to a remote ip address 192.168.254.10 (sstp interface).
I could implement second tunnel and assign to another sstp interface another ip, for example 192.168.254.9.
My service will continue to try to connect to 192.168.254.10 that is down. In this case I should implement also a procedure server side to swith to other ip address. Right?
Since years the problem of short outages of the primary path and the associated transient effects is solved by a "WTR timer" (Wait to Restore) which is used to delay the swicthover back to the primary path after it becomes OK. This is normally used to avoid falling back on false OKs, but could help a bit to you in terms that the total time of the outage would be shorter.

However, my approach would be to use one SSTP tunnel via each LTE interface and use a failover between these two tunnels for the critical data. It could then detect failures much faster than once in a minute.
Thanks a lot for your suggestion! A question: if I implement 2 sstp tunnel I have 2 different ip address client side. right?
 
sindy
Forum Guru
Forum Guru
Posts: 4218
Joined: Mon Dec 04, 2017 9:19 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Thu Jul 12, 2018 4:57 pm

Both tunnels will be internally on private addresses, so no NAT will interfere with routing operation. This permits you to create a bridge with no member ports on the remote device (the one with 2 LTE uplinks), assign a third IP address to that bridge, and on the SSTP server side, create two routes to that third address, each using another SSTP client interface address as a gateway.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
leon84
Member Candidate
Member Candidate
Topic Author
Posts: 184
Joined: Wed Dec 02, 2009 12:15 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Thu Jul 12, 2018 5:19 pm

Please check if I understood:
I create 2 sstp-client interface and sstp server asssign (for example) 192.168.254.9 at one and 192.168.254.10 at another one.
Then I create a bridge and add to the bridge both lte interface; I assign to the bridge an ip address for example 192.168.254.7.

On server side I create 2 route for 192.168.254.7 using sstp interface connected to 192.168.254.9 and interface connected to 192.168.254.10.

My service will contact always 192.168.254.7 that will be contactet trough sstp tunnel 1 or sstp tunnel 2 in function for example of a route distance. When an sstp tunnel go down the relative route will be disabled and the traffic go to other sstp tunnel.

Right? Sorry but in this case I could create also a simple bridge without any interface or a loopback interface. Right?

Both tunnels will be internally on private addresses, so no NAT will interfere with routing operation. This permits you to create a bridge with no member ports on the remote device (the one with 2 LTE uplinks), assign a third IP address to that bridge, and on the SSTP server side, create two routes to that third address, each using another SSTP client interface address as a gateway.
 
sindy
Forum Guru
Forum Guru
Posts: 4218
Joined: Mon Dec 04, 2017 9:19 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Thu Jul 12, 2018 5:55 pm

Your idea assumes creation of a WAN bridge on L2, but SSTP tunnels do not transport L2 frames, only L3 packets. So if you would like to take that way, you would have to set up EoIP tunnels inside the SSTP tunnels, i.e. the server would also have to be a Mikrotik device, and you'd waste a painful amount of packet space on tunnel headers. But yes, in such case, you could create a bond of the two EoIP tunnel interfaces on each end, or bridge them together with STP protocol enabled at the bridge.

My suggestion was an L3 one. I'm not sure how exactly SSTP tunnels work so maybe the two client addresses cannot be in the same subnet, but that's not the key here, so let's assume they can be 192.168.254.9 and 192.168.254.10 as you suggest.

The key is that the bridge at the client side would have no member ports at all (neither physical nor virtual), it will only be created in order to hold an IP address unrelated to any existing subnet, such as 10.22.33.44/32, and the routes at the server side will say (using Mikrotik syntax):

/ip route
add dst=address=10.22.33.44/32 distance=1 gateway=192.168.254.9
add dst=address=10.22.33.44/32 distance=2 gateway=192.168.254.10


At the client side, the routes will be

/ip route
add dst=address=the.data.collector.ip/32 distance=1 gateway=sstp-out1
add dst=address=the.data.collector.ip/32 distance=2 gateway=sstp-out2


There seems to be nothing to configure about tunnel state monitorinig on the /interface sstp-client, so either it is always on or it doesn't exist at all. Depending on that, you may need to use a script to monitor both tunnels' transparency at both ends and disable the route if the tunnel gets down.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
leon84
Member Candidate
Member Candidate
Topic Author
Posts: 184
Joined: Wed Dec 02, 2009 12:15 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Thu Jul 12, 2018 6:12 pm

Perfect I think to understand.
Then I should force sstp-out1 to use only lte1 and sstp-out2 to use only lte2. Right?
In which way? With policy based route?

Please check if I understood:
I create 2 sstp-client interface and sstp server asssign (for example) 192.168.254.9 at one and 192.168.254.10 at another one.
Then I create a bridge and add to the bridge both lte interface; I assign to the bridge an ip address for example 192.168.254.7.

On server side I create 2 route for 192.168.254.7 using sstp interface connected to 192.168.254.9 and interface connected to 192.168.254.10.

My service will contact always 192.168.254.7 that will be contactet trough sstp tunnel 1 or sstp tunnel 2 in function for example of a route distance. When an sstp tunnel go down the relative route will be disabled and the traffic go to other sstp tunnel.

Right? Sorry but in this case I could create also a simple bridge without any interface or a loopback interface. Right?

Both tunnels will be internally on private addresses, so no NAT will interfere with routing operation. This permits you to create a bridge with no member ports on the remote device (the one with 2 LTE uplinks), assign a third IP address to that bridge, and on the SSTP server side, create two routes to that third address, each using another SSTP client interface address as a gateway.
 
sindy
Forum Guru
Forum Guru
Posts: 4218
Joined: Mon Dec 04, 2017 9:19 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Thu Jul 12, 2018 6:29 pm

Then I should force sstp-out1 to use only lte1 and sstp-out2 to use only lte2. Right?
In which way? With policy based route?
This is the question I was afraid of :-D

One of problems associated to use of SSTP is that there is no way to configure anything but the remote address and port in Mikrotik's SSTP client configuration, so you have nothing but the dst-address and dst-port as a base for policy-based routing (a simple one, using just mark=routing rules in chain=output and the routes with routing-marks and a distance=2 route with type=blackhole to prevent failover if the required WAN is down).

So on the server, you have to set port forwarding from some other port to the one on which it normally listens for incoming SSTP connections, so that it would effectively accept them on both.

If it is not possible, there is a way to achieve that effect on the client side, but it is a brain damaging one.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
leon84
Member Candidate
Member Candidate
Topic Author
Posts: 184
Joined: Wed Dec 02, 2009 12:15 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Thu Jul 12, 2018 6:57 pm

:-)

Then I can use another port on server side and redirect traffic to the standard 443 port of sstp-server. In this way I could force sstp-out1 on lte1 and sstp-out2 on lte2.
But the very important question is: suppose I configured all correctly; I have my service connected to 192.168.254.7; the traffic go from my service to sstp-server (that has 2 route for 192.168.254.7) and through the route with minor distance. If this route goes down, sstp server select the other route for 192.168.254.7 that use other lte interface on client side. In this case I have in any way an interruption of tcp traffic from my service to 192.168.254.7 or it is transparent?




Then I should force sstp-out1 to use only lte1 and sstp-out2 to use only lte2. Right?
In which way? With policy based route?
This is the question I was afraid of :-D

One of problems associated to use of SSTP is that there is no way to configure anything but the remote address and port in Mikrotik's SSTP client configuration, so you have nothing but the dst-address and dst-port as a base for policy-based routing (a simple one, using just mark=routing rules in chain=output and the routes with routing-marks and a distance=2 route with type=blackhole to prevent failover if the required WAN is down).

So on the server, you have to set port forwarding from some other port to the one on which it normally listens for incoming SSTP connections, so that it would effectively accept them on both.

If it is not possible, there is a way to achieve that effect on the client side, but it is a brain damaging one.
 
sindy
Forum Guru
Forum Guru
Posts: 4218
Joined: Mon Dec 04, 2017 9:19 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Thu Jul 12, 2018 9:11 pm

suppose I configured all correctly; I have my service connected to 192.168.254.7; the traffic go from my service to sstp-server (that has 2 route for 192.168.254.7) and through the route with minor distance. If this route goes down, sstp server select the other route for 192.168.254.7 that use other lte interface on client side. In this case I have in any way an interruption of tcp traffic from my service to 192.168.254.7 or it is transparent?
There are two factors:
  • there are no settings in the SSTP configuration related to keepalives, so I don't know how quickly a failure of the SSTP tunnel will be detected and whether it will be detected at all before sending of a payload packet fails. So it may happen that the tunnel interface will be up long after the carrier path goes down, and only the first transport packet not to get acknowledged within timeout will cause the tunnel interface state change to down. If so, the TCP protocol in the payload will be heavily affected - google for "tcp meltdown". That's why I wrote before about "one of problems associated to SSTP".
  • RouterOS uses a routing cache which is on by default. I assume that if the gateway in use becomes unavailable, the cached route is replaced by a new one, but I'm not sure, so maybe you'll have to disable routing cache.
If not for the above - yes, even if each of the packets of the same TCP connection would randomly choose one of the tunnels, as there is no NAT, the receiving side wouldn't notice a difference, except that as the paths are very likely to have a different transport delay, the packets would be coming in shuffled order.

So all in all I'm afraid that the TCP session providing the SSTP transport will give up retransmitting and shut the tunnel interface down at about the same time when the payload TCP session will also give up retransmitting, so the route failover will come too late for the payload session to survive. But that's a property of all VPN solutions relying on TCP transport. So if devices at both ends of the tunnels are under your control, you may prefer to use IPsec to create the tunnels, as it uses transport protocols without retransmission control and you can use e.g. pinging of the remote end to detect a breakdown of the tunnel much faster than in the SSTP case. This way, the payload TCP session would encounter some lost packets but wouldn't break unless both paths would be down.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
leon84
Member Candidate
Member Candidate
Topic Author
Posts: 184
Joined: Wed Dec 02, 2009 12:15 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Fri Jul 13, 2018 9:40 am

Dear sindy,
congratulations for your knowledge :-). Great!
I would to do a test also if I prefer a more simple configuration :-).
In any case I have sstp server on TCP port 443 and 444 (When I receive SYN on 444 I forward it to 444).
Then in client side I have sstp-out1 connected to 443 and sstp-out2 connected to 444. Now I'm trying to force sstp-out1 to use only lte1 and sstp-out2 to use only lte2. And I have some problem.
I configured a routing mark on mangle table.
In prerouting chain I have dst-address=public ip of sstp server, protocol=tcp and dst-port 443 and in this case the action is to mark-routing=to-443
A similar rule for the same dst-address, same protocol but dst-port=444 and in this case the action is to mark-routing=to-444
Then in routing table I added to rules for the same destination address (the sstp server public ip) but with different gateway interface (lte1 for one with routing-mark to-443 and lte2 for other one with routing mark to-444). Both sstp-out interface are connected but through the same lte interface. Where is the issue?
 
sindy
Forum Guru
Forum Guru
Posts: 4218
Joined: Mon Dec 04, 2017 9:19 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Fri Jul 13, 2018 9:50 am

The issue is that mangle rules for packets originating from the Mikrotik itself have to be placed in chain=output.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
leon84
Member Candidate
Member Candidate
Topic Author
Posts: 184
Joined: Wed Dec 02, 2009 12:15 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Fri Jul 13, 2018 10:00 am

Perfect. I add mangle rules in output chain and now I can see traffic counter on these rules but now only sstp-out1 can connect to server because it use default gateway. Then I disabled default gateway but there are the 2 specific rules to sstp-server public ip but they (sstp-out1 and sstp-out2) can't connect.
 
sindy
Forum Guru
Forum Guru
Posts: 4218
Joined: Mon Dec 04, 2017 9:19 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Fri Jul 13, 2018 10:06 am

Show me /ip route print detail and /ip firewall mangle print.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
leon84
Member Candidate
Member Candidate
Topic Author
Posts: 184
Joined: Wed Dec 02, 2009 12:15 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Fri Jul 13, 2018 10:19 am

[admin@MikroTik] > ip firewall mangle print
Flags: X - disabled, I - invalid, D - dynamic
0 chain=output action=mark-routing new-routing-mark=to-443 passthrough=no protocol=tcp dst-address=SSTP-SERVER-IP-PUBLIC dst-port=443 log=no log-prefix=""

1 chain=output action=mark-routing new-routing-mark=to-444 passthrough=no protocol=tcp dst-address=SSTP-SERVER-IP-PUBLIC dst-port=444 log=no log-prefix=""


[admin@MikroTik] > ip route print detail
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit
0 A S dst-address=sstp-server-ip-public/32 gateway=lte1 gateway-status=lte1 reachable distance=1 scope=30 target-scope=10 routing-mark=to-443

1 A S dst-address=sstp-server-ip-public/32 gateway=lte2 gateway-status=lte2 reachable distance=1 scope=30 target-scope=10 routing-mark=to-444

2 A S ;;; default-lte2
dst-address=0.0.0.0/0 gateway=lte2 gateway-status=lte2 reachable distance=1 scope=30 target-scope=10

3 S ;;; default-lte1
dst-address=0.0.0.0/0 gateway=lte1 gateway-status=lte1 reachable distance=3 scope=30 target-scope=10

4 A S dst-address=8.8.4.4/32 gateway=lte2 gateway-status=lte2 reachable distance=1 scope=30 target-scope=10

5 A S dst-address=8.8.8.8/32 gateway=lte1 gateway-status=lte1 reachable distance=1 scope=30 target-scope=10

6 ADC dst-address=10.92.119.88/30 pref-src=10.92.119.89 gateway=lte1 gateway-status=lte1 reachable distance=0 scope=10

7 ADC dst-address=10.122.174.196/30 pref-src=10.122.174.197 gateway=lte2 gateway-status=lte2 reachable distance=0 scope=10

8 ADC dst-address=192.168.0.0/24 pref-src=192.168.0.1 gateway=bridge1 gateway-status=bridge1 reachable distance=0 scope=10

9 ADC dst-address=192.168.254.1/32 pref-src=192.168.254.9 gateway=sstp-out2 gateway-status=sstp-out2 reachable distance=0 scope=10
 
sindy
Forum Guru
Forum Guru
Posts: 4218
Joined: Mon Dec 04, 2017 9:19 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Fri Jul 13, 2018 10:33 am

No idea, everything seems correct to me. Try /ip firewall connection remove [find dst-address~"^SSTP-SERVER-IP-PUBLIC:44[34]"], but it is just a blind shot.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
leon84
Member Candidate
Member Candidate
Topic Author
Posts: 184
Joined: Wed Dec 02, 2009 12:15 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Fri Jul 13, 2018 11:08 am

Please,
consider this new routing table because I thinl there is some issue here. I have always the same mangle routing mark as previous scenario but this new routing table. In this case if I disable "default-lte1" it use only default-lte2 and the only interface can connect is sstp-out2. If I disable "default-lte2" the only interface can connect is "sstp-out1".

[admin@MikroTik] > ip route print detail
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit
0 A S dst-address=0.0.0.0/0 gateway=lte1 gateway-status=lte1 reachable distance=1 scope=30 target-scope=10 routing-mark=to-443

1 A S dst-address=0.0.0.0/0 gateway=lte2 gateway-status=lte2 reachable distance=1 scope=30 target-scope=10 routing-mark=to-444

2 A S ;;; default-lte1
dst-address=0.0.0.0/0 gateway=lte1 gateway-status=lte1 reachable distance=3 scope=30 target-scope=10

3 X S ;;; default-lte2
dst-address=0.0.0.0/0 gateway=lte2 gateway-status=lte2 inactive distance=3 scope=30 target-scope=10

4 A S dst-address=8.8.4.4/32 gateway=lte2 gateway-status=lte2 reachable distance=1 scope=30 target-scope=10

5 A S dst-address=8.8.8.8/32 gateway=lte1 gateway-status=lte1 reachable distance=1 scope=30 target-scope=10

6 ADC dst-address=10.92.119.88/30 pref-src=10.92.119.89 gateway=lte1 gateway-status=lte1 reachable distance=0 scope=10

7 ADC dst-address=10.122.174.196/30 pref-src=10.122.174.197 gateway=lte2 gateway-status=lte2 reachable distance=0 scope=10

8 ADC dst-address=192.168.0.0/24 pref-src=192.168.0.1 gateway=bridge1 gateway-status=bridge1 reachable distance=0 scope=10

9 ADC dst-address=192.168.254.1/32 pref-src=192.168.254.10 gateway=sstp-out1 gateway-status=sstp-out1 reachable distance=0 scope=10
 
leon84
Member Candidate
Member Candidate
Topic Author
Posts: 184
Joined: Wed Dec 02, 2009 12:15 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Fri Jul 13, 2018 11:14 am

Dear sindy,
I think I solved in this way:

[admin@MikroTik] > ip route print detail
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit
0 A S dst-address=0.0.0.0/0 gateway=lte1 gateway-status=lte1 reachable distance=1 scope=30 target-scope=10 routing-mark=to-443

1 A S dst-address=0.0.0.0/0 gateway=lte2 gateway-status=lte2 reachable distance=1 scope=30 target-scope=10 routing-mark=to-444

2 A S dst-address=0.0.0.0/0 gateway=lte1,lte2 gateway-status=lte1 reachable,lte2 reachable distance=1 scope=30 target-scope=10

3 A S dst-address=8.8.4.4/32 gateway=lte2 gateway-status=lte2 reachable distance=1 scope=30 target-scope=10

4 A S dst-address=8.8.8.8/32 gateway=lte1 gateway-status=lte1 reachable distance=1 scope=30 target-scope=10

5 ADC dst-address=10.92.119.88/30 pref-src=10.92.119.89 gateway=lte1 gateway-status=lte1 reachable distance=0 scope=10

6 ADC dst-address=10.122.174.196/30 pref-src=10.122.174.197 gateway=lte2 gateway-status=lte2 reachable distance=0 scope=10

7 ADC dst-address=192.168.0.0/24 pref-src=192.168.0.1 gateway=bridge1 gateway-status=bridge1 reachable distance=0 scope=10

8 ADC dst-address=192.168.254.1/32 pref-src=192.168.254.9 gateway=sstp-out2,sstp-out1 gateway-status=sstp-out2 reachable,sstp-out1 reachable distance=0 scope=10
 
sindy
Forum Guru
Forum Guru
Posts: 4218
Joined: Mon Dec 04, 2017 9:19 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Fri Jul 13, 2018 11:28 am

Please follow the instructions in my automatic signature... the mangle rules and routes as they were in the previous post seemed correct to me, so maybe there is something elsewhere in the configuration. I suppose you don't use any special handling on the server side (like port-forwarding only if the packet comes fom a particular IP address).

The way you've modified the default route causes even connections to use lte1 and odd connections use lte2 or vice versa, so it is not a solution of the issue, it just hides it.

Another point, once the issue of not establishing a connection via the marked routes is resolved, is that you have to prevent the SSTP sessions from getting established via the "wrong" WAN. The routing normally uses a fallback to routing table "main" (consisting of routes without routing-mark) if no active route with the routing-mark is found; you have to prevent this behaviour, because if the proper WAN is down, all routes via that WAN become inactive so the above would happen. So add also the following routes:
dst-address=0.0.0.0/0 routing-mark=to-443 distance=2 type=unreachable
dst-address=0.0.0.0/0 routing-mark=to-444 distance=2 type=unreachable
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
leon84
Member Candidate
Member Candidate
Topic Author
Posts: 184
Joined: Wed Dec 02, 2009 12:15 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Fri Jul 13, 2018 12:00 pm

Dear sindy,
resetted router and reconfigured it for simplicity. Now this is the configuration and the only interface connected is sstp-out1
/interface bridge
add fast-forward=no name=bridge1
add fast-forward=no name=bridge2
/interface lte apn
add apn=mobile.vodafone.it name=lte1
add apn=mobile.vodafone.it name=lte2
/interface lte
set [ find ] apn-profiles=lte1 mac-address=02:1E:10:1F:00:00 name=lte1
set [ find ] apn-profiles=lte2 mac-address=02:1E:10:1F:00:00 name=lte2
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip hotspot profile
set [ find default=yes ] html-directory=flash/hotspot
/interface sstp-client
add connect-to=sstp-server-public-ip disabled=no keepalive-timeout=10 name=sstp-out1 profile=default-encryption user=client1 verify-server-certificate=yes
add connect-to=sstp-server-public-ip:444 disabled=no keepalive-timeout=10 name=sstp-out2 profile=default-encryption user=client1bk verify-server-certificate=yes
/interface bridge port
add bridge=bridge1 interface=ether1
add bridge=bridge1 interface=ether2
add bridge=bridge1 interface=ether3
/ip address
add address=192.168.0.1/24 interface=bridge1 network=192.168.0.0
add address=192.168.254.7 interface=bridge2 network=192.168.254.7
/ip firewall mangle
add action=mark-routing chain=output dst-address=sstp-server-public-ip dst-port=443 new-routing-mark=to-443 passthrough=no protocol=tcp
add action=mark-routing chain=output dst-address=sstp-server-public-ip dst-port=444 new-routing-mark=to-444 passthrough=no protocol=tcp
/ip route
add distance=1 gateway=lte1 routing-mark=to-443
add distance=2 routing-mark=to-443 type=unreachable
add distance=1 gateway=lte2 routing-mark=to-444
add distance=2 routing-mark=to-444 type=unreachable
add distance=1 gateway=lte1
add distance=1 gateway=lte2
/ip service
set www port=8080
/system clock
set time-zone-name=Europe/Rome
/system ntp client
set enabled=yes primary-ntp=193.204.114.232 secondary-ntp=93.204.114.233
/system routerboard settings
set silent-boot=no
 
sindy
Forum Guru
Forum Guru
Posts: 4218
Joined: Mon Dec 04, 2017 9:19 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Fri Jul 13, 2018 12:12 pm

It cannot be simpler... what is the software version and what does /ip firewall connection print detail say?
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
sindy
Forum Guru
Forum Guru
Posts: 4218
Joined: Mon Dec 04, 2017 9:19 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Fri Jul 13, 2018 12:35 pm

Well, one more idea... you probably have to add a srcnat rule. I don't know at what step the local address for the outgoing packets is chosen, so maybe the packets are sent via lte2 with source IP address matching lte1.

/ip firewall nat
add chain=srcnat action=masquerade out-interface=lte1
add chain=srcnat action=masquerade out-interface=lte2
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
leon84
Member Candidate
Member Candidate
Topic Author
Posts: 184
Joined: Wed Dec 02, 2009 12:15 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Fri Jul 13, 2018 12:38 pm

I'm using RouterOS 6.42.5 and this is output of ip firewall connections:

[admin@MikroTik] >  /ip firewall connection print detail 
Flags: E - expected, S - seen-reply, A - assured, C - confirmed, D - dying, F - fasttrack, s - srcnat, d - dstnat 
 0    C     protocol=udp src-address=169.254.125.216:53141 dst-address=255.255.255.255:20561 reply-src-address=255.255.255.255:20561 reply-dst-address=169.254.125.216:53141 timeout=9s orig-packets=238 580 orig-bytes=14 629 788 orig-fasttrack-packets=0 
            orig-fasttrack-bytes=0 repl-packets=0 repl-bytes=0 repl-fasttrack-packets=0 repl-fasttrack-bytes=0 orig-rate=8.3kbps repl-rate=0bps 

 1  SAC     protocol=tcp src-address=10.92.119.89:41381 dst-address=sstp-server-public-ip:443 reply-src-address=sstp-server-public-ip:443 reply-dst-address=10.92.119.89:41381 tcp-state=established timeout=23h59m59s orig-packets=3 851 orig-bytes=496 389 
            orig-fasttrack-packets=0 orig-fasttrack-bytes=0 repl-packets=8 969 repl-bytes=783 323 repl-fasttrack-packets=0 repl-fasttrack-bytes=0 orig-rate=1224bps repl-rate=2.0kbps 

 2    C     protocol=tcp src-address=10.92.119.89:58355 dst-address=sstp-server-public-ip:444 reply-src-address=sstp-server-public-ip:444 reply-dst-address=10.92.119.89:58355 tcp-state=syn-sent timeout=2s orig-packets=2 orig-bytes=120 orig-fasttrack-packets=0 
            orig-fasttrack-bytes=0 repl-packets=0 repl-bytes=0 repl-fasttrack-packets=0 repl-fasttrack-bytes=0 orig-rate=0bps repl-rate=0bps 
[admin@MikroTik] > 

 
sindy
Forum Guru
Forum Guru
Posts: 4218
Joined: Mon Dec 04, 2017 9:19 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Fri Jul 13, 2018 12:41 pm

This supports the idea in my post above, so add the masquerade rules and try again.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
leon84
Member Candidate
Member Candidate
Topic Author
Posts: 184
Joined: Wed Dec 02, 2009 12:15 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Fri Jul 13, 2018 12:55 pm

Great! Now is running correctly! Now I try with failover! Thanks a lot!
This supports the idea in my post above, so add the masquerade rules and try again.
 
sindy
Forum Guru
Forum Guru
Posts: 4218
Joined: Mon Dec 04, 2017 9:19 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Fri Jul 13, 2018 1:20 pm

The explanation is that the packet is first routed before passing through the mangle rules, and gets a source address corresponding to the route. Then, the "routing adjustment" (see this picture) takes place if a mangle rule assigns a routing-mark, but the source address remains the initially assigned one unless it is explicitly changed in the next step using a srcnat rule. In most setups everything that leaves the box via the WANs is srcnated because traffic forwarded from LAN wouldn't ever get any responses otherwise, so this subtle aspect of redirection of local output traffic remains unnoticed.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
leon84
Member Candidate
Member Candidate
Topic Author
Posts: 184
Joined: Wed Dec 02, 2009 12:15 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Fri Jul 13, 2018 1:25 pm

Very good! Now I have both sstp-out connected but I noticed that I have more latency in ping from 192.168.254.1 (sstp-server private ip in the tunnel) and 192.168.254.9 or 192.168.254.10. I have a latency of about 700ms but if I disable one of the sstp-out the latency go to about 70 ms. Have you some explaination?
 
sindy
Forum Guru
Forum Guru
Posts: 4218
Joined: Mon Dec 04, 2017 9:19 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Fri Jul 13, 2018 2:24 pm

None if the effect is symmetric (both tunnel 1 and tunnel 2 give 70 ms while the other one is disabled, and you only get 700 ms if both are enabled).

Also, I don't know how the routing looks like when both tunnels are up (/ip route print detail).
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
leon84
Member Candidate
Member Candidate
Topic Author
Posts: 184
Joined: Wed Dec 02, 2009 12:15 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Fri Jul 13, 2018 3:38 pm

Dear sindy,
this is route print whene both sstp interface are connected.
[admin@MikroTik] > ip route print detail 
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit 
 0 A S  dst-address=0.0.0.0/0 gateway=lte1 gateway-status=lte1 reachable distance=1 scope=30 target-scope=10 routing-mark=to-443 

 1   SU dst-address=0.0.0.0/0 type=unreachable distance=2 routing-mark=to-443 

 2 A S  dst-address=0.0.0.0/0 gateway=lte2 gateway-status=lte2 reachable distance=1 scope=30 target-scope=10 routing-mark=to-444 

 3   SU dst-address=0.0.0.0/0 type=unreachable distance=2 routing-mark=to-444 

 4 A S  dst-address=0.0.0.0/0 gateway=lte1 gateway-status=lte1 reachable distance=1 scope=30 target-scope=10 

 5   S  dst-address=0.0.0.0/0 gateway=lte2 gateway-status=lte2 reachable distance=1 scope=30 target-scope=10 

 6 ADC  dst-address=10.8.28.60/30 pref-src=10.8.28.62 gateway=lte2 gateway-status=lte2 reachable distance=0 scope=10 

 7 ADC  dst-address=10.64.118.204/30 pref-src=10.64.118.206 gateway=lte1 gateway-status=lte1 reachable distance=0 scope=10 

 8 ADC  dst-address=192.168.0.0/24 pref-src=192.168.0.1 gateway=bridge1 gateway-status=bridge1 reachable distance=0 scope=10 

 9 ADC  dst-address=192.168.254.0/24 pref-src=192.168.254.7 gateway=bridge2 gateway-status=bridge2 reachable distance=0 scope=10 

10 ADC  dst-address=192.168.254.1/32 pref-src=192.168.254.10 gateway=sstp-out1,sstp-out2 gateway-status=sstp-out1 reachable,sstp-out2 reachable distance=0 scope=10 

 
leon84
Member Candidate
Member Candidate
Topic Author
Posts: 184
Joined: Wed Dec 02, 2009 12:15 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Fri Jul 13, 2018 3:49 pm

I attach you also sstp-server route print :-) :
[admin@SSTP-SERVER] > ip route print detail 
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit 
 0 A S  dst-address=0.0.0.0/0 gateway=public-ip gateway-status=public-ip reachable via  ether1 distance=1 scope=30 target-scope=10 

 1 A S  dst-address=192.168.254.7/32 gateway=sstp-client1bk,sstp-client1 gateway-status=sstp-client1bk reachable,sstp-client1 reachable distance=1 scope=30 target-scope=10 

 2 ADC  dst-address=192.168.254.9/32 pref-src=192.168.254.1 gateway=sstp-client1bk gateway-status=sstp-client1bk reachable distance=0 scope=10 

 3 ADC  dst-address=192.168.254.10/32 pref-src=192.168.254.1 gateway=sstp-client1 gateway-status=sstp-client1 reachable distance=0 scope=10 

 
sindy
Forum Guru
Forum Guru
Posts: 4218
Joined: Mon Dec 04, 2017 9:19 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Fri Jul 13, 2018 7:58 pm

I'm afraid I'd start from assigning distinct local and remote addresses to the two SSTP user "secrets", so that the dynamically generated routes would not merge into one, and place these addresses outside any server or client subnet, to act only as interconnection networks. And then I would create routes via these interconnection networks (with different distance values) to other local address on the opposite router so that I could choose the usage strategy of these routes, instead of having to use round-robin as two point-to-point connected subnets with the same remote IP address inevitably cause.

And finally I would test pinging to the destination address of these routes, not any of the addresses associated to the tunnel interfaces, from source address different from those associated to tunnel interfaces.

You can use tunnel interface names as route gateways, so using distinct tunnel interface addresses at server side may not be necessary to split the routes to non-tunnel addresses, but doing so doesn't allow you to use the scriptless failover based on recursive next-hop.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
leon84
Member Candidate
Member Candidate
Topic Author
Posts: 184
Joined: Wed Dec 02, 2009 12:15 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Mon Jul 16, 2018 12:43 pm

Dear sindy,
now I have two tunnel sstp in this way:
sstp-out1 with ip address 192.168.254.10 and remtoe sstp server as 192.168.254.1
sstp-out2 with ip address 192.168.254.9 and remote sstp server as 192.168.254.2

Now the route table of client is:
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit 
 0 A S  dst-address=0.0.0.0/0 gateway=lte1 gateway-status=lte1 reachable distance=1 scope=30 target-scope=10 routing-mark=to-443 

 1   SU dst-address=0.0.0.0/0 type=unreachable distance=2 routing-mark=to-443 

 2 A S  dst-address=0.0.0.0/0 gateway=lte2 gateway-status=lte2 reachable distance=1 scope=30 target-scope=10 routing-mark=to-444 

 3   SU dst-address=0.0.0.0/0 type=unreachable distance=2 routing-mark=to-444 

 4 A S  dst-address=0.0.0.0/0 gateway=lte1 gateway-status=lte1 reachable distance=1 scope=30 target-scope=10 

 5   S  dst-address=0.0.0.0/0 gateway=lte2 gateway-status=lte2 reachable distance=1 scope=30 target-scope=10 

 6 ADC  dst-address=10.68.177.64/29 pref-src=10.68.177.68 gateway=lte1 gateway-status=lte1 reachable distance=0 scope=10 

 7 ADC  dst-address=10.119.181.56/30 pref-src=10.119.181.57 gateway=lte2 gateway-status=lte2 reachable distance=0 scope=10 

 8 ADC  dst-address=192.168.0.0/24 pref-src=192.168.0.1 gateway=bridge1 gateway-status=bridge1 reachable distance=0 scope=10 

 9 ADC  dst-address=192.168.254.0/24 pref-src=192.168.254.7 gateway=bridge2 gateway-status=bridge2 reachable distance=0 scope=10 

10 ADC  dst-address=192.168.254.1/32 pref-src=192.168.254.10 gateway=sstp-out1 gateway-status=sstp-out1 reachable distance=0 scope=10 

11 ADC  dst-address=192.168.254.2/32 pref-src=192.168.254.9 gateway=sstp-out2 gateway-status=sstp-out2 reachable distance=0 scope=10 

And the route table of sstp-server is:
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit 
 0 A S  dst-address=0.0.0.0/0 gateway=public-ip gateway-status=public-ip reachable via  ether1 distance=1 scope=30 target-scope=10 

 1 A S  dst-address=192.168.254.7/32 gateway=sstp-client1 gateway-status=sstp-client1 reachable distance=1 scope=30 target-scope=10 

 2   S  dst-address=192.168.254.7/32 gateway=sstp-client1bk gateway-status=sstp-client1bk reachable distance=2 scope=30 target-scope=10 

 3 ADC  dst-address=192.168.254.9/32 pref-src=192.168.254.2 gateway=sstp-client1bk gateway-status=sstp-client1bk reachable distance=0 scope=10 

 4 ADC  dst-address=192.168.254.10/32 pref-src=192.168.254.1 gateway=sstp-client1 gateway-status=sstp-client1 reachable distance=0 scope=10 

 5 ADC  dst-address=public-subnet/27 pref-src=public-ip gateway=ether1 gateway-status=ether1 reachable distance=0 scope=10 
[admin@SSTP-SERVER] > 

About latency sorry but it was the provider connection. In any case I ask me if there is a really advantage in this configuration because it introduce a major complex client configuration. Whene there is an lte down and the relative sstp-out interface the sstp-server wait for sstp keepalive (configured to 10s) and then swith to the other route. When interface come back up it newly switch to primary route and in this case there isn't any disconnection. It is gret but as I mentioned it introduce a more complex client configuration and I think to come back at initial configuration. :-) THANKS A LOT
 
sindy
Forum Guru
Forum Guru
Posts: 4218
Joined: Mon Dec 04, 2017 9:19 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Mon Jul 16, 2018 1:04 pm

Redundancy always comes at a cost of complexity. So it is your decision what you prefer. I prefer simplicity of maintenance to simplicity of configuration.

And most important for your case, regardless what redundancy model you finally choose, I hope that the two Vodafone SIMs were used only for testing and for production you'll use SIMs from different operators not sharing network infrastructure (which, I admit, may not be an easy task in Italy in particular).
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
leon84
Member Candidate
Member Candidate
Topic Author
Posts: 184
Joined: Wed Dec 02, 2009 12:15 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Mon Jul 16, 2018 1:08 pm

Oh yes I used 2 Vodafone sim only as test. I will use 2 different operators like Vodafone and TIM that have indipendent network infrastructure.

Redundancy always comes at a cost of complexity. So it is your decision what you prefer. I prefer simplicity of maintenance to simplicity of configuration.

And most important for your case, regardless what redundancy model you finally choose, I hope that the two Vodafone SIMs were used only for testing and for production you'll use SIMs from different operators not sharing network infrastructure (which, I admit, may not be an easy task in Italy in particular).
 
leon84
Member Candidate
Member Candidate
Topic Author
Posts: 184
Joined: Wed Dec 02, 2009 12:15 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Mon Jul 16, 2018 3:23 pm

Sorry cindy,
do you know what is the correct way to distribute some routes when an sstp tunnel come up ? There is the parameter "routes" in PPP -> Secrects but it doesn't distribute the route to to remote sstp client? In which way Can I specify route in this parameter?
 
sindy
Forum Guru
Forum Guru
Posts: 4218
Joined: Mon Dec 04, 2017 9:19 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Tue Jul 17, 2018 12:23 am

I've never tried that, but as far as I know, the route parameter in the /ppp secret is there to add routes at the server, not at the client. The PPP (which constitutes the transport underneath SSTP) can configure only the default route, and although it should be possible to use DHCP over a PPP channel to deliver a routing table to the client, Mikrotik seems to ignore the relevant RFC, insisting that DHCP can only be used on L2 interfaces.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
User avatar
SergeiShablovsky
just joined
Posts: 2
Joined: Mon Dec 16, 2019 2:01 pm
Location: San Francisco/Kyiv
Contact:

Re: RBM33G with 2 LTE module (how to manage failover)

Mon Dec 16, 2019 3:06 pm

Dear sindy & leon84 !

Could You be so please to explain how to make failover in case using RBM33G as out-of-band management access device:
- 2 x R11e-LTE as 2 Cellular Operator with physically separate infrastructur
- 2 x Eth - for WAN connectivity as 2 ISP with physically separate infrastructure
and last
- 1 Eth used for connection to Ethernet-accessible KVM (like Aten, StarTech) or to RS232 Port Server (like Digi, Lantronix) for out-of-band management.

Because out-of-band management used rarely and only something on a remote site goes wrong, basically, RBM33G must be remote accessible in any moment of time thru each of 4 connections:
each of 2 static IPs from ISPs, and
each of 2 dynamic IPs from Cellular Operators (DYNdns or other service used for IP, no matter).

I thinking the solution for this must be some script to periodically checking each of all of 4 WAN uplink connections (2 Eth + 2 LTE) and changing routing path inside RBM33G in case one (or many) of WANs not able to check certain IP (for example CloudFlare 1.1.1.1 or Google 8.8.8.8/8.8.4.4) within certain period of time (for example 10 sec).
Some sort of WAN uplink failover for 4 WANs.

Am I right?

P.S. Several notes about tech support without head pain.
From technical support side would be best to have only one point of entry (vpnmgmt.example.com for example) on cloud provider, and delegate this cloud provider the periodically checking in cycle each of 4 IPs of device and notifying admins in case 1 or more of device's IP is unreachable for a certain period of time.
As I know CloudFlare doing this balancing from 2017, please read https://blog.cloudflare.com/introducing ... loudflare/ if You need to know more. And this is really not costly - starting from $5/month for 2 origins (https://support.cloudflare.com/hc/en-us ... -Balancing).
At end 2019 CloudFlare able to notify only on one email, but You may using this “alarm-notify” email to redirect to operator-depending SMS service or to Your support team Telegram channel.

You may ask “why I need to pay extra for external monitoring instead to doing the same by my own servers?”
The answer are easy to understand: because this is more costly in $ and time (and also add some complexity and point of failure to your infrastructure), and You also need to place Your Monitoring servers (no matter physical or virtual) in different geolocations. CloudFlare, for example, ask only $15 for 8(!) different geolocations on different continents. More than sure, if Your client have small/middle business inside one country, 2 different points inside country + 1 point outside country is ok in most case.
With best wishes to You!
 
User avatar
SiB
Member
Member
Posts: 358
Joined: Sun Jan 06, 2013 11:19 pm
Location: Poland

Re: RBM33G with 2 LTE module (how to manage failover)

Tue Dec 17, 2019 7:50 pm

Could You be so please to explain how to make failover in case using RBM33G as out-of-band management access device:
- 2 x R11e-LTE as 2 Cellular Operator with physically separate infrastructur
- 2 x Eth - for WAN connectivity as 2 ISP with physically separate infrastructure
and last
- 1 Eth used for connection to Ethernet-accessible KVM (like Aten, StarTech) or to RS232 Port Server (like Digi, Lantronix) for out-of-band management.

Because out-of-band management used rarely and only something on a remote site goes wrong, basically, RBM33G must be remote accessible in any moment of time thru each of 4 connections:
each of 2 static IPs from ISPs, and
each of 2 dynamic IPs from Cellular Operators (DYNdns or other service used for IP, no matter).

I thinking the solution for this must be some script to periodically checking each of all of 4 WAN uplink connections (2 Eth + 2 LTE) and changing routing path inside RBM33G in case one (or many) of WANs not able to check certain IP (for example CloudFlare 1.1.1.1 or Google 8.8.8.8/8.8.4.4) within certain period of time (for example 10 sec).
Some sort of WAN uplink failover for 4 WANs.

Am I right?
You can use all WAN at the same time to outgoing and/or incomming traffic.
ETH ports you can assign into bridge/bonding with VLAN's for LAN & OOB/MGMT subnet. This one ETH1 can have VLANs to separate KVM/OOB etc.
ROS have got own DDNS (IP>Cloud who use main route table to do outgoing checking).

By CLI and scripting you can do any-way checking of WAN's.
At this moment please do lecture of this:
Bandwidth-based load-balancing with failover. This presentation also covers Mangle.
This was presented at the MUM (MikroTik User Meeting) in New Orelans, USA.
Tomas Kirnak - YouTube: https://www.youtube.com/watch?v=67Dna_ffCvc
http://mum.mikrotik.com/presentations/US12/tomas.pdf

.
P.S. Several notes about tech support without head pain.
From technical support side would be best to have only one point of entry (vpnmgmt.example.com for example) on cloud provider, and delegate this cloud provider the periodically checking in cycle each of 4 IPs of device and notifying admins in case 1 or more of device's IP is unreachable for a certain period of time.
As I know CloudFlare doing this balancing from 2017, please read https://blog.cloudflare.com/introducing ... loudflare/ if You need to know more. And this is really not costly - starting from $5/month for 2 origins (https://support.cloudflare.com/hc/en-us ... -Balancing).
At end 2019 CloudFlare able to notify only on one email, but You may using this “alarm-notify” email to redirect to operator-depending SMS service or to Your support team Telegram channel.
You can use DNS like [ isp1 | isp2 | isp3 | isp4 ].yourdomain.tld and isp.yourdomain.tld with 4x A record - know as RoundRobin.
Many ways of this. You can even connect 4x VPNs via all ISP to one CHR at one of VPS providers and do your own CHR/LB with one DNS entry into CHR.
.
You may ask “why I need to pay extra for external monitoring instead to doing the same by my own servers?”
The answer are easy to understand: because this is more costly in $ and time (and also add some complexity and point of failure to your infrastructure), and You also need to place Your Monitoring servers (no matter physical or virtual) in different geolocations. CloudFlare, for example, ask only $15 for 8(!) different geolocations on different continents. More than sure, if Your client have small/middle business inside one country, 2 different points inside country + 1 point outside country is ok in most case.
I not ask, this is not related with MikroTik. This paragraph shoud have P.S. incarnation :).
P.S. Check OMD todo distributed monitoring.
MTCNA + MTCRE + MTCINE | ~600 users at ~150 RouterBoards in EMEA | Telegram: @SiB_PL
WinBox Tip: F6 works as ALT+TAB | Gliffy.com - free network schematic | prnt.sc - free ScreenShot software
I will be at MUMEUROPE Prague on 26-27 march 2020
 
sindy
Forum Guru
Forum Guru
Posts: 4218
Joined: Mon Dec 04, 2017 9:19 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Tue Dec 17, 2019 11:39 pm

To access the Mikrotik remotely via its public IP addresses, you need to set up policy routing so that responses to requests coming via a given WAN are routed out through the same WAN. Dynamically switching a single default route between WANs would be a bad idea. If those public addresses are dynamic, and you thus need to use a DDNS system to keep track of the current IP address, I suspect that the Mikrotik cloud is not the best choice as I assume it can work only with a single FQDN. So maybe you could use a script to regularly update the single fqdn via another WAN every 5 minutes, so if one WAN would go down, you would have to wait for just 5 minutes until the FQDN would start being resolved to the other one. Or, of course, use a 3rd party DDNS with one FQDN per each WAN.

To access the Mikrotik remotely via its private or shared (100.64.0.0/10) addresses (mobile connections), the only way is to set up a VPN tunnel to some device on a public IP via each of these WANs (and of course, use policy routing again for the same reason as above) and access the Mikrotik via this other device. It is not clear from your desription whether the addresses assigned by the mobile operators are directly accessible public ones, it depends on your contract. I've seen mobile operators NATing public IPs assigned to LTE modems to other public IPs, so the fact that you get a public IP does not necessarily mean that it is reachable from outside.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
User avatar
SergeiShablovsky
just joined
Posts: 2
Joined: Mon Dec 16, 2019 2:01 pm
Location: San Francisco/Kyiv
Contact:

Re: RBM33G with 2 LTE module (how to manage failover)

Wed Dec 18, 2019 1:33 pm

You can use all WAN at the same time to outgoing and/or incomming traffic.
ETH ports you can assign into bridge/bonding with VLAN's for LAN & OOB/MGMT subnet. This one ETH1 can have VLANs to separate KVM/OOB etc.
ROS have got own DDNS (IP>Cloud who use main route table to do outgoing checking).

By CLI and scripting you can do any-way checking of WAN's.
At this moment please do lecture of this:
Bandwidth-based load-balancing with failover. This presentation also covers Mangle.
This was presented at the MUM (MikroTik User Meeting) in New Orelans, USA.
Tomas Kirnak - YouTube: https://www.youtube.com/watch?v=67Dna_ffCvc
http://mum.mikrotik.com/presentations/US12/tomas.pdf
Thank You so much!

Because I am a little bit frustrating after reading how Juniper describe High Availability and Redundancy:
https://www.juniper.net/documentation/s ... ce_all.pdf (From page 1651)

https://www.juniper.net/documentation/e ... bility.pdf

https://www.juniper.net/documentation/p ... bility.pdf

Even the Juniper conception are a little bit heavy to understanding, but from other side, - not linked to Juniper devices so hard.

How You comment the Redundancy described in "Concepts & Examples...." ? Sorry for dumb question, I am a newbe in MikroTik platform. Tnx!
With best wishes to You!
 
User avatar
SiB
Member
Member
Posts: 358
Joined: Sun Jan 06, 2013 11:19 pm
Location: Poland

Re: RBM33G with 2 LTE module (how to manage failover)

Wed Dec 18, 2019 2:24 pm

Thank You so much!
You welcome

I correct your quote:
Because I am a little bit frustrating after reading how Juniper describe High Availability and Redundancy:
screenos6.3.0/630_ce_all.pdf (From page 1651)
high-availability.pdf
config-guide-high-availability.pdf
Even the Juniper conception are a little bit heavy to understanding, but from other side, - not linked to Juniper devices so hard.
How You comment the Redundancy described in "Concepts & Examples...." ? Sorry for dumb question, I am a newbe in MikroTik platform. Tnx!
Easy, in ROSv8 means MikroTik RouterOSv8 the all bug's and not exist features will be included. We can be sure that :)
HA between two routers with shared one Conntract table what give us possibility to build LoadBalance devices.... IPSec with VTI... this is future what be included into ROSv8 :p
MTCNA + MTCRE + MTCINE | ~600 users at ~150 RouterBoards in EMEA | Telegram: @SiB_PL
WinBox Tip: F6 works as ALT+TAB | Gliffy.com - free network schematic | prnt.sc - free ScreenShot software
I will be at MUMEUROPE Prague on 26-27 march 2020
 
leon84
Member Candidate
Member Candidate
Topic Author
Posts: 184
Joined: Wed Dec 02, 2009 12:15 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Wed Dec 18, 2019 3:48 pm

Dear all sorry for the latency.
Then I implemented 2 sstp tunnel (sstp-out1 and sstp-out2) to the same sstp-server having an ip public and accepting connection on TCP port 443 ant TCP port 444. In this way I can apply routing mark in mangle table using different TCP port. In the next step I added routes for each routing mark and redirected the traffic of sstp-out1 on LTE1 and sstp-out2 on LTE2. In my case sstp-out1 is it possibile only if LTE1 is up and running and sstp-out2 only if LTE2 is up and running. I chosed this strategy because I want 2 concurrent connections with remote router.
 
leon84
Member Candidate
Member Candidate
Topic Author
Posts: 184
Joined: Wed Dec 02, 2009 12:15 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Tue Jan 14, 2020 10:20 am

Dear sindy,
I have a question for you. I implemented two separate VPN tunnel sstp-out1 and sstp-out2 with a central SSTP Server. sstp-out1 connect to tcp 443 and sstp-out2 connect to tcp 444 port of the same server. On the server I have a dnat from 444 to 443. In this way I can apply routing mark because I want sstp-out1 only on lte1 interface and sstp-out2 only on lte2 interface.

All seems to be ok. From server I run continously ping to 172.18.0.203 and 172.18.0.204 that are private ip address assigned from vpn server.
I have this problem. Why when an LTE interface goes down I have a repercussion on both ping?
Example: If i put down lte1 interface ping stop for both sstp-out interface and after some seconds restart obviously for sstp-out2. I want that lte down doesn't affect both sstp-out but only sstp-out interface using that lte.
I attach you my mangle table (please consider xxx.xxx.xxx.xxx.public ip of sstp server:

 [admin@MikroTik] > ip firewall mangle print 
Flags: X - disabled, I - invalid, D - dynamic 
 0    chain=output action=mark-routing new-routing-mark=to-443 passthrough=yes protocol=tcp dst-address=xxx.xxx.xxx.xxx dst-port=443 log=no log-prefix="" 

 1    chain=output action=mark-routing new-routing-mark=to-444 passthrough=yes protocol=tcp dst-address=xxx.xxx.xxx.xxx dst-port=444 log=no log-prefix="" 
 
 
I attach also routing table:

[admin@MikroTik] > ip route print detail    
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit 
 0 A SU dst-address=0.0.0.0/0 type=unreachable distance=1 routing-mark=to-443 

 1 A S  dst-address=xxx.xxx.xxx.xxx/32 gateway=lte1 gateway-status=lte1 reachable distance=1 scope=30 target-scope=10 routing-mark=to-443 

 2 A SU dst-address=0.0.0.0/0 type=unreachable distance=1 routing-mark=to-444 

 3 A S  dst-address=xxx.xxx.xxx.xxx/32 gateway=lte2 gateway-status=lte2 reachable distance=1 scope=30 target-scope=10 routing-mark=to-444 

 4 A S  dst-address=0.0.0.0/0 gateway=lte2,lte1 gateway-status=lte2 reachable,lte1 reachable distance=1 scope=30 target-scope=10 

 5 ADC  dst-address=yyy.yyy.yyy.yyy/24 pref-src=zzz.zzz.zzz.zzz gateway=lte1 gateway-status=lte1 reachable distance=0 scope=10 

 6 ADC  dst-address=10.167.111.100/30 pref-src=10.167.111.101 gateway=lte2 gateway-status=lte2 reachable distance=0 scope=10 

 7 A SU dst-address=172.18.0.0/24 type=unreachable distance=1 

 8 ADC  dst-address=172.18.0.1/32 pref-src=172.18.0.203 gateway=sstp-out1 gateway-status=sstp-out1 reachable distance=0 scope=10 

 9 ADC  dst-address=172.18.0.254/32 pref-src=172.18.0.204 gateway=sstp-out2 gateway-status=sstp-out2 reachable distance=0 scope=10 

10 ADC  dst-address=192.168.0.0/24 pref-src=192.168.0.1 gateway=bridge1 gateway-status=bridge1 reachable distance=0 scope=10 

 
sindy
Forum Guru
Forum Guru
Posts: 4218
Joined: Mon Dec 04, 2017 9:19 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Tue Jan 14, 2020 6:09 pm

As the first step I would check, using /tool sniffer quick, that the ping requests and responses really take the routes you expect them to take when both LTEs and their associated SSTP tunnels are up. It may be that there is some hole in your policy routing and the requests or the responses actually take a different route than you expect, so when the LTE goes down, it takes the SSTP some time to notice that and put down its virtual interface so that the other route could come into use.

So while both LTEs and SSTPs are up, run /tool sniffer quick ip-address=172.18.0.203 protocol=icmp and see the interface names in the packet list; then change ip-address to 172.18.0.204 and look again.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
leon84
Member Candidate
Member Candidate
Topic Author
Posts: 184
Joined: Wed Dec 02, 2009 12:15 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Tue Jan 14, 2020 6:50 pm

As the first step I would check, using /tool sniffer quick, that the ping requests and responses really take the routes you expect them to take when both LTEs and their associated SSTP tunnels are up. It may be that there is some hole in your policy routing and the requests or the responses actually take a different route than you expect, so when the LTE goes down, it takes the SSTP some time to notice that and put down its virtual interface so that the other route could come into use.

So while both LTEs and SSTPs are up, run /tool sniffer quick ip-address=172.18.0.203 protocol=icmp and see the interface names in the packet list; then change ip-address to 172.18.0.204 and look again.
Dear sindy,
the sniffer seems to be correct:
[admin@MikroTik] /tool> /tool sniffer quick ip-protocol=icmp ip-address=172.18.0.203
INTERFACE                                                                                        TIME    NUM DIR SRC-MAC           DST-MAC           VLAN   SRC-ADDRESS                         DST-ADDRESS                         PROTOCOL   SIZE CPU FP 
sstp-out1                                                                                      16.078     33 <-                                             172.18.0.1                          172.18.0.203                        ip:icmp      56   0 no 
sstp-out1                                                                                      16.078     34 ->                                             172.18.0.203                        172.18.0.1                          ip:icmp      56   0 no 
sstp-out1                                                                                      17.084     35 <-                                             172.18.0.1                          172.18.0.203                        ip:icmp      56   0 no 
sstp-out1                                                                                      17.084     36 ->                                             172.18.0.203                        172.18.0.1                          ip:icmp      56   0 no 
sstp-out1                                                                                       18.08     37 <-                                             172.18.0.1                          172.18.0.203                        ip:icmp      56   0 no 
sstp-out1                                                                                       18.08     38 ->                                             172.18.0.203                        172.18.0.1                          ip:icmp      56   0 no 
sstp-out1                                                                                       19.08     39 <-                                             172.18.0.1                          172.18.0.203                        ip:icmp      56   0 no 
sstp-out1                                                                                       19.08     40 ->                                             172.18.0.203                        172.18.0.1                          ip:icmp      56   0 no 
sstp-out1                                                                                       20.08     41 <-                                             172.18.0.1                          172.18.0.203                        ip:icmp      56   0 no 
sstp-out1                                                                                       20.08     42 ->                                             172.18.0.203                        172.18.0.1                          ip:icmp      56   0 no 
sstp-out1                                                                                       21.08     43 <-                                             172.18.0.1                          172.18.0.203                        ip:icmp      56   0 no 
sstp-out1                                                                                       21.08     44 ->                                             172.18.0.203                        172.18.0.1                          ip:icmp      56   0 no 
sstp-out1                                                                                      22.078     45 <-                                             172.18.0.1                          172.18.0.203                        ip:icmp      56   0 no 
sstp-out1                                                                                      22.078     46 ->                                             172.18.0.203                        172.18.0.1                          ip:icmp      56   0 no 
sstp-out1                                                                                      23.078     47 <-                                             172.18.0.1                          172.18.0.203                        ip:icmp      56   0 no 
sstp-out1                                                                                      23.078     48 ->                                             172.18.0.203                        172.18.0.1                          ip:icmp      56   0 no 
sstp-out1                                                                                      24.084     49 <-                                             172.18.0.1                          172.18.0.203                        ip:icmp      56   0 no 
sstp-out1                                                                                      24.084     50 ->                                             172.18.0.203                        172.18.0.1                          ip:icmp      56   0 no 
sstp-out1                                                                                      25.078     51 <-                                             172.18.0.1                          172.18.0.203                        ip:icmp      56   0 no 
sstp-out1                                                                                      25.078     52 ->                                             172.18.0.203                        172.18.0.1                          ip:icmp      56   0 no 

[admin@MikroTik] /tool> /tool sniffer quick ip-protocol=icmp ip-address=172.18.0.204
INTERFACE                                                                                        TIME    NUM DIR SRC-MAC           DST-MAC           VLAN   SRC-ADDRESS                         DST-ADDRESS                         PROTOCOL   SIZE CPU FP 
sstp-out2                                                                                        0.92      1 <-                                             172.18.0.254                        172.18.0.204                        ip:icmp      56   0 no 
sstp-out2                                                                                        0.92      2 ->                                             172.18.0.204                        172.18.0.254                        ip:icmp      56   0 no 
sstp-out2                                                                                        1.92      3 <-                                             172.18.0.254                        172.18.0.204                        ip:icmp      56   0 no 
sstp-out2                                                                                        1.92      4 ->                                             172.18.0.204                        172.18.0.254                        ip:icmp      56   0 no 
sstp-out2                                                                                        2.92      5 <-                                             172.18.0.254                        172.18.0.204                        ip:icmp      56   0 no 
sstp-out2                                                                                        2.92      6 ->                                             172.18.0.204                        172.18.0.254                        ip:icmp      56   0 no 
sstp-out2                                                                                       3.904      7 <-                                             172.18.0.254                        172.18.0.204                        ip:icmp      56   0 no 
sstp-out2                                                                                       3.904      8 ->                                             172.18.0.204                        172.18.0.254                        ip:icmp      56   0 no 
sstp-out2                                                                                           5      9 <-                                             172.18.0.254                        172.18.0.204                        ip:icmp      56   0 no 
sstp-out2                                                                                           5     10 ->                                             172.18.0.204                        172.18.0.254                        ip:icmp      56   0 no 
sstp-out2                                                                                       5.959     11 <-                                             172.18.0.254                        172.18.0.204                        ip:icmp      56   0 no 
sstp-out2                                                                                        5.96     12 ->                                             172.18.0.204                        172.18.0.254                        ip:icmp      56   0 no 
sstp-out2                                                                                        6.92     13 <-                                             172.18.0.254                        172.18.0.204                        ip:icmp      56   0 no 
sstp-out2                                                                                        6.92     14 ->                                             172.18.0.204                        172.18.0.254                        ip:icmp      56   0 no 
sstp-out2                                                                                       7.911     15 <-                                             172.18.0.254                        172.18.0.204                        ip:icmp      56   0 no 
sstp-out2                                                                                       7.912     16 ->                                             172.18.0.204                        172.18.0.254                        ip:icmp      56   0 no 
sstp-out2                                                                                        9.16     17 <-                                             172.18.0.254                        172.18.0.204                        ip:icmp      56   0 no 
sstp-out2                                                                                        9.16     18 ->                                             172.18.0.204                        172.18.0.254                        ip:icmp      56   0 no 
sstp-out2                                                                                       10.12     19 <-                                             172.18.0.254                        172.18.0.204                        ip:icmp      56   0 no 
sstp-out2                                                                                       10.12     20 ->                                             172.18.0.204                        172.18.0.254                        ip:icmp      56   0 no 

 
sindy
Forum Guru
Forum Guru
Posts: 4218
Joined: Mon Dec 04, 2017 9:19 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Tue Jan 14, 2020 7:20 pm

In this case, and supposing the outer policy routing (of the SSTP transport packets) is correct as well, the only thing left which comes to my mind is that the processing of the shutdown of the LTE takes so much CPU that the handling of the pings is affected.

How many pings are lost on the tunnel which stays up? If more than two (assuming you send them in one second intervals), you might be able to spot a spike in CPU load if you run /tool profile (while connected to the Tik using its LAN interface) and disable one of the LTEs. But it sounds weird to me, given that the CPU is not among the weakest ones used.

Also, if you run the tool sniffer pointing to the internal IP of the tunnel which remains up, you may see whether the ping requests and/or responses are really lost or just delayed (if a response is delayed past the expectation window, the recipient ignores it). Plus I would do the same also at server side.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
leon84
Member Candidate
Member Candidate
Topic Author
Posts: 184
Joined: Wed Dec 02, 2009 12:15 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Tue Jan 14, 2020 7:36 pm

In this case, and supposing the outer policy routing (of the SSTP transport packets) is correct as well, the only thing left which comes to my mind is that the processing of the shutdown of the LTE takes so much CPU that the handling of the pings is affected.

How many pings are lost on the tunnel which stays up? If more than two (assuming you send them in one second intervals), you might be able to spot a spike in CPU load if you run /tool profile (while connected to the Tik using its LAN interface) and disable one of the LTEs. But it sounds weird to me, given that the CPU is not among the weakest ones used.

Also, if you run the tool sniffer pointing to the internal IP of the tunnel which remains up, you may see whether the ping requests and/or responses are really lost or just delayed (if a response is delayed past the expectation window, the recipient ignores it). Plus I would do the same also at server side.
Dear sindy,
I noticed this one: If I run a continuos ping from sstp-server to 172.18.0.203 and 172.18.0.204 and try to shutdown lte interface I have no problem.
If i try to do lte shutdown without a continous ping I have interruption on both sstp tunnel also if other LTE is up.
 
sindy
Forum Guru
Forum Guru
Posts: 4218
Joined: Mon Dec 04, 2017 9:19 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Tue Jan 14, 2020 7:42 pm

What exactly means "continuous ping"? Something like interval=50ms? If you don't run this "continuous ping" while shutting down the LTE, how do you check the transparency of the second tunnel?
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
leon84
Member Candidate
Member Candidate
Topic Author
Posts: 184
Joined: Wed Dec 02, 2009 12:15 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Tue Jan 14, 2020 7:51 pm

What exactly means "continuous ping"? Something like interval=50ms? If you don't run this "continuous ping" while shutting down the LTE, how do you check the transparency of the second tunnel?
Sorry it is strange: now I have problem also with continous ping.

Then: I'm connected to sstp-server trhough winbox and I opened 2 terminals. On one terminal I launched simply ping 172.18.0.203 and on other one I launched ping 172.18.0.204. Both ping correctly.
172.18.0.203 and 172.18.0.204 are the same remote mikrotik device. Then I connected to it locally trhough ethernet interface and when I shutdown one lte interface some time it is all ok but in other case when I shutdown for example only one lte interface both ping goes in timeout.
 
leon84
Member Candidate
Member Candidate
Topic Author
Posts: 184
Joined: Wed Dec 02, 2009 12:15 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Wed Jan 15, 2020 1:06 pm

Dear sindy,
another log file information:
When I shutdown interface lte1, sstp-out1 goes down (and it is ok) but also sstp-out2 for connection timeout:
12:02:36 interface,info lte1 link down 
12:02:36 system,info device changed by admin 
12:02:36 dhcp,info dhcp-client on lte1 lost IP address xxx.xxx.xxx.xxx - lease stopped locally 
12:03:00 sstp,ppp,info sstp-out1: terminating... - connection timeout 
12:03:00 sstp,ppp,info sstp-out1: disconnected 
12:03:00 sstp,ppp,info sstp-out1: initializing... 
12:03:00 sstp,ppp,info sstp-out1: connecting... 
12:03:13 sstp,ppp,info sstp-out2: terminating... - connection timeout 
12:03:13 sstp,ppp,info sstp-out2: disconnected 

 
sindy
Forum Guru
Forum Guru
Posts: 4218
Joined: Mon Dec 04, 2017 9:19 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Wed Jan 15, 2020 9:25 pm

So are you really sure that the transport connections (TCP sessions to ports 443 and 444 of the SSTP server) do use the paths you expect them to use? Use the tool sniffer again while both LTEs are up and check that packets to/from SSTP server's IP address and port 443 use LTE1 and packets to/from SSTP server's IP address and port 444 use LTE2.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
leon84
Member Candidate
Member Candidate
Topic Author
Posts: 184
Joined: Wed Dec 02, 2009 12:15 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Wed Jan 15, 2020 10:14 pm

So are you really sure that the transport connections (TCP sessions to ports 443 and 444 of the SSTP server) do use the paths you expect them to use? Use the tool sniffer again while both LTEs are up and check that packets to/from SSTP server's IP address and port 443 use LTE1 and packets to/from SSTP server's IP address and port 444 use LTE2.
Sure. Each tcp session use different lte interface.
 
sindy
Forum Guru
Forum Guru
Posts: 4218
Joined: Mon Dec 04, 2017 9:19 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Wed Jan 15, 2020 11:32 pm

So are you really sure that the transport connections (TCP sessions to ports 443 and 444 of the SSTP server) do use the paths you expect them to use?
Sure. Each tcp session use different lte interface.
OK. Before I start more speculations, try to replace the two type=unreachable routes with routing-mark=to-44x by type=blackhole ones, change their distance to 2, and try again.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
leon84
Member Candidate
Member Candidate
Topic Author
Posts: 184
Joined: Wed Dec 02, 2009 12:15 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Thu Jan 16, 2020 10:25 am

So are you really sure that the transport connections (TCP sessions to ports 443 and 444 of the SSTP server) do use the paths you expect them to use?
Sure. Each tcp session use different lte interface.
OK. Before I start more speculations, try to replace the two type=unreachable routes with routing-mark=to-44x by type=blackhole ones, change their distance to 2, and try again.
I tried but It's the same ... I think there is something that drop tcp session 443 or 444 when I shutdown an interface lte...
 
leon84
Member Candidate
Member Candidate
Topic Author
Posts: 184
Joined: Wed Dec 02, 2009 12:15 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Thu Jan 16, 2020 1:50 pm

Dear sindy,
I think I solved! :-)

I disabled this src-nat rule in table table :
ip firewall nat print 
Flags: X - disabled, I - invalid, D - dynamic 
 0 X  chain=srcnat action=masquerade log=no log-prefix="" 

Probably when there was an interface shutdown it has using not correct ip address to src-nat connection with remote tcp 443 or 444 and then vpn tunnel with remote server did go down.
What do you think?
 
leon84
Member Candidate
Member Candidate
Topic Author
Posts: 184
Joined: Wed Dec 02, 2009 12:15 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Thu Jan 16, 2020 2:54 pm

I'm doing a lot of test and now I can confirm you that without src-nat it run correctly :-). But with this configuration I noticed that:

1) when an lte interface goes down and then become up the relative sstp-out interface can use also 2 minute before restabilish connection. In the log I see sstp-out[1-2] connecting ... and it seems to await a lot before retry.
2) in the previous configuration when LTE interface goes up rhe sstp-out interface become up after about 5-6 seconds.

Is it possible there is some NAT cache for example?
 
sindy
Forum Guru
Forum Guru
Posts: 4218
Joined: Mon Dec 04, 2017 9:19 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Thu Jan 16, 2020 3:26 pm

This was the speculation I mentioned yesterday. There is no "NAT cache", but the NAT is a property of a connection. So the NAT tables of the firewall are only consulted when the first packet of each connection is handled; the NAT processing of all the subsequent packets of the same connection depends on what the handling of the first packet has set up in that connection's properties. But since you use masquerade, as soon as the interface through which a connection was established loses or changes its IP address (or goes down), all connections using that address as reply-dst (src-nat IP) are removed. So the subsequent packet of the same TCP session eventually creates another connection, but at first place it should not be routed anywhere thanks to the blackhole routes. And even if it is routed via a wrong LTE (and thus a wrong IP), the server should just ignore it.

So as you describe it, it sounds as if when any of the interfaces loses/changes its IP, all connections whose NAT handling has been triggered by action=masquerade get dropped, i.e. also those unrelated to the interface which went down.

I'm not sure how it is possible that it works without the srcnat rules, given that you assign the routing-mark in chain=postrouting - for locally originated packets, the route is chosen, and thus a local source address assigned, already before the routing-mark is assigned, so the src-nat (or masquerade) should be necessary for the packets to leave with the proper source IP because the marked route changes the gateway but not the source address.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
leon84
Member Candidate
Member Candidate
Topic Author
Posts: 184
Joined: Wed Dec 02, 2009 12:15 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Thu Jan 16, 2020 3:43 pm

I understand your explanation and this is very strange. Now the only problem I have is that after an lte interface become up relative sstp interface has a delay of about 2 minutes ....

Who is online

Users browsing this forum: Bing [Bot], George90 and 129 guests