Community discussions

 
leon84
Member Candidate
Member Candidate
Topic Author
Posts: 155
Joined: Wed Dec 02, 2009 12:15 pm

RBM33G with 2 LTE module (how to manage failover)

Tue Jul 10, 2018 2:12 pm

Dear all,
I'm using RBM33G with 2 Huawei 909s-120 LTE module. I would to configure a failover internet connection on lte. Could you help me?
I tried this one https://wiki.mikrotik.com/wiki/Advanced ... _Scripting but I didn't help me because with LTE interface I don't have a default gateway specified as ip address but only with name of the interface. In my case lte1 and lte2.
I tried also with static route and different distance but It run correctly only if the lte interface goes down. In many case the interface remain up but I don't have internet connectivity and then the failover doesn't run.
Have you some suggestion?
Thanks in advance.
 
krafg
Frequent Visitor
Frequent Visitor
Posts: 77
Joined: Sun Jun 28, 2015 7:36 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Wed Jul 11, 2018 10:42 pm

Play with route distances. 1 for main LTE module and 2 for backup LTE module.

Regards.
 
sindy
Forum Guru
Forum Guru
Posts: 2514
Joined: Mon Dec 04, 2017 9:19 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Wed Jul 11, 2018 11:17 pm

To what I know you cannot do that without scripting. You can use various setups to force pinging of some reference IP addresses on the internet via lte1 and other reference IP addresses via lte2 so that you knew that not only the link is up but you can actually get somewhere, but for gateways other than IP addresses, it is not possible to translate this information into route availability without scripting.

The script might look like this.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
leon84
Member Candidate
Member Candidate
Topic Author
Posts: 155
Joined: Wed Dec 02, 2009 12:15 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Thu Jul 12, 2018 9:55 am

Dear all,
I realized a little script to manage failover but on lte interface is more complicated.
I have a configure 2 routes and some firewall rule to allow ping to 8.8.8.8 only for LTE1 interface and 8.8.4.4 for LTE2 interface.
Then I have 2 default routes, one through LTE1 (with distance 1 because for me is primary link) and another one through LTE2 (with distance of 2).
With a script and the scheduler I check ping to 8.8.8.8 and 8.8.4.4 every 1 minute and if one of them fails I disable relative default route. For example if 8.8.8.8 ping fails I disable default route on LTE1 interface.
Where is the problem? I have to contact from remote my RBM33G and I'm not using SIM that allow bidirectional traffic on their APN. Then my routerboard can generate trafffic to INTERNET but it is not contactable from internet. In the past I used specific APN (more expensive for application machine 2 machine) but now I implemented a VPN with SSTP protocol.
When I switch from LTE1 to LTE2 the sstp client on my rotuerboard disconnect and immediatly create a new connection. Because I use RBM33G for transmit real data stream this disconnections can create some problem. Often disconnection of LTE1 interface can be very short and then the connection swith on LTE2 interface and after few seconds (when LTE1 become up) another one on LTE1. This create 2 sstp client disconnection. In case of using only one sim (in this case) I would have had only one disconnection :-).
Have you some idea?
 
sindy
Forum Guru
Forum Guru
Posts: 2514
Joined: Mon Dec 04, 2017 9:19 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Thu Jul 12, 2018 1:53 pm

Since years the problem of short outages of the primary path and the associated transient effects is solved by a "WTR timer" (Wait to Restore) which is used to delay the swicthover back to the primary path after it becomes OK. This is normally used to avoid falling back on false OKs, but could help a bit to you in terms that the total time of the outage would be shorter.

However, my approach would be to use one SSTP tunnel via each LTE interface and use a failover between these two tunnels for the critical data. It could then detect failures much faster than once in a minute.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
leon84
Member Candidate
Member Candidate
Topic Author
Posts: 155
Joined: Wed Dec 02, 2009 12:15 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Thu Jul 12, 2018 2:00 pm

Since years the problem of short outages of the primary path and the associated transient effects is solved by a "WTR timer" (Wait to Restore) which is used to delay the swicthover back to the primary path after it becomes OK. This is normally used to avoid falling back on false OKs, but could help a bit to you in terms that the total time of the outage would be shorter.

However, my approach would be to use one SSTP tunnel via each LTE interface and use a failover between these two tunnels for the critical data. It could then detect failures much faster than once in a minute.
Thanks a lot for your suggestion! A question: if I implement 2 sstp tunnel I have 2 different ip address client side. right?
 
sindy
Forum Guru
Forum Guru
Posts: 2514
Joined: Mon Dec 04, 2017 9:19 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Thu Jul 12, 2018 4:02 pm

Yes, you need to create two SSTP client interfaces, each will have another user and thus get another IP addresses. Only if each SSTP client would connect to a different SSTP server, there would be a danger that both interfaces would get the same IP address assigned.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
leon84
Member Candidate
Member Candidate
Topic Author
Posts: 155
Joined: Wed Dec 02, 2009 12:15 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Thu Jul 12, 2018 4:19 pm

Actually I have a service that acquire realtime data from remote RBM33G through a unique SSTP Tunnel. This service is connected to a remote ip address 192.168.254.10 (sstp interface).
I could implement second tunnel and assign to another sstp interface another ip, for example 192.168.254.9.
My service will continue to try to connect to 192.168.254.10 that is down. In this case I should implement also a procedure server side to swith to other ip address. Right?
Since years the problem of short outages of the primary path and the associated transient effects is solved by a "WTR timer" (Wait to Restore) which is used to delay the swicthover back to the primary path after it becomes OK. This is normally used to avoid falling back on false OKs, but could help a bit to you in terms that the total time of the outage would be shorter.

However, my approach would be to use one SSTP tunnel via each LTE interface and use a failover between these two tunnels for the critical data. It could then detect failures much faster than once in a minute.
Thanks a lot for your suggestion! A question: if I implement 2 sstp tunnel I have 2 different ip address client side. right?
 
sindy
Forum Guru
Forum Guru
Posts: 2514
Joined: Mon Dec 04, 2017 9:19 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Thu Jul 12, 2018 4:57 pm

Both tunnels will be internally on private addresses, so no NAT will interfere with routing operation. This permits you to create a bridge with no member ports on the remote device (the one with 2 LTE uplinks), assign a third IP address to that bridge, and on the SSTP server side, create two routes to that third address, each using another SSTP client interface address as a gateway.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
leon84
Member Candidate
Member Candidate
Topic Author
Posts: 155
Joined: Wed Dec 02, 2009 12:15 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Thu Jul 12, 2018 5:19 pm

Please check if I understood:
I create 2 sstp-client interface and sstp server asssign (for example) 192.168.254.9 at one and 192.168.254.10 at another one.
Then I create a bridge and add to the bridge both lte interface; I assign to the bridge an ip address for example 192.168.254.7.

On server side I create 2 route for 192.168.254.7 using sstp interface connected to 192.168.254.9 and interface connected to 192.168.254.10.

My service will contact always 192.168.254.7 that will be contactet trough sstp tunnel 1 or sstp tunnel 2 in function for example of a route distance. When an sstp tunnel go down the relative route will be disabled and the traffic go to other sstp tunnel.

Right? Sorry but in this case I could create also a simple bridge without any interface or a loopback interface. Right?

Both tunnels will be internally on private addresses, so no NAT will interfere with routing operation. This permits you to create a bridge with no member ports on the remote device (the one with 2 LTE uplinks), assign a third IP address to that bridge, and on the SSTP server side, create two routes to that third address, each using another SSTP client interface address as a gateway.
 
sindy
Forum Guru
Forum Guru
Posts: 2514
Joined: Mon Dec 04, 2017 9:19 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Thu Jul 12, 2018 5:55 pm

Your idea assumes creation of a WAN bridge on L2, but SSTP tunnels do not transport L2 frames, only L3 packets. So if you would like to take that way, you would have to set up EoIP tunnels inside the SSTP tunnels, i.e. the server would also have to be a Mikrotik device, and you'd waste a painful amount of packet space on tunnel headers. But yes, in such case, you could create a bond of the two EoIP tunnel interfaces on each end, or bridge them together with STP protocol enabled at the bridge.

My suggestion was an L3 one. I'm not sure how exactly SSTP tunnels work so maybe the two client addresses cannot be in the same subnet, but that's not the key here, so let's assume they can be 192.168.254.9 and 192.168.254.10 as you suggest.

The key is that the bridge at the client side would have no member ports at all (neither physical nor virtual), it will only be created in order to hold an IP address unrelated to any existing subnet, such as 10.22.33.44/32, and the routes at the server side will say (using Mikrotik syntax):

/ip route
add dst=address=10.22.33.44/32 distance=1 gateway=192.168.254.9
add dst=address=10.22.33.44/32 distance=2 gateway=192.168.254.10


At the client side, the routes will be

/ip route
add dst=address=the.data.collector.ip/32 distance=1 gateway=sstp-out1
add dst=address=the.data.collector.ip/32 distance=2 gateway=sstp-out2


There seems to be nothing to configure about tunnel state monitorinig on the /interface sstp-client, so either it is always on or it doesn't exist at all. Depending on that, you may need to use a script to monitor both tunnels' transparency at both ends and disable the route if the tunnel gets down.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
leon84
Member Candidate
Member Candidate
Topic Author
Posts: 155
Joined: Wed Dec 02, 2009 12:15 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Thu Jul 12, 2018 6:12 pm

Perfect I think to understand.
Then I should force sstp-out1 to use only lte1 and sstp-out2 to use only lte2. Right?
In which way? With policy based route?

Please check if I understood:
I create 2 sstp-client interface and sstp server asssign (for example) 192.168.254.9 at one and 192.168.254.10 at another one.
Then I create a bridge and add to the bridge both lte interface; I assign to the bridge an ip address for example 192.168.254.7.

On server side I create 2 route for 192.168.254.7 using sstp interface connected to 192.168.254.9 and interface connected to 192.168.254.10.

My service will contact always 192.168.254.7 that will be contactet trough sstp tunnel 1 or sstp tunnel 2 in function for example of a route distance. When an sstp tunnel go down the relative route will be disabled and the traffic go to other sstp tunnel.

Right? Sorry but in this case I could create also a simple bridge without any interface or a loopback interface. Right?

Both tunnels will be internally on private addresses, so no NAT will interfere with routing operation. This permits you to create a bridge with no member ports on the remote device (the one with 2 LTE uplinks), assign a third IP address to that bridge, and on the SSTP server side, create two routes to that third address, each using another SSTP client interface address as a gateway.
 
sindy
Forum Guru
Forum Guru
Posts: 2514
Joined: Mon Dec 04, 2017 9:19 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Thu Jul 12, 2018 6:29 pm

Then I should force sstp-out1 to use only lte1 and sstp-out2 to use only lte2. Right?
In which way? With policy based route?
This is the question I was afraid of :-D

One of problems associated to use of SSTP is that there is no way to configure anything but the remote address and port in Mikrotik's SSTP client configuration, so you have nothing but the dst-address and dst-port as a base for policy-based routing (a simple one, using just mark=routing rules in chain=output and the routes with routing-marks and a distance=2 route with type=blackhole to prevent failover if the required WAN is down).

So on the server, you have to set port forwarding from some other port to the one on which it normally listens for incoming SSTP connections, so that it would effectively accept them on both.

If it is not possible, there is a way to achieve that effect on the client side, but it is a brain damaging one.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
leon84
Member Candidate
Member Candidate
Topic Author
Posts: 155
Joined: Wed Dec 02, 2009 12:15 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Thu Jul 12, 2018 6:57 pm

:-)

Then I can use another port on server side and redirect traffic to the standard 443 port of sstp-server. In this way I could force sstp-out1 on lte1 and sstp-out2 on lte2.
But the very important question is: suppose I configured all correctly; I have my service connected to 192.168.254.7; the traffic go from my service to sstp-server (that has 2 route for 192.168.254.7) and through the route with minor distance. If this route goes down, sstp server select the other route for 192.168.254.7 that use other lte interface on client side. In this case I have in any way an interruption of tcp traffic from my service to 192.168.254.7 or it is transparent?




Then I should force sstp-out1 to use only lte1 and sstp-out2 to use only lte2. Right?
In which way? With policy based route?
This is the question I was afraid of :-D

One of problems associated to use of SSTP is that there is no way to configure anything but the remote address and port in Mikrotik's SSTP client configuration, so you have nothing but the dst-address and dst-port as a base for policy-based routing (a simple one, using just mark=routing rules in chain=output and the routes with routing-marks and a distance=2 route with type=blackhole to prevent failover if the required WAN is down).

So on the server, you have to set port forwarding from some other port to the one on which it normally listens for incoming SSTP connections, so that it would effectively accept them on both.

If it is not possible, there is a way to achieve that effect on the client side, but it is a brain damaging one.
 
sindy
Forum Guru
Forum Guru
Posts: 2514
Joined: Mon Dec 04, 2017 9:19 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Thu Jul 12, 2018 9:11 pm

suppose I configured all correctly; I have my service connected to 192.168.254.7; the traffic go from my service to sstp-server (that has 2 route for 192.168.254.7) and through the route with minor distance. If this route goes down, sstp server select the other route for 192.168.254.7 that use other lte interface on client side. In this case I have in any way an interruption of tcp traffic from my service to 192.168.254.7 or it is transparent?
There are two factors:
  • there are no settings in the SSTP configuration related to keepalives, so I don't know how quickly a failure of the SSTP tunnel will be detected and whether it will be detected at all before sending of a payload packet fails. So it may happen that the tunnel interface will be up long after the carrier path goes down, and only the first transport packet not to get acknowledged within timeout will cause the tunnel interface state change to down. If so, the TCP protocol in the payload will be heavily affected - google for "tcp meltdown". That's why I wrote before about "one of problems associated to SSTP".
  • RouterOS uses a routing cache which is on by default. I assume that if the gateway in use becomes unavailable, the cached route is replaced by a new one, but I'm not sure, so maybe you'll have to disable routing cache.
If not for the above - yes, even if each of the packets of the same TCP connection would randomly choose one of the tunnels, as there is no NAT, the receiving side wouldn't notice a difference, except that as the paths are very likely to have a different transport delay, the packets would be coming in shuffled order.

So all in all I'm afraid that the TCP session providing the SSTP transport will give up retransmitting and shut the tunnel interface down at about the same time when the payload TCP session will also give up retransmitting, so the route failover will come too late for the payload session to survive. But that's a property of all VPN solutions relying on TCP transport. So if devices at both ends of the tunnels are under your control, you may prefer to use IPsec to create the tunnels, as it uses transport protocols without retransmission control and you can use e.g. pinging of the remote end to detect a breakdown of the tunnel much faster than in the SSTP case. This way, the payload TCP session would encounter some lost packets but wouldn't break unless both paths would be down.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
leon84
Member Candidate
Member Candidate
Topic Author
Posts: 155
Joined: Wed Dec 02, 2009 12:15 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Fri Jul 13, 2018 9:40 am

Dear sindy,
congratulations for your knowledge :-). Great!
I would to do a test also if I prefer a more simple configuration :-).
In any case I have sstp server on TCP port 443 and 444 (When I receive SYN on 444 I forward it to 444).
Then in client side I have sstp-out1 connected to 443 and sstp-out2 connected to 444. Now I'm trying to force sstp-out1 to use only lte1 and sstp-out2 to use only lte2. And I have some problem.
I configured a routing mark on mangle table.
In prerouting chain I have dst-address=public ip of sstp server, protocol=tcp and dst-port 443 and in this case the action is to mark-routing=to-443
A similar rule for the same dst-address, same protocol but dst-port=444 and in this case the action is to mark-routing=to-444
Then in routing table I added to rules for the same destination address (the sstp server public ip) but with different gateway interface (lte1 for one with routing-mark to-443 and lte2 for other one with routing mark to-444). Both sstp-out interface are connected but through the same lte interface. Where is the issue?
 
sindy
Forum Guru
Forum Guru
Posts: 2514
Joined: Mon Dec 04, 2017 9:19 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Fri Jul 13, 2018 9:50 am

The issue is that mangle rules for packets originating from the Mikrotik itself have to be placed in chain=output.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
leon84
Member Candidate
Member Candidate
Topic Author
Posts: 155
Joined: Wed Dec 02, 2009 12:15 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Fri Jul 13, 2018 10:00 am

Perfect. I add mangle rules in output chain and now I can see traffic counter on these rules but now only sstp-out1 can connect to server because it use default gateway. Then I disabled default gateway but there are the 2 specific rules to sstp-server public ip but they (sstp-out1 and sstp-out2) can't connect.
 
sindy
Forum Guru
Forum Guru
Posts: 2514
Joined: Mon Dec 04, 2017 9:19 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Fri Jul 13, 2018 10:06 am

Show me /ip route print detail and /ip firewall mangle print.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
leon84
Member Candidate
Member Candidate
Topic Author
Posts: 155
Joined: Wed Dec 02, 2009 12:15 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Fri Jul 13, 2018 10:19 am

[admin@MikroTik] > ip firewall mangle print
Flags: X - disabled, I - invalid, D - dynamic
0 chain=output action=mark-routing new-routing-mark=to-443 passthrough=no protocol=tcp dst-address=SSTP-SERVER-IP-PUBLIC dst-port=443 log=no log-prefix=""

1 chain=output action=mark-routing new-routing-mark=to-444 passthrough=no protocol=tcp dst-address=SSTP-SERVER-IP-PUBLIC dst-port=444 log=no log-prefix=""


[admin@MikroTik] > ip route print detail
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit
0 A S dst-address=sstp-server-ip-public/32 gateway=lte1 gateway-status=lte1 reachable distance=1 scope=30 target-scope=10 routing-mark=to-443

1 A S dst-address=sstp-server-ip-public/32 gateway=lte2 gateway-status=lte2 reachable distance=1 scope=30 target-scope=10 routing-mark=to-444

2 A S ;;; default-lte2
dst-address=0.0.0.0/0 gateway=lte2 gateway-status=lte2 reachable distance=1 scope=30 target-scope=10

3 S ;;; default-lte1
dst-address=0.0.0.0/0 gateway=lte1 gateway-status=lte1 reachable distance=3 scope=30 target-scope=10

4 A S dst-address=8.8.4.4/32 gateway=lte2 gateway-status=lte2 reachable distance=1 scope=30 target-scope=10

5 A S dst-address=8.8.8.8/32 gateway=lte1 gateway-status=lte1 reachable distance=1 scope=30 target-scope=10

6 ADC dst-address=10.92.119.88/30 pref-src=10.92.119.89 gateway=lte1 gateway-status=lte1 reachable distance=0 scope=10

7 ADC dst-address=10.122.174.196/30 pref-src=10.122.174.197 gateway=lte2 gateway-status=lte2 reachable distance=0 scope=10

8 ADC dst-address=192.168.0.0/24 pref-src=192.168.0.1 gateway=bridge1 gateway-status=bridge1 reachable distance=0 scope=10

9 ADC dst-address=192.168.254.1/32 pref-src=192.168.254.9 gateway=sstp-out2 gateway-status=sstp-out2 reachable distance=0 scope=10
 
sindy
Forum Guru
Forum Guru
Posts: 2514
Joined: Mon Dec 04, 2017 9:19 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Fri Jul 13, 2018 10:33 am

No idea, everything seems correct to me. Try /ip firewall connection remove [find dst-address~"^SSTP-SERVER-IP-PUBLIC:44[34]"], but it is just a blind shot.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
leon84
Member Candidate
Member Candidate
Topic Author
Posts: 155
Joined: Wed Dec 02, 2009 12:15 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Fri Jul 13, 2018 11:08 am

Please,
consider this new routing table because I thinl there is some issue here. I have always the same mangle routing mark as previous scenario but this new routing table. In this case if I disable "default-lte1" it use only default-lte2 and the only interface can connect is sstp-out2. If I disable "default-lte2" the only interface can connect is "sstp-out1".

[admin@MikroTik] > ip route print detail
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit
0 A S dst-address=0.0.0.0/0 gateway=lte1 gateway-status=lte1 reachable distance=1 scope=30 target-scope=10 routing-mark=to-443

1 A S dst-address=0.0.0.0/0 gateway=lte2 gateway-status=lte2 reachable distance=1 scope=30 target-scope=10 routing-mark=to-444

2 A S ;;; default-lte1
dst-address=0.0.0.0/0 gateway=lte1 gateway-status=lte1 reachable distance=3 scope=30 target-scope=10

3 X S ;;; default-lte2
dst-address=0.0.0.0/0 gateway=lte2 gateway-status=lte2 inactive distance=3 scope=30 target-scope=10

4 A S dst-address=8.8.4.4/32 gateway=lte2 gateway-status=lte2 reachable distance=1 scope=30 target-scope=10

5 A S dst-address=8.8.8.8/32 gateway=lte1 gateway-status=lte1 reachable distance=1 scope=30 target-scope=10

6 ADC dst-address=10.92.119.88/30 pref-src=10.92.119.89 gateway=lte1 gateway-status=lte1 reachable distance=0 scope=10

7 ADC dst-address=10.122.174.196/30 pref-src=10.122.174.197 gateway=lte2 gateway-status=lte2 reachable distance=0 scope=10

8 ADC dst-address=192.168.0.0/24 pref-src=192.168.0.1 gateway=bridge1 gateway-status=bridge1 reachable distance=0 scope=10

9 ADC dst-address=192.168.254.1/32 pref-src=192.168.254.10 gateway=sstp-out1 gateway-status=sstp-out1 reachable distance=0 scope=10
 
leon84
Member Candidate
Member Candidate
Topic Author
Posts: 155
Joined: Wed Dec 02, 2009 12:15 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Fri Jul 13, 2018 11:14 am

Dear sindy,
I think I solved in this way:

[admin@MikroTik] > ip route print detail
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit
0 A S dst-address=0.0.0.0/0 gateway=lte1 gateway-status=lte1 reachable distance=1 scope=30 target-scope=10 routing-mark=to-443

1 A S dst-address=0.0.0.0/0 gateway=lte2 gateway-status=lte2 reachable distance=1 scope=30 target-scope=10 routing-mark=to-444

2 A S dst-address=0.0.0.0/0 gateway=lte1,lte2 gateway-status=lte1 reachable,lte2 reachable distance=1 scope=30 target-scope=10

3 A S dst-address=8.8.4.4/32 gateway=lte2 gateway-status=lte2 reachable distance=1 scope=30 target-scope=10

4 A S dst-address=8.8.8.8/32 gateway=lte1 gateway-status=lte1 reachable distance=1 scope=30 target-scope=10

5 ADC dst-address=10.92.119.88/30 pref-src=10.92.119.89 gateway=lte1 gateway-status=lte1 reachable distance=0 scope=10

6 ADC dst-address=10.122.174.196/30 pref-src=10.122.174.197 gateway=lte2 gateway-status=lte2 reachable distance=0 scope=10

7 ADC dst-address=192.168.0.0/24 pref-src=192.168.0.1 gateway=bridge1 gateway-status=bridge1 reachable distance=0 scope=10

8 ADC dst-address=192.168.254.1/32 pref-src=192.168.254.9 gateway=sstp-out2,sstp-out1 gateway-status=sstp-out2 reachable,sstp-out1 reachable distance=0 scope=10
 
sindy
Forum Guru
Forum Guru
Posts: 2514
Joined: Mon Dec 04, 2017 9:19 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Fri Jul 13, 2018 11:28 am

Please follow the instructions in my automatic signature... the mangle rules and routes as they were in the previous post seemed correct to me, so maybe there is something elsewhere in the configuration. I suppose you don't use any special handling on the server side (like port-forwarding only if the packet comes fom a particular IP address).

The way you've modified the default route causes even connections to use lte1 and odd connections use lte2 or vice versa, so it is not a solution of the issue, it just hides it.

Another point, once the issue of not establishing a connection via the marked routes is resolved, is that you have to prevent the SSTP sessions from getting established via the "wrong" WAN. The routing normally uses a fallback to routing table "main" (consisting of routes without routing-mark) if no active route with the routing-mark is found; you have to prevent this behaviour, because if the proper WAN is down, all routes via that WAN become inactive so the above would happen. So add also the following routes:
dst-address=0.0.0.0/0 routing-mark=to-443 distance=2 type=unreachable
dst-address=0.0.0.0/0 routing-mark=to-444 distance=2 type=unreachable
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
leon84
Member Candidate
Member Candidate
Topic Author
Posts: 155
Joined: Wed Dec 02, 2009 12:15 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Fri Jul 13, 2018 12:00 pm

Dear sindy,
resetted router and reconfigured it for simplicity. Now this is the configuration and the only interface connected is sstp-out1
/interface bridge
add fast-forward=no name=bridge1
add fast-forward=no name=bridge2
/interface lte apn
add apn=mobile.vodafone.it name=lte1
add apn=mobile.vodafone.it name=lte2
/interface lte
set [ find ] apn-profiles=lte1 mac-address=02:1E:10:1F:00:00 name=lte1
set [ find ] apn-profiles=lte2 mac-address=02:1E:10:1F:00:00 name=lte2
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip hotspot profile
set [ find default=yes ] html-directory=flash/hotspot
/interface sstp-client
add connect-to=sstp-server-public-ip disabled=no keepalive-timeout=10 name=sstp-out1 profile=default-encryption user=client1 verify-server-certificate=yes
add connect-to=sstp-server-public-ip:444 disabled=no keepalive-timeout=10 name=sstp-out2 profile=default-encryption user=client1bk verify-server-certificate=yes
/interface bridge port
add bridge=bridge1 interface=ether1
add bridge=bridge1 interface=ether2
add bridge=bridge1 interface=ether3
/ip address
add address=192.168.0.1/24 interface=bridge1 network=192.168.0.0
add address=192.168.254.7 interface=bridge2 network=192.168.254.7
/ip firewall mangle
add action=mark-routing chain=output dst-address=sstp-server-public-ip dst-port=443 new-routing-mark=to-443 passthrough=no protocol=tcp
add action=mark-routing chain=output dst-address=sstp-server-public-ip dst-port=444 new-routing-mark=to-444 passthrough=no protocol=tcp
/ip route
add distance=1 gateway=lte1 routing-mark=to-443
add distance=2 routing-mark=to-443 type=unreachable
add distance=1 gateway=lte2 routing-mark=to-444
add distance=2 routing-mark=to-444 type=unreachable
add distance=1 gateway=lte1
add distance=1 gateway=lte2
/ip service
set www port=8080
/system clock
set time-zone-name=Europe/Rome
/system ntp client
set enabled=yes primary-ntp=193.204.114.232 secondary-ntp=93.204.114.233
/system routerboard settings
set silent-boot=no
 
sindy
Forum Guru
Forum Guru
Posts: 2514
Joined: Mon Dec 04, 2017 9:19 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Fri Jul 13, 2018 12:12 pm

It cannot be simpler... what is the software version and what does /ip firewall connection print detail say?
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
sindy
Forum Guru
Forum Guru
Posts: 2514
Joined: Mon Dec 04, 2017 9:19 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Fri Jul 13, 2018 12:35 pm

Well, one more idea... you probably have to add a srcnat rule. I don't know at what step the local address for the outgoing packets is chosen, so maybe the packets are sent via lte2 with source IP address matching lte1.

/ip firewall nat
add chain=srcnat action=masquerade out-interface=lte1
add chain=srcnat action=masquerade out-interface=lte2
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
leon84
Member Candidate
Member Candidate
Topic Author
Posts: 155
Joined: Wed Dec 02, 2009 12:15 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Fri Jul 13, 2018 12:38 pm

I'm using RouterOS 6.42.5 and this is output of ip firewall connections:

[admin@MikroTik] >  /ip firewall connection print detail 
Flags: E - expected, S - seen-reply, A - assured, C - confirmed, D - dying, F - fasttrack, s - srcnat, d - dstnat 
 0    C     protocol=udp src-address=169.254.125.216:53141 dst-address=255.255.255.255:20561 reply-src-address=255.255.255.255:20561 reply-dst-address=169.254.125.216:53141 timeout=9s orig-packets=238 580 orig-bytes=14 629 788 orig-fasttrack-packets=0 
            orig-fasttrack-bytes=0 repl-packets=0 repl-bytes=0 repl-fasttrack-packets=0 repl-fasttrack-bytes=0 orig-rate=8.3kbps repl-rate=0bps 

 1  SAC     protocol=tcp src-address=10.92.119.89:41381 dst-address=sstp-server-public-ip:443 reply-src-address=sstp-server-public-ip:443 reply-dst-address=10.92.119.89:41381 tcp-state=established timeout=23h59m59s orig-packets=3 851 orig-bytes=496 389 
            orig-fasttrack-packets=0 orig-fasttrack-bytes=0 repl-packets=8 969 repl-bytes=783 323 repl-fasttrack-packets=0 repl-fasttrack-bytes=0 orig-rate=1224bps repl-rate=2.0kbps 

 2    C     protocol=tcp src-address=10.92.119.89:58355 dst-address=sstp-server-public-ip:444 reply-src-address=sstp-server-public-ip:444 reply-dst-address=10.92.119.89:58355 tcp-state=syn-sent timeout=2s orig-packets=2 orig-bytes=120 orig-fasttrack-packets=0 
            orig-fasttrack-bytes=0 repl-packets=0 repl-bytes=0 repl-fasttrack-packets=0 repl-fasttrack-bytes=0 orig-rate=0bps repl-rate=0bps 
[admin@MikroTik] > 

 
sindy
Forum Guru
Forum Guru
Posts: 2514
Joined: Mon Dec 04, 2017 9:19 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Fri Jul 13, 2018 12:41 pm

This supports the idea in my post above, so add the masquerade rules and try again.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
leon84
Member Candidate
Member Candidate
Topic Author
Posts: 155
Joined: Wed Dec 02, 2009 12:15 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Fri Jul 13, 2018 12:55 pm

Great! Now is running correctly! Now I try with failover! Thanks a lot!
This supports the idea in my post above, so add the masquerade rules and try again.
 
sindy
Forum Guru
Forum Guru
Posts: 2514
Joined: Mon Dec 04, 2017 9:19 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Fri Jul 13, 2018 1:20 pm

The explanation is that the packet is first routed before passing through the mangle rules, and gets a source address corresponding to the route. Then, the "routing adjustment" (see this picture) takes place if a mangle rule assigns a routing-mark, but the source address remains the initially assigned one unless it is explicitly changed in the next step using a srcnat rule. In most setups everything that leaves the box via the WANs is srcnated because traffic forwarded from LAN wouldn't ever get any responses otherwise, so this subtle aspect of redirection of local output traffic remains unnoticed.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
leon84
Member Candidate
Member Candidate
Topic Author
Posts: 155
Joined: Wed Dec 02, 2009 12:15 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Fri Jul 13, 2018 1:25 pm

Very good! Now I have both sstp-out connected but I noticed that I have more latency in ping from 192.168.254.1 (sstp-server private ip in the tunnel) and 192.168.254.9 or 192.168.254.10. I have a latency of about 700ms but if I disable one of the sstp-out the latency go to about 70 ms. Have you some explaination?
 
sindy
Forum Guru
Forum Guru
Posts: 2514
Joined: Mon Dec 04, 2017 9:19 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Fri Jul 13, 2018 2:24 pm

None if the effect is symmetric (both tunnel 1 and tunnel 2 give 70 ms while the other one is disabled, and you only get 700 ms if both are enabled).

Also, I don't know how the routing looks like when both tunnels are up (/ip route print detail).
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
leon84
Member Candidate
Member Candidate
Topic Author
Posts: 155
Joined: Wed Dec 02, 2009 12:15 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Fri Jul 13, 2018 3:38 pm

Dear sindy,
this is route print whene both sstp interface are connected.
[admin@MikroTik] > ip route print detail 
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit 
 0 A S  dst-address=0.0.0.0/0 gateway=lte1 gateway-status=lte1 reachable distance=1 scope=30 target-scope=10 routing-mark=to-443 

 1   SU dst-address=0.0.0.0/0 type=unreachable distance=2 routing-mark=to-443 

 2 A S  dst-address=0.0.0.0/0 gateway=lte2 gateway-status=lte2 reachable distance=1 scope=30 target-scope=10 routing-mark=to-444 

 3   SU dst-address=0.0.0.0/0 type=unreachable distance=2 routing-mark=to-444 

 4 A S  dst-address=0.0.0.0/0 gateway=lte1 gateway-status=lte1 reachable distance=1 scope=30 target-scope=10 

 5   S  dst-address=0.0.0.0/0 gateway=lte2 gateway-status=lte2 reachable distance=1 scope=30 target-scope=10 

 6 ADC  dst-address=10.8.28.60/30 pref-src=10.8.28.62 gateway=lte2 gateway-status=lte2 reachable distance=0 scope=10 

 7 ADC  dst-address=10.64.118.204/30 pref-src=10.64.118.206 gateway=lte1 gateway-status=lte1 reachable distance=0 scope=10 

 8 ADC  dst-address=192.168.0.0/24 pref-src=192.168.0.1 gateway=bridge1 gateway-status=bridge1 reachable distance=0 scope=10 

 9 ADC  dst-address=192.168.254.0/24 pref-src=192.168.254.7 gateway=bridge2 gateway-status=bridge2 reachable distance=0 scope=10 

10 ADC  dst-address=192.168.254.1/32 pref-src=192.168.254.10 gateway=sstp-out1,sstp-out2 gateway-status=sstp-out1 reachable,sstp-out2 reachable distance=0 scope=10 

 
leon84
Member Candidate
Member Candidate
Topic Author
Posts: 155
Joined: Wed Dec 02, 2009 12:15 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Fri Jul 13, 2018 3:49 pm

I attach you also sstp-server route print :-) :
[admin@SSTP-SERVER] > ip route print detail 
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit 
 0 A S  dst-address=0.0.0.0/0 gateway=public-ip gateway-status=public-ip reachable via  ether1 distance=1 scope=30 target-scope=10 

 1 A S  dst-address=192.168.254.7/32 gateway=sstp-client1bk,sstp-client1 gateway-status=sstp-client1bk reachable,sstp-client1 reachable distance=1 scope=30 target-scope=10 

 2 ADC  dst-address=192.168.254.9/32 pref-src=192.168.254.1 gateway=sstp-client1bk gateway-status=sstp-client1bk reachable distance=0 scope=10 

 3 ADC  dst-address=192.168.254.10/32 pref-src=192.168.254.1 gateway=sstp-client1 gateway-status=sstp-client1 reachable distance=0 scope=10 

 
sindy
Forum Guru
Forum Guru
Posts: 2514
Joined: Mon Dec 04, 2017 9:19 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Fri Jul 13, 2018 7:58 pm

I'm afraid I'd start from assigning distinct local and remote addresses to the two SSTP user "secrets", so that the dynamically generated routes would not merge into one, and place these addresses outside any server or client subnet, to act only as interconnection networks. And then I would create routes via these interconnection networks (with different distance values) to other local address on the opposite router so that I could choose the usage strategy of these routes, instead of having to use round-robin as two point-to-point connected subnets with the same remote IP address inevitably cause.

And finally I would test pinging to the destination address of these routes, not any of the addresses associated to the tunnel interfaces, from source address different from those associated to tunnel interfaces.

You can use tunnel interface names as route gateways, so using distinct tunnel interface addresses at server side may not be necessary to split the routes to non-tunnel addresses, but doing so doesn't allow you to use the scriptless failover based on recursive next-hop.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
leon84
Member Candidate
Member Candidate
Topic Author
Posts: 155
Joined: Wed Dec 02, 2009 12:15 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Mon Jul 16, 2018 12:43 pm

Dear sindy,
now I have two tunnel sstp in this way:
sstp-out1 with ip address 192.168.254.10 and remtoe sstp server as 192.168.254.1
sstp-out2 with ip address 192.168.254.9 and remote sstp server as 192.168.254.2

Now the route table of client is:
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit 
 0 A S  dst-address=0.0.0.0/0 gateway=lte1 gateway-status=lte1 reachable distance=1 scope=30 target-scope=10 routing-mark=to-443 

 1   SU dst-address=0.0.0.0/0 type=unreachable distance=2 routing-mark=to-443 

 2 A S  dst-address=0.0.0.0/0 gateway=lte2 gateway-status=lte2 reachable distance=1 scope=30 target-scope=10 routing-mark=to-444 

 3   SU dst-address=0.0.0.0/0 type=unreachable distance=2 routing-mark=to-444 

 4 A S  dst-address=0.0.0.0/0 gateway=lte1 gateway-status=lte1 reachable distance=1 scope=30 target-scope=10 

 5   S  dst-address=0.0.0.0/0 gateway=lte2 gateway-status=lte2 reachable distance=1 scope=30 target-scope=10 

 6 ADC  dst-address=10.68.177.64/29 pref-src=10.68.177.68 gateway=lte1 gateway-status=lte1 reachable distance=0 scope=10 

 7 ADC  dst-address=10.119.181.56/30 pref-src=10.119.181.57 gateway=lte2 gateway-status=lte2 reachable distance=0 scope=10 

 8 ADC  dst-address=192.168.0.0/24 pref-src=192.168.0.1 gateway=bridge1 gateway-status=bridge1 reachable distance=0 scope=10 

 9 ADC  dst-address=192.168.254.0/24 pref-src=192.168.254.7 gateway=bridge2 gateway-status=bridge2 reachable distance=0 scope=10 

10 ADC  dst-address=192.168.254.1/32 pref-src=192.168.254.10 gateway=sstp-out1 gateway-status=sstp-out1 reachable distance=0 scope=10 

11 ADC  dst-address=192.168.254.2/32 pref-src=192.168.254.9 gateway=sstp-out2 gateway-status=sstp-out2 reachable distance=0 scope=10 

And the route table of sstp-server is:
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit 
 0 A S  dst-address=0.0.0.0/0 gateway=public-ip gateway-status=public-ip reachable via  ether1 distance=1 scope=30 target-scope=10 

 1 A S  dst-address=192.168.254.7/32 gateway=sstp-client1 gateway-status=sstp-client1 reachable distance=1 scope=30 target-scope=10 

 2   S  dst-address=192.168.254.7/32 gateway=sstp-client1bk gateway-status=sstp-client1bk reachable distance=2 scope=30 target-scope=10 

 3 ADC  dst-address=192.168.254.9/32 pref-src=192.168.254.2 gateway=sstp-client1bk gateway-status=sstp-client1bk reachable distance=0 scope=10 

 4 ADC  dst-address=192.168.254.10/32 pref-src=192.168.254.1 gateway=sstp-client1 gateway-status=sstp-client1 reachable distance=0 scope=10 

 5 ADC  dst-address=public-subnet/27 pref-src=public-ip gateway=ether1 gateway-status=ether1 reachable distance=0 scope=10 
[admin@SSTP-SERVER] > 

About latency sorry but it was the provider connection. In any case I ask me if there is a really advantage in this configuration because it introduce a major complex client configuration. Whene there is an lte down and the relative sstp-out interface the sstp-server wait for sstp keepalive (configured to 10s) and then swith to the other route. When interface come back up it newly switch to primary route and in this case there isn't any disconnection. It is gret but as I mentioned it introduce a more complex client configuration and I think to come back at initial configuration. :-) THANKS A LOT
 
sindy
Forum Guru
Forum Guru
Posts: 2514
Joined: Mon Dec 04, 2017 9:19 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Mon Jul 16, 2018 1:04 pm

Redundancy always comes at a cost of complexity. So it is your decision what you prefer. I prefer simplicity of maintenance to simplicity of configuration.

And most important for your case, regardless what redundancy model you finally choose, I hope that the two Vodafone SIMs were used only for testing and for production you'll use SIMs from different operators not sharing network infrastructure (which, I admit, may not be an easy task in Italy in particular).
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
leon84
Member Candidate
Member Candidate
Topic Author
Posts: 155
Joined: Wed Dec 02, 2009 12:15 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Mon Jul 16, 2018 1:08 pm

Oh yes I used 2 Vodafone sim only as test. I will use 2 different operators like Vodafone and TIM that have indipendent network infrastructure.

Redundancy always comes at a cost of complexity. So it is your decision what you prefer. I prefer simplicity of maintenance to simplicity of configuration.

And most important for your case, regardless what redundancy model you finally choose, I hope that the two Vodafone SIMs were used only for testing and for production you'll use SIMs from different operators not sharing network infrastructure (which, I admit, may not be an easy task in Italy in particular).
 
leon84
Member Candidate
Member Candidate
Topic Author
Posts: 155
Joined: Wed Dec 02, 2009 12:15 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Mon Jul 16, 2018 3:23 pm

Sorry cindy,
do you know what is the correct way to distribute some routes when an sstp tunnel come up ? There is the parameter "routes" in PPP -> Secrects but it doesn't distribute the route to to remote sstp client? In which way Can I specify route in this parameter?
 
sindy
Forum Guru
Forum Guru
Posts: 2514
Joined: Mon Dec 04, 2017 9:19 pm

Re: RBM33G with 2 LTE module (how to manage failover)

Tue Jul 17, 2018 12:23 am

I've never tried that, but as far as I know, the route parameter in the /ppp secret is there to add routes at the server, not at the client. The PPP (which constitutes the transport underneath SSTP) can configure only the default route, and although it should be possible to use DHCP over a PPP channel to deliver a routing table to the client, Mikrotik seems to ignore the relevant RFC, insisting that DHCP can only be used on L2 interfaces.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.

Who is online

Users browsing this forum: No registered users and 40 guests