Request: FEC tunnel types

I don’t know (or care) about how LTE handles loss, etc, but I think we’ve got offtopic comparing this to LTE.
LTE is just one type of Internet access and is irrelevant to the topic at hand IMHO.

I can see this technology having significant benefits even if your uplinks are fiber based with 99.999% uptime.

The internet is a best effort network. If a destination is reachable OK (=no packet loss/jitter) on one ISP now, might not work OK later, while at the same time the same destination might be perfectly reachable without any issues from another ISP.

Having a solution like this is perfect for those times when your primary ISP has an issue, or more commonly the path between you and the destination has an issue through your primary ISP.
At the moment there is no fully automated method in ROS to switch from a bad uplink (bad as in slight packet loss or increased latency somewhere between you and the destination - but still functional as far as ROS is concerned) to a backup/good one.
You either have to resort to cumbersome scripts and/or netwatch or simply wait for a problem to rise and manually intervene.

So +1 from me for integrating FEC tunnel to ROS.

I guess what you guys missed is that I’m using this to great effect already, I just want to get it included in Mikrotik to remove an extra piece of hardware and simplfy design.

The internet itself is slightly lossy. Pick ANY connection and ping essentially any off-net resource and wait. Their will be a packet lost at some point and there will be a high latency packet at some point.

If I have a circuit that is doing 40ms to an EC2 instance, I can tune my FEC parameters to queue 40ms of packets and if anything is lost or just late beyond the buffer, rebuild. That random packet that takes 160ms is because of packet loss and another link in the path rebuilding it in it’s EC code (most likely), but I don’t care and I’d be just as happy to drop that late packet as get it late because FEC will allow me to rebuild it at 80ms per the example above.

LTE is completely irrelevant to the conversation. It’s the equivalent of saying you have a perfect driveway with no cracks and if there are cracks they are instantly rebuilt and claiming that means you’ll have a smooth ride to work everyday as a result. The path between A and Z is unknown and you’re focusing on the path from A-B only, the rest of the path could change at any point, and there are points [B-Y].* in between that could be saturated at any point along the way causing buffer bloat and latency spikes, even outages that have to be routed around.

If we make some assumptions:
80% of packets will arrive on time on at least ONE of the links on the network.

That 80% will be spread somewhat evenly over the 100%, ie packet loss and latency spikes are generally random
OR
A link fails causing a number of sequentially dropped packets and a failover event.

So, if I plan on 20% FEC redundancy I should be able to rebuild missing packets from the random loss/jitter AND handle the lost packets that would cause a failover event.

There is no ‘double fec’ because for VoIP needs, all EC in the middle is useless if I have end-to-end FEC. I’d rather forget that packet and rebuild it with RS than wait for it. That makes my latency predictable, just a function of the % of packets queued in my FEC buffer. If typical latency is 40ms, FEC will increase that to say 55ms

Any argument specifying any specific links in the path is moot, it’s the aggregate of all links, most of the completely unknown. MOST of the data path wont have any substantial EC.

Right now over a tinyfecvpn link I can get 65ms from a site through to an EC2 instance consistently with not more than about 20ms jitter (FEC takes ~20ms to rebuild a packet) and I can failover from cable to LTE without loosing a packet. Latency moves up to about 160ms with that same 20ms jitter until the cable comes back and it gracefully moves back down (no packets lost in the route update). I can yank the ethernet cable out of the cable modem and I get something like 65ms,65ms, 84ms, 65ms, 85ms, 180ms, 160ms, 160ms. Looking at a pcap of the SIP and RTP traffic, it matches with no lost packets in the sequence. Plug back in and wait for my tunnel to come back up and it drops right back down to 65ms. Customers on the phone don’t lose a call or lose a bit of audio. They might be able to perceive that latency move up but it’s still within comfortable VoIP margins.

The problem is the data path is more complex than necessary. phone > computer(tinyfecvpn) > mikrotik w/ dual wan and dual tunnels configured for failover > internet > CHR in EC2 > instance w/ tinyfecvpn > PBX

Secondly, I’m taking a hit on MTU putting a tunnel in a tunnel.

This works, and it works very well. It’s just missing in routeros.

While I agree in principal, I’m running 2 tunnels and BGP over them. The ‘unlimited’ WAN connection gets BFD. The metered backup doesn’t. <500ms failover time.

It would be really nice to see an implementation of multi-hop BFD or something that works similarly over WAN links. That’s another topic/request though.

Drones with video downlink used for control (vs. using it to send nice video streams to viewers on internet) is usually still done using analog video over FM links, because any appreciable latency (which you normally have when doing a digital solution) makes it difficult to control the drone.

Yes, I also use BFD the same way. It works ok for when a link goes completely down, but when there’s packet loss it either flaps up and down over and over, or doesn’t even bother (if the loss is too low). So, again manual intervention is needed for cases like that.
I haven’t used tinyfecvpn, but from what I understand, it can handle this exact scenario more gracefully. Correct?

P.S.: I think I saw a multihop BFD option on the v7.0 beta that was released a few days ago.

Yes, the FEC tunnel should smooth all that out.

Also, I would suggest altering your BFD timers so it’s less willing to consider a link up.

With FEC, that flapping is just an annoyance in your logs and maybe some extra data usage. End users wouldn’t really see a difference.

Again totally agree this approach be useful. As noted, it’s working for @syadnom. Some builtin approach to trading latency for reliability to even out the performance of divergent upstream path would be handy.

With Mikrotik making more mobile devices with multiple uplink paths – e.g. the newer LtAP that has two miniPCIe slots (e.g. 2 LTE modems), and the M33 RB953 have for a long time…the use cases described in the thread are going to be more common.

I did read that OpenVPN support in ROSv7 includes UDP tunneling now, so one step closer…and MBIM support in ROSv7 will open the door to more modern LTE modems, …but if you run into intermittent packet loss, there isn’t a good solution builtin to Mikrotik that can “smooth” that out (with FEC being one approach).

speaking of openvpn, https://community.openvpn.net/openvpn/ticket/949
there’s a FEC enabled build with results. This is exactly what I’m wanting to see.

telecommunications companies would really appreciate this feature and it would be a huge differentiator for Mikrotik.
If you don’t fully understand why folks are asking for this and what FEC can do to improve voice & other streaming applications, please please please go look up silver-peak - they do a pretty good job explaining it here: https://www.silver-peak.com/sd-wan/voice-video-over-broadband. Once you understand the benefits of FEC, whether for a single WAN or dual WAN, the debate whether to add it or not should be over and the new debate should be… how quickly can we get it added.
Personally, I’d love to see FEC available for any and all tunnel types that can support it - specifically, l2tp/ipsec if at all possible. OpenVPN would be a good start though.

1 Like

agreed. I’m thinking something almost like ‘ip packing’ where you add it to an existing tunnel.

Pepwave is getting pretty serious in the product lines as well.

The key here is ‘end to end’, not just some EC on any given leg of the link, but between two devices that may have many paths including failover links.

What would be absolutely sick is a new wireguard tunnel and FEC on that.