I'm using a kit called tinyfecvpn, it's based on UDPSpeeder which is a UDP tunnel with forward error correction via reed-solomon encoding.

I would love to see a tunnel type that included FEC. Maybe a check box in EoIP and GRE tunnels for this.

The usefulness of FEC on tunnels is as follows, especially when combined with dynamic routing.

Redundant paths with failover. FEC can handle very slow packets and lost packets to make the failover process incredibly smooth. I've tested this with clients on either side of mikrotik's with a WAN and an LTE path using OSPF and BFD (EoIP tunnels) and tinyfecvpn. Pull the WAN, LTE takes over within a second or so and tinyfecvpn re-assembles the lost-in-transit packets for a zero downtime transition.

This is a feature other products are built around (think peplink).

Another mode is simple WAN smoothing. Maybe you have a lossy link that drops 5% of packets or has latency spikes. tinyfecvpn can be configured for virtually any amount of redundancy up to your latency ceiling. lost packets are re-assembled, but late packets are also reassembled by specifying max jitter. Setting it at 8ms would rebuild any packet that is 8ms behind the last instead of waiting. I've tested this with 100% redundancy and you really can have 50-60% packet loss completely absorbed into the tunnel.

Anyone with hosted VoIP could use this. I'm using GCE for 3CX and FreePBX installs and a raspberry pi with tinyfecvpn on-site linking up to a GCE hosted server. I route the local subnet to the PBX subnet over tinyfecvpn and have rock solid links. No more choppy VoIP from a little packet loss or jitter, it's gone.

This seems like something a lot of people could use, especially those using VoIP. No interest?

Really like the idea! Thanks for pointing out that an implementation already exists!!

Interesting idea...

I can see FEC-based tunneling might help with LTE, where there is little you can do to "tweak" the network to avoid packet loss, and FEC tunneling might help deal smooth out packet loss exposed due to RF reception issues (but not congestion issues...). Essentially it be "double FEC" since the LTE layer has already applied FEC to the LTE packets before decapsulating them to IP for the Mikrotik.

But the usefulness might limited: for FEC to help, the Layer 3 packet loss have a somewhat predicable rate, otherwise you'd still have packet loss, or just be wasting bandwidth with unneeded FEC codes. Also, if the packet loss is actually related to congestion, FEC actually would be harmful over other approaches...And for VoIP quality issues, there are already a variety (too many in fact) to fix them than implementing FEC-based UDP tunneling as some magic panacea. For example, making sure RTCP is utilized by SIP and/or your media server would help, since that give real-time feed back to RTP based on *actual* network conditions vs. a fixed FEC rate. Also, not sure how it's widely supported in IP PBXes, but RFC 5109 describes using FEC inside the RTP stream (e.g. at the ISO Layer 4 level): https://tools.ietf.org/html/rfc5109 (also in WebRTC, see https://tools.ietf.org/html/draft-ietf-rtcweb-fec-07 )...

Now where FEC-based UDP tunnel in a Mikrotik makes a lot more sense is potentially avoiding TCP retransmissions across a lossy network (such as LTE), since typical TCP congestion control mechanism kinda assume packet loss is always related to congestion.

When errors are unavoidable and not related to congestion, certainly FEC has a lot of merit but it's ain't just a checkbox on a EoIP tunnel...

I would think using a fullblown FEC like reed-solomon is a bit wasteful for this purpose, as you can expect that you receive either correct packets or nothing.
Sure the reed-solomon code would be able to fix some corrupted packets, but do they really occur in practice? As written above, that is already covered at lower layers.

What you need here is more like a parity scheme like RAID-5.

tinyfecvpn is great and yes, I would love to see it in the ROS
There are many scenarios where you may find it useful: VOIP, GSM/LTE, links with bit of interference and packet loss, mesh networking, AP handovers, link redundancy

Used it to get live video from the portable sender as it was roaming across multiple APs. Normally this is a problem because roaming is not seamless and you get some UDP packet loss. But with FEC, this can be handled perfectly with zero loss and you have total control of maximum latency and other settings.
I also experimented with RAW mode where sender was just spitting out UDP packets with no AP association (sending RAW wlan frames from custom program running on OpenWRT) and having multiple receivers running in wlan monitor mode, waiting for the data and forwarding it to the server for reassembly and FEC processing. This worked really well for 4K video streaming. There are some proprietary video streaming systems that work like this costing a lot of $$$$, but thanks to the tinyfec you can make one for cheap. And this is just one niche use...

I think many would avoid running VoIP trough tunneling (because it will ad a new layer of jitter and delay). Re-transmisson should always be handled in the application layer, since the application is the only one that knows what to be re-sent.

But I's a cool solution indeed.

I agree that the application should be the 'smart' part. But in the case of Wifi, there's CSMA that wreaks havoc on any sort of predictability...
UDP for instance has a checksum(optional) but no CRC. some implementations of L0 discard them, some pass along the degraded packets.
There's no black and white here, and i think this may be a great tool to play with.

Hearing the monitor mode history got me just the more interested in this thing. Gonna have to mess with it a bit now.

@Amm0: what makes you claim that LTE is lossy?

@mkx, let's call it "noisy" - mainly suggest that with LTE, or Wi-Fi, the L2/L1 stuff that deals with the noise (e.g. ACM) does have side-effects at L3, and that's where FEC might be able to mitigate frame loss. @rOOt points "a bit of interference and packet loss" and "AP handovers" as example and those be helped by tinyfecvpn's approach. Now if the RF space LTE is using is "clean" (say, directly licensed), tinyfecvpn would seem to little help.

I think many would avoid running VoIP trough tunneling (because it will ad a new layer of jitter and delay). Re-transmisson should always be handled in the application layer, since the application is the only one that knows what to be re-sent.

Seem like a bad idea for VoIP...Kinda why I commented here in the first place... But I'm old school on VoIP: DSCP marking on L2, some kinda queue on L3 – and let the jitter buffer to do it's job on L4+.

tinyfecvpn is great and yes, I would love to see it in the ROS

+1

Related questions, I'm curious if anyone here has tried to using to address the kinda problems @rOOt describes above?

Was thinking Mikrotik could add FEC (e.g. reed-solomon or similar) to /ip packing... Never used the feature myself (but now curious) but would seem to solve similar problems to tinyfecvpn. And, If you think about it since /ip packing supports IP header (and body?) compression already...that reduces the bits need to be carried over a link, which could help offset the additional bandwidth needed by FEC parity bits.

/ip packing adds latency as noted in the wiki...but again that's the same bargin tinyfecvpn and UDPspeeder offer — trade latency/bandwidth for more consistent data stream at L3. But judging from the dirth of information on /ip packing, seem like it pretty disfavored by Mikrotik/others for some reason?

IP compression (packing) was useful in the past, when network traffic often consisted of uncompressed data.
E.g. in my network it would be SMB traffic reading uncompressed files, or HTML traffic. These would be highly compressible and there would be some gain.

Today most traffic is encrypted or at least already compressed, and network data compression brings nothing anymore.

I’m not sure why the narrow focus on the last mile.

MOST isps are going to be some 6-15 hops on 3-5 different carries to get to a remote resource.

So who cares that LTE has built in error correction that converts loss into latency.

The delivery tech has no bearing on the goal.

End to end FEC to reduce packet loss and latency spikes across the entire path.

LTE has a lot of built-in latency due to ptmp scheduling (i run LTE APs...) and when there is some retransmission the latency spikes significantly. Those systems are tuned for data, not real-time communications.

If LTE rebuilds a packer on a link and adds 80ms latency, that packet is as good as lost as far as I’m concerned.

Agree with @mkx on LTE – it's extremely robust – and the usefulness of "double FEC" is rightfully questionable. If your WISP, carrier, or ISP deploying a network and want to use an FEC tunnel that be crazy talk.

I don't mean to pick on LTE here again...an FEC feature be potentially useful when you don't control the links between your network tunnels, and can't just fix some underlying problem that's causing packet loss at L3. This could be any type of link, although wireless links would be more likely.

But still think @rOOt's live video using someone else's LTE network "use case" is where an FEC-enabled tunnel on a Mikrotik might have some value. No doubt overall packet loss for LTE is low, but in dense environment with a lot of LTE users, say in an indoor sport arena, I can say packet loss for LTE becomes more than just 0.001%. Packet loss for an UDP MPEG-TS stream means a poorer quality picture, and most video equipment can't just use TCP so simply. While FEC adds latency&overhead, broadcast video latency tolerances are all built around satellite links (500ms) [and the sat link is where FEC is applied by DVB-S], so if an FEC tunnel over LTE adds say 50-100ms latency and end-to-end IP packet loss is lower because of added FEC, that be a win since the receiving end has a higher latency tolerance the LTE. So the "belt-and-suspenders approach" of adding additional FEC (on top LTE's "built-in" FEC) in very congested LTE environments where this might make some sense...even if it ain't pretty.

That is a pretty specialist use case which would be better solved in a dedicated LTE router which also has multiple SIMs and radios.
You can undoubtedly get these on the market already.

We also came forward to many "unusual" requests from some customers (radio and TV stations to mention some) ...

I thought that is what he was referring to. These days Electronic News Gathering is often done using portable setups with like 4 LTE radios+cards to transmit the live video, instead of a satellite truck.
For sure there is equipment to do that out of the box. This is not a market for $100-$400 boxes like MikroTik.

I see two really good and interesting use cases:

1) VOIP over challenging links. Like from a car, driving in a country side, where LTE may not be available and drops down to 3G or GSM. This is challenging even for VoLTE and normal cellphone handovers. But it could be even as simple thing as trying to use VOIP in a crowded hotel on a public wifi. Packet loss will be horrible and then you may add VPN as well, making any missing packets even worse...
With FEC, you can trade a bit of extra latency for perfect transmission. Set it to 100ms latency + big overhead, so that any lost or late packets are reconstructed from remaining data. This works really well and bit of latency is worth the improvement in overall quality.

2) Video streaming. This is not as niche application as it may seems. People really like to stream various events to Youtube, be it conferences, school or company events... You don't need super expensive camera gear either, you can get close to cinematic quality from $5000 gear. But often upload is where things break. Professional solutions use multiple LTE modems in parallel, with FEC for redundancy. And with FEC tunnel, you can easily use multiple connections at the same time for seamless handover and reliability. You can put together setup that's as good as the expensive professional broadcast solution with OpenWRT router, LTE modems with unlimited data plans and a bit of know-how.
And then you have drones and UAVs, this market is booming and sending live video is one of the biggest challenges. Mikrotik could really gain some share here because current solutions are often very bandwidth limited (ugly overcompressed video) or very short-range. And people want to stream 4K stereoscopic video at 60fps or simply multiple video streams at once (wideangle camera, zoomed one, thermal IR, ...). Perfect use case for 60GHz stuff if the handovers worked reliably and fast enough, maybe with 5GHz as a backup or long range. And that's where FEC tunnel might help. Also for signal diversity (multiple receivers on ground, like sector antennas) and redundancy (backup radio with seamless switch-over) etc. Demoing system like this, in nice ruggedized version, on one of the military trade shows, would pretty much guarantee "shut up and take my money" reactions, no matter how crazy the price would be... there is currently nothing close commercially available.

I don't know (or care) about how LTE handles loss, etc, but I think we've got offtopic comparing this to LTE.
LTE is just one type of Internet access and is irrelevant to the topic at hand IMHO.

I can see this technology having significant benefits even if your uplinks are fiber based with 99.999% uptime.

The internet is a best effort network. If a destination is reachable OK (=no packet loss/jitter) on one ISP now, might not work OK later, while at the same time the same destination might be perfectly reachable without any issues from another ISP.

Having a solution like this is perfect for those times when your primary ISP has an issue, or more commonly the path between you and the destination has an issue through your primary ISP.
At the moment there is no fully automated method in ROS to switch from a bad uplink (bad as in slight packet loss or increased latency somewhere between you and the destination - but still functional as far as ROS is concerned) to a backup/good one.
You either have to resort to cumbersome scripts and/or netwatch or simply wait for a problem to rise and manually intervene.

So +1 from me for integrating FEC tunnel to ROS.

I guess what you guys missed is that I'm using this to great effect already, I just want to get it included in Mikrotik to remove an extra piece of hardware and simplfy design.

The internet itself is slightly lossy. Pick ANY connection and ping essentially any off-net resource and wait. Their will be a packet lost at some point and there will be a high latency packet at some point.

If I have a circuit that is doing 40ms to an EC2 instance, I can tune my FEC parameters to queue 40ms of packets and if anything is lost or just late beyond the buffer, rebuild. That random packet that takes 160ms is because of packet loss and another link in the path rebuilding it in it's EC code (most likely), but I don't care and I'd be just as happy to drop that late packet as get it late because FEC will allow me to rebuild it at 80ms per the example above.

LTE is completely irrelevant to the conversation. It's the equivalent of saying you have a perfect driveway with no cracks and if there are cracks they are instantly rebuilt and claiming that means you'll have a smooth ride to work everyday as a result. The path between A and Z is unknown and you're focusing on the path from A-B only, the rest of the path could change at any point, and there are points [B-Y].* in between that could be saturated at any point along the way causing buffer bloat and latency spikes, even outages that have to be routed around.

If we make some assumptions:
80% of packets will arrive on time on at least ONE of the links on the network.

That 80% will be spread somewhat evenly over the 100%, ie packet loss and latency spikes are generally random
*OR*
A link fails causing a number of sequentially dropped packets and a failover event.

So, if I plan on 20% FEC redundancy I should be able to rebuild missing packets from the random loss/jitter AND handle the lost packets that would cause a failover event.

There is no 'double fec' because for VoIP needs, all EC in the middle is useless if I have end-to-end FEC. I'd rather forget that packet and rebuild it with RS than wait for it. That makes my latency predictable, just a function of the % of packets queued in my FEC buffer. If typical latency is 40ms, FEC will increase that to say 55ms

Any argument specifying any specific links in the path is moot, it's the aggregate of all links, most of the completely unknown. MOST of the data path wont have any substantial EC.

Right now over a tinyfecvpn link I can get 65ms from a site through to an EC2 instance consistently with not more than about 20ms jitter (FEC takes ~20ms to rebuild a packet) and I can failover from cable to LTE without loosing a packet. Latency moves up to about 160ms with that same 20ms jitter until the cable comes back and it gracefully moves back down (no packets lost in the route update). I can yank the ethernet cable out of the cable modem and I get something like 65ms,65ms, 84ms, 65ms, 85ms, 180ms, 160ms, 160ms. Looking at a pcap of the SIP and RTP traffic, it matches with no lost packets in the sequence. Plug back in and wait for my tunnel to come back up and it drops right back down to 65ms. Customers on the phone don't lose a call or lose a bit of audio. They might be able to perceive that latency move up but it's still within comfortable VoIP margins.

The problem is the data path is more complex than necessary. phone > computer(tinyfecvpn) > mikrotik w/ dual wan and dual tunnels configured for failover > ~~internet~~ > CHR in EC2 > instance w/ tinyfecvpn > PBX

Secondly, I'm taking a hit on MTU putting a tunnel in a tunnel.

This works, and it works very well. It's just missing in routeros.

At the moment there is no fully automated method in ROS to switch from a bad uplink (bad as in slight packet loss or increased latency somewhere between you and the destination - but still functional as far as ROS is concerned) to a backup/good one.
You either have to resort to cumbersome scripts and/or netwatch or simply wait for a problem to rise and manually intervene.

While I agree in principal, I'm running 2 tunnels and BGP over them. The 'unlimited' WAN connection gets BFD. The metered backup doesn't. <500ms failover time.

It would be really nice to see an implementation of multi-hop BFD or something that works similarly over WAN links. That's another topic/request though.

And then you have drones and UAVs, this market is booming and sending live video is one of the biggest challenges.

Drones with video downlink used for control (vs. using it to send nice video streams to viewers on internet) is usually still done using analog video over FM links, because any appreciable latency (which you normally have when doing a digital solution) makes it difficult to control the drone.

At the moment there is no fully automated method in ROS to switch from a bad uplink (bad as in slight packet loss or increased latency somewhere between you and the destination - but still functional as far as ROS is concerned) to a backup/good one.
You either have to resort to cumbersome scripts and/or netwatch or simply wait for a problem to rise and manually intervene.

While I agree in principal, I'm running 2 tunnels and BGP over them. The 'unlimited' WAN connection gets BFD. The metered backup doesn't. <500ms failover time.

It would be really nice to see an implementation of multi-hop BFD or something that works similarly over WAN links. That's another topic/request though.

Yes, I also use BFD the same way. It works ok for when a link goes completely down, but when there's packet loss it either flaps up and down over and over, or doesn't even bother (if the loss is too low). So, again manual intervention is needed for cases like that.
I haven't used tinyfecvpn, but from what I understand, it can handle this exact scenario more gracefully. Correct?

P.S.: I think I saw a multihop BFD option on the v7.0 beta that was released a few days ago.

Yes, I also use BFD the same way. It works ok for when a link goes completely down, but when there's packet loss it either flaps up and down over and over, or doesn't even bother (if the loss is too low). So, again manual intervention is needed for cases like that.
I haven't used tinyfecvpn, but from what I understand, it can handle this exact scenario more gracefully. Correct?

P.S.: I think I saw a multihop BFD option on the v7.0 beta that was released a few days ago.

Yes, the FEC tunnel should smooth all that out.

Also, I would suggest altering your BFD timers so it's less willing to consider a link up.

With FEC, that flapping is just an annoyance in your logs and maybe some extra data usage. End users wouldn't really see a difference.

Again totally agree this approach be useful. As noted, it's working for @syadnom. *Some* builtin approach to trading latency for reliability to even out the performance of divergent upstream path would be handy.

With Mikrotik making more mobile devices with multiple uplink paths – e.g. the newer LtAP that has two miniPCIe slots (e.g. 2 LTE modems), and the M33 RB953 have for a long time...the use cases described in the thread are going to be more common.

I did read that OpenVPN support in ROSv7 includes UDP tunneling now, so one step closer...and MBIM support in ROSv7 will open the door to more modern LTE modems, ...but if you run into intermittent packet loss, there isn't a good solution builtin to Mikrotik that can "smooth" that out (with FEC being one approach).

speaking of openvpn, https://community.openvpn.net/openvpn/ticket/949
there's a FEC enabled build with results. This is exactly what I'm wanting to see.

telecommunications companies would really appreciate this feature and it would be a huge differentiator for Mikrotik.
If you don't fully understand why folks are asking for this and what FEC can do to improve voice & other streaming applications, please please please go look up silver-peak - they do a pretty good job explaining it here: https://www.silver-peak.com/sd-wan/voic ... -broadband. Once you understand the benefits of FEC, whether for a single WAN or dual WAN, the debate whether to add it or not should be over and the new debate should be... how quickly can we get it added.
Personally, I'd love to see FEC available for any and all tunnel types that can support it - specifically, l2tp/ipsec if at all possible. OpenVPN would be a good start though.

agreed. I'm thinking something almost like 'ip packing' where you add it to an existing tunnel.

Pepwave is getting pretty serious in the product lines as well.

The key here is 'end to end', not just some EC on any given leg of the link, but between two devices that may have many paths including failover links.

What would be absolutely sick is a new wireguard tunnel and FEC on that.