I’m looking for a bit of shared experience here, from wireless wizards, wherever you may live…
What’s the most links in an end-to-end PtP link that you’ve done? (Mine is 3 -to date)
Each link a different frequency, or did you use WDS and all the same frequency?
What limitations/issues have you discovered?
What air interface did you use? I see nv2 issues mentioned 8 months ago…
Thanks in advance for the input. 
I have six links (seven sites) in a row: A-B-C-D-E-F-G
Total of about 175mi of wireless links.
I don’t use WDS or MPLS. I route each hop and use OSPF to tie it all together. I have a IP tunnel between site A and site G that kicks in when a link goes down. Site A is a cable modem and Site G is Verizon 4G w/ static IP.
I use Nstreme because I like the low latency and jitter. Never had much use for NV2 as I always had enough bandwidth. My application is a private backhaul for my company rather than a WISP.
I started out with RB4xx boards and XR5 cards. I got tired of the power offset circus between UBNT and Mikrotik so I’m slowly migrating my equipment to Metal 5s. So half are now 802.11n and the other half are 802.11a. Never had any problem with 802.11n and nstreme.
I use 40MHz of spectrum for each link and planned it so the two radios at each site do not overlap. There is a 10Mhz or 20Mhz buffer in the middle.
Note that all my equipment is indoors and I use ew63/ew52 waveguide so the radios are much closer to each other than they might be out on the tower.
Great input, Colebert.
a couple of questions…
When you say you have an IP tunnel that kicks in when a link goes down, do you mean a separate land-line link?
Using Metal 5s in 802.11n - no MIMO. If you could get it end-to-end, do you think MIMO could potentially increase performance/throughput? I realize that you still have XR5s in the link as well…
It does seem that Nstreme is the air interface of choice, from what I’m gathering on the forum, for multilink PtP runs. Did you try nv2?
Colebert, if I could ask, what sort of latency and jitter results are you seeing? From your results, do you have thought on what a ‘per link’ approximation of latency and/or jitter might be? If you could guess, what do you think would be the results if you doubled your total links to 12 - end-to-end - from six?
Thanks for the input. My links under consideration are very short - but underground in long corridors ‘channeling’ line-of-sight. Certainly issues with multipath are likely, so frequency selection will be a challenge. (see my post on UNDERGROUND).
Thanks again for the input!
My sites are in a line. A is HQ and G is the farthest site about 200 miles away. At A we have a Cable Modem connected to the router there. At G we have Verizon 4G LTE w/ a static IP connected to the router there. The mikrotik router at A and G connect to each other over these internet connections and form an IP Tunnel. Then I encrypt the tunnel traffic with IPSEC. Then I add the link into the OSPF area and assign it a very high cost. So if any of my six links between A and G go down the network converges and sends traffic over the IP tunnel until the link is restored. Mostly this is only neccessary during brief microwave fades at night or a major power outage.
MIMO would improve performance for sure. I don’t know about reliability. MIMO isn’t much of an option for me because some of my sites are single polarity dishes or other shortcut decisions previous engineering staff made. Other sites I could literally do 6x6 MIMO if the technology existed and was practical. But for my purposes 1x1 is sufficent.
Actually, I think nstreme isn’t the obviously popular protocol. I love it and have had no problems but alot of people here complain about it not working with their 802.11n. I haven’t played with NV2 since the ROS5 beta but I didn’t much care for it. It increased jitter and for my purposes didn’t seem as consistent as nstreme. But if I was trying to get all the bandwidth possible I definitely would be using it. My links are now 100% production so I don’t have much room to fix things that aren’t broken so really I can’t say what advances have been made on nv2 since ROS5.
On my six hop, seven site link roundtrip latency is 7ms with about 800kbps of steady traffic going across the system. As I built out the system, I found each additional hop added about 1ms of latency. Keep in mind most of my hops are about 26mi long. One is 37 and another is 18. This is with nstreme. 802.11n or a made no difference in latency.
I am no expert to be sure. When I first start in 2009 I was a clown. I connected radios back to back to test and knew barely anything. I really only know about things that relate to my situation which is somewhat unique since I am not a WISP and my system is 100% indoor waveguide based.
Ahh, now i understand the IP tunnel. Basically a ring with path weighting. Good idea.
Are you using 5GHz-only-N with Nstreme (1X1)? You’re having no issues with it, but you’ve heard others are? I read in the forum that when nv2 is used for multilink PtP routes, it has problems with N.
Wow - that latency is really low. What was the ‘range’ of latency? Kind of a guess about jitter.
My limited testing so far shows a bit of latency spread on nv2 - not sure yet my setup is giving me solid results but I see a range of nearly double the average - with an occasional result much larger. I’m going to work more with Nstreme - maybe see about Nstreme Dual, although I’m not crazy about the additional complexity and cost.
Great signature line, colebert! Hard to argue with that!
Here’s some Karma for being so helpful - makes forum’s like this work.
5GHz upper band only. Everything is 1x1.
Well, I just see forum post here with people crying about how nstreme doesn’t work with this or nv2 doesn’t work with that. I don’t have a lot of time to test and my application is very limited (5ghz, long range ptp, all indoor) so I just take note of all the noise out there.
I just ran a quick ping test from A to G and am attaching it to the bottom. Right now it’s middle of the day and I’ve got three engineers out in the field screwing around on their PCs and about 1mbps of telemetry data streaming in. Basically, it’s pretty stable around 7ms most of the time with a few jumps occasionally. I can totally saturate it and push 50mbps through end to end on a btest and the latency only goes up to a stable 40-50ms.
Pinging 192.168.201.1 with 32 bytes of data:
Reply from 192.168.201.1: bytes=32 time=8ms TTL=51
Reply from 192.168.201.1: bytes=32 time=7ms TTL=51
Reply from 192.168.201.1: bytes=32 time=7ms TTL=51
Reply from 192.168.201.1: bytes=32 time=12ms TTL=51
Reply from 192.168.201.1: bytes=32 time=7ms TTL=51
Reply from 192.168.201.1: bytes=32 time=6ms TTL=51
Reply from 192.168.201.1: bytes=32 time=7ms TTL=51
Reply from 192.168.201.1: bytes=32 time=7ms TTL=51
Reply from 192.168.201.1: bytes=32 time=7ms TTL=51
Reply from 192.168.201.1: bytes=32 time=7ms TTL=51
Reply from 192.168.201.1: bytes=32 time=7ms TTL=51
Reply from 192.168.201.1: bytes=32 time=7ms TTL=51
Reply from 192.168.201.1: bytes=32 time=33ms TTL=51
Reply from 192.168.201.1: bytes=32 time=7ms TTL=51
Reply from 192.168.201.1: bytes=32 time=7ms TTL=51
Reply from 192.168.201.1: bytes=32 time=8ms TTL=51
Reply from 192.168.201.1: bytes=32 time=10ms TTL=51
Reply from 192.168.201.1: bytes=32 time=10ms TTL=51
Reply from 192.168.201.1: bytes=32 time=11ms TTL=51
Reply from 192.168.201.1: bytes=32 time=8ms TTL=51
Reply from 192.168.201.1: bytes=32 time=7ms TTL=51
Reply from 192.168.201.1: bytes=32 time=7ms TTL=51
Reply from 192.168.201.1: bytes=32 time=6ms TTL=51
Reply from 192.168.201.1: bytes=32 time=7ms TTL=51
Reply from 192.168.201.1: bytes=32 time=7ms TTL=51
Reply from 192.168.201.1: bytes=32 time=8ms TTL=51
Reply from 192.168.201.1: bytes=32 time=33ms TTL=51
Reply from 192.168.201.1: bytes=32 time=7ms TTL=51
Reply from 192.168.201.1: bytes=32 time=6ms TTL=51
Reply from 192.168.201.1: bytes=32 time=8ms TTL=51
Reply from 192.168.201.1: bytes=32 time=8ms TTL=51
Reply from 192.168.201.1: bytes=32 time=7ms TTL=51
Reply from 192.168.201.1: bytes=32 time=7ms TTL=51
Reply from 192.168.201.1: bytes=32 time=6ms TTL=51
Reply from 192.168.201.1: bytes=32 time=8ms TTL=51
Reply from 192.168.201.1: bytes=32 time=7ms TTL=51
Reply from 192.168.201.1: bytes=32 time=7ms TTL=51
Reply from 192.168.201.1: bytes=32 time=6ms TTL=51
Reply from 192.168.201.1: bytes=32 time=7ms TTL=51
Reply from 192.168.201.1: bytes=32 time=8ms TTL=51
Reply from 192.168.201.1: bytes=32 time=7ms TTL=51
Reply from 192.168.201.1: bytes=32 time=27ms TTL=51
Reply from 192.168.201.1: bytes=32 time=7ms TTL=51
Reply from 192.168.201.1: bytes=32 time=7ms TTL=51
Reply from 192.168.201.1: bytes=32 time=7ms TTL=51
Reply from 192.168.201.1: bytes=32 time=6ms TTL=51
Reply from 192.168.201.1: bytes=32 time=6ms TTL=51
Reply from 192.168.201.1: bytes=32 time=7ms TTL=51
Reply from 192.168.201.1: bytes=32 time=7ms TTL=51
Reply from 192.168.201.1: bytes=32 time=7ms TTL=51
Reply from 192.168.201.1: bytes=32 time=7ms TTL=51
Reply from 192.168.201.1: bytes=32 time=7ms TTL=51
Reply from 192.168.201.1: bytes=32 time=7ms TTL=51
Reply from 192.168.201.1: bytes=32 time=9ms TTL=51
Reply from 192.168.201.1: bytes=32 time=47ms TTL=51
Reply from 192.168.201.1: bytes=32 time=7ms TTL=51
Reply from 192.168.201.1: bytes=32 time=6ms TTL=51
Reply from 192.168.201.1: bytes=32 time=6ms TTL=51
Reply from 192.168.201.1: bytes=32 time=7ms TTL=51
Reply from 192.168.201.1: bytes=32 time=6ms TTL=51
Reply from 192.168.201.1: bytes=32 time=8ms TTL=51
Reply from 192.168.201.1: bytes=32 time=7ms TTL=51
Reply from 192.168.201.1: bytes=32 time=7ms TTL=51
Reply from 192.168.201.1: bytes=32 time=10ms TTL=51
Reply from 192.168.201.1: bytes=32 time=8ms TTL=51
Reply from 192.168.201.1: bytes=32 time=8ms TTL=51
Reply from 192.168.201.1: bytes=32 time=7ms TTL=51
Reply from 192.168.201.1: bytes=32 time=8ms TTL=51
Reply from 192.168.201.1: bytes=32 time=8ms TTL=51
Ping statistics for 192.168.201.1:
Packets: Sent = 69, Received = 69, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 6ms, Maximum = 47ms, Average = 8ms