EOIP tunneling and routing for Radio over IP

RouterOS Beginner Basics
1

I have posted a few different problems recently and I know I am probably in over my head but I am trying to youtube/Google my way to solutions that are probably simple enough for the learned, so I would like to put out there what the purpose and limitations of the equipment I am working with so I can ask a few questions and i’m sure more as the system expands. I appreciate any input on the issues that follow.

Scope of the Issue
We have audio circuits to control our radio system running on copper that have been in place for decades. The phone companies seem hesitant to repair or replace copper as time goes on because it is outdated tech. We don’t have the budget to fully update our radio system so we are trying to migrate away from copper lines as funding is available.

I am testing a set of ROIP devices that have two IP addresses per device, the standard address and an audio engine IP address. Each device has up to 8 analog audio ports, each analog port can be pointed at an analog port on another device, creating a point-point audio circuit. Each port can be pointed at a different IP address so Site A-1 can talk to Site B-1, site A-2 can talk to C-1, Site A-3 can talk to Site D-1…

One of the main limitations that we have is latency(being an Audio device), the maximum latency we can have and still have a useable system is 300ms.




Initially I set up the equipment in the office using a switch, assigned the equipment IP addresses, made sure they were all on the same network, Things worked flawlessly. But our sites are 100+ miles apart so the LAN was only an option for testing. I was asked to test them in the field using LTE modems(we have a contract with a Cellular carrier that makes these fairly cost effective compared to other WAN options). The issue I ran into with this option, was the IP address’s that the equipment required. Each device required two addresses so the single WAN static IP that the LTE modem passed through wasn’t going to do the job. After some googling and youtubing, I learned of the EOIP tunnels that Mikrotik offers, and the affordability of the equipment is a huge plus.

I currently have 3 of the audio devices connected to 3 Mikrotik HAP AC Lite’s, my wan connections are coming through the three LTE cellular modems with IP pass through enabled, I have EOIP tunnels set up between the three mikrotik’s in a Triangle set-up. Site A-Site B, Site A-Site C, Site B - Site C. As it stands now, I have a couple issues.

Issue 1 ; Site A tunnel to site C… when the router at Site A loses wan connection even momentarily the EOIP tunnel drops and doesn’t recover. on the same switch the EOIP tunnel from A to B does recover.
I don’t know the solution here, resetting the router at site C doesn’t bring the tunnel back up, disabling and re-enabling the tunnel on either end doesn’t recover the tunnel, removing and rebuilding the tunnel on both ends does not work either. the only thing that brings the tunnel back up is a reboot at of the router at Site A.

The equipment is in the field and it’s a 200 mile round trip so I am hesitant to do anything that will cause me to lose the ability to talk to the router remotely, so I haven’t tried resetting the router to the default config and rebuilding everything, short of that I may just head out there and swap out the equipment with another router if there isn’t an easier solution.


Issue 2 : routing… I have been keeping an eye on the traffic when things are up and running. With EOIP tunnels A-B and A-C active only, My audio links between site A-B and A-C work great(until issue 1 causes the A-C to drop out). However when I try to add EOIP tunnel B-C, things start to go bad. It looks like the traffic for EOIP tunnel B-C takes the long way around going from B-A-C instead of the direct path B-C. In a 5g world this wouldn’t be an issue, but we are stuck in a 4gLTE land. Ping times between each site directly hover around 200ms. So when the traffic between A-C takes the long way around it doubles the latency to 400+ pushing us outside of the audio equipment’s working specs. I haven’t been successful at googling the application of static routes for this situation, I think I need to set up static routes for traffic from A-B going through EOIP tunnel A-B, traffic for A-C going through EOIP tunnel A-C and so forth, but I am not sure how or where to start.

Issue 3 : Future… I have an IP scheme/layout in mind for a much larger system, and plan on upgrading the equipment from the HAP AC Lite’s to a rack mount unit when/if things are fully deployed. I am looking at having 4 remote radio sites, and 4 office locations, that all inter-mingle at times. so if there are any solutions it would be really helpful if things are fairly copy and paste-able with minor tweaks here and there.

Again I appreciate any assistance
routerconfigs.txt (14.2 KB)
ROIP over LTE.jpg

2

Wow, that’s an interesting topic…

Some questions:

  • Do you really need EOIP (L2 level connectivity) or would a TCP/IP tunneling also do?
  • Do you need a secured/encrypted connection or could a regular firewall do the job?

Because, depending on the answer on these questions, there could other solutions available, e.g. a full mesh IPIP setup, maybe with dynamic OSPF routing. IPIP being stateless, there is also no issues with recovering a lost connection.

3

To be completely honest, using anything other than an EOIP tunnel hadn’t crossed my mind. The manual states that the Audio device’s IP and Audio Engine IP need to be in the same network, but I can’t see where it specifically says that Device A and Device B need to be in the same network. We took a run at these devices awhile back and I remember heading towards the EOIP tunneling for some reason. I believe they are designed for a campus wide network, which is why(the non IT guy googled LAN connections over WAN) and was led down the EOIP path. The initial attempts resulted in failure due to latency limitations but they addressed that by adding a 0 to their buffer window so we now fall within the average 4gLTE Latency range.

in my uneducated mind, eoip would allow all IP addresses to be in the same network over a large area/multiple sites connected by 4gLTE modems, I could also plug in a laptop at site A and talk to all the equipment at Site B or Site C. Other options may be better suited, and I am open to suggestions.

We aren’t passing any real sensitive data or audio so I don’t think anything other than the basic firewall is necessary. If someone really wanted to hear our guys in the field talk about grabbing beers after work, I am sure there are easier ways.

4

For the issue 1 it looks like a firewall drop to me…
Did you also check your log?

For issue 2 since you use EoIP bpdu packets for rstp are crossing the tunnels.. Between sites a,b and c, who is your root bridge? Check all bridges status tab…

5

Going back through earlier emails, it looks like one of the engineers for the devices stated I would need a layer2 device, so google led me here.



For Site A, Root Bridge box is unchecked, and Route Port is currently Tunnel A-C
For Site B, it says that the Root Port is currently tunnel B-C
Site C, Shows - Root Bridge Box Checked, and Root Port is None

Currently it looks like all of the traffic is indeed flowing through site C.
All options there are greyed out, I assume because RSTP is enabled on the bridge at each site.


When I built the third tunnel B-C, I had to enable the RSTP, because,I Assume, I am firewall/routing illiterate. The loop caused constant drops due to devices seeing their own address’s. or something along those lines.

6

You can try to lower the bridge bpdu from 8000 to 7000 at your very first site, i guess its the A and see if that resolves the issue…

7

I am going to head out on the road to try rebuilding the config at “site A” to see if I can stabilize the A-C tunnel.


on the routing question -

Could input filtering at each site be a possible solution? If say at Site A I dropped all traffic that is addressed from WAN IP-B to WAN IP-C?

Would that effectively shut down the “Long way around”?

8

Spanning tree will not do what you want - it is designed to block redundant paths. As Mikrotik do not implement SPB (shortest path bridging) you might be able to do something with bridge split horizon, or failing that static bridge filters.

Ideally being able to use IP (i.e. layer 3) instead of Ethernet (i.e. layer 2) would resolve having to resort to tricks. The product website is vague - there are mentions of both IP and Ethernet, but no detail as to what is possible.

9

I couldn’t figure out the “issue 1” so I brought the equipment back for some homework, rebuilding completely and starting from scratch on a different HAP AC Lite still resulted in one tunnel recovering but second EOIP tunnel wouldn’t recover. LOG does show the LINK staying up for a period of time before it shows it going down even without any wan, I assume that is part of the keep alive. But once the link drops it acts like it doesn’t even try to recover the second bridge, even though the LOG does say it is initializing Phase 1… just shows the link failed due to time-up. A reboot cures it, until the next time the wan is dropped.

The routing issue, reading through the wiki on the bridge split horizon, looks promising if it can be applied to the EOIP tunnels. I appreciate the pointers and will look into that.

10

I would probably consider MPLS with VPLS if you know that you need L2 adjacency. EoIP is great for very simple applications but if i’m going to be running an L2 overlay for any length of time, I use VPLS for stability and scalability.

11

I am trying to follow the example of MPLSvpls, it appears that they are connecting the routers via Ethernet. How do I apply this method to my routers that are/will be connected via tunnels instead of an Ethernet connection?

I feel like they are jumping straight into the routing and in my case I need to set up the tunneling first and then apply the routing on top of the tunneling. My apologies for being illiterate on this.

12

In the Mikrotik MPLS/VPLS example the routers are interconnected with IP running over direct ethernet connections, in your case they would be interconnected with IP running over your LTE modem connections.

As each of your sites has a single WAN connection, i.e. you don’t have redundant paths as in the example, I’m not convinced MPLS/VPLS offers anything over a mesh of EoIP connections - you would still need to implement a mesh of VPLS connections and split horizon, instead of a mesh of EoIP connections and split horizon, so traffic can pass directly between sites rather than looping through a central one.

13

OK…

in my current attempt, kind of frankensteined from several places.

I have the EOIP tunnels set up with my ISP IP addresses, Local and remote, IPSEC is enabled and secrets match.(the tunnels come up)
All tunnels are added to the bridge-to equipment to use the tunnels
I have added a pair of IP addresses to each tunnel interface (1.1.1.1$1.1.1.2… 2.2.2.1&2.2.2.2…ect.)
I then added, through a lot of spreadsheet copy and pasting
/ip route add dst-address=(all ip’s from remote site pools) gateway=x.x.x.x(ip address of remote site’s EOIP tunnel interface added previous step)


my thought on this is anything from site a that is addressed to site b will go through gateway b.

AM I remotely Close?


with STP enabled on the bridge I still get a-c traffic going from a-b-c while tunnel a-c sits idle.
and with no protocol on the bridges I get loops detected still.

14

What you are attempting is similar to what we did for our RoIP system. You A-C link is down because RSTP is protecting you from loops. Let it do it’s job. Being a non IT guy, I wouldn’t go down the MPLS/VPLS road, you’ll just get frustrated. EOIP with RSTP is beautifully simple for what you are trying to do.

What we did, is set up a hub site (2 actually for redundancy, but that you can do later). Have all your EOIP tunnels bridge at the hub site. To keep latency down, make the hub site a reasonably fast wired site. That way the most LTE modems you have between any 2 sites is 2. Also, specify a low MAC address for your hub bridge to insure it is always root. We assign MAC addresses to all our bridges to guarantee the hub(s) are always the root.

Also a word about latency. Most vendors are referring to one-way latency. Pings, measuring round trip time, is double your latency time. Our MotoTRBO systems request no more than 120ms. latency, and yes, we are using LTE modems at some sites. We have about 70 sites across 4 states in our system and have not had any audio issues using this scheme.

About using IP for wireline replacement. I wouldn’t use tone control unless you use the 64kb codec. It will distort you tones causing intermittent operation at best. Use E&M if you can.

Cheers

15

This is probably, most likely the answer I have been waiting for… I have been pulling my hair out for a couple weeks reading definitions of acronyms defined by acronyms. This idea makes sense to the layman… I really appreciate all the answers and will mark it solved when I can put this to the test.