RouterOS Version: 2.9.43
Router Board: RB532

I have a problem and I need some help on what to do/try next. I will try to be as brief as possible, but it will be hard since this has been an ongoing issue for months now. The problem I am having is Packet Loss from a Mikrotik backbone at the top of two towers down to a switch at the base of the tower.

We have two towers out of about 20 that are having the exact same issue. Each tower has 2.4GHz using Mikrotiks, 900 using Canopy, and 5.7 backhauls using Mikrotik, one of the two towers has 5.7 Canopy. At almost all of our towers we are the only carrier of any type accept for a few. From what I can tell the only thing the two problematic towers have in common is high RF.

One of the towers has been seeing problems for a few months. The only thing significant that happened around the time it started was a second Cell phone company setup shop right below us. So now there are two cell phone companies, us, and the local ambulance service on this water tower. As I said, the problem started about the time the second phone company turned service on, we worked with their RF and tower engineers, they were very helpful, and even turned the equipment off for about 30 minutes so we could test. Even when they turned their equipment off we were still seeing the packet loss. So we had to rule them out as the cause….

The second tower, the problem started last month. The problem started when we switched from one Mikrotik handling all backbones and the customer AP on the same board. We swapped to one Mikrotik for each backhaul and one for the customers. The only thing on this tower other than us is a high powered FM station; we are about 30-40 feet below their antennas. It should be noted though we had a number of issues at the beginning when we first turned this tower up and we had to go through a lot of reengineering to get service stable.

Basically, I need some ideas on what we can try to stop our packet loss. My only guess at this point is the high RF environment at each location is wreaking havoc on our Ethernet or Mikrotik. Below I summarized everything we have tried, but I wanted to cut this message off at this point, because I’m sure some people have already left me. If anyone has dealt with high RF environments and have had similar problems please share some ideas so I can fix this problem. If you are interested in helping, please read below, I swear I have done just about everything at this point, but there is always more to do.

Thank You!
Brandon Toyzan
Technical Operations Manager
Mercury Network - Michigan/Wisconsin

First here is some more information describing the problem.

Only one Mikrotik, the main backhaul, has loss. On top of that, we only see the loss going one way (keep in mind pings are two way...). If I ping from our main router, to the customer AP on the tower, I see the loss. The loss can range from 2% on up to 10%, there is no pattern to the loss. Sometimes you won’t see loss for 5 minutes, other times you will see loss every 30 seconds for 5 minutes. This is not a loading issue, as we have much busier and bandwidth intensive towers in our network.

The only customers who notice any thing are bandwidth intensive businesses or VOIP customers; we get no other complaints from anyone else.

If we ping from the Mikrotik at the bottom of the tower up to the backhaul Mikrotik at the top of the tower we do NOT see loss. We do see the loss, pinging from the Mikrotik at the top of the tower down the Mikrotik at the bottom of the tower.

MAC pings reveal the same thing, loss going down the tower, but not back up.

We do not see any packet loss going from the bottom of the tower up to the Customer AP or the other backhaul on the tower. Also there is no packet loss up to our Motorola 900 or 5.7 equipment. But there is loss, when pinging from our central colo router over the backhaul to the Motorola equipment, just like each other piece of equipment. Unfortunately, at one tower the back-up backhaul is on a much longer loop, and response times are higher, so switching to that backhaul would be noticed even by our web browsing customers. The other tower, unfortunately we just repositioned the backhaul to a new tower that has not gone active, so moving over to that backhaul is not an option yet.

Here is a list of things we have tried, I have been keeping pretty good track of them and adding them to a list as we go. Little things may have been done in frustration and I may not have kept track of them. I will summarize each as best as I can. Almost all of the steps below have been done to both locations at this point.
1. Removed Gel filled outdoor Cat 5, replaced with Shielded Cat 5
2. Grounded Shielded Cable
3. Replaced 48 volt PoE multiple times
4. Replaced switch multiples times
5. Changed the switch brands we were using
6. Replaced Mikrotiks
7. Replaced Radio Cards
8. Added Ferrite Beads, tested with them, moved them around, up down, side to side….
9. Grounded Cable to grounding rod, instead of our grounded cabinet
10. Re-grounded Cabinet, moved grounding back to cabinet
11. Tested with other local services (Had Cell carrier turn off equipment, had ambulance service talk on their equipment).
12. Wrapped Electrical Tape around the metal poles that our Mikrotik outdoor enclosures are on.
13. Replaced the Back-haul antenna.
14. Moved the positioning of our cables from one leg of tower to another, moved cables as far away from other equipment as possible.
15. Upgraded Mikrotik OS multiple times, currently on at least 2.9.43, one ap has 2.9.45, but it is a losing battle and a waste of time to upgrade every week.

Thanks…look forward to ideas.

We did come up with one more today while I was writing this, if truly I don’t have packet loss to the other Mikrotik unit at the top, I’m just going to get longer Coax to run from the antenna over to the Mikrotik that doesn’t have packet loss.

Brandon

I’m just going to get longer Coax to run from the antenna over to the Mikrotik that doesn’t have packet loss.

Brandon

Try that , if it works move the other board over there . I'm sure high rf can induce a lot of power into a board

In what cases are your MT boxes ?
Try metal ones, properly grounded.
Try to isolate power sources in metal cases also. Come back and tell us if this helps.

What OS level are you running?
What was it upgraded from (upgrade history, etc.)?

Thanks for the replies.

Some of these are simple fixes that I will try asap (I really need this fixed).

We are in Metal Cases already, they are not individually grounded, but the shielded cable coming down the tower is grounded.

The PoEs is down at the base of the tower, they are in a metal cabinet that is grounded. I could try isolating them separately if anyone thought it might help.

The water tower we are having problems with is running OS 2.9.43, previously 2.9.40, before that 2.9.38.

The tower that just started doing having this problem was previously an Intel board, now it is a Routerboard so that was a major change. So that was a major jump in version history, I know we made no upgrades to that Mikrotik OS for probably 2 years. So it was a big disappointment to get the Routerboards with the newest version up in the air to suddenly start seeing packet loss.

I appreciate the help, I think I have someone available this weekend to do some climbing and make some of the changes proposed here. I'm hoping with the longer Coax cable maybe we will at least prove something.

If anyone else has any ideas chime in, much easier to find someone to climb one day, instead of day after day after day. I'll probably try most of these changes on the Water tower first, since it is a much easier climb.
Brandon

Are the RB boards in the metal boxes grounded to the cases as per the RB manual?

How long are your cable runs. On many commercial installs the shield is grounded every 100ft up the tower this is to make sure the potential in the cable is the as that on the tower. When grounding the cable shield are you grounding top and bottom or both? Depending on the source and product some recommend grounding both other warn against doing it. What kind of filter to do you have on your PS. If you have not tried a Tripplitte ultra power bar/filter they have worked wonders for me it the issue is coming in on the AC.

Hope this helps


Erik

Grounding should always be at the the bottom closest to the ground.
The grounding should also occur outdoors.

This is standard grounding practice.

The tower, will have the majority of power induced into it, and you do not want to give electricity an alternate method to ground, other than the tower. Let any induced current induced into the cables, flow to ground through the cables as it is minimal compared to what the tower will carry.

[EDIT] I am refering to ethernet and not coax.
Coax I would recommend grounding as frequently as possible, and use a lightning arrestor even if being used just for DC power.

Thanks guys for the suggestions, there are few things here that I don't do with the towers that I probably should. I want to start reviewing how we are grounding things a little better, it might be a weak point, but we learned our lesson early about grounding and since that time 2 years ago we have had very little problems with equipment getting blown or fried.

As for the original problem I posted about....it appears that it was not a high RF issue. It was actually a misconfiguration on our backbone network.

We use RSTP through out our network, we also use WDS on our backbones. The change that fixed everything? I changed the "WDS Cost Range" from "50-150" to just "50." That cleared up everything on both towers, when applied to both sides of the backbone and each of the redundant paths along the way. I have since gone through every tower on our network and made this change.

Now that I have the critical issue fixed I am going to go through and see about some of the other suggestions in regards to my question about high RF towers.

Have a good one everyone, I always enjoy reading the posts on this message board, everyone is helpful and has good ideas.

Brandon Toyzan
Mercury Network