I'm really confused so I don't even know how to word this.

I have a RB2011 as my main router with 6 devices hardwired to it, one of which is a WAP (which may have around a dozen clients) and another is a point to point wireless bridge to building "B". Building B then has a WAP with another dozen or so devices.

Everything works... most of the time. Then some strangeness happens, seemingly random. Building B loses all connectivity. Rebooting everything does not resolve it, however UNPLUGGING and power cycling Building B's bridge resolves it.

When the connection is "lost", it is only lost one way - which is really weird. From Building A, I can ping both sides of the bridge. I can log into both sides of the bridge. But when I use the bridge's onboard diagnostic tools (ping, etc), I can't ping the router (or anything). Again, resetting/rebooting via firmware doesn't do anything. Powering off/on does.

I thought I had a bad bridge device, so I replaced it. And it just happened again. That made me start wondering if it is something with the router?

But where would I even start?

Your description does not sound like "one-way comm". This sounds like loss of connectivity between the remote bridge and the remote router. From your description, it is difficult to divine where the problem is. You glossed over a few details.

Lets make sure we have the logical layout right.

Router1 (main router) connects to BridgeA via ethernet, ( or is there a switch in between? )

BridgeA connects to BridgeB via radio,

BridgeB connects to WAP2 (other building) via ethernet. ( or is there a switch in between? )

When you have problems, the devices at the other building lose connectivity to all devices at the main building, and you can ping the IP address of BridgeB from Router1 or devices near Router1. Is that correct?

You can log into BridgeA and BridgeB but using the tools on the Bridge devices you cannot ping Router1, WAP2, or the other bridge device. Is that correct?

A "yes" to the last question leads to much confusion.

I presume you are using private IP space for all of this. In that case, it shouldn't be compromising to tell us what they are. Please, show us the IP address configurations (ip address, subnet mask, and default gateway) used for Router1's ethernet interface which connects to the Bridges, the Bridges, and the WAP at the other building. Are all of the devices in the main building in the same subnet with the bridge and all the devices at the second building?

Also, tell us what the Bridges and the WAP are. Someone may recognize a known problem with the devices. Tell us what version of software all of the devices are using.

There are several possibilities. Before sending you down all of those paths, more information may help us narrow it down for you.

Your description does not sound like "one-way comm". This sounds like loss of connectivity between the remote bridge and the remote router.

There is no "remote router", if by "remote" you mean the equipment at Building B. There is a WAP. DHCP, routing, etc is all on the main router.

From your description, it is difficult to divine where the problem is. You glossed over a few details.

I'm sure I did. I'm not a pro at this by any means.

Lets make sure we have the logical layout right.

Router1 (main router) connects to BridgeA via ethernet,

Correct

BridgeA connects to BridgeB via radio,

Correct. Two Engenius Enstation5 dishes. A in AP mode and B in client bridge mode.

BridgeB connects to WAP2 (other building) via ethernet.

Correct.

When you have problems, the devices at the other building lose connectivity to all devices at the main building, and you can ping the IP address of BridgeB from Router1 or devices near Router1. Is that correct?

Correct. Since I'm not physically there, all I know of devices in Building B is that they can't connect to the Internet and they can't ping the router. i'm not sure if they have a valid IP (more investigation required). I can ping BridgeB from Router1 and all devices "this side of" BridgeA. I can log into BridgeB. I can't, however, ping back to the router or any other devices from BridgeB. Hence my description of "one-way"

You can log into BridgeA and BridgeB but using the tools on the Bridge devices you cannot ping Router1, WAP2, or the other bridge device. Is that correct?

For BridgeB, yes. For BridgeA I can ping just fine. B can ping A, A can't ping B. A can ping router, router can ping A. B can't ping router, router can ping B. Nothing "upstream" of B can ping anything "downstream" of B (for example, WAP2).

A "yes" to the last question leads to much confusion.

Yep, that's why I'm here.

I presume you are using private IP space for all of this.

Correct.

Please, show us the IP address configurations (ip address, subnet mask, and default gateway) used for Router1's ethernet interface which connects to the Bridges, the Bridges, and the WAP at the other building.

I may not have understood the question, but its a pretty basic setup. 192.168.1.xxx for everything. router is 192.168.1.1. DHCP for everything, although most devices I've used "static" within the Microtik (so device is set to DHCP, router keeps it static). For example, BridgeA is 192.168.1.55 and BridgeB is 192.168.1.56. The WAP2 is dynamic (currently pulled 192.168.1.15 if that somehow matters)

Are all of the devices in the main building in the same subnet with the bridge and all the devices at the second building?

Yes

Also, tell us what the Bridges and the WAP are. Someone may recognize a known problem with the devices. Tell us what version of software all of the devices are using.

I can tell you the bridges are both Engenius Enstation5 FW. 1.5.96 (I recently upgraded from older Engenius 2.4Ghz thinking this problem was caused by failing equipment)

I'm not entirely sure what WAP2 is right now, but I think I've ruled that out. Prior to changing the bridge equipment, I changed that equipment 3x. It kept happening and I eventually saw that power cycling the BridgeB (sometimes twice) resolved it, which is why I swapped out the Bridges. If it is important, I can get the WAP2 info.

There are several possibilities. Before sending you down all of those paths, more information may help us narrow it down for you.

Thanks for your help.

I believe I've ruled out the WAP2 and I've ruled out the Bridge hardware, at least in terms of a faulty unit. I *think* it has to either be a problem with the Engenius line in general (something that affected my old ENS200 as well as the newer Enstation5) or else it has something to do with the Mikrotik or network configuration. I just find it so odd that I can happily log into the BridgeB interface and do whatever I want there, but it won't ping ANYTHING. Just as odd, is that rebooting it doesn't fix anything, but power cycling does. I suppose that could be coincidence. This latest time I rebooted remotely 2x and then had the person there power cycle it 1x. Maybe had I rebooted it a third time it would have worked? In the past, I've power cycled it once to no effect but a 2nd power cycling does the trick. How in the world can that be the case? Beats me.

Just to be clear, once I've power cycled BridgeB (such that the problem is fixed)... at that point, I *CAN* ping from that device's admin back to the router or the Internet or whatever. I say that just in case there was any question if for some reason BridgeB's ping utility simply doesn't work in this network config. It does (when it is working).

I also just want to be absolutely clear that this is an intermittent problem. For example, right now it is working just fine. I can log into either Bridge and ping anything. I can ping any of the devices connected to WAP2 (and WAP2 itself, of course). Everything's just fine. And the only things I did to get it working was the following sequence: Reboot BridgeB (unfixed); Reboot Router (unfixed); Reboot BridgeB then A then router (unfixed). Unplug BridgeB, wait 20s, plug in BridgeB (fixed).

But then one day... it will all stop working suddenly. I won't know when, I won't know why.

Look at the ARP table on the router for the bridge and other devices at the other building. Are they the mac addresses all different? There may be several IPs showing as having the same MAC address. Also look at the log on the router for anything unusual, maybe loops or something.

When you replaced the Engenious devices, did you change the power supplies? The cabling? I assume they are PoE powered, so between the PoE and the radio and between the PoE and the WAP.

Are there any switches at the remote building? Or is the WAP the only cabled device? A layer 2 loop at the remote building could cause spanning tree to shut down a port somewhere. Most likely, this is an Engenious or cabling issue, not a MikroTik issue.

Look at the ARP table on the router for the bridge and other devices at the other building. Are they the mac addresses all different? There may be several IPs showing as having the same MAC address.

BridgeA one IP, one MAC

BridgeB (remote) 6 IP, one MAC.

Problem?

Also look at the log on the router for anything unusual, maybe loops or something.

Nothing looks "unusual" to me, but I don't know what I'm looking for

When you replaced the Engenious devices, did you change the power supplies? The cabling? I assume they are PoE powered, so between the PoE and the radio and between the PoE and the WAP.

The new devices came with different PoE injectors vs the old ones, so I swapped it. Same cabling. Changing the cable from the bridge to the injector would be problematic, but possible. Changing the cable from injector to WAP is simple

Are there any switches at the remote building? Or is the WAP the only cabled device? A layer 2 loop at the remote building could cause spanning tree to shut down a port somewhere. Most likely, this is an Engenious or cabling issue, not a MikroTik issue.

No switches on that side. Its just the WAP and wireless clients over there.

Your bridge is not in a "transparent" mode. Look through Engenious documentation for how to make the bridge fully transparent.

nothing in the admin relates to "transparent", however I'll dig through docs and I'll contact Engenius as well. Thanks for your suggestion.

In your opinion, is the intermittent nature of this explained by your conclusion? Any idea why rebooting doesn't fix while power cycling does?


ps.

There is a setting, which is defaulted to "off" and I've always left it that way, relating to "Spanning Tree Protocol". It can be set to On, which then brings up the following:

Bridge Hello Time (default 2 seconds. Valid 1-10)
Bridge Max Age (default 20 seconds. Valid 6-40)
Bridge Forward Delay (default 4 seconds. Valid 4-30)
Priority (default 32768, valid 0-65535)

Can you also confirm what you dhcp configuration is. ie what the dhcp range is.
ensure that the static ips are reserved in the dhcp pool

Well, keep in mind they are only "static" insofar as the router keeps a persistent IP for a given MAC... on the client side, they are all set up to receive a dynamic address.

The pool is 192.168.1.10 - 192.168.1.254

All ips are in that range.

Interesting (maybe?)

When I earlier looked at the ARP list I saw one mac having 6 IP addresses, including (for example) X.15 and X.18. Now in the DHCP pool screen, when I click on "used addresses" I see those IPs, but with unique MAC addresses.

Does that tell us anything?

ok so you have a reservation for each unit in place. can confirm are you using the engenius units in wds bridge mode? what access point you using at building b for distributing wireless

can confirm are you using the engenius units in wds bridge mode?

No. In System->Operation Mode I have 4 choices, which are Access Point, Client Bridge, WDS and Client Router. The "local" side is set to Access Point and the "remote" side is set to Client Bridge. That was the recommended setup for my situation in the included documentation that came with the equipment.

Choosing WDS gives me 3 sub-options of Access Point, Bridge and Station.

what access point you using at building b for distributing wireless

Currently it is an Actiontec WCB3000N. The fibre service provided it free of charge, so I used it to test in place of the DLink DIR-895 which was there (which replaced an older Asus while trying to troubleshoot this problem). Since it worked just as well, we opted to use it and return the expensive DLink. The issues described here existed regardless of which AP/router was in place there. To be clear, when this "condition" exists, we can unplug the WAP entirely and hardwire a laptop to the bridge and it is the same scenario. So I don't think the WAP has anything to do with this (??).

Engenius suggest WDS bridge for a point to point link. https://www.youtube.com/watch?v=Wz7d9MQ2hLU
Another thing you need to check is that the engenius units firmware is up to date.

That does seem like the recommended setup for my situation, thank you.

However, is there any reason to believe that would solve the problem I intermittently experience?

As it appears that the problem is resolved when you reboot the bridge unit it appears to be a problem on the wireless side. With the engenius in AP mode there is potential for other people to try and connect to it or something similar. I have had a lot of issues in the past with them only allowing a maximum of 20 to 25 clients connected at one time and then they stopping passing traffic through the wireless interface or if they do it is very slow. Sometimes the engenius waps will pass traffic to some devices and not others.I find them very temperamental at times. Saying that I have over 20 engenius access points, some that have been working for over 5 years without issue as long as I keep the number of clients down. I would change to WDS bridge as this will help eliminate that as a potential problem.

This is now off topic... but I tried setting it up in WDS bridge mode, and it didn't exactly go well.

I got the WDS Link to go... both devices showed a solid link and no problems were reported. However, the remote device wouldn't pull an address from the DHCP. It actually behaved exactly like it did when I was describing the original problem (which, now that I think about it... I can't discount that it may have just "f'd up" again. Sigh). I put everything back the way it was and it was fine. I'm wondering what I'm missing. Is there anything I have to change on the DHCP server for a WDS bridge to work? The local side of the bridge got its address fine.

I manually assigned the remote unit its normal address and it still would not ping the router (again, consistent with the original problem, so I can't rule that out).

How are configured and connecting to your RB2011 . WDS bridge should be transparent. WHen setting up a ptp link I would always use static ip's on the links. If you can share more info about how things are connected we can assist further.

I don't know what info you need me to share

Pretty simple setup. BridgeA (local) is plugged into one of the ports on the Mikrotik. No switch, etc, in between. The Mikrotik is mostly out-of-the-box settings. I've setup a simple queue but that should have no bearing on this. Both bridges have "static" (I would call it "reserved") IP in the router, but each are set to get their IP dynamically. When that didn't work, I assigned the IP but still no connection. That is, the WDS link showed as being good but BridgeB could not ping BridgeA or router or anything else.

Tomorrow I'm going to take BridgeB from Building B and set it up in my office in a controlled environment and see if I can get it working before returning it to the wall.

Sounds like something on the switch side of the remote building. When you reboot, the port on the switch does not down/up, but when you power cycle it does, so it gets reset.

Do you have a manageable switch by any chance? Check/replace the switch with a dumb switch.


Sent from my iPhone using Tapatalk

I just ran a bench test to confirm that the WDS bridge should do what you are requiring. Please note it will not pass VLAN tags. I had an issue in the begining as the power was to high on the engenius units and they say they are connected but they will not pass data and further investigation showed one side was up the otherside showed the wds link as being down. So you may find the issue you were having was your wireless was being over powered on the remote unit causing the link to go down. my test show the wds should do what you want and it just requires you to set the transmit power at a suitable level.

Sounds like something on the switch side of the remote building.

No switch involved.

I just ran a bench test to confirm that the WDS bridge should do what you are requiring. Please note it will not pass VLAN tags. I had an issue in the begining as the power was to high on the engenius units and they say they are connected but they will not pass data and further investigation showed one side was up the otherside showed the wds link as being down. So you may find the issue you were having was your wireless was being over powered on the remote unit causing the link to go down. my test show the wds should do what you want and it just requires you to set the transmit power at a suitable level.

No Vlan? That must be uniquely Engenius. That sucks. I'm not using Vlan right now, but I had plans to. Maybe I should stick to AP/Client mode and deal with power cycling once in awhile. As far as power goes... Due to the short distance, it was already set to lowest.

Doing more testing today

It may be that some device needed its ARP table cleared after switching to WDS due to the MAC address translation which the bridge uses when not in WDS mode.

I would try making sure lease times are short, like 10 minutes maximum. Then while at the other building, probably after business hours, convert to WDS. Power cycle everything at that building.

Go back to the main building. Power cycle the engenious and the 2011.

See what it looks like then, or 10 minutes after then. You may also want to change your DHCP server to 'Add ARP'. That can help when a rogue device starts ARPing for other IPs. We don't know that is happening yet. But I haven't seen that setting cause harm in an all DHCP network.

If everything is working, you can increase your lease times to what they were, or not.

The last Engenious devices I used were 802.11b only. We removed them from service as soon as we found something better. The hardware was alright. Our problems were with the firmware. Things may have changed with their newer models. We haven't bothered to find out.

In the USA, MikroTik doesn't have enough FCC approvals for my company to be willing to use them for 5 GHz outdoor links, UNI-II band, DFS. We primarily use vendor U's gear for what you're doing outdoors. They've been predictable and as reliable as the installation. For a 2.4 GHz link between buildings, MikroTik devices would be a solid choice. 2.4GHz is too trashed in many places, making 5GHz a better option.

A hAP ac lite would be a good choice if you ever decide to replace your WAP.

I will run a test tomorrow to check vlan passing, but all the forums i have read they have some serious issues. we have moved away from them as they just dont work at the levels most of us require. Can suggest a number of manufacturers with better hardware for a lower price. Their firware is really buggy and their processors underpowered which could also be an issue you experiencing. keep us posted on your bench tests. I am also unsure if they pass vlans in the ap client mode. According to tests wds will give you best peformance.

Thank for all your help and suggestions!

probably after business hours

Not really a business. Small non-profit. That's why there's only me - a non expert - dealing with this Just trying my best to make things work and getting help from you guys is greatly appreciated.

re: choosing Engenius... I had the old Engenius devices so I was more familiar with their setup. That and price (awesome deal on Amazon) was the only reason I went with them.

Oh, I also chose them partly because of a typo by Amazon. The Amazon listing said the Enstation5 has 2 Gigabit ethernet ports. So when I was comparing to similar priced units from UBNT and Mikrotik... which of course have fast ethernet... it seemed like the right decision.

A hAP ac lite would be a good choice if you ever decide to replace your WAP.

I appreciate that... and once I know we have the bridge sorted we'll certainly look at that. I kind of doubt single hAP ac lite would be "good enough" by itself for the whole building. It has been frustrating getting the wifi in there to be acceptable. It is 2 floors and only about 2400 s.f. but I swear they have lead in the floors or something. Getting a signal downstairs has been... an exercise. First we had an old Asus that I borrowed from my house. That worked fine on the floor it was installed, but horribly in the floor below. We tried the UBNT "Lite" WAP and that was even worse (after I over-promised that moving to that "enterprise" equipment would solve all our problems). I went to Costco and got some "top of the line" gawdy red DLink router thinking that would be overkill, but meh... it was better, but not fantastic. It is leased space, so running wires is not ideal. Anyway, a problem for another day. If you think the hAP is a superior device though, I have no problem suggesting that when the time comes.

Thanks again.

Are you running any QOS or traffic shaping on then engenius AP's

No. As far as I can tell, they don't have any QoS. The traffic shaping appears to be extremely limited. I figured it was best to let the router handle that kind of stuff.

Are you running encryption on the engenius?

Yes. WPA2-PSK AES

Ran a test this morning to see if vlans will pass over the WDS bridge. Definately don't on the versions of hardware I have. But the Bridge was solid not drops for the last 16 hours.