My Problem:

AT&T Network Cisco Microcell won't work. Specifically, AT&T support says the ports are not open for the microcell and thus it won't register. What is odd is that the Microcell was installed without port forwarding and worked fine for months. After an outage with our tower where this person is connected, the Microcell would not reconnect again. This is actually not the first time I've seen this and I'm about at my wits end going back and forth between the customers, AT&T and around again in circles! (there's my gripe,I'm done. let's continue...)

I setup the same rules that work for DVR / NVR / IP Cameras, Xbox etc... But the same rules will not work with the Microcell.

I have set my firewall rules and checked them twice. My NMAP says naughty even though they are nice.

I checked the rules will work by forwarding to several other devices on the local network (also beyond whatever switch and wiring at the home) and scanning with NMAP. Ports show as opened only when not pointed to the Microcell.

Setup and Equipment and Etc...
This is a RB941 hAP Lite for the router, on the latest 6.7 Router OS and 3.33 Firmware. The connection is a wISP fixed wireless, bridged UBNT 5GHz PtMP setup. The router has a Static Public IP set static on the WAN gateway. I added another Public IP to try and bypass the Masquerade NAT and effectively DMZ the microcell. AT&T said it was a hardened device and this is safe to do.

Here's a compact of the Config Export. Right now I have it setup as a "DMZ" style which was what the last AT&T support suggested we try. You'll see the other rules I disabled trying different combinations. The other rules for port 4369 is for a "Home Automation" system unrelated and it's working without a hitch.
Steps Taken So Far

First I setup your typical dst-nat for the ports to the local device IP and confirmed it is in fact the right device and IP, and reserved the DHCP lease for the microcell. I played around with the rules.

No luck. For giggles I tested forwarding to other devices on the local network, and scanning with nmap showed the ports open. When I set the rules back to the Microcell nmap shows no ports reachable. Furthermore I can not even ping the microcell on the local network, which seems strange. I see the following.

ping 192.168.88.108
SEQ HOST SIZE TTL TIME STATUS
0 192.168.88.108 84 64 0ms port unreachable

I had our network admin check my dst-nat port forwarding rules forwarding all ports (minus winbox) to the darn thing, he said they looked fine and should work. I thought to try this myself and then when I spoke to the AT&T support he said if the forwarding doesn't work try to DMZ it.

Then I tried a different public IP altogether and just tried to SRC-NAT and DST-NAT to hopefully bypass the masquerade and effectively DMZ. (please forgive me I'm a total n00blette over here...)

And finally I realized several things. We supplied the router, but someone else did the internal networking and there is a switch or hub and wiring between where the microcell is. The Mikrotik hAP Lite is in a garage with thick walls. I will have him tomorrow plug it in directly to the port on the mikrotik and see if we have joy port forwarding to it then. Problem is GPS might not sync inside the garage :/

Second, the ATampersandT support said sometimes the problem is the nat itself. I tried to bypass it but could not as-is. My admin suggested making port 4 a slave to the WAN ether1-gateway so that hardware switching would fully bypass the routerOS and then I could assign a public IP directly to the Microcell using the Tower's router to issue it via a custom DHCP network. (The tower is MIkrotik )

I am probably over thinking this, and that's why I am coming to you guys on bended knee and humbly asking for any advice you might have. I did search forums but could not find a similar question or answer for this.

Other Notes
Several things the support people shared with me, for what it's worth.
1. Dropping Fragmented Packets should be turned off. I found a rule to do that I tried, no dice.
2. IPSec Passthrough should be off? I honestly don't know what this is.
3. MTU should be 1492. and I did try switching the interfaces all to 1492 and 1480. I switched it back though as it didn't fix it and I don't know what I'm doing enough to mess with it.
4. Latency should be under 50ms. I was pinging google DNS at 10ms-17ms-21ms responses.
5. Also I've been up for two days straight and I can't remember what else but I wrote it down, somewhere... I will probably realize later when I read this it's totally incoherent.
6.Support said the error code he was receiving from the microcell (i guess it calls home, often) only happens when ports are blocked. not the error for a bad unit etc.

The device is showing internet status and GPS status indicators so *should* work.

Summary
could be a bad microcell
could be I'm an idiot I am missing something glaringly obvious to you guys
could be some unknown reason why mikrotik and microcell won't play nice. Might just have to get him a different router?
could be a local device causing conflict (that's right, I'm looking at you DirecTV box!! o.0 )

While I originally became obsessed with and doubted my firewall rules, I am not sure if it's actually the culprit and tomorrow will have them plug in directly to the dish feed and assign a Public IP directly to the Microcell. If that works i'll try the port on the actual Mikrotik, so I know it's not the switch in the dudes house. I will keep you apprised.

In the meantime, ya'll can tell me what the best way I might setup this here fancy thingymajigger to work best?

Furthermore I can not even ping the microcell on the local network, which seems strange. I see the following.

ping 192.168.88.108
SEQ HOST SIZE TTL TIME STATUS
0 192.168.88.108 84 64 0ms port unreachable

[...]
We supplied the router, but someone else did the internal networking and there is a switch or hub and wiring between where the microcell is.

Do you have access to the microcell? can you post its relevant tcp/ip configuration?

First thing to get straight is routing between the mikrotik and the microcell, until that ping works no dst-nat is going to solve anything. Seems to be microcell related: could be firewalling, a static route, wrong wiring...

Default route through 192.168.88.1 as handed by DHCP should be fine, did you notice in the logs and DHCP leases if it was being assigned?.

BTW, 192.168.88.1 should be assigned to the bridge-local interface, not any of its ports. Check if it makes a difference.

Where's the microcell connected (ether port)?

1. I did not know one could log into the microcell, or do you mean the online management portal? I figured that AT&T doesn't want you to log into it, I can't find any documentation though I'd be willing to try if anyone knows how. I can have them log in to the AT&T portal. Otherwise, I was told the device is DHCP so must have a DHCP server. I could go on site it's just an hour away and physically access it. They did try deregistering and registering on the portal yesterday several times.

2. From what I can gather, these devices are supposed to be "hardened" but not responding to ping made me think maybe the unit has "locked up" or gone haywire. But I also wouldn't be surprised if they just have icmp ping disabled by default on all of them for security reasons.

3. I can see in the log it is assigning DHCP Lease IP to the MAC.

4.That is just the default config, I switched it to the bridge-local and tested, no ping and NMAP scan showing ports closed.

5. The microcell is connected to a wall jack in the home, to a switch, and the switch to ether2-master-local. I tried turning on proxy-arp for the port as well, did not help.

1. I did not know one could log into the microcell, or do you mean the online management portal? I figured that AT&T doesn't want you to log into it, I can't find any documentation though I'd be willing to try if anyone knows how. I can have them log in to the AT&T portal. Otherwise, I was told the device is DHCP so must have a DHCP server. I could go on site it's just an hour away and physically access it. They did try deregistering and registering on the portal yesterday several times.

Ask them about the TCP/IP config then: make sure IP address/mask, and default gateway are being properly set by DHCP.

2. From what I can gather, these devices are supposed to be "hardened" but not responding to ping made me think maybe the unit has "locked up" or gone haywire. But I also wouldn't be surprised if they just have icmp ping disabled by default on all of them for security reasons.

IT can be that (blocked ICMP), though I think it's not a very clever idea in terms of troubleshooting... I'd further investigate this area as it is the most obvious anomaly, that could also explain your problems (reactive hardening, config reset...).

3. I can see in the log it is assigning DHCP Lease IP to the MAC.

Ok, so it's clear it is something related to the microcell: it may have another default gateway, or it could be set to not set it from DHCP. As you don't have access this is something that should be checked with AT&T or whoever manages the microcell.

4.That is just the default config, I switched it to the bridge-local and tested, no ping and NMAP scan showing ports closed.

Ok, expected that. That's the right way anyhow.

5. The microcell is connected to a wall jack in the home, to a switch, and the switch to ether2-master-local. I tried turning on proxy-arp for the port as well, did not help.

The fact DHCP is working between microcell and mikrotik shows this isn't a L2 or wiring issue.

No ports need to be "opened" for the AT&T Microcells to work. They only need to be allows to exit the network. They create an IPSec tunnel between themselves and the closed AT&T "POP". You will likely have a hard time with the GPS signal than anything else. As long as you are allowing the device to have internet access, that is all you need to do.

From the AT&T Microcell user manual http://www.att.com/media/att/2014/suppo ... Manual.pdf:

Installing the MicroCell behind a firewall, or behind a router with firewall capabilities, requires the following ports be opened to prevent the firewall from blocking communication with the network

123/UDP: NTP timing (NTP traffic) 443/TCP: HTTPS over TLS/SSL for provisioning and management traffic
4500/UDP: IPSec NAT Traversal (for all signaling, data, voice traffic)
500/UDP: IPSec Phase 1 prior to NAT detection (after NAT detection, 4500/UDP is used)
4500/UDP: After NAT detection, 4500/ UDP is used
NOTE: Customers attempting to connect a MicroCell on their corporate Internet connection may experience connection issues. The MicroCell is designed
to function using a direct Internet connection.

Make sure the ports are open outgoing and Natted to the micorcell IP address incoming in the firewall rules.


Hope this helps.

When we turned on any firewall rules on our mikrotik routed network it broke all the microcells. Esentially this started blocking fragmented packets therefore breaking the ipsec tunnels. Turning off all firewall rules fixed it. Not sure why Mikrotik starts disallowing fragmented packets once firewall rules are created - even with connection tracking off. So - it probably has to do with conn-track and firewall. Try newest mikrotik routeros version with RAW firewall and tell it to allow fragmented packets and see if that helps.

Make sure the ports are open outgoing and Natted to the micorcell IP address incoming in the firewall rules.
Hope this helps.

DO NOT NAT the ports to the Microcell. It does NOT need incoming connections. The Microcell makes an outgoing Ipsec tunnel to the AT&T servers. You only need to make sure that it does not have outbound connections blocked.

When we turned on any firewall rules on our mikrotik routed network it broke all the microcells. Esentially this started blocking fragmented packets therefore breaking the ipsec tunnels. Turning off all firewall rules fixed it. Not sure why Mikrotik starts disallowing fragmented packets once firewall rules are created - even with connection tracking off. So - it probably has to do with conn-track and firewall. Try newest mikrotik routeros version with RAW firewall and tell it to allow fragmented packets and see if that helps.

Try disabling the rule that blocks fragmented packets: the forward chain, drop invalid one and see if it makes a difference. Make sure you're using latest ROS and firmware, there have been fixes solving packet loss/reordering/fragmentation lately.

You guys didn't understand. If you have firewall rules at all - even a single one that has nothing to do with ATT microcells - and connection tracking is off it will break fragmented packets. You cannot have a single firewall rule if you disable connection tracking and wish to have att microcells or other fragmented packets work.

@changeip you cannot disable connection tracking as the router needs to source nat outgoing traffic.

No connection tracking = no NAT, (and no firewall filter rules).

If your Tracking settings Enabled Parameter is set to auto, setting up any Firewall filter, mangle, or nat rules will enable it.

Disabling connection tracking is not an option, unless you assign a public IP directly onto the microcell, and leave it unfirewalled.

If you don't want RouterOS to "scrub" fragmented, out of order, etc packets, then disabling the drop invalid rule on both the input and forwarding chains will prevent that. (Shouldn't be necessary)

Did any one solve this as we have the same issue with a bunch of our clients. Only the IPSec connections of ATT microcells ever seem affected. Any help in resolving this would be appreciated.

You have to allow fragments to pass ... or turn off connection-tracking so its ignored.

We have a bunch of routers at various sites that feed our core router at the head end that does NAT and mangle. Obviously we need connection tracking at the core, so how do we allow fragmented packets to pass through the firewall and is there any downside to this and would it break anything else?

I was never able to find a solution for this, unfortunately. We changed our policy to state that we do not support Microcells.

At this point I am convinced it does not have anything to do with port forwarding. As others in this thread have pointed out, it's a SRC-NAT connection and we still have people using them just fine on parts of our network behind double NATs with no port forwarding whatsoever.

I suspect that the packets are being fragmented somewhere on or off of our network that is breaking the IPsec tunnel back to AT&T. I worked with multiple users in three way calls with AT&T support and wrote to their engineers to see if they could work with us but had no response or escalation and no resolution on this. I used DHCP to issue a public IP directly to a microcell to see them try and say we were still blocking the ports >:) - still didn't work. The response from their CS was they are designed to be used BEHIND a router and not on a public IP. SMH...

The problem here is they keep telling the users we are blocking the ports and not allowing the devices to work, when in fact, we opened all of the ports specified and they just don't work. I tried different (non mikrotik) routers at the premises to no avail - but our core is all mikrotik routers and I wouldn't have it any other way.

I am not sure if this is the same issue you are having, or not, @wispvt. Good luck!

I wish I knew more but these devices are locked down you can't even ping them.

I discussed turning up MTU on some of the equipment on our network. My administrator did not want to tinker with it, understandably.

Most of the users in question have moved to using "WiFi Calling" in lieu of the microcells. Hope this helps...

It has to do with packet fragments not making it. I finally was able to track this down by running a packet capture at the very first router closest to the customer, and then running one successively each hop out and figuring out where things broke. It came to the first router that had connection-tracking disabled, BUT with a single firewall drop rule. That activated the firewall module and in turn stopped fragments.

HOWEVER, the problem might not be in your control. I would packet capture as it is leaving and see if the fragments are all in tact on the way out to make sure you aren't losing them before they even leave your network. If at that same first hop you aren't receiving them all properly then you need to take it up with your ISP.

PS- you absolutely do not need to forward port 80,443,53,etc like it says you do. Thats obsurd and we all know you don't need to do that.