Community discussions

MUM Europe 2020
 
User avatar
horse1bun
just joined
Topic Author
Posts: 13
Joined: Wed Jul 29, 2015 1:05 am

ATT Microcell Port Forwarding difficulties

Sat Sep 24, 2016 11:02 am

My Problem:

AT&T Network Cisco Microcell won't work. Specifically, AT&T support says the ports are not open for the microcell and thus it won't register. What is odd is that the Microcell was installed without port forwarding and worked fine for months. After an outage with our tower where this person is connected, the Microcell would not reconnect again. This is actually not the first time I've seen this and I'm about at my wits end going back and forth between the customers, AT&T and around again in circles! (there's my gripe,I'm done. let's continue...)

I setup the same rules that work for DVR / NVR / IP Cameras, Xbox etc... But the same rules will not work with the Microcell.
I have set my firewall rules and checked them twice. My NMAP says naughty even though they are nice.
I checked the rules will work by forwarding to several other devices on the local network (also beyond whatever switch and wiring at the home) and scanning with NMAP. Ports show as opened only when not pointed to the Microcell.

Setup and Equipment and Etc...
This is a RB941 hAP Lite for the router, on the latest 6.7 Router OS and 3.33 Firmware. The connection is a wISP fixed wireless, bridged UBNT 5GHz PtMP setup. The router has a Static Public IP set static on the WAN gateway. I added another Public IP to try and bypass the Masquerade NAT and effectively DMZ the microcell. AT&T said it was a hardened device and this is safe to do.

Here's a compact of the Config Export. Right now I have it setup as a "DMZ" style which was what the last AT&T support suggested we try. You'll see the other rules I disabled trying different combinations. The other rules for port 4369 is for a "Home Automation" system unrelated and it's working without a hitch.
# sep/24/2016 07:08:45 by RouterOS 6.37
# software id = G0UK-LSFL
#
/interface bridge
add admin-mac=E4:8D:8C:8B:07:6C auto-mac=no name=bridge-local
/interface wireless
set [ find default-name=wlan1 ] band=2ghz-b/g/n channel-width=20/40mhz-Ce \
    country="united states" distance=indoors frequency=auto mode=ap-bridge \
    ssid=meow wireless-protocol=802.11
/interface ethernet
set [ find default-name=ether1 ] name=ether1-gateway
set [ find default-name=ether2 ] arp=proxy-arp name=ether2-master-local
set [ find default-name=ether3 ] arp=proxy-arp master-port=\
    ether2-master-local name=ether3-slave-local
set [ find default-name=ether4 ] master-port=ether2-master-local name=\
    ether4-slave-local
/interface wireless security-profiles
set [ find default=yes ] authentication-types=wpa-psk,wpa2-psk mode=\
    dynamic-keys supplicant-identity=MikroTik
/ip ipsec proposal
set [ find default=yes ] enc-algorithms=aes-128-cbc
/ip pool
add name=dhcp ranges=192.168.88.10-192.168.88.254
/ip dhcp-server
add address-pool=dhcp disabled=no interface=bridge-local name=default
/interface bridge port
add bridge=bridge-local interface=ether2-master-local
add bridge=bridge-local interface=wlan1
/ip address
add address=192.168.88.1/24 comment="default configuration" interface=\
    ether2-master-local network=192.168.88.0
add address=176.59.114.50/30 interface=ether1-gateway network=176.59.114.48
/ip dhcp-client
add comment="default configuration" dhcp-options=hostname,clientid disabled=\
    no interface=ether1-gateway
/ip dhcp-server lease
add address=192.168.88.108 client-id="69:70:2e:61:63:63:65:73:73:2d:33:47:2d:4\
    1:50:2d:37:34:35:34:37:44:2d:30:30:32:37:35:33:39:37:33:33" comment=\
    microcell mac-address=74:54:7D:FC:1B:F8 server=default
/ip dhcp-server network
add address=192.168.88.0/24 comment="default configuration" gateway=\
    192.168.88.1
/ip dns
set servers=8.8.8.8,8.8.4.4
/ip dns static
add address=192.168.88.1 name=router
/ip firewall filter
add action=accept chain=forward comment="default configuration" \
    connection-state=established,related disabled=yes
add action=drop chain=forward comment="default configuration" \
    connection-state=invalid disabled=yes
add action=drop chain=forward comment="default configuration" \
    connection-nat-state=!dstnat connection-state=new disabled=yes \
    in-interface=ether1-gateway
add action=accept chain=input/forward fragment=yes protocol=tcp
add action=accept chain=input/forward fragment=yes protocol=udp
/ip firewall nat
add action=masquerade chain=srcnat comment="default configuration" \
    out-interface=ether1-gateway
add action=accept chain=dstnat dst-port=8291 protocol=tcp
add action=src-nat chain=srcnat disabled=yes out-interface=ether1-gateway \
    src-address=192.168.88.108 to-addresses=176.59.114.50
add action=accept chain=dstnat disabled=yes dst-address=176.59.114.50 \
    to-addresses=192.168.88.108
add action=src-nat chain=srcnat disabled=yes src-address=192.168.88.108 \
    to-addresses=176.59.114.50
add action=dst-nat chain=dstnat in-interface=ether1-gateway protocol=tcp \
    to-addresses=192.168.88.108 to-ports=0-65535
add action=dst-nat chain=dstnat in-interface=ether1-gateway protocol=udp \
    to-addresses=192.168.88.108 to-ports=0-65535
add action=dst-nat chain=dstnat dst-address=176.59.113.98 dst-port=4369 \
    protocol=tcp to-addresses=192.168.88.101 to-ports=4369
add action=dst-nat chain=dstnat dst-address=176.59.113.98 dst-port=4369 \
    protocol=udp to-addresses=192.168.88.101 to-ports=4369
add action=dst-nat chain=dstnat disabled=yes dst-port=123 in-interface=\
    ether1-gateway protocol=udp to-addresses=192.168.88.108 to-ports=123
add action=dst-nat chain=dstnat disabled=yes dst-port=443 in-interface=\
    ether1-gateway protocol=tcp to-addresses=192.168.88.108 to-ports=123
add action=dst-nat chain=dstnat disabled=yes dst-port=500 in-interface=\
    ether1-gateway protocol=udp to-addresses=192.168.88.108 to-ports=500
add action=dst-nat chain=dstnat disabled=yes dst-port=4500 in-interface=\
    ether1-gateway protocol=udp to-addresses=192.168.88.108 to-ports=4500
/ip firewall service-port
set ftp disabled=yes
set tftp disabled=yes
set irc disabled=yes
set h323 disabled=yes
set sip disabled=yes
set pptp disabled=yes
set udplite disabled=yes
set dccp disabled=yes
set sctp disabled=yes
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www disabled=yes
set ssh disabled=yes
set api disabled=yes
set api-ssl disabled=yes
/ip upnp
set enabled=yes
/ip upnp interfaces
add interface=bridge-local type=internal
add interface=ether1-gateway type=external
/system clock
set time-zone-autodetect=no
/system identity
set name=01165-person-name
/system routerboard settings
set cpu-frequency=650MHz protected-routerboot=disabled
Steps Taken So Far

First I setup your typical dst-nat for the ports to the local device IP and confirmed it is in fact the right device and IP, and reserved the DHCP lease for the microcell. I played around with the rules.

No luck. For giggles I tested forwarding to other devices on the local network, and scanning with nmap showed the ports open. When I set the rules back to the Microcell nmap shows no ports reachable. Furthermore I can not even ping the microcell on the local network, which seems strange. I see the following.

ping 192.168.88.108
SEQ HOST SIZE TTL TIME STATUS
0 192.168.88.108 84 64 0ms port unreachable

I had our network admin check my dst-nat port forwarding rules forwarding all ports (minus winbox) to the darn thing, he said they looked fine and should work. I thought to try this myself and then when I spoke to the AT&T support he said if the forwarding doesn't work try to DMZ it.

Then I tried a different public IP altogether and just tried to SRC-NAT and DST-NAT to hopefully bypass the masquerade and effectively DMZ. (please forgive me I'm a total n00blette over here...)

And finally I realized several things. We supplied the router, but someone else did the internal networking and there is a switch or hub and wiring between where the microcell is. The Mikrotik hAP Lite is in a garage with thick walls. I will have him tomorrow plug it in directly to the port on the mikrotik and see if we have joy port forwarding to it then. Problem is GPS might not sync inside the garage :/

Second, the ATampersandT support said sometimes the problem is the nat itself. I tried to bypass it but could not as-is. My admin suggested making port 4 a slave to the WAN ether1-gateway so that hardware switching would fully bypass the routerOS and then I could assign a public IP directly to the Microcell using the Tower's router to issue it via a custom DHCP network. (The tower is MIkrotik :P )

I am probably over thinking this, and that's why I am coming to you guys on bended knee and humbly asking for any advice you might have. I did search forums but could not find a similar question or answer for this.

Other Notes
Several things the support people shared with me, for what it's worth.
1. Dropping Fragmented Packets should be turned off. I found a rule to do that I tried, no dice.
2. IPSec Passthrough should be off? I honestly don't know what this is.
3. MTU should be 1492. and I did try switching the interfaces all to 1492 and 1480. I switched it back though as it didn't fix it and I don't know what I'm doing enough to mess with it.
4. Latency should be under 50ms. I was pinging google DNS at 10ms-17ms-21ms responses.
5. Also I've been up for two days straight and I can't remember what else but I wrote it down, somewhere... I will probably realize later when I read this it's totally incoherent.
6.Support said the error code he was receiving from the microcell (i guess it calls home, often) only happens when ports are blocked. not the error for a bad unit etc.

The device is showing internet status and GPS status indicators so *should* work.

Summary
could be a bad microcell
could be I'm an idiot I am missing something glaringly obvious to you guys
could be some unknown reason why mikrotik and microcell won't play nice. Might just have to get him a different router?
could be a local device causing conflict (that's right, I'm looking at you DirecTV box!! o.0 )

While I originally became obsessed with and doubted my firewall rules, I am not sure if it's actually the culprit and tomorrow will have them plug in directly to the dish feed and assign a Public IP directly to the Microcell. If that works i'll try the port on the actual Mikrotik, so I know it's not the switch in the dudes house. I will keep you apprised.

In the meantime, ya'll can tell me what the best way I might setup this here fancy thingymajigger to work best?
 
User avatar
pukkita
Trainer
Trainer
Posts: 2986
Joined: Wed Dec 04, 2013 11:09 am
Location: Spain

Re: ATT Microcell Port Forwarding difficulties

Sat Sep 24, 2016 12:12 pm

Furthermore I can not even ping the microcell on the local network, which seems strange. I see the following.

ping 192.168.88.108
SEQ HOST SIZE TTL TIME STATUS
0 192.168.88.108 84 64 0ms port unreachable

[...]
We supplied the router, but someone else did the internal networking and there is a switch or hub and wiring between where the microcell is.
Do you have access to the microcell? can you post its relevant tcp/ip configuration?

First thing to get straight is routing between the mikrotik and the microcell, until that ping works no dst-nat is going to solve anything. Seems to be microcell related: could be firewalling, a static route, wrong wiring...

Default route through 192.168.88.1 as handed by DHCP should be fine, did you notice in the logs and DHCP leases if it was being assigned?.

BTW, 192.168.88.1 should be assigned to the bridge-local interface, not any of its ports. Check if it makes a difference.

Where's the microcell connected (ether port)?
Simplicity is the Ultimate Sophistication - Da Vinci
Getting the most out of this forum
 
User avatar
horse1bun
just joined
Topic Author
Posts: 13
Joined: Wed Jul 29, 2015 1:05 am

Re: ATT Microcell Port Forwarding difficulties

Sat Sep 24, 2016 1:04 pm

1. I did not know one could log into the microcell, or do you mean the online management portal? I figured that AT&T doesn't want you to log into it, I can't find any documentation though I'd be willing to try if anyone knows how. I can have them log in to the AT&T portal. Otherwise, I was told the device is DHCP so must have a DHCP server. I could go on site it's just an hour away and physically access it. They did try deregistering and registering on the portal yesterday several times.

2. From what I can gather, these devices are supposed to be "hardened" but not responding to ping made me think maybe the unit has "locked up" or gone haywire. But I also wouldn't be surprised if they just have icmp ping disabled by default on all of them for security reasons.

3. I can see in the log it is assigning DHCP Lease IP to the MAC.

4.That is just the default config, I switched it to the bridge-local and tested, no ping and NMAP scan showing ports closed.

5. The microcell is connected to a wall jack in the home, to a switch, and the switch to ether2-master-local. I tried turning on proxy-arp for the port as well, did not help.
 
User avatar
pukkita
Trainer
Trainer
Posts: 2986
Joined: Wed Dec 04, 2013 11:09 am
Location: Spain

Re: ATT Microcell Port Forwarding difficulties

Sat Sep 24, 2016 1:53 pm

1. I did not know one could log into the microcell, or do you mean the online management portal? I figured that AT&T doesn't want you to log into it, I can't find any documentation though I'd be willing to try if anyone knows how. I can have them log in to the AT&T portal. Otherwise, I was told the device is DHCP so must have a DHCP server. I could go on site it's just an hour away and physically access it. They did try deregistering and registering on the portal yesterday several times.
Ask them about the TCP/IP config then: make sure IP address/mask, and default gateway are being properly set by DHCP.
2. From what I can gather, these devices are supposed to be "hardened" but not responding to ping made me think maybe the unit has "locked up" or gone haywire. But I also wouldn't be surprised if they just have icmp ping disabled by default on all of them for security reasons.
IT can be that (blocked ICMP), though I think it's not a very clever idea in terms of troubleshooting... I'd further investigate this area as it is the most obvious anomaly, that could also explain your problems (reactive hardening, config reset...).
3. I can see in the log it is assigning DHCP Lease IP to the MAC.
Ok, so it's clear it is something related to the microcell: it may have another default gateway, or it could be set to not set it from DHCP. As you don't have access this is something that should be checked with AT&T or whoever manages the microcell.
4.That is just the default config, I switched it to the bridge-local and tested, no ping and NMAP scan showing ports closed.
Ok, expected that. That's the right way anyhow.
5. The microcell is connected to a wall jack in the home, to a switch, and the switch to ether2-master-local. I tried turning on proxy-arp for the port as well, did not help.
The fact DHCP is working between microcell and mikrotik shows this isn't a L2 or wiring issue.
Simplicity is the Ultimate Sophistication - Da Vinci
Getting the most out of this forum
 
IntrusDave
Forum Guru
Forum Guru
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: ATT Microcell Port Forwarding difficulties

Sat Sep 24, 2016 8:46 pm

No ports need to be "opened" for the AT&T Microcells to work. They only need to be allows to exit the network. They create an IPSec tunnel between themselves and the closed AT&T "POP". You will likely have a hard time with the GPS signal than anything else. As long as you are allowing the device to have internet access, that is all you need to do.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
villageworker
newbie
Posts: 38
Joined: Fri Nov 11, 2011 9:54 pm

Re: ATT Microcell Port Forwarding difficulties

Tue Sep 27, 2016 8:23 am

From the AT&T Microcell user manual http://www.att.com/media/att/2014/suppo ... Manual.pdf:
Installing the MicroCell behind a firewall, or behind a router with firewall capabilities, requires the following ports be opened to prevent the firewall from blocking communication with the network

123/UDP: NTP timing (NTP traffic) 443/TCP: HTTPS over TLS/SSL for provisioning and management traffic
4500/UDP: IPSec NAT Traversal (for all signaling, data, voice traffic)
500/UDP: IPSec Phase 1 prior to NAT detection (after NAT detection, 4500/UDP is used)
4500/UDP: After NAT detection, 4500/ UDP is used
NOTE: Customers attempting to connect a MicroCell on their corporate Internet connection may experience connection issues. The MicroCell is designed
to function using a direct Internet connection.
Make sure the ports are open outgoing and Natted to the micorcell IP address incoming in the firewall rules.


Hope this helps.
Provide robust Network, IoT & Building management systems
 
changeip
Forum Guru
Forum Guru
Posts: 3804
Joined: Fri May 28, 2004 5:22 pm

Re: ATT Microcell Port Forwarding difficulties

Tue Sep 27, 2016 6:39 pm

When we turned on any firewall rules on our mikrotik routed network it broke all the microcells. Esentially this started blocking fragmented packets therefore breaking the ipsec tunnels. Turning off all firewall rules fixed it. Not sure why Mikrotik starts disallowing fragmented packets once firewall rules are created - even with connection tracking off. So - it probably has to do with conn-track and firewall. Try newest mikrotik routeros version with RAW firewall and tell it to allow fragmented packets and see if that helps.
Colo and Wholesale Bandwidth Available! Sales at SanDiegoBroadband dot com
 
IntrusDave
Forum Guru
Forum Guru
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: ATT Microcell Port Forwarding difficulties

Wed Sep 28, 2016 2:45 am

Make sure the ports are open outgoing and Natted to the micorcell IP address incoming in the firewall rules.
Hope this helps.

DO NOT NAT the ports to the Microcell. It does NOT need incoming connections. The Microcell makes an outgoing Ipsec tunnel to the AT&T servers. You only need to make sure that it does not have outbound connections blocked.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
pukkita
Trainer
Trainer
Posts: 2986
Joined: Wed Dec 04, 2013 11:09 am
Location: Spain

Re: ATT Microcell Port Forwarding difficulties

Wed Sep 28, 2016 12:18 pm

When we turned on any firewall rules on our mikrotik routed network it broke all the microcells. Esentially this started blocking fragmented packets therefore breaking the ipsec tunnels. Turning off all firewall rules fixed it. Not sure why Mikrotik starts disallowing fragmented packets once firewall rules are created - even with connection tracking off. So - it probably has to do with conn-track and firewall. Try newest mikrotik routeros version with RAW firewall and tell it to allow fragmented packets and see if that helps.
Try disabling the rule that blocks fragmented packets: the forward chain, drop invalid one and see if it makes a difference. Make sure you're using latest ROS and firmware, there have been fixes solving packet loss/reordering/fragmentation lately.
Simplicity is the Ultimate Sophistication - Da Vinci
Getting the most out of this forum
 
changeip
Forum Guru
Forum Guru
Posts: 3804
Joined: Fri May 28, 2004 5:22 pm

Re: ATT Microcell Port Forwarding difficulties

Wed Sep 28, 2016 6:22 pm

You guys didn't understand. If you have firewall rules at all - even a single one that has nothing to do with ATT microcells - and connection tracking is off it will break fragmented packets. You cannot have a single firewall rule if you disable connection tracking and wish to have att microcells or other fragmented packets work.
Colo and Wholesale Bandwidth Available! Sales at SanDiegoBroadband dot com
 
User avatar
pukkita
Trainer
Trainer
Posts: 2986
Joined: Wed Dec 04, 2013 11:09 am
Location: Spain

Re: ATT Microcell Port Forwarding difficulties

Thu Sep 29, 2016 2:26 pm

@changeip you cannot disable connection tracking as the router needs to source nat outgoing traffic.

No connection tracking = no NAT, (and no firewall filter rules).

If your Tracking settings Enabled Parameter is set to auto, setting up any Firewall filter, mangle, or nat rules will enable it.

Disabling connection tracking is not an option, unless you assign a public IP directly onto the microcell, and leave it unfirewalled.

If you don't want RouterOS to "scrub" fragmented, out of order, etc packets, then disabling the drop invalid rule on both the input and forwarding chains will prevent that. (Shouldn't be necessary)
Simplicity is the Ultimate Sophistication - Da Vinci
Getting the most out of this forum
 
wispvt
Frequent Visitor
Frequent Visitor
Posts: 66
Joined: Tue May 10, 2011 4:20 pm

Re: ATT Microcell Port Forwarding difficulties

Mon Nov 21, 2016 11:00 pm

Did any one solve this as we have the same issue with a bunch of our clients. Only the IPSec connections of ATT microcells ever seem affected. Any help in resolving this would be appreciated.
 
changeip
Forum Guru
Forum Guru
Posts: 3804
Joined: Fri May 28, 2004 5:22 pm

Re: ATT Microcell Port Forwarding difficulties

Tue Nov 22, 2016 12:36 am

You have to allow fragments to pass ... or turn off connection-tracking so its ignored.
Colo and Wholesale Bandwidth Available! Sales at SanDiegoBroadband dot com
 
wispvt
Frequent Visitor
Frequent Visitor
Posts: 66
Joined: Tue May 10, 2011 4:20 pm

Re: ATT Microcell Port Forwarding difficulties

Mon Nov 28, 2016 3:53 pm

We have a bunch of routers at various sites that feed our core router at the head end that does NAT and mangle. Obviously we need connection tracking at the core, so how do we allow fragmented packets to pass through the firewall and is there any downside to this and would it break anything else?
 
User avatar
horse1bun
just joined
Topic Author
Posts: 13
Joined: Wed Jul 29, 2015 1:05 am

Re: ATT Microcell Port Forwarding difficulties

Tue Apr 25, 2017 9:00 am

I was never able to find a solution for this, unfortunately. We changed our policy to state that we do not support Microcells.

At this point I am convinced it does not have anything to do with port forwarding. As others in this thread have pointed out, it's a SRC-NAT connection and we still have people using them just fine on parts of our network behind double NATs with no port forwarding whatsoever.

I suspect that the packets are being fragmented somewhere on or off of our network that is breaking the IPsec tunnel back to AT&T. I worked with multiple users in three way calls with AT&T support and wrote to their engineers to see if they could work with us but had no response or escalation and no resolution on this. I used DHCP to issue a public IP directly to a microcell to see them try and say we were still blocking the ports >:) - still didn't work. The response from their CS was they are designed to be used BEHIND a router and not on a public IP. SMH...

The problem here is they keep telling the users we are blocking the ports and not allowing the devices to work, when in fact, we opened all of the ports specified and they just don't work. I tried different (non mikrotik) routers at the premises to no avail - but our core is all mikrotik routers and I wouldn't have it any other way.

I am not sure if this is the same issue you are having, or not, @wispvt. Good luck!

I wish I knew more but these devices are locked down you can't even ping them.

I discussed turning up MTU on some of the equipment on our network. My administrator did not want to tinker with it, understandably.

Most of the users in question have moved to using "WiFi Calling" in lieu of the microcells. Hope this helps...
 
changeip
Forum Guru
Forum Guru
Posts: 3804
Joined: Fri May 28, 2004 5:22 pm

Re: ATT Microcell Port Forwarding difficulties

Tue Apr 25, 2017 9:28 am

It has to do with packet fragments not making it. I finally was able to track this down by running a packet capture at the very first router closest to the customer, and then running one successively each hop out and figuring out where things broke. It came to the first router that had connection-tracking disabled, BUT with a single firewall drop rule. That activated the firewall module and in turn stopped fragments.

HOWEVER, the problem might not be in your control. I would packet capture as it is leaving and see if the fragments are all in tact on the way out to make sure you aren't losing them before they even leave your network. If at that same first hop you aren't receiving them all properly then you need to take it up with your ISP.

PS- you absolutely do not need to forward port 80,443,53,etc like it says you do. Thats obsurd and we all know you don't need to do that.
Colo and Wholesale Bandwidth Available! Sales at SanDiegoBroadband dot com

Who is online

Users browsing this forum: Google [Bot] and 97 guests