Port Forwarding Woes

RouterOS Beginner Basics
1

Good evening,
I recently purchased my first routerboard, an RB2011UiAS-RM, and am very pleased with it except for one challenge that I haven’t been able to resolve. Please forgive me if I’ve overlooked something obvious.

I’m replacing a router running DD-WRT that performs very poorly, but as I’m trying to avoid disrupting services, I’ve placed the MikroTik in front of the existing router (between it and the cable modem), double-NATing everything on the existing network. The existing router is hosting the 192.168.0.0/24 subnet and I’m moving to the 10.0.0.0/16 subnet on the MikroTik. The WAN interface of the existing router is assigned the 10.0.0.37 address on the MikroTik.

I have several servers behind the previous router to which I forward several ports. I was able to move all but one of these servers onto the 10.0.0.0/16 subnet hosted by the MikroTik and configure the port forwards to point to their new addresses and all is well.

When I try to configure the last server similarly, however, I am unable to access the services on the destination server from the Internet. This server is a Cent OS 6 server that is dual homed on both the old (192.168.0.10) and new (10.0.0.11) subnets. This server exposes SSH (22) and HTTP (80) to the Internet.

  • I can access the services by connecting to both the old and new IP from their respective networks. That is, I can connect to the services at 10.0.0.11 when connected to the 10.0.0.0/16 network, and I can connect to the 192.168.0.10 services when connected to the 192.168.0.0/24 network. I can even access the 10.0.0.11 services when connected to the 192.168.0.0/24 network.

  • I can configure port forwarding on the existing DD-WRT router to forward to the 192.168.0.10 address and configure the MikroTik to forward the ports for the outside world to the existing router’s IP (10.0.0.37), and the services function as expected. Using Steve Gibson’s Sheilds Up! tool, these ports are listed as “Open”.

  • If, however, I configure the MikroTik to forward to the 10.0.0.11 address directly, the services are completely inaccessible to the Internet. Using Steve Gibson’s Shields Up! tool, these ports are listed as “Stealth”.

I have verified this behavior by switching amongst both configurations. Services accessible when doubly-forwarded, inaccessible when only traversing the MikroTik. The behavior is completely consistent.

My forwarding rules in the working configuration appear as follows:

[user@MikroTik] > /ip firewall nat print detail 
Flags: X - disabled, I - invalid, D - dynamic 
 0    ;;; default configuration
      chain=srcnat action=masquerade out-interface=ether1-gateway log=no log-prefix="" 

 1    chain=dstnat action=dst-nat to-addresses=10.0.0.37 to-ports=80 protocol=tcp 
      in-interface=ether1-gateway dst-port=80 log=no log-prefix="" 

 2    chain=dstnat action=dst-nat to-addresses=10.0.1.1 to-ports=443 protocol=tcp 
      in-interface=ether1-gateway dst-port=443 log=no log-prefix="" 

 3    chain=dstnat action=dst-nat to-addresses=10.0.0.37 to-ports=22 protocol=tcp 
      in-interface=ether1-gateway dst-port=22 log=no log-prefix=""

And in the non-working configuration:

[user@MikroTik] > /ip firewall nat print detail 
Flags: X - disabled, I - invalid, D - dynamic 
 0    ;;; default configuration
      chain=srcnat action=masquerade out-interface=ether1-gateway log=no log-prefix="" 

 1    chain=dstnat action=dst-nat to-addresses=10.0.0.11 to-ports=80 protocol=tcp 
      in-interface=ether1-gateway dst-port=80 log=no log-prefix="" 

 2    chain=dstnat action=dst-nat to-addresses=10.0.1.1 to-ports=443 protocol=tcp 
      in-interface=ether1-gateway dst-port=443 log=no log-prefix="" 

 3    chain=dstnat action=dst-nat to-addresses=10.0.0.11 to-ports=22 protocol=tcp 
      in-interface=ether1-gateway dst-port=22 log=no log-prefix=""

What I’ve Done:

  • My initial thought was that the system firewall on the problematic server was blocking access on that subnet, but this does not appear to be the case as I can access the services from the local network. A visual inspection of the iptables rules does not indicate anything out of place to me. Also, completely disabling the firewall temporarily has no effect.

The iptables rules appear as follows:

# Firewall configuration maintained by puppet.
# Manual customization of this file is not recommended.
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
# SSH
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
# NTP
-A INPUT -p udp -m state --state NEW --dport 123 -j ACCEPT
# DNS
-A INPUT -p udp -m state --state NEW --dport 53 -j ACCEPT
-A INPUT -p tcp -m state --state NEW --dport 53 -j ACCEPT
# LDAP
-A INPUT -p tcp -m state --state NEW --dport 389 -j ACCEPT
-A INPUT -p tcp -m state --state NEW --dport 636 -j ACCEPT
# Kerberos
-A INPUT -p tcp -m state --state NEW --dport 88 -j ACCEPT
-A INPUT -p tcp -m state --state NEW --dport 749 -j ACCEPT
# NFS
-A INPUT -p tcp -m state --state NEW --dport 111   -j ACCEPT
-A INPUT -p udp -m state --state NEW --dport 111   -j ACCEPT
-A INPUT -p tcp -m state --state NEW --dport 2049  -j ACCEPT
-A INPUT -p udp -m state --state NEW --dport 2049  -j ACCEPT
-A INPUT -p tcp -m state --state NEW --dport 32803 -j ACCEPT
-A INPUT -p udp -m state --state NEW --dport 32769 -j ACCEPT
-A INPUT -p tcp -m state --state NEW --dport 892   -j ACCEPT
-A INPUT -p udp -m state --state NEW --dport 892   -j ACCEPT
-A INPUT -p tcp -m state --state NEW --dport 875   -j ACCEPT
-A INPUT -p udp -m state --state NEW --dport 875   -j ACCEPT
-A INPUT -p tcp -m state --state NEW --dport 662   -j ACCEPT
-A INPUT -p udp -m state --state NEW --dport 662   -j ACCEPT
# Samba
-A INPUT -p tcp -m state --state NEW --dport 137 -j ACCEPT
-A INPUT -p tcp -m state --state NEW --dport 138 -j ACCEPT
-A INPUT -p tcp -m state --state NEW --dport 139 -j ACCEPT
-A INPUT -p tcp -m state --state NEW --dport 445 -j ACCEPT
# mDNS
-A INPUT -p udp -m state --state NEW --dport 5353 -j ACCEPT
-A INPUT -p tcp -m state --state NEW --dport 5353 -j ACCEPT
-A INPUT -p udp -m state --state NEW --dport 5354 -j ACCEPT
-A INPUT -p tcp -m state --state NEW --dport 5354 -j ACCEPT
# HTTP
-A INPUT -p tcp -m state --state NEW --dport 80 -j ACCEPT
-A INPUT -p tcp -m state --state NEW --dport 443 -j ACCEPT
# Puppet
-A INPUT -p tcp -m state --state NEW --dport 8140 -j ACCEPT
## MySQL
-A INPUT -p tcp -m state --state NEW --dport 3306 -s 10.0.1.2 -j ACCEPT

## Reverse SSH Tunnels
-A INPUT -p tcp -m state --state NEW --dport  3001 -j ACCEPT
-A INPUT -p tcp -m state --state NEW --dport  4201 -j ACCEPT
-A INPUT -p tcp -m state --state NEW --dport 31337 -j ACCEPT

# Default
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT
  • I have confirmed that the services are listening on both interfaces.
$ sudo netstat -tunelp | grep -E ':22|:80 '
tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      0          111935568  2265/sshd           
tcp        0      0 0.0.0.0:80                  0.0.0.0:*                   LISTEN      0          20671      2973/nginx          
tcp        0      0 :::22                       :::*                        LISTEN      0          111935570  2265/sshd
  • I have also confirmed that I can forward the same services to another host and they function. This host is running debian stable.
[user@MikroTik] > /ip firewall nat print detail 
Flags: X - disabled, I - invalid, D - dynamic 
 0    ;;; default configuration
      chain=srcnat action=masquerade out-interface=ether1-gateway log=no log-prefix="" 

 1    chain=dstnat action=dst-nat to-addresses=10.0.1.1 to-ports=80 protocol=tcp 
      in-interface=ether1-gateway dst-port=80 log=no log-prefix="" 

 2    chain=dstnat action=dst-nat to-addresses=10.0.1.1 to-ports=443 protocol=tcp 
      in-interface=ether1-gateway dst-port=443 log=no log-prefix="" 

 3    chain=dstnat action=dst-nat to-addresses=10.0.1.1 to-ports=22 protocol=tcp 
      in-interface=ether1-gateway dst-port=22 log=no log-prefix=""

All this leads me to believe that there is something strange that the MikroTik is doing to the forwarded packets that is causing them to be rejected by the CentOS box. I’ve been banging my head on this for three days and am completely stumped. This is the last hold up to getting rid of the double NAT configuration and routing all of my traffic through the MikroTik.

Sorry for the long post. Any ideas?

2

To add even more confusion, I can SSH from the MikroTik to the troublesome server.

(Slightly sanitized)

[user@MikroTik] > system ssh 10.0.0.11 src-address=10.0.0.1
Last login: Tue Sep  8 14:11:25 2015 from mikrotik
Loading configuration for Linux
user@10.0.0.11:~$ ifconfig xenbr1
xenbr1    Link encap:Ethernet  HWaddr XX:XX:XX:XX:XX:XX  
          inet addr:10.0.0.11  Bcast:10.0.255.255  Mask:255.255.0.0
          inet6 addr: fe80::XXXX:XXXX:XXXX:XXXX/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:5751049 errors:0 dropped:0 overruns:0 frame:0
          TX packets:5233757 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:777765223 (741.7 MiB)  TX bytes:846133448 (806.9 MiB)

user@10.0.0.11:~$ logout

Welcome back!
[user@MikroTik] >

So it’s just the port forward that doesn’t work, which leads me even more to believe there is something weird happening to the packets getting mangled by the dst-nat rule. Any ideas?

3

Hello and thank you for the excellent description and for posting the relevant configurations.

In RouterOS, to forward a port to a device, you need two entries in the firewall area: one for NAT and one for the actual firewall itself.

You have the correct NAT rule, you are just missing the ip firewall rule to ‘allow forward dst-ip=lan-ip protocol=tcp dst-port=*destination-port" in-interface=ISP action=accept’

Tony

4

I’m confused. I don’t see any rules like that for forwarding to any of the other hosts where port forwarding is functioning correctly.

[user@MikroTik] > /ip firewall export               
# sep/09/2015 07:50:21 by RouterOS 6.23
# software id = 9V5D-YDMK
#
/ip firewall filter
add chain=input comment="default configuration" protocol=icmp
add chain=input comment="default configuration" connection-state=\
    established,related
add action=drop chain=input comment="default configuration" in-interface=\
    ether1-gateway
add chain=forward comment="default configuration" connection-state=\
    established,related
add action=drop chain=forward comment="default configuration" \
    connection-state=invalid
add action=drop chain=forward comment="default configuration" \
    connection-nat-state=!dstnat connection-state=new in-interface=\
    ether1-gateway
/ip firewall nat
add action=masquerade chain=srcnat comment="default configuration" \
    out-interface=ether1-gateway
add action=dst-nat chain=dstnat dst-port=80 in-interface=ether1-gateway \
    protocol=tcp to-addresses=10.0.1.1 to-ports=80
add action=dst-nat chain=dstnat dst-port=443 in-interface=ether1-gateway \
    protocol=tcp to-addresses=10.0.1.1 to-ports=443
add action=dst-nat chain=dstnat dst-port=22 in-interface=ether1-gateway \
    protocol=tcp to-addresses=10.0.0.37 to-ports=22
[user@MikroTik] >

Am I missing something? Where should I add the forwarding rule?

Thanks for your help!

5

Hi,

check ip->services. Change these “service” ports, or disable for test purposes. You should be good to go.

6

And server’s default gateway is still set to DD-WRT (I suppose 192.168.0.1)? If so, it can be blocked by DD-WRT’s firewall. Because when you forward port to 10.0.0.11, it goes there directly from RB, bypassing DD-WRT. But since it has some public IP as source, reply packets will go via default route, i.e. to DD-WRT. And depending on how its firewall is configured, it might see those packets as invalid, because they don’t belong to any known connection.

7

That makes perfect sense and I bet you’re right! It definitely still has 192.168.0.1 as the default route. I’ll try moving it solely onto the 10.0.0.0/16 network this evening and let you know if that solves it!

Thanks for the idea!

8

Mikrotik also offer a nice feature for dst-nat rules. You can create one firewall rule to allow all dst-nat rules:-

/ip firewall filter
add chain=forward connection-nat-state=dstnat in-interface=pppoe-out1

9

And we have a winner! Thank you so much, Sob! In hindsight, this is obvious, but I was completely stymied. I sincerely appreciate the help!

For what it’s worth, I haven’t actually taken it off the 192.168.0.0/24 network yet. All I had to do was change the default route and the forwarding worked. I’ll be able to get rid of the double-NAT this weekend without affecting anyone!