Port forwarding not working from Public IP ranges

Floydian · April 13, 2021, 12:00pm

Hello,

I’m trying to create a port forwarding set up, which currently is working when accessing from an IP in the 192.168.x.x range, but not working when trying to access from a public IP.
I’m issuing a curl command to fetch a website. This works from the local IP, but not when coming from a public IP.

The setup is:
Apple AirPort → OldLAN (192.168.x.x) → RB4011 (192.168.0.102) → NewLan (10.2.x.x)

When I issue the curl command from a PC on the OldLAN, I can access the NewLAN and fetch data from the server.
When I issue the same curl command from outside, through the Apple AirPort, the connection comes through to the RB4011, however no data is returned from the NewLAN server.
To rule out the AirPort I have also tried connecting the RB4011 directly to the internet, which gives the same result - not possible to connect.

I see the following in the log file:
Access from Private IP in the 192.168.x.x range:

13:22:40 firewall,info [httpnat] dstnat: in:outside out:(unknown 0), src-mac xx:xx:xx:xx:xx:xx, proto TCP (SYN), 192.168.0.20:57585->192.168.0.102:80, len 64
13:22:40 firewall,info [http] forward: in:outside out:dmzbridge, src-mac xx:xx:xx:xx:xx:xx, proto TCP (SYN), 192.168.0.20:57585->10.2.0.10:80, NAT 192.168.0.20:57585->(192.168.0.102:80->10.2.0.10:80), len 64

Access from a Public IP:

13:26:27 firewall,info [httpnat] dstnat: in:outside out:(unknown 0), src-mac 24:a0:74:73:a7:e8, proto TCP (SYN), 206.189.180.4:34026->192.168.0.102:80, len 60
13:26:27 firewall,info [http] forward: in:outside out:dmzbridge, src-mac 24:a0:74:73:a7:e8, proto TCP (SYN), 206.189.180.4:34026->10.2.0.10:80, NAT 206.189.180.4:34026->(192.168.0.102:80->10.2.0.10:80), len 60

Only apparent difference is the “len 64” when it’s working and “len 60” when it’s not working (?).

Have tried to enable logging on the different rules, but not been able to get a hit on what the difference is between the two connections.

Below is my current config:

# apr/13/2021 13:45:44 by RouterOS 6.48.1
# software id = IJJI-4YGG
#
# model = RB4011iGS+
# serial number = xxxxxx
/interface bridge
add comment="Admin bridge" name=admbridge
add comment=DMZ name=dmzbridge protocol-mode=none
add comment="IoT Bridge" name=iotbridge
add admin-mac=xx:xx:xx:xx:xx:xx auto-mac=no comment=defconf name=lanbridge
/interface ethernet
set [ find default-name=ether10 ] comment="Admin port" name=admin poe-out=off
set [ find default-name=ether2 ] comment=DMZ name=dmz1
set [ find default-name=ether3 ] comment=DMZ name=dmz2
set [ find default-name=ether4 ] comment="IoT Interface" name=iot1
set [ find default-name=ether1 ] comment="Wan Interface" name=outside
/interface ethernet switch port
set 0 default-vlan-id=0
set 1 default-vlan-id=0
set 2 default-vlan-id=0
set 3 default-vlan-id=0
set 4 default-vlan-id=0
set 5 default-vlan-id=0
set 6 default-vlan-id=0
set 7 default-vlan-id=0
set 8 default-vlan-id=0
set 9 default-vlan-id=0
set 10 default-vlan-id=0
set 11 default-vlan-id=0
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
add name=DMZ
add name=IOT
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=lan_dhcp_pool ranges=10.1.0.10-10.1.0.254
add name=dmz_dhcp_pool ranges=10.2.0.200-10.2.0.250
add name=iot_dhcp_pool ranges=10.3.0.200-10.3.0.250
/ip dhcp-server
add address-pool=lan_dhcp_pool disabled=no interface=lanbridge name=\
    lan_dhcp_server
add address-pool=lan_dhcp_pool disabled=no interface=admbridge name=\
    admin_dhcp_server
add address-pool=dmz_dhcp_pool disabled=no interface=dmzbridge name=\
    dmz_dhcp_server
add address-pool=iot_dhcp_pool disabled=no interface=iotbridge name=\
    iot_dhcp_server
/interface bridge port
add bridge=dmzbridge comment=dmz interface=dmz1
add bridge=dmzbridge comment=dmz interface=dmz2
add bridge=iotbridge comment=defconf interface=iot1
add bridge=lanbridge comment=defconf interface=ether5
add bridge=lanbridge comment=defconf interface=ether6
add bridge=lanbridge comment=defconf interface=ether7
add bridge=lanbridge comment=defconf interface=ether8
add bridge=lanbridge comment=defconf interface=ether9
add bridge=admbridge comment=defconf interface=admin
add bridge=lanbridge comment=defconf interface=sfp-sfpplus1
/ip neighbor discovery-settings
set discover-interface-list=none
/ip settings
set tcp-syncookies=yes
/interface list member
add comment=defconf interface=lanbridge list=LAN
add comment=defconf interface=outside list=WAN
add interface=dmzbridge list=LAN
/ip address
add address=10.1.0.1/24 comment="LAN Bridge" interface=lanbridge network=\
    10.1.0.0
add address=10.2.0.1/24 comment="DMZ Bridge" interface=dmzbridge network=\
    10.2.0.0
add address=10.3.0.1/24 comment="IOT bridge" interface=iotbridge network=\
    10.3.0.0
/ip cloud
set update-time=no
/ip dhcp-client
add comment=defconf disabled=no interface=outside
/ip dhcp-server network
add address=10.1.0.0/24 comment="LAN DHCP" dns-server=\
    192.168.0.y,192.168.0.x domain=example.com gateway=10.1.0.1 netmask=24
add address=10.2.0.0/24 comment="DMZ DHCP" domain=example.com gateway=10.2.0.1 \
    netmask=24
add address=10.3.0.0/24 comment="IOT DHCP" dns-server=\
    192.168.0.x,192.168.0.y domain=example.com gateway=10.3.0.1 netmask=24
/ip dns static
add address=10.4.0.1 comment=defconf name=router.lan
/ip firewall address-list
add address=10.1.0.10-10.1.0.254 list=allowed_to_router
add address=10.4.0.10-10.4.0.254 list=allowed_to_router
add address=0.0.0.0/8 comment=RFC6890 list=not_in_internet
add address=172.16.0.0/12 comment=RFC6890 list=not_in_internet
add address=192.168.0.0/16 comment=RFC6890 disabled=yes list=not_in_internet
add address=10.0.0.0/8 comment=RFC6890 list=not_in_internet
add address=169.254.0.0/16 comment=RFC6890 list=not_in_internet
add address=127.0.0.0/8 comment=RFC6890 list=not_in_internet
add address=224.0.0.0/4 comment=Multicast list=not_in_internet
add address=198.18.0.0/15 comment=RFC6890 list=not_in_internet
add address=192.0.0.0/24 comment=RFC6890 list=not_in_internet
add address=192.0.2.0/24 comment=RFC6890 list=not_in_internet
add address=198.51.100.0/24 comment=RFC6890 list=not_in_internet
add address=203.0.113.0/24 comment=RFC6890 list=not_in_internet
add address=100.64.0.0/10 comment=RFC6890 list=not_in_internet
add address=240.0.0.0/4 comment=RFC6890 list=not_in_internet
add address=192.88.99.0/24 comment="6to4 relay Anycast [RFC 3068]" list=\
    not_in_internet
add list=ddos-attackers
add list=ddos-target
/ip firewall filter
add action=accept chain=input comment="Accept Established, Related" \
    connection-state=established,related
add action=drop chain=input comment="Drop all invalid packets from WAN" \
    connection-state=invalid log=yes log-prefix="[drop invalid] "
add action=jump chain=forward comment="DDOS protection" connection-state=new \
    jump-target=detect-ddos log-prefix="[ddos jump] "
add action=fasttrack-connection chain=forward comment=FastTrack \
    connection-state=established,related log-prefix="[fasttrack] "
add action=accept chain=forward comment="Allow Estab & Related" \
    connection-state=established,related
add action=accept chain=forward comment=HTTP dst-port=80 in-interface=outside \
    log=yes log-prefix="[http] " protocol=tcp
add action=accept chain=forward comment=HTTPS dst-port=443 in-interface=\
    outside protocol=tcp
add action=accept chain=input comment="Allow ICMP" protocol=icmp
add action=accept chain=input comment="allow admin access" src-address-list=\
    allowed_to_router
add action=drop chain=input comment="drop all not coming from admbridge"
add action=accept chain=forward comment="defconf: accept in ipsec policy" \
    ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" \
    ipsec-policy=out,ipsec
add action=drop chain=forward comment="Drop Invalid Forward" \
    connection-state=invalid log=yes log-prefix="[invalid fwd]"
add action=accept chain=forward comment="Screen sharing from LAN to DMZ" \
    dst-port=5900 in-interface=lanbridge log-prefix=screen_sharing protocol=\
    tcp
add action=accept chain=forward in-interface=dmzbridge out-interface=\
    lanbridge protocol=tcp src-port=5900
add action=accept chain=forward comment="MariaDB Lan to DMZ" \
    connection-state=established,related,new,untracked dst-port=3306 \
    protocol=tcp
add action=drop chain=forward comment=\
    "defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat \
    connection-state=new in-interface-list=WAN log=yes log-prefix=\
    "[drop inv wan]"
add action=drop chain=forward comment=\
    "Drop tries to reach not public addresses from LAN" disabled=yes \
    dst-address-list=not_in_internet in-interface=lanbridge log=yes \
    log-prefix="[!public_from_LAN]" out-interface=!lanbridge
add action=drop chain=forward comment=\
    "Drop tries to reach not public addresses from IOT" dst-address-list=\
    not_in_internet in-interface=iotbridge log=yes log-prefix=\
    "[!public_from_IOT] " out-interface=!iotbridge
add action=drop chain=forward comment=\
    "Drop tries to reach not public addresses from DMZ" disabled=yes \
    dst-address-list=not_in_internet in-interface=dmzbridge log=yes \
    log-prefix=!public_from_DMZ out-interface=!dmzbridge
add action=drop chain=forward comment=\
    "Drop tries to reach not public addresses from ADM" disabled=yes \
    dst-address-list=not_in_internet in-interface=admbridge log=yes \
    log-prefix=!public_from_ADM out-interface=!admbridge
add action=drop chain=forward comment=\
    "Drop incoming packets that are not NATed" connection-nat-state=!dstnat \
    connection-state=new in-interface=outside log=yes log-prefix="[!NAT] "
add action=drop chain=forward comment=\
    "Drop incoming from internet which is not public IP" in-interface=outside \
    log=yes log-prefix="[!public]" src-address-list=not_in_internet
add action=drop chain=forward comment=\
    "Drop packets from LAN that do not have LAN IP" in-interface=lanbridge \
    log=yes log-prefix="[LAN_!LAN]" src-address=!10.1.0.0/24
add action=drop chain=forward comment=\
    "Drop packets from DMZ which does not have DMZ IP" in-interface=dmzbridge \
    log=yes log-prefix="[DMZ_!DMZ]" src-address=!10.2.0.0/24
add action=drop chain=forward comment=\
    "Drop packets from IOT which does not have IOT IP" in-interface=iotbridge \
    log=yes log-prefix="[IOT_!IOT]" src-address=!10.3.0.0/24
add action=return chain=detect-ddos dst-limit=32,32,src-and-dst-addresses/10s
add action=add-dst-to-address-list address-list=ddos-target \
    address-list-timeout=10m chain=detect-ddos
add action=add-src-to-address-list address-list=ddos-attackers \
    address-list-timeout=10m chain=detect-ddos
add action=return chain=detect-ddos dst-limit=32,32,src-and-dst-addresses/10s \
    protocol=tcp tcp-flags=syn,ack
/ip firewall mangle
add action=log chain=prerouting connection-state=new disabled=yes dst-port=80 \
    log=yes log-prefix="[port80] " protocol=tcp
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" \
    ipsec-policy=out,none log-prefix="[masq]" out-interface-list=WAN
add action=masquerade chain=srcnat disabled=yes dst-address=10.2.0.10 \
    out-interface=lanbridge protocol=tcp src-address=10.1.0.0/24
add action=dst-nat chain=dstnat dst-port=80 in-interface-list=WAN log=yes \
    log-prefix="[httpnat] " protocol=tcp to-addresses=10.2.0.10
add action=dst-nat chain=dstnat dst-port=443 in-interface=outside protocol=\
    tcp to-addresses=10.2.0.10 to-ports=443
add action=src-nat chain=srcnat out-interface=outside src-address=10.1.0.0/24 \
    to-addresses=84.x.y.z
add action=src-nat chain=srcnat log=yes log-prefix="[srcnat]" out-interface=\
    outside src-address=10.2.0.0/24 to-addresses=84.x.y.z
add action=src-nat chain=srcnat out-interface=outside src-address=10.3.0.0/24 \
    to-addresses=84.x.y.z
/ip firewall raw
add action=drop chain=prerouting dst-address-list=ddos-target log=yes \
    log-prefix="[ddos attacker] " src-address-list=ddos-attackers
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www address=10.1.0.0/24 port=8080
set ssh port=22
set api disabled=yes
set winbox disabled=yes
set api-ssl disabled=yes
/system clock
set time-zone-name=Europe/Copenhagen
/system identity
set name=RB4011
/system ntp client
set enabled=yes server-dns-names=0.dk.pool.ntp.org
/tool bandwidth-server
set enabled=no
/tool mac-server
set allowed-interface-list=none
/tool mac-server mac-winbox
set allowed-interface-list=none
/tool mac-server ping
set enabled=no

Thank you in advance for any assistance/pointers that can help me solve this.

Floydian · April 15, 2021, 9:53pm

Now, this is driving me nuts.

Had a MT hEX (RB750Gr3) lying around, and tried to use this for testing.

After reading up on stuff, decided to use VLAN’s instead of Bridges to segment the network.

Followed the guide discussed here: http://forum.mikrotik.com/t/using-routeros-to-vlan-your-network/126489/4 more specific the Router-Switch-AP (all in one) part.

Now I got this working on the RB750Gr3 - I can access the server behind it both from the intermediate lan (IP: 192.168.0.x), as well as from outside through the Apple AirPort.
So far so good.
Dumped my config, and followed the guide + config on the RB4011iGS+ thinking that switching to VLAN’s actually solved the issue, however, once done configuring, I’m at the exact same point as I was with the Bridges. I can access the server from the Intermediate network, but not when accessing through the Apple AirPort.

Below the config from the RB4011 which is not working. Attached is the config from th RB750Gr3, which is working.

Am I missing something with regards to the differences in hardware that I should be aware of on the RB4011 (Having two switches compared to the 750’s one)?

# apr/15/2021 23:08:14 by RouterOS 6.48
# software id = IJJI-4YGG
#
# model = RB4011iGS+
# serial number = 
/interface bridge
add admin-mac=xx:xx:xx:xx:xx:xx auto-mac=no comment="Main Bridge" name=MainBridge \
    protocol-mode=none vlan-filtering=yes
/interface ethernet
set [ find default-name=ether2 ] name=admin
set [ find default-name=ether3 ] name=dmz1
set [ find default-name=ether4 ] name=dmz2
set [ find default-name=ether5 ] name=iot1
set [ find default-name=ether6 ] name=lan1
set [ find default-name=ether7 ] name=lan2
set [ find default-name=ether8 ] name=lan3
set [ find default-name=ether9 ] name=lan4
set [ find default-name=ether10 ] name=lan5
set [ find default-name=ether1 ] name=outside
/interface vlan
add interface=MainBridge name=ADM_VLAN vlan-id=99
add interface=MainBridge name=DMZ_VLAN vlan-id=20
add interface=MainBridge name=IOT_VLAN vlan-id=30
add interface=MainBridge name=LAN_VLAN vlan-id=10
/interface ethernet switch port
set 0 default-vlan-id=0
set 1 default-vlan-id=0
set 2 default-vlan-id=0
set 3 default-vlan-id=0
set 4 default-vlan-id=0
set 5 default-vlan-id=0
set 6 default-vlan-id=0
set 7 default-vlan-id=0
set 8 default-vlan-id=0
set 9 default-vlan-id=0
set 10 default-vlan-id=0
set 11 default-vlan-id=0
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
add name=VLAN
add name=ADMIN
add name=IOT
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name="Bridge Pool" ranges=10.1.0.10-10.1.0.250
add name=LAN_POOL ranges=10.1.0.100-10.1.0.250
add name=DMZ_POOL ranges=10.2.0.100-10.2.0.250
add name=IOT_POOL ranges=10.3.0.100-10.3.0.250
add name=ADM_POOL ranges=10.4.0.100-10.4.0.250
/ip dhcp-server
add address-pool="Bridge Pool" disabled=no interface=MainBridge name=MB_DHCP
add address-pool=LAN_POOL disabled=no interface=LAN_VLAN name=LAN_DHCP
add address-pool=DMZ_POOL disabled=no interface=DMZ_VLAN name=DMZ_DHCP
add address-pool=IOT_POOL disabled=no interface=IOT_VLAN name=IOT_DHCP
add address-pool=ADM_POOL disabled=no interface=ADM_VLAN name=ADM_DHCP
/interface bridge port
add bridge=MainBridge comment=ADM frame-types=\
    admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=\
    admin pvid=99
add bridge=MainBridge comment="DMZ 1" frame-types=\
    admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=\
    dmz1 pvid=20
add bridge=MainBridge comment="DMZ 2" frame-types=\
    admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=\
    dmz2 pvid=20
add bridge=MainBridge comment="IOT 1" frame-types=\
    admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=\
    iot1 pvid=30
add bridge=MainBridge comment="LAN 1" frame-types=\
    admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=\
    lan1 pvid=10
add bridge=MainBridge comment="LAN 2" frame-types=\
    admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=\
    lan2 pvid=10
add bridge=MainBridge comment="LAN 3" frame-types=\
    admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=\
    lan3 pvid=10
add bridge=MainBridge comment="LAN 4" frame-types=\
    admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=\
    lan4 pvid=10
add bridge=MainBridge comment="LAN 5" frame-types=\
    admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=\
    lan5 pvid=10
add bridge=MainBridge comment=defconf interface=sfp-sfpplus1
/ip neighbor discovery-settings
set discover-interface-list=ADMIN
/interface bridge vlan
add bridge=MainBridge tagged=MainBridge untagged=lan1,lan2,lan3,lan4,lan5 \
    vlan-ids=10
add bridge=MainBridge tagged=MainBridge untagged=dmz1,dmz2 vlan-ids=20
add bridge=MainBridge tagged=MainBridge untagged=iot1 vlan-ids=30
add bridge=MainBridge tagged=MainBridge untagged=admin vlan-ids=99
/interface list member
add comment=defconf interface=MainBridge list=LAN
add comment=defconf interface=outside list=WAN
add interface=ADM_VLAN list=ADMIN
add interface=DMZ_VLAN list=VLAN
add interface=IOT_VLAN list=VLAN
add interface=LAN_VLAN list=VLAN
add interface=ADM_VLAN list=VLAN
/ip address
add address=10.1.0.1/24 comment="Main bridge" interface=MainBridge network=\
    10.1.0.0
add address=10.4.0.1/24 interface=ADM_VLAN network=10.4.0.0
add address=10.1.0.1/24 interface=LAN_VLAN network=10.1.0.0
add address=10.2.0.1/24 interface=DMZ_VLAN network=10.2.0.0
add address=10.3.0.1/24 interface=IOT_VLAN network=10.3.0.0
/ip dhcp-client
add comment=defconf disabled=no interface=outside
/ip dhcp-server network
add address=10.2.0.0/24 comment="DMZ Network" dns-server=192.168.0.x \
    gateway=10.2.0.1
add address=10.3.0.0/24 comment="IOT Network" dns-server=192.168.0.x \
    gateway=10.3.0.1
add address=10.4.0.0/24 comment="ADM Network" dns-server=192.168.0.x \
    gateway=10.4.0.1
add address=10.1.0.0/24 comment="LAN Network" dns-server=192.168.0.x \
    gateway=10.1.0.1
/ip dns
set allow-remote-requests=yes servers=192.168.0.x,192.168.0.y
/ip dns static
add address=10.1.0.1 comment=defconf name=router.lan
/ip firewall filter
add action=accept chain=input comment=\
    "Accept established,related & untracked" connection-state=\
    established,related,untracked
add action=drop chain=input comment="Drop invalid" connection-state=invalid
add action=accept chain=input comment="Allow VLAN to router services" \
    in-interface-list=VLAN
add action=accept chain=input comment="Allow Admin_VLAN Full Access" \
    in-interface=ADM_VLAN
add action=accept chain=input comment="Accept ICMP" protocol=icmp
add action=accept chain=input comment=\
    "Accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=drop chain=input comment="Drop everything else"
add action=accept chain=forward comment="HTTP 8081" dst-address=192.168.0.z \
    dst-port=8081 in-interface=outside log=yes log-prefix="[fw8081]" \
    protocol=tcp
add action=accept chain=forward comment="Accept in ipsec policy" \
    ipsec-policy=in,ipsec
add action=accept chain=forward comment="Accept out ipsec policy" \
    ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="Fasttrack Forward" \
    connection-state=established,related
add action=accept chain=forward comment=\
    "Accept established, related & untracked" connection-state=\
    established,related,untracked
add action=accept chain=forward comment="VLAN Internet Access only" \
    connection-state=new in-interface-list=VLAN out-interface-list=WAN
add action=accept chain=forward comment="Allow Port Forwarding" \
    connection-nat-state=dstnat connection-state=new in-interface-list=WAN \
    log=yes log-prefix="[fw dstnat]"
add action=drop chain=forward comment="Drop invalid forward"
add action=drop chain=forward comment="Drop all from WAN not DSTNATed" \
    connection-nat-state=!dstnat connection-state=new in-interface-list=WAN
/ip firewall nat
add action=masquerade chain=srcnat comment="Default masquerade" \
    out-interface-list=WAN
add action=dst-nat chain=dstnat comment="NAT test" dst-address=192.168.0.z \
    dst-port=8081 log=yes log-prefix="[NAT8081]" protocol=tcp to-addresses=\
    10.2.0.x to-ports=80
/system clock
set time-zone-name=Europe/Copenhagen
/system identity
set name=fldMTRB4011
/tool mac-server
set allowed-interface-list=ADMIN
/tool mac-server mac-winbox
set allowed-interface-list=ADMIN

RB750Gr3 20210415_working.rsc (5.28 KB)

anav · April 15, 2021, 10:36pm

Not sure why you think port forwarding is going to work through an ISP modem/rouer, followed by an apple device, then some other router, and then the MT device.
Draw a network diagram to show what is reallly going on!

Port forwarding works just fine if the ISP modem is feeding the MT, or even aN ISP router modem is providing a private IP, as long as one has access to the ISP modem router to make any necessary changes on the router side of the device.

Floydian · April 16, 2021, 7:24pm

Attached a sketch of the current setup - green dotted lines indicates what is working, red dotted what is not.

There’s no Modem in the mix. I get a direct public IP connection from my provider.

This is fed into the Apple AirPort, and from there I have my current LAN.
Intention is to replace the AirPort with the MT RB4011.

As mentioned in the original post, I have already tried to have the RB4011 connected to my ISP after having tested that I could get through from the 192.168.0.x LAN, but as soon as it got the public IP nothing was working - same as what is happening with the current double-NAT’ing.
If the double-NAT’ing was an issue, I would expect that I would see the same behaviour with the RB750Gr3, but that is behaving as expected, even with double NAT’ing (?).

Hope this help clarify the issue I’m having.

Again, thank you for any help.
Network - Test Setup.png

anav · April 16, 2021, 7:35pm

Is the switch a managed switch or unmanaged (dumb).
I would put the the 4011 in place of the apple…
Its like towing a 2020 land rover with a 2002 vw beetle…

Floydian · April 16, 2021, 8:53pm

It’s a managed switch, but currently used in “unmanaged” mode - no filtering implemented (yet).

I would put the the 4011 in place of the apple…
Its like towing a 2020 land rover with a 2002 vw beetle…

That is also the plan - just need to sort out this issue.

Floydian · April 17, 2021, 9:11am

Did another test, removing both the Apple AirPort and switch out of the equation with the RB4011 connected directly to the Internet, with a public IP on the WAN, and same result.
Can see packets reach the NAT and Firewall rules, but there’s no data being returned.

Found a few quirks in my configuration, which started causing issues as soon as there was no connection on the WAN port.
Noticed I had assigned same IP range to the MainBridge and the LAN_VLAN (10.1.0.x).

/ip address
add address=10.1.0.1/24 comment=“Main bridge” interface=MainBridge network=10.1.0.0
add address=10.1.0.1/24 interface=LAN_VLAN network=10.1.0.0

/ip pool
add name=dhcp ranges=10.1.0.10-10.1.0.250
add name=LAN_POOL ranges=10.1.0.100-10.1.0.250

/ip dhcp-server network
add address=10.1.0.0/24 comment=“LAN Network” dns-server=192.168.0.18 gateway=10.1.0.1

Thinking that could be the culprit I changed the VLAN range to 10.5.0.x which resolved some routing issues between the VLAN’s when having a public IP (or with no connection on the WAN port).

/ip address
add address=10.1.0.1/24 comment=“Main bridge” interface=MainBridge network=10.1.0.0
add address=10.5.0.1/24 interface=LAN_VLAN network=10.5.0.0

/ip pool
add name=LAN_POOL ranges=10.5.0.100-10.5.0.250

/ip dhcp-server network
add address=10.5.0.0/24 comment=“LAN Network” dns-server=192.168.0.18 gateway=10.5.0.1

However even with this change, I still can’t get through RB4011 when coming from a public IP.
It still works when coming from an IP in the 192.168.0.x range.

The above discovery made me think that I’m missing some understanding on what happens between the Bridge and VLAN’s.

mkx · April 17, 2021, 9:22am

When vlan-filtering=yes on bridge, it’s a bit debatable whether untagged frames are actually passing bridge (the switch-like entity). Which means you have to folow one of the following:

set pvid on bridge interface and use bridge interface as untagged (or hybrid) interface. If you don’t set pvid explicitly, it’ll use VID=1 (implicit default in ROS)
better option IMO is to use bridge interface explicitly as trunk (all tagged) interface. In this case you have to use appropriate vlan interfaces for all uses. Also set bridge interface as taged member of all vlans

Your latest configuration export (post from 2021-04-15) shows you’re using bridge as hybrid interface but most member ports are using explicitly set pvids …

Floydian · April 17, 2021, 3:07pm

@mkx Thank you for taking the time to look at this.

Think I’m getting closer to figuring out the culprit of my issues.
Watched the Mikrotik presentation on VLAN processing in New bridge implementation, and noticed their mention of the 4 bytes added to the IP Frame (https://youtu.be/ZMMpza-O7_w?t=684).

Those 4 bytes corresponds to what I’ve noticed in the log file.
When my setup works (that is when I’m using the intermediate lan 192.168.0.x) I get the following entry in my logs.

[NAT8081] dstnat: in:outside out:(unknown 0), src-mac xx:xx:xx:xx:xx:xx, proto TCP (SYN), 192.168.0.20:59304->192.168.0.102:8081, len 64
[fw8081] forward: in:outside out:DMZ_VLAN, src-mac xx:xx:xx:xx:xx:xx, proto TCP (SYN), 192.168.0.20:59304->10.2.0.10:8081, NAT 192.168.0.20:59304->(192.168.0.102:8081->10.2.0.10:8081), len 64

When it’s not working I see this (When connection comes directly from the Internet):

[NAT8081] dstnat: in:outside out:(unknown 0), src-mac xx:xx:xx:xx:xx:xx, proto TCP (SYN), 139.59.147.204:39312->192.168.0.102:8081, len 60
[fw8081] forward: in:outside out:DMZ_VLAN, src-mac xx:xx:xx:xx:xx:xx, proto TCP (SYN), 139.59.147.204:39312->10.2.0.10:8081, NAT 139.59.147.204:39312->(192.168.0.102:8081->10.2.0.10:8081), len 60

Am aware that the packages that comes from the Internet connection will not contain a VLAN tag.

Then the question is how do I make it work when there’s no VLAN tag.
Would your suggestions address this? Or am I’m staring myself blind looking at the wrong stuff?

I’m trying to wrap my mind around your suggestion b.
Could you provide some example of how to implement the “trunk (all tagged) interface” part.

As for the bridge being a tagged member of all VLANS, below is what I have in my current config.

/interface bridge vlan
add bridge=MainBridge tagged=MainBridge untagged=lan1,lan2,lan3,lan4,lan5 vlan-ids=10
add bridge=MainBridge tagged=MainBridge untagged=dmz1,dmz2 vlan-ids=20
add bridge=MainBridge tagged=MainBridge untagged=iot1 vlan-ids=30
add bridge=MainBridge tagged=MainBridge untagged=admin vlan-ids=99

Do I need to do something else to full fill the “set bridge interface as tagged member of all vlans” ?

Once again, thank you for taking your time to look at this.

mkx · April 17, 2021, 3:47pm

Make sure this line is gone:

/ip address
add address=10.1.0.1/24 comment="Main bridge" interface=MainBridge network=10.1.0.0

Then

/interface list member
add comment=defconf interface=MainBridge list=LAN

Interface LAN_VLAN should be member of LAN interface list rather than MainBridge.

BTW,

/ip firewall filter
add action=accept chain=input comment="Allow VLAN to router services" in-interface-list=VLAN
add action=accept chain=input comment="Allow Admin_VLAN Full Access" in-interface=ADM_VLAN

I don’t see how these two treat different VLANs differently. You’ll have to work on rules allowing access to router from VLANs (other than ADM_VLAN) only for certain services, not for all of them.

mkx · April 17, 2021, 3:56pm

For your router, packets coming from 192.168.0.20 or from random internet host are just the same. They enter router through interface outside which expects untagged frames. So either your CP (192.168.0.20) is using VLAN tags (but how would then it access internet if your current internet gateway works without VLAN tags) or the 4 bytes of difference are something else. And it’s impossible to tell what is it without seeing traffic dump with full analysis of packets and headers.

Floydian · April 17, 2021, 4:41pm

It was just a guess as to what could be causing the problem.
I tried to create a “fake” internet with the following setup, which also appear to rule out that the 4 bytes difference should be the issue.
Network - Test Setup v2.png
Giving the network on the “outside” interface of my RB4011 a public IP, gave the same result as when connected to the real internet - not possible to get through to the the server behind the RB4011.
However giving the network a “private” IP, this case in the 192.168.10.x range, allowed data through.
In both scenarios I can see data hit both the firewall and nat rules on the RB4011.

So there’s clearly some internal routing on the RB4011 being messed up as soon as the connection is not coming from a private IP range.

Have also just tried to follow your suggestions for “/IP address” and “/interface list member”, though that still haven’t made a change.
Removing the “MainBridge” from the “/IP address” gave me a new entry in the config:

/interface detect-internet
set detect-interface-list=static

Now when I look at the “Quick Set” page in the webadmin “IP address” for Local Network is shown in red with the ip 0.0.0.0.

Below the config as it looks at the moment. (Am aware of the double “firewall filters” you mentioned).

# apr/17/2021 17:16:10 by RouterOS 6.48
# software id = IJJI-4YGG
#
# model = RB4011iGS+
# serial number =
/interface bridge
add admin-mac=xx:xx:xx:xx:xx:xx auto-mac=no comment="Main Bridge" name=\
    MainBridge protocol-mode=none vlan-filtering=yes
/interface ethernet
set [ find default-name=ether2 ] name=admin
set [ find default-name=ether3 ] name=dmz1
set [ find default-name=ether4 ] name=dmz2
set [ find default-name=ether5 ] name=iot1
set [ find default-name=ether6 ] name=lan1
set [ find default-name=ether7 ] name=lan2
set [ find default-name=ether8 ] name=lan3
set [ find default-name=ether9 ] name=lan4
set [ find default-name=ether10 ] name=lan5 poe-out=off
set [ find default-name=ether1 ] name=outside
/interface vlan
add interface=MainBridge name=ADM_VLAN vlan-id=99
add interface=MainBridge name=DMZ_VLAN vlan-id=20
add interface=MainBridge name=IOT_VLAN vlan-id=30
add interface=MainBridge name=LAN_VLAN vlan-id=10
/interface ethernet switch port
set 0 default-vlan-id=0
set 1 default-vlan-id=0
set 2 default-vlan-id=0
set 3 default-vlan-id=0
set 4 default-vlan-id=0
set 5 default-vlan-id=0
set 6 default-vlan-id=0
set 7 default-vlan-id=0
set 8 default-vlan-id=0
set 9 default-vlan-id=0
set 10 default-vlan-id=0
set 11 default-vlan-id=0
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
add name=VLAN
add name=ADMIN
add name=IOT
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=dhcp ranges=10.1.0.10-10.1.0.250
add name=LAN_POOL ranges=10.5.0.100-10.5.0.250
add name=DMZ_POOL ranges=10.2.0.100-10.2.0.250
add name=IOT_POOL ranges=10.3.0.100-10.3.0.250
add name=ADM_POOL ranges=10.4.0.100-10.4.0.250
/ip dhcp-server
add address-pool=dhcp disabled=no interface=MainBridge name=MB_DHCP
add address-pool=LAN_POOL disabled=no interface=LAN_VLAN name=LAN_DHCP
add address-pool=DMZ_POOL disabled=no interface=DMZ_VLAN name=DMZ_DHCP
add address-pool=IOT_POOL disabled=no interface=IOT_VLAN name=IOT_DHCP
add address-pool=ADM_POOL disabled=no interface=ADM_VLAN name=ADM_DHCP
/interface bridge port
add bridge=MainBridge comment=ADM frame-types=\
    admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=\
    admin pvid=99
add bridge=MainBridge comment="DMZ 1" frame-types=\
    admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=\
    dmz1 pvid=20
add bridge=MainBridge comment="DMZ 2" frame-types=\
    admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=\
    dmz2 pvid=20
add bridge=MainBridge comment="IOT 1" frame-types=\
    admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=\
    iot1 pvid=30
add bridge=MainBridge comment="LAN 1" frame-types=\
    admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=\
    lan1 pvid=10
add bridge=MainBridge comment="LAN 2" frame-types=\
    admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=\
    lan2 pvid=10
add bridge=MainBridge comment="LAN 3" frame-types=\
    admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=\
    lan3 pvid=10
add bridge=MainBridge comment="LAN 4" frame-types=\
    admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=\
    lan4 pvid=10
add bridge=MainBridge comment="LAN 5" frame-types=\
    admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=\
    lan5 pvid=10
add bridge=MainBridge comment=defconf interface=sfp-sfpplus1
/ip neighbor discovery-settings
set discover-interface-list=ADMIN
/interface bridge vlan
add bridge=MainBridge tagged=MainBridge untagged=lan1,lan2,lan3,lan4,lan5 \
    vlan-ids=10
add bridge=MainBridge tagged=MainBridge untagged=dmz1,dmz2 vlan-ids=20
add bridge=MainBridge tagged=MainBridge untagged=iot1 vlan-ids=30
add bridge=MainBridge tagged=MainBridge untagged=admin vlan-ids=99
/interface detect-internet
set detect-interface-list=static
/interface list member
add comment=defconf interface=outside list=WAN
add interface=ADM_VLAN list=ADMIN
add interface=DMZ_VLAN list=VLAN
add interface=IOT_VLAN list=VLAN
add interface=LAN_VLAN list=VLAN
add interface=ADM_VLAN list=VLAN
add interface=LAN_VLAN list=LAN
add interface=DMZ_VLAN list=LAN
/ip address
add address=10.4.0.1/24 interface=ADM_VLAN network=10.4.0.0
add address=10.5.0.1/24 interface=LAN_VLAN network=10.5.0.0
add address=10.2.0.1/24 interface=DMZ_VLAN network=10.2.0.0
add address=10.3.0.1/24 interface=IOT_VLAN network=10.3.0.0
/ip dhcp-client
add comment=defconf disabled=no interface=outside
/ip dhcp-server network
add address=10.2.0.0/24 comment="DMZ Network" dns-server=192.168.0.18 \
    gateway=10.2.0.1
add address=10.3.0.0/24 comment="IOT Network" dns-server=192.168.0.18 \
    gateway=10.3.0.1
add address=10.4.0.0/24 comment="ADM Network" dns-server=192.168.0.18 \
    gateway=10.4.0.1
add address=10.5.0.0/24 comment="LAN Network" dns-server=192.168.0.18 \
    gateway=10.5.0.1
/ip dns
set allow-remote-requests=yes servers=192.168.0.18,192.168.0.20
/ip dns static
add address=10.1.0.1 comment=defconf name=router.lan
/ip firewall filter
add action=accept chain=input comment=\
    "Accept established,related & untracked" connection-state=\
    established,related,untracked
add action=drop chain=input comment="Drop invalid" connection-state=invalid
add action=accept chain=input comment="Allow VLAN to router services" \
    in-interface-list=VLAN
add action=accept chain=input comment="Allow Admin_VLAN Full Access" \
    in-interface=ADM_VLAN
add action=accept chain=input comment="Accept ICMP" protocol=icmp
add action=accept chain=input comment=\
    "Accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=drop chain=input comment="Drop everything else"
add action=accept chain=forward comment="HTTP 8081" dst-address=10.2.0.10 \
    dst-port=8081 in-interface-list=WAN log=yes log-prefix="[fw8081]" \
    protocol=tcp
add action=accept chain=forward dst-port=8081 in-interface-list=VLAN \
    protocol=tcp
add action=accept chain=forward comment="Accept in ipsec policy" \
    ipsec-policy=in,ipsec
add action=accept chain=forward comment="Accept out ipsec policy" \
    ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="Fasttrack Forward" \
    connection-state=established,related
add action=accept chain=forward comment=\
    "Accept established, related & untracked" connection-state=\
    established,related,untracked
add action=accept chain=forward comment="VLAN Internet Access only" \
    connection-state=new in-interface-list=VLAN out-interface-list=WAN
add action=accept chain=forward comment="Allow Port Forwarding" \
    connection-nat-state=dstnat connection-state=new in-interface-list=WAN \
    log=yes log-prefix="[fw dstnat]"
add action=drop chain=forward comment="Drop invalid forward" log-prefix=\
    "[invfwd]"
add action=drop chain=forward comment="Drop all from WAN not DSTNATed" \
    connection-nat-state=!dstnat connection-state=new in-interface-list=WAN
/ip firewall nat
add action=masquerade chain=srcnat comment="Default masquerade" \
    out-interface-list=WAN
add action=dst-nat chain=dstnat comment="NAT test" dst-address=192.168.0.102 \
    dst-port=8081 log=yes log-prefix="[NAT8081]" protocol=tcp to-addresses=\
    10.2.0.10 to-ports=8081
add action=dst-nat chain=dstnat dst-address=84.x.x.x dst-port=8081 \
    protocol=tcp to-addresses=10.2.0.10
/system clock
set time-zone-name=Europe/Copenhagen
/system identity
set name=fldMTRB4011
/tool mac-server
set allowed-interface-list=ADMIN
/tool mac-server mac-winbox
set allowed-interface-list=ADMIN

mkx · April 17, 2021, 5:29pm

After you start configuring things outside Quickset, never ever use it again. At best it’ll display misleading information, at worst it’ll mess with configuration in some random ways.

Floydian · April 17, 2021, 5:49pm

Didn’t intend to use the quick set, just observed this behaviour.

mkx · April 18, 2021, 8:05am

I’m out of ideas as to what prevents your setup to perform correctly. There were cases where seemingly correct config did not work right and solution was factory reset, followed by application of very same config. It seems that sometimes there’s some configuration burried somewhere and not shown in UI.
If I were in your place, I’d export current config (the text version), reset RB4011 with no config, use winbox MAC connection and set things up again, step by step. And verify proper operation as soon as possible without too much config in place just in case some settings actually break things.

Floydian · April 18, 2021, 12:22pm

On one hand I had hoped that I had missed something obvious in my config, on the other hand I’m happy that what I’ve made so far haven’t made a complete fool of me.

Since my similar setup on the RB750 is working, I also have a feeling that something is stuck somewhere in the software, so I’ll take your suggestion to start from scratch and maybe try with a much simpler setup, just to see that I can get data through from a public IP.

Once again, thank you very much for taking your time to have a look at this. It’s highly appreciated.

anav · April 18, 2021, 6:00pm

Yup time to reload the firmware!

Floydian · April 20, 2021, 5:45pm

Just a short update from here.

A reset and a step-by-step implementation of my configuration via WinBox/terminal so far appears to be working.
Will need to to do some more testing, but so far I can get through from the Internet to my server behind the RB4011 so NAT and Routing is currently working as intended.

Once again, thank you @mkx & @anav for your assistance.

Zacharias · April 20, 2021, 6:47pm

It seems that sometimes there’s some configuration burried somewhere and not shown in UI.

Not shown on configuration export as well ?
How is that actually possible ?

mkx · April 20, 2021, 8:19pm

I’ve never experienced such case myself and I’ve no idea how configuration shown in UI (any of them) correlates to actual configuration. But there were cases (MT confirmed) that there were some exploits which couldn’t be eradicated simply by clearing (apparent) configuration.

Think like this: it’s known that firewall filter rules in ROS are a sort of UI for linux kernel iptables. Who knows what UI does when printing or exporting rules, could be that some rules actually configured in device’s iptables are not shown (because de-compiler skips them for some reason)? Or that rules shown are not actual rules running but rather some list stored in device’s file system which ideally would be in sync with running setup but could get out of sync for some reason? With ROS closed in a black-box it’s impossible to tell …