Problem: reply packets from internet request are ignored and not routed to subnet.

Hi everyone, i’m upgrading my room with some network controlled devices (smart lights etc.) and because i share the network with some other people i wanted to divide the network into subnets for security.

The network:
Subnet A - 192.168.88.0/24
Gateway: 192.168.88.1
Open for all users.

Subnet B - 192.168.80.0/24
Gateway: 192.168.80.1
Private network.

The internet router is located in subnet A with the static IP address 192.168.88.254 and running NAT.

The problem
Locally it all works perfectly, subnet A can ping subnet B and vice versa. Both can ping and access the internet router. But i only have internet on subnet A, on subnet B i can’t get a internet connection. I checked with torch and the internet traffic is going out because when i ping 8.8.8.8 from subnet B - 192.168.80.2, i see a reply coming back with src 8.8.8.8 and dst 192.168.80.2 on the subnet A bridge interface. But it isn’t routed to subnet B for some reason.

When i masquerade the traffic comming from subnet B the internet is working, but i don’t want double NAT.

I’ve been working on this for the last couple of days and i’m all out of ideas. So if someone has a solution or a push in the right direction i would really appreciate it! :slight_smile:.

(As a side note: when everything works i will block traffic between subnets)

Configuration
I have a RB951G-2HnD running version 6.29.1 as the main router and a Cisco EPC3925 as the internet gateway.

I divided the network into 2 bridges:

[admin@MikroTik] > interface bridge print     
Flags: X - disabled, R - running 
 0  R name="bridge-local" mtu=1500 actual-mtu=1500 l2mtu=1598 arp=enabled 
      mac-address=D4:CA:6D:F2:CD:B9 protocol-mode=rstp priority=0x8000 
      auto-mac=no admin-mac=D4:CA:6D:F2:CD:B9 max-message-age=20s 
      forward-delay=15s transmit-hold-count=6 ageing-time=5m 

 1  R name="bridge-local-private" mtu=1500 actual-mtu=1500 l2mtu=1598 
      arp=enabled mac-address=D4:CA:6D:F2:CD:BC protocol-mode=rstp 
      priority=0x8000 auto-mac=yes admin-mac=00:00:00:00:00:00 
      max-message-age=20s forward-delay=15s transmit-hold-count=6 
      ageing-time=5m



[admin@MikroTik] > interface bridge port print
Flags: X - disabled, I - inactive, D - dynamic 
 #    INTERFACE                                     BRIDGE                                    PRIORITY  PATH-COST    HORIZON
 0    ether2-master-local                           bridge-local                                  0x80         10       none
 1    wlan1                                         bridge-local                                  0x80         10       none
 2    MyPrivateNetwork                              bridge-local-private                          0x80         10       none
 3 I  ether5-master-local-private                   bridge-local-private                          0x80         10       none



[admin@MikroTik] > interface wireless print
Flags: X - disabled, R - running 
 0  R name="MyPrivateNetwork" mtu=1500 mac-address=D6:CA:6D:F2:CD:BD 
      arp=enabled interface-type=virtual-AP master-interface=wlan1 
      ssid="MyPrivateNetwork" wds-mode=disabled wds-default-bridge=none 
      wds-ignore-ssid=no bridge-mode=enabled default-authentication=yes 
      default-forwarding=yes default-ap-tx-limit=0 default-client-tx-limit=0 
      hide-ssid=no security-profile=private 

 1  R name="wlan1" mtu=1500 mac-address=D4:CA:6D:F2:CD:BD arp=enabled 
      interface-type=Atheros AR9300 mode=ap-bridge ssid="PucksPartyHouse" 
      frequency=2412 band=2ghz-onlyn channel-width=20mhz scan-list=default 
      wireless-protocol=802.11 wds-mode=disabled wds-default-bridge=none 
      wds-ignore-ssid=no bridge-mode=enabled default-authentication=yes 
      default-forwarding=yes default-ap-tx-limit=0 default-client-tx-limit=0 
      hide-ssid=no security-profile=default compression=no

Each bridge has is own subnet and DHCP server:

[admin@MikroTik] > ip address print
Flags: X - disabled, I - invalid, D - dynamic 
 #   ADDRESS            NETWORK         INTERFACE                                  
 0   ;;; default configuration
     192.168.88.1/24    192.168.88.0    bridge-local                               
 1   ;;; Private network configuration
     192.168.80.1/24    192.168.80.0    bridge-local-private



[admin@MikroTik] > ip pool print
 # NAME                                             RANGES                         
 0 default-dhcp                                     192.168.88.10-192.168.88.254   
 1 private-dhcp                                     192.168.80.2-192.168.80.254



[admin@MikroTik] > ip dhcp-server print
Flags: X - disabled, I - invalid 
 #   NAME             INTERFACE             RELAY           ADDRESS-POOL             LEASE-TIME ADD-ARP
 0   default          bridge-local                          default-dhcp             3d        
 1   private          bridge-local-private                  private-dhcp             3d



[admin@MikroTik] > ip dhcp-server network print
 # ADDRESS            GATEWAY         DNS-SERVER      WINS-SERVER     DOMAIN                           
 0 ;;; private network configuration
   192.168.80.0/24    192.168.80.1    192.168.80.1   
 1 ;;; default configuration
   192.168.88.0/24    192.168.88.1    192.168.88.1



[admin@MikroTik] > ip dns print                  
                servers: 8.8.8.8,8.8.4.4,192.168.88.254
        dynamic-servers: 
  allow-remote-requests: yes
    max-udp-packet-size: 4096
   query-server-timeout: 2s
    query-total-timeout: 10s
             cache-size: 2048KiB
          cache-max-ttl: 1w
             cache-used: 79KiB

Most routes were added dynamically and i only added a static route for the internet gateway:

[admin@MikroTik] > ip route print
Flags: X - disabled, A - active, D - dynamic, 
C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, 
B - blackhole, U - unreachable, P - prohibit 
 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 0 A S  0.0.0.0/0                          192.168.88.254            1
 1 ADC  192.168.80.0/24    192.168.80.1    bridge-local-pr...        0
 2 ADC  192.168.88.0/24    192.168.88.1    bridge-local              0

I have no mangle, firewall and NAT rules enabled:

[admin@MikroTik] > ip firewall filter print
Flags: X - disabled, I - invalid, D - dynamic 

[admin@MikroTik] > ip firewall nat print
Flags: X - disabled, I - invalid, D - dynamic 
 0 X  ;;; default configuration
      chain=srcnat action=masquerade to-addresses=192.168.88.0/24 
      src-address=192.168.80.0/24 out-interface=bridge-local log=no 
      log-prefix="" 
      
[admin@MikroTik] > ip firewall mangle  print
Flags: X - disabled, I - invalid, D - dynamic

I read your post a few times, config all sounds correct. Tools > Packet Sniffer is the only thing I can think of for troubleshooting but it may not show much more than Torch does. Capture packets to file and post it on here.

IP > Arp
Does it show the arp entry of your PC, 192.168.80.2?

You have two options:

a) Put the cisco in bridge mode so that the mikrotik gets the public IP, and set the mikrotik to masquerade both networks (by dst-address = !local and output-interface=wan for example)

b) If you want to keep the public IP on the cisco, set LAN config on the cisco so that it also masquerades 192.168.80.x network, and a route so that it knows how to reach 192.168.80.0 subnet (i.e. destination=192.168.80.0/24 gateway 192.168.88.1).

I tried the Packet Sniffer and now i’m not sure anymore if the router is receiving reply packets. When i use Torch i see a reply coming back with src 8.8.8.8 and dest 192.168.80.10 (i changed from *.80.2 to *.80.10). But when i use PacketSniffer i can only see packets going out but nothing coming back from 8.8.8.8, when i change to masquerade the traffic from 192.168.80.0/24, i can see packets from 8.8.8.8 coming back, so i guess my packet tracer configuration is correct.

Can Torch see traffic that packet tracer doesn’t?

I did a additional check to make sure traffic was going out, i pinged my VPS and monitored incoming ping request there, so traffic is definitely leaving my network.
Torch.png
Packetsniffer.png
I added the packet trace as an attachment:
PacketSniffer.pcap.tar.gz
When i was using 192.168.80.2 it was also in the ARP list.

[admin@MikroTik] > ip arp print
Flags: X - disabled, I - invalid, H - DHCP, D - dynamic, P - published 
 #   ADDRESS         MAC-ADDRESS       INTERFACE                                                                                                      
 0 D 192.168.88.254  18:59:33:B0:B4:4B bridge-local                                                                                                   
 1 D 192.168.88.15   EC:9A:74:94:EF:61 bridge-local                                                                                                   
 2 D 192.168.80.3    B8:27:EB:45:86:D8 bridge-local-private                                                                                           
 3 D 192.168.80.10   CC:AF:78:75:2C:FF bridge-local-private                                                                                           
 4 D 192.168.88.31   7C:D1:C3:76:6A:2A bridge-local                                                                                                   
 5 D 192.168.88.101  48:5A:3F:6C:6B:C9 bridge-local



I use the Cisco router for a additional 5Ghz access point, because the Mikrotik only has 2.4Ghz. The Cisco router is pretty locked up, i can’t add routes and the only changes i can do to the NAT configuration is some port forwarding etc.

Ok so routing is out of the equation then.

Your only resort is NATting 192.168.80.0

Try

/ip firewall nat
add chain=srcnat action=masquerade src-address=192.168.80.0/24 dst-address=192.168.88.254 out-interface=bridge-local log=no log-prefix="" comment="Masquerade private lan traffic to cisco router"

add chain=srcnat src-address=192.168.80.0/24 dst-address=192.168.88.0/24 out-interface=bridge-local log=no log-prefix="" comment="Leave Inter-LAN traffic unmasqueraded"

add chain=srcnat action=masquerade src-address=192.168.80.0/24 dst-address=0.0.0.0/0 out-interface=bridge-local log=no log-prefix="" comment="masquerade private LAN traffic to Internet"

Yeah that works :slight_smile:, that’s what i’m running now for the moment. The double NAT is not that big of a problem, but i’m still curious why it isn’t working when i’m not masquerading the traffic from 192.168.80.0/24 and why its possible that Torch sees traffic that the packet sniffer doesn’t.

Thanks for helping me :wink:

It doesn’t work because for routing to work both parties have to know how to route to the other; the rb “knows” how to reach the cisco (192.168.88.254), and its default route (0.0.0.0/0) through it.

But the cisco doesn’t have a route to .80.0 through RB’s 192.168.88.1, so it doesn’t know how to reach .80.0/24 ; that’s why masquerading as if everything came from 88.0/24 solves the issue.