received dst-nat packet not seen after dst-nat and mangle

Have two routers on different subnets, linked by cable configured as 10.10.10.0

internet ----(WAN)<R1>(..88.0 bridge)-------hosts1
                       (..10.1 eth3) -----(WAN ..10.10)<R2>(..55.0 bridge)---Hosts2[

The goal is to reach Hosts2 from the internet using dst-nat in R1.
Using ssh on a cell phone (termux) I can see packets addressed to R1 arrive on WAN interface (Torch and Sniffer). Passthrough counter rules in NAT and Mangle show packets arrive. Firewall rules to accept dst-nat’ed packets in the forward chain show nothing. Torch and Sniffer on ethe 3 of R1 don’t see anything that looks like the ssh packet. Firewall has rules to forward/accept dst-nat packets from WAN. Where do I look next for these packets?
Config file to follow in next message after I redact it. Pardon the multiple rules to drop traffic or ports 21, 22, 23, and 80–was getting over a million attempts in a couple of days to login to the router.

config of R1 attached–replacing incorrect file
Thanks for your attention.
gwconfig20240528redact.txt (9.35 KB)

The config seems to be r2 as it has a 192.168.55.0/24 on the bridge interface. Can you check?

Sorry, missed the selection by a line. Correct file uploaded.
Thanks for the catch.

NP.

Here are my comments -

/interface bridge port
   add bridge=bridge comment=defconf disabled=yes interface=ether3

It is better to remove the interface from the bridge instead of disabling it.

/ip dhcp-server network
   add address=192.168.55.0/24 gateway=10.10.10.10 netmask=24

Not sure what you try to do there - is this router the DHCP server for 192.168.55.0/24 as well?

/ip route rule
   add action=unreachable dst-address=192.168.55.0/24 interface=ether1

This may be an issue - the dst-nat comes before the routing (See here). Not absolutely certain and I need to test, but can you disable it and check if you see a different result?

Thanks for the input.
Removed eth3 from the bridge, with the “-” in the header.
The inclusion of ..55.0 in IP/DHCP/Networks I thought is how the router knows this subnet exists and where to reach it. If not here, where does this info come from?
The “not reachable” entry for ..55.0 via eth1 was an attempt to keep traceroute from wandering onto the internet when I did a TR to that subnet from the other TR returned the next router and then 29 lines of “* * *” which I originally thought indicated a rabbit-hole detour on the internet. This rule deleted.
Torch and Sniffer show packets arriving on eth1, but no activity on eth3. Will work on it more tomorrow.
Thanks again.

Knowledge of “foreign” IP subnets is passed to router through proper configuration in routing table, e.g.

/ip/route
add dst-address=192.168.55.0/24 gateway=10.10.10.10

(and that’s the whole story, nothing else about that particular “foreign” IP subnet needs to be configured).

When it comes to routing, proper netmasks are paramount for getting it right. When router searches for next hop in own routing table, it selects entry which matches and has the longest netmask. The default route can be written as “dst-address=0.0.0.0/0 gateway=” … and any specific route (e.g. the one written above) will have precedence over the default one due to length of netmask (24 v.s. 0).

@mkx beat me to it. The same applies to 192.168.2.0/24: unless R1 provides DHCP services to that network, you can remove it and add a route instead.

After removing these, the route “unreachable” and the interface from the bridge, the config looks better. Some notes regarding the order of rules but nothing that would block traffic.

Can you use the packet sniffer instead of torch?

Changed the config to put “other” subnets in the route list only.
Put Sniffer on R1 to watch for TCP packet to port 22: nothing shows for eth1, eth3, or ‘all’ interfaces. The dst-nat rule increases the packet count by 3 for each attempt to ssh. Mangle rule in prerouting chain to passthrough tcp:xxxx03 shows 3 packets arriving. Mangle in all 5 chains show no activity when looking for port 22. I expected a packet to show up someplace :frowning:
Torch on eh3:22 shows no indication of my ssh attempts, only dns, bootp, discovery, etc.
Where to look for the lost packet?

The thing is, there is nothing obviously wrong in your config. So let’s do this systematically.

From R2, can you SSH into the backend server on 192.168.55.0/24?

/system/ssh address=<dst address> user=<user>

If so, from R1, can you SSH into the backend server on 192.168.55.0/24?

Successfully logged in to the server on ..55. from hosts on both ..88. and .55. subnets, that is from a host on each router.

OK. So R1 knows where to send the packets, and R2 accepts them.

Can you send a fresh config from R1? There were a few changes now.

As I redacted the config file I noticed

/ip service
set ssh address=0.0.0.0/0
set api disabled=yes
set api-ssl disabled=yes

My understanding is that ssl and ssh are different protocols so ssh isn’t affected.
Current config attached (let me know if you prefer in-line).
gwconfig20240531redact.txt (9.27 KB)

Hi,

Could you -

Remove the route rule

/ip route rule
   add action=unreachable dst-address=192.168.55.0/24 interface=ether1

Edit the firewall filter to have these three rules on top of the rule base (chain=forward)

   add action=fasttrack-connection chain=forward comment="defconf: fasttrack" \
       connection-state=established,related disabled=yes
   add action=accept chain=forward comment=\
       "defconf: accept established,related, untracked" connection-state=\
       established,related,untracked
   add action=drop chain=forward comment="defconf: drop invalid" \
       connection-state=invalid

Move the passthrough rule (chain=forward, out-interface=ether3 …) above this rule. Otherwise the accept will take precedence and exit before the passthrough is evaluated.

   add action=accept chain=forward comment="pass dst-nat" connection-nat-state=\
       dstnat in-interface=ether1

And sniff with no interface

/tool sniffer
   set filter-interface=bridge filter-ip-protocol=tcp filter-port=ssh \
       streaming-server=0.0.0.0:ssh

Changed order of firewall filter rules and removed the route rule. Put Sniffer on R1 with blank Interface field. Sniffer shows packets from the ssh client (on my phone, whose IP changes between attempts). These packets are addressed to the proper host on R2 (..55. and port 22). Get the same result when Sniffer is placed on eth3 (he link to R2). SSH still times out. I include the user name@IP address -pxxx03 from the ssh client on Termux.
NAT rule for dst-nat shows 3 packets. Three Mangle passthroughs show packet counts in prerouting, forwarding, and postrouting chains.
gwconfig20240531bF-Wfilters.txt (6.73 KB)

Nice, so packets are going through r1 now, correct?

Can you send r2’s config?

Yes, ssh packets from internet are going through R1. Sniffer on R2 sees packets arriving on eth1, from R1, addressed to the server on port 22. SSH session over the internet from cell phone still times out.
The attached config for R2 doesn’t include the items I deleted from the IP/DHCP/Networks list (all the subnets not serviced by a DHCP server on R2). The large number of filter rules in the firewall resulted from searching for specific packets while troubleshooting.
LabConfig20240531redact.txt (6.36 KB)

So, here are

Move these firewall rules (chain=forward) to the top of the rulebase

   add action=fasttrack-connection chain=forward comment="defconf: fasttrack" \
       connection-state=established,related disabled=yes
   add action=accept chain=forward comment=\
       "defconf: accept established,related, untracked" connection-state=\
       established,related,untracked
   add action=drop chain=forward comment="defconf: drop invalid" \
       connection-state=invalid

Create a firewall address list with the IP 192.168.55.247, 192.168.55.248, 192.168.55.249, 192.168.55.228, 192.168.55.244

/ip firewall address-list
   add address=192.168..247 list=SSH_SERVERS
   add address=192.168.55.247 list=SSH_SERVERS
   add address=192.168.55.248 list=SSH_SERVERS
   add address=192.168.55.249 list=SSH_SERVERS
   add address=192.168.55.228 list=SSH_SERVERS
   add address=192.168.55.244 list=SSH_SERVERS

Create a firewall rule to permit SSH connections to the members of that list from the WAN interface

/ip/firewall/filter
 add chain=forward action=accept comment="Permit SSH to the servers" protocol=tcp dst-port=22 in-interface-list=WAN dst-address-list=SSH_SERVERS

Move that rule above this one:

   add action=drop chain=forward comment=\
       "defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat \
       connection-state=new in-interface-list=WAN

And test.

Good news: again the PC on R1 can connect an instance of Winbox to R2.
The top item in the firewall rule base on R2 can’t be moved–special dummy rule for Fasttrack… If I open that rule it can’t be edited. It is Passthrough in the Forward chain. I created a new Fasttrack rule as you described (disabled) and placed it second. Close enough to the top?
In IP/Firewall/AddressList the form for a new entry accepted an address range. Is a range OK, or does it need individual entry for each address? After testing, which timed out, changed range to individual IP. Still timed out. Added a similar forward rule with ssh-servers as the source address list–no effect and doesn’t increment packet count.
Disabled all NAT in R2. R1 performs masquerading and DST-NAT.
In R2: Mangle in prerouting sees packets with dst port 22 when I SSH from cell phone. FW rule to allow ssh to servers shows 1 packet per attempt while Torch and Sniffer on eth1 show R2 receives 3 packets from the ssh client, which times out. No packets transmitted, per Sniffer (can’t tell on Torch). Host on R1 can ssh to server on R2.

When there is only one interface in the WAN list, is it better to use that one address or the i/f list?
LabConfig20240601.txt (6.85 KB)

That’s fine. The idea is that the fasttrack, established/related, and invalid rules are as high as possible in the rulebase.

Whichever works the best for you, use it. I tend to use separate IP for servers as it makes it more visual what is included when reading the configuration, but that is purely aesthetics. Functionally, they are the same.

But you don’t see the traffic in postrouting when trying from the phone, correct?

It will work the same. I tend to use lists to prepare for more than one interface, or interface changes. I also use the interface name to describe the technology, and the list to designate the security zone/function.

On R1, re-reading the configuration, I noticed that your DHCP network entry for 10.10.10.0/24 may contain a mistake:

/ip dhcp-server network
   add address=10.10.10.0/24 gateway=10.10.10.10 netmask=24

The gateway parameter is to give the DHCP client the default gateway. In your case, it should be set to 10.10.10.1.