Hello!!
I have a MikroTik hAP ax², I`m very happy with it. However, I am having trouble connecting devices on different subnets of my home LAN. As shown in the following diagram:
I have a first Proxmox hypervisor where I virtualize a Sophos firewall and a Proxmox Backup Server. These devices have an address belonging to the 192.168.2.0/24 network and are part of the WAN zone. In the LAN network, with addressing 192.168.3.0/24, I have a second Proxmox hypervisor where I have virtualized a Checkmk and another Proxmox Backup Server.
In between, to connect both networks, I have a MikroTik router with the following characteristics:
This is the MikroTik configuration:
These are my problems:
On one hand, I cannot send SNMP traps from the Sophos firewall (192.168.2.1 in the WAN zone) to the Checkmk (192.168.3.5 in the LAN zone).
I also cannot link a datastore from Proxmox (192.168.2.3 in the WAN zone) with the Proxmox Backup Server (192.168.3.8 in the LAN zone).
I have been trying to solve this for several days, I have read the official MikroTik documentation, I have searched on Google, but I cannot get it to work, neither does Checkmk receive traps from the Sophos, nor can I mount the datastore in Proxmox-1. Curiously, I have noticed that:
From any machine in the WAN network, I can ping any machine in the LAN network.
If I disable srcnat, Checkmk does receive traps from Sophos, but I cannot ping any machine in the WAN network from the LAN network.
I’m out of ideas, some help would be really apreciated…
You mention that your Sophos Firewall cannot send traps to a server on 192.168.3.0/24: does your Sophos firewall has a route for 192.168.3.0/24 via 192.168.2.2?
Here is your diagram a bit flattened out.
The first issue I see is that machines on the 192.168.2.0/24 will route their traffic via their default gateway, likely the Sophos Firewall, to reach the 192.168.3.0/24. If that is not a big deal for ICMP ECHO/ECHO reply, anything with more than 2 packets will have issues: the Sophos firewall will see only half of the traffic and will likely drop it as invalid. Solution: either redesign your network so the devices are on leaf networks (i.e. networks with only the default gateway as a router), or distribute the route to 192.168.3.0/24 (gateway is your mikrotik router) via DHCP.
The second issue: why are you doing NAT on the Mikrotik - if machines are to be presented with their real IP, simply disable all the NAT rules including the masquerading one. Your configuration shows that on the way out you NAT behind the IP of the router, on the way in you keep the real IP.
The third (potential) issue: can you verify the conditions for the WAN to LAN rule? Typically, that rule says to permit everything that went through a NAT rule. If you are not doing NAT, you can skip the conditions, and have something saying “from interface list WAN to interface list LAN, permit”
Also, if you can, please export the /ip/firewall section in text and post here. That is a lot easier for me to troubleshoot than screenshots.
You are right, in the original post I’ve forgotten to mention that I already have a static route configured in the Sophos:
Sorry, but could you explain what you mean when you mention that I should redesign my network so that the devices are in leaf networks?
I also don’t understand about the DHCP: I now have DHCP configured on the MikroTik to serve IPs from the 192.168.3.0/24 network with the gateway 192.168.3.1. Do you mean that in this DHCP I should configure the gateway with the IP of the WAN side of the MikroTik, 192.168.2.2?
I have srcnat active because if I disable it, I can’t reach the machines on the 192.168.2.0/24 network from the LAN network 192.168.3.0/24. It is also true that if I do this, Checkmk does receive the SNMP traps sent by the Sophos.
Finally, when I check the rule that allows traffic from the WAN to the LAN, I don’t see the list of LAN interfaces, only the WAN. Should I choose the “all” option?
Here is the result of the export related to firewall, I’ve added some extra sections in case they are helpful.
As @vingjfg already wrote, but omitted to mention another solution to the problem:
When you disable NAT on mikrotik, you imediately step into routing triangle: when a 3rd machine from subnet 192.168.2.0/24 (let’s call it A) tries to communicate with a machine in subnet 192.168.3.0/24 (let’s call it B), this packet flow happens:
A determines that B is not in same subnet, hence it finds best route towards it. And that will be default gateway (sophos). So it delivers packet, destined for B, to sophos
sophos receives packet, analyses dst-address, consults its routing information and selects MT as next hop
MT receives packet, analyses dst-address, consults its routing information, notices it’s destined to a host inside one of directly attached networks and delivers packet to B
B receives a packet and (after a lenghty path via application layers) prepares a return packet. B determines that A is not in same subnet, hence it finds best route towards it. And that will be default gateway (MR). So it delivers packet, destined for A, to MT
MT receives packet, analyses dst-address, consults its routing information, notices it’s destined to a host inside one of directly attached networks and delivers packet to A
any further packet exchange follows the same pattern. As you can see, only “forward” packets pass sophos, return packets bypass it. While for touting (and any other stateless operatiobs, such as firewall raw rules) this is not a problem, it becomes a problem for firewall, running on sophos.
Firewall keeps connection state so it can detetmine, if an ingress packet belongs to already established connection or it belongs to related teaffic (e.g. ICMP packets) or is a grnuine initial packet for a new connection … or it’s invalid because its indicated state doesn’t correspond with connection tracking machinery’s state. In the later case packet will be dropped. And this happens to your sophos FW in the routing triangle case: it sees initial packet, it misses return packet and then it sees second forward packet which in case of TCP connection (with its mandatory 3-way handshake) shoukd not happen until after the first return packet. So it’s only normal for sophos to block such connection. And very similar thing hapoens if connection is initiated from 192.168.3.0/24 /+(soohis won’t see initial forward packet, it’ll see first return packet which to sophos doesn’t make any sense) and by doing SRC-NAT on MT you’re hiding the fact that there’s 192.168.3.0 in the game, 192.168.2.0/24 devices will think they are communicating with MT at 192.168.2.2 and they can do it directly, bypassing sophos in both directions.
Now the solutions: apart from running NAT (both SRC and DST), you have 2 options, both involving config on sophos:
set FW on sophos so that it won’t block between 192.168.2.0/24 and 192.168.3.0/24. In MT world it xan be done either by excluding this traffic from being conntracked (default firewall config allows untracked traffic to pass) or by explicitly allowing this traffic before any dtop rules may interfere. I don’t know what kind of pissibilities sophos is offering.
create a separate “routing” subnet for connecting MT to sophos. This way MT won’t be member of 192.168.2.0/24 and will have to srnd all traffic towards this subnet to sophos. This kind of layout will allow sophos to see both direction of traffic and properly firewall it.
Another possibility is to anchor 192.168.3.0/24 off sophos and use MT as a switch … IMO this solution would be the best if it’s feasible for you (in your particular use case).
Thank you all for the information provided, it is very interesting. After several readings trying to understand what you are explaining to me, I have come to the following conclusion:
MikroTik => Configure NAT on the MikroTik, both SRC and DST.
Sophos => Configure a static route to the network 192.168.3.0/24 with gateway 192.168.2.2 and create a rule that allows traffic between both subnets.
If you’re doing SRC and DST NAT on your MT, then in principle you don’t need routing set up on sophos … because your MT will act the same as if it was border gateway of your LAN … none of internet routers know your LAN IP address space. So the second line is not necessary. However, if you want to keep addressing 192.168.3.0/24 from your 192.168.2.0/24 hosts, then NAT won’t help you completely, you’ll have to do something about firewalling on sophos.
A quite obvious problem: remove firewall rule properties where you have empty values (connection-nat-state=“” connection-state=“”) … having them set with empty value is not the same as not setting them at all, they will match packets with empty state (but every packet has some state, “untracked” if not anything else).
Next: when doing both SRC-NAT and DST-NAT, the bare minimum FW filter rule set for chain=forward would look something like this:
If oyu don’t want to work with interface lists (I always recommend doing it, it’s easier to move some function from one port to another, the necessary change is only in the interface list, no need to change potentially tens or hundreds of firewall rules), then replace “in-interface-list=WAN” with “in-interface=etherX” (and likewise out-interface-list vs. out-interface).
The NAT rules seem fine. The use from 192.168.2.0/24 subnet is to use (e.g. checkmk) 192.168.2.2:161 (so you don’t reference any address inside 192.168.3.0/24 in your 192.168.2.0/24 subnet, it doesn’t exist for those hosts). BTW, not sure about UDP port 161, it’s standard SNMP port (not specific to checkmk) and you PFW this port to many “internal” (192.168.3.0/24) hosts, all of which you want to monitor using SNMP. Your setup (checkmk server on 192.168.3.5) likely implies that you’d like to send SNMP traps to checkmk server, this is usually done via UDP port 162 (but depending on SNMP agent config it may be possible to use other ports or even TCP).
Hello!!
Please, could you explain me this again? Thank you very much!!
The NAT rules seem fine. The use from 192.168.2.0/24 subnet is to use (e.g. checkmk) 192.168.2.2:161 (so you don’t reference any address inside 192.168.3.0/24 in your 192.168.2.0/24 subnet, it doesn’t exist for those hosts).
Hello!!
I have been thinking about how to solve this problem, I have realized that I have a very easy solution: Change from router with AP to switch with AP
This involves me migrating the machines from the 192.168.2.0/24 network to 192.168.3.0/24, and then eliminating it.
Another steps I must do are move the ether1 interface to the LAN bridge, enable DHCP client, disable DHCP server, and disable firewall. I think that should work
This new design of my network is already fine for me…
