Problems with Checkmk and Proxmox Backup Server

mrbyte · Sat Jul 13, 2024 12:01 am

Hello!!
I have a MikroTik hAP ax², I`m very happy with it. However, I am having trouble connecting devices on different subnets of my home LAN. As shown in the following diagram:

LAN - Mikrotik.png

I have a first Proxmox hypervisor where I virtualize a Sophos firewall and a Proxmox Backup Server. These devices have an address belonging to the 192.168.2.0/24 network and are part of the WAN zone. In the LAN network, with addressing 192.168.3.0/24, I have a second Proxmox hypervisor where I have virtualized a Checkmk and another Proxmox Backup Server.

In between, to connect both networks, I have a MikroTik router with the following characteristics:

[admin@MikroTik] > system routerboard print
       routerboard: yes
        board-name: hAP ax^2
             model: C52iG-5HaxD2HaxD
     serial-number: HEE08KQ96H3
     firmware-type: ipq6000
  factory-firmware: 7.7
  current-firmware: 7.12.1
  upgrade-firmware: 7.14.3

This is the MikroTik configuration:

IP -> Adresses.png

IP -> Filter Rules.png

IP -> NAT.png

IP -> Routes.png

These are my problems:
- On one hand, I cannot send SNMP traps from the Sophos firewall (192.168.2.1 in the WAN zone) to the Checkmk (192.168.3.5 in the LAN zone).
- I also cannot link a datastore from Proxmox (192.168.2.3 in the WAN zone) with the Proxmox Backup Server (192.168.3.8 in the LAN zone).

I have been trying to solve this for several days, I have read the official MikroTik documentation, I have searched on Google, but I cannot get it to work, neither does Checkmk receive traps from the Sophos, nor can I mount the datastore in Proxmox-1. Curiously, I have noticed that:
- From any machine in the WAN network, I can ping any machine in the LAN network.
- If I disable srcnat, Checkmk does receive traps from Sophos, but I cannot ping any machine in the WAN network from the LAN network.

I'm out of ideas, some help would be really apreciated...

Thank you very much and best regards!!

vingjfg · Sun Jul 14, 2024 9:28 am

Hi there,

I see a few of issues.

You mention that your Sophos Firewall cannot send traps to a server on 192.168.3.0/24: does your Sophos firewall has a route for 192.168.3.0/24 via 192.168.2.2?

Here is your diagram a bit flattened out.

Network Flattened Diagram.drawio.png

The first issue I see is that machines on the 192.168.2.0/24 will route their traffic via their default gateway, likely the Sophos Firewall, to reach the 192.168.3.0/24. If that is not a big deal for ICMP ECHO/ECHO reply, anything with more than 2 packets will have issues: the Sophos firewall will see only half of the traffic and will likely drop it as invalid. Solution: either redesign your network so the devices are on leaf networks (i.e. networks with only the default gateway as a router), or distribute the route to 192.168.3.0/24 (gateway is your mikrotik router) via DHCP.

The second issue: why are you doing NAT on the Mikrotik - if machines are to be presented with their real IP, simply disable all the NAT rules including the masquerading one. Your configuration shows that on the way out you NAT behind the IP of the router, on the way in you keep the real IP.

The third (potential) issue: can you verify the conditions for the WAN to LAN rule? Typically, that rule says to permit everything that went through a NAT rule. If you are not doing NAT, you can skip the conditions, and have something saying "from interface list WAN to interface list LAN, permit"

Also, if you can, please export the /ip/firewall section in text and post here. That is a lot easier for me to troubleshoot than screenshots.

mrbyte · Sun Jul 14, 2024 5:28 pm

Hello,
First, one million thanks for your help!!

You are right, in the original post I've forgotten to mention that I already have a static route configured in the Sophos:

Captura de pantalla_20240714_155253.png

Sorry, but could you explain what you mean when you mention that I should redesign my network so that the devices are in leaf networks?

I also don't understand about the DHCP: I now have DHCP configured on the MikroTik to serve IPs from the 192.168.3.0/24 network with the gateway 192.168.3.1. Do you mean that in this DHCP I should configure the gateway with the IP of the WAN side of the MikroTik, 192.168.2.2?

I have srcnat active because if I disable it, I can't reach the machines on the 192.168.2.0/24 network from the LAN network 192.168.3.0/24. It is also true that if I do this, Checkmk does receive the SNMP traps sent by the Sophos.

Finally, when I check the rule that allows traffic from the WAN to the LAN, I don't see the list of LAN interfaces, only the WAN. Should I choose the "all" option?

Captura de pantalla_20240714_161753.png

Here is the result of the export related to firewall, I've added some extra sections in case they are helpful.

/ip pool
add name=dhcp_pool0 ranges=192.168.3.25-192.168.3.240

/ip dhcp-server
add address-pool=dhcp_pool0 interface=LAN name=server1

/snmp community
set [ find default=yes ] addresses=192.168.3.0/24

/ip firewall connection tracking
set udp-timeout=10s

/interface list member
add interface=ether1 list=WAN
add interface=LAN list=*FFFFFFFF

/ip address
add address=192.168.3.1/24 comment=LAN interface=LAN network=192.168.3.0
add address=192.168.2.2/24 comment=WAN interface=ether1 network=192.168.2.0

/ip dhcp-client
add disabled=yes interface=ether1

/ip dhcp-server lease
add address=192.168.3.21 mac-address=D8:8C:79:6A:82:B3 server=server1

/ip dhcp-server network
add address=192.168.3.0/24 dns-server=192.168.3.3,208.67.222.222 gateway=192.168.3.1 netmask=24

/ip dns
set allow-remote-requests=yes servers=208.67.222.222,208.67.222.220

/ip firewall filter
add action=accept chain=forward comment="WAN to LAN" dst-address=192.168.3.0/24 in-interface-list=WAN out-interface-list=all src-address=192.168.2.0/24

/ip firewall nat
add action=masquerade chain=srcnat out-interface=ether1
add action=dst-nat chain=dstnat comment="Sophos to Checkmk" disabled=yes dst-port=1-65535 in-interface-list=WAN protocol=udp to-addresses=192.168.3.5 to-ports=161
add action=dst-nat chain=dstnat comment="Proxmox 1 to Proxmox Backup Server 2" disabled=yes dst-port=1-65535 in-interface-list=WAN protocol=tcp to-addresses=192.168.3.5 to-ports=161

/ip route
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.2.1 pref-src="" routing-table=main suppress-hw-offload=no

/snmp
set enabled=yes src-address=192.168.3.1 trap-generators=start-trap,interfaces,temp-exception trap-interfaces=all trap-target=192.168.3.5 trap-version=2

Again, thank you very much for your help!!

Kind regards

mkx · Sun Jul 14, 2024 7:09 pm

As @vingjfg already wrote, but omitted to mention another solution to the problem:

The first issue I see is that machines on the 192.168.2.0/24 will route their traffic via their default gateway, likely the Sophos Firewall, to reach the 192.168.3.0/24. If that is not a big deal for ICMP ECHO/ECHO reply, anything with more than 2 packets will have issues: the Sophos firewall will see only half of the traffic and will likely drop it as invalid.

When you disable NAT on mikrotik, you imediately step into routing triangle: when a 3rd machine from subnet 192.168.2.0/24 (let's call it A) tries to communicate with a machine in subnet 192.168.3.0/24 (let's call it B), this packet flow happens:

A determines that B is not in same subnet, hence it finds best route towards it. And that will be default gateway (sophos). So it delivers packet, destined for B, to sophos
sophos receives packet, analyses dst-address, consults its routing information and selects MT as next hop
MT receives packet, analyses dst-address, consults its routing information, notices it's destined to a host inside one of directly attached networks and delivers packet to B
B receives a packet and (after a lenghty path via application layers) prepares a return packet. B determines that A is not in same subnet, hence it finds best route towards it. And that will be default gateway (MR). So it delivers packet, destined for A, to MT
MT receives packet, analyses dst-address, consults its routing information, notices it's destined to a host inside one of directly attached networks and delivers packet to A

any further packet exchange follows the same pattern. As you can see, only "forward" packets pass sophos, return packets bypass it. While for touting (and any other stateless operatiobs, such as firewall raw rules) this is not a problem, it becomes a problem for firewall, running on sophos.
Firewall keeps connection state so it can detetmine, if an ingress packet belongs to already established connection or it belongs to related teaffic (e.g. ICMP packets) or is a grnuine initial packet for a new connection ... or it's invalid because its indicated state doesn't correspond with connection tracking machinery's state. In the later case packet will be dropped. And this happens to your sophos FW in the routing triangle case: it sees initial packet, it misses return packet and then it sees second forward packet which in case of TCP connection (with its mandatory 3-way handshake) shoukd not happen until after the first return packet. So it's only normal for sophos to block such connection. And very similar thing hapoens if connection is initiated from 192.168.3.0/24 /+(soohis won't see initial forward packet, it'll see first return packet which to sophos doesn't make any sense) and by doing SRC-NAT on MT you're hiding the fact that there's 192.168.3.0 in the game, 192.168.2.0/24 devices will think they are communicating with MT at 192.168.2.2 and they can do it directly, bypassing sophos in both directions.

Now the solutions: apart from running NAT (both SRC and DST), you have 2 options, both involving config on sophos:

set FW on sophos so that it won't block between 192.168.2.0/24 and 192.168.3.0/24. In MT world it xan be done either by excluding this traffic from being conntracked (default firewall config allows untracked traffic to pass) or by explicitly allowing this traffic before any dtop rules may interfere. I don't know what kind of pissibilities sophos is offering.
create a separate "routing" subnet for connecting MT to sophos. This way MT won't be member of 192.168.2.0/24 and will have to srnd all traffic towards this subnet to sophos. This kind of layout will allow sophos to see both direction of traffic and properly firewall it.

Another possibility is to anchor 192.168.3.0/24 off sophos and use MT as a switch ... IMO this solution would be the best if it's feasible for you (in your particular use case).

mrbyte · Sun Jul 14, 2024 10:46 pm

Hello!!

Thank you all for the information provided, it is very interesting. After several readings trying to understand what you are explaining to me, I have come to the following conclusion:
- MikroTik => Configure NAT on the MikroTik, both SRC and DST.
- Sophos => Configure a static route to the network 192.168.3.0/24 with gateway 192.168.2.2 and create a rule that allows traffic between both subnets.

Is this correct?

Kind regards!!

mkx · Mon Jul 15, 2024 7:57 am

If you're doing SRC and DST NAT on your MT, then in principle you don't need routing set up on sophos ... because your MT will act the same as if it was border gateway of your LAN ... none of internet routers know your LAN IP address space. So the second line is not necessary. However, if you want to keep addressing 192.168.3.0/24 from your 192.168.2.0/24 hosts, then NAT won't help you completely, you'll have to do something about firewalling on sophos.

mrbyte · Tue Jul 16, 2024 12:14 am

Hello,
I've tried Port Forwarding with this configuration:

[admin@MikroTik] /ip/firewall> export
/ip firewall connection tracking
set udp-timeout=10s
/ip firewall filter
add action=accept chain=forward comment="LAN TO WAN" connection-nat-state="" connection-state="" disabled=yes dst-address=192.168.2.0/24 out-interface=ether1 src-address=192.168.3.0/24
add action=accept chain=forward comment="WAN TO LAN" disabled=yes dst-address=192.168.3.0/24 in-interface-list=WAN src-address=192.168.2.0/24
/ip firewall nat
add action=masquerade chain=srcnat out-interface=ether1
add action=dst-nat chain=dstnat comment=Checkmk dst-port=161 in-interface-list=WAN protocol=udp to-addresses=192.168.3.5 to-ports=161
add action=dst-nat chain=dstnat comment=PBS-2 dst-port=8007 in-interface-list=WAN protocol=tcp to-addresses=192.168.3.8 to-ports=8007

However, it doesn't work:

Captura de pantalla_20240715_230514.png

I also have to say that I've deleted the static route in the Sophos.

What am I doing wrong?

Kind regards!!

mkx · Tue Jul 16, 2024 11:47 am

A quite obvious problem: remove firewall rule properties where you have empty values (connection-nat-state="" connection-state="") ... having them set with empty value is not the same as not setting them at all, they will match packets with empty state (but every packet has some state, "untracked" if not anything else).

Next: when doing both SRC-NAT and DST-NAT, the bare minimum FW filter rule set for chain=forward would look something like this:

/ip/firewall/filter
add action=accept chain=forward comment=\
    "defconf: accept established,related, untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid
add action=accept chain=forward comment="allow dst-nat connections from WAN" \
    connection-nat-state=dstnat connection-state=new in-interface-list=WAN
add action=drop chain=forward comment="drop anything else from WAN" \
    in-interface-list=WAN

If oyu don't want to work with interface lists (I always recommend doing it, it's easier to move some function from one port to another, the necessary change is only in the interface list, no need to change potentially tens or hundreds of firewall rules), then replace "in-interface-list=WAN" with "in-interface=etherX" (and likewise out-interface-list vs. out-interface).

The NAT rules seem fine. The use from 192.168.2.0/24 subnet is to use (e.g. checkmk) 192.168.2.2:161 (so you don't reference any address inside 192.168.3.0/24 in your 192.168.2.0/24 subnet, it doesn't exist for those hosts). BTW, not sure about UDP port 161, it's standard SNMP port (not specific to checkmk) and you PFW this port to many "internal" (192.168.3.0/24) hosts, all of which you want to monitor using SNMP. Your setup (checkmk server on 192.168.3.5) likely implies that you'd like to send SNMP traps to checkmk server, this is usually done via UDP port 162 (but depending on SNMP agent config it may be possible to use other ports or even TCP).

mrbyte · Wed Jul 17, 2024 12:30 am

Hello!!
Please, could you explain me this again? Thank you very much!!

The NAT rules seem fine. The use from 192.168.2.0/24 subnet is to use (e.g. checkmk) 192.168.2.2:161 (so you don't reference any address inside 192.168.3.0/24 in your 192.168.2.0/24 subnet, it doesn't exist for those hosts).

I tried these configs, none of them works:

/ip firewall connection tracking
set udp-timeout=10s
/ip firewall filter
add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=forward comment="allow dst-nat connections from WAN" connection-nat-state=dstnat connection-state=new in-interface=ether1
add action=drop chain=forward comment="drop anything else from WAN" in-interface=ether1
/ip firewall nat
add action=masquerade chain=srcnat out-interface=ether1
add action=dst-nat chain=dstnat comment=Checkmk dst-address=192.168.2.2 dst-port=161 in-interface-list=WAN protocol=udp to-addresses=192.168.3.5 to-ports=161
add action=dst-nat chain=dstnat comment=PBS-2 dst-address=192.168.2.2 dst-port=8007 in-interface-list=WAN protocol=tcp to-addresses=192.168.3.8 to-ports=8007
add action=dst-nat chain=dstnat comment=Checkmk_2 dst-address=192.168.2.2 dst-port=162 in-interface-list=WAN protocol=udp to-addresses=192.168.3.5 to-ports=162

=============================================================================================================

/ip firewall connection tracking
set udp-timeout=10s
/ip firewall filter
add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=forward comment="allow dst-nat connections from WAN" connection-nat-state=dstnat connection-state=new in-interface=ether1
add action=drop chain=forward comment="drop anything else from WAN" in-interface=ether1
/ip firewall nat
add action=masquerade chain=srcnat out-interface=ether1
add action=dst-nat chain=dstnat comment=Checkmk dst-port=161 in-interface-list=WAN protocol=udp to-addresses=192.168.3.5 to-ports=161
add action=dst-nat chain=dstnat comment=PBS-2 dst-port=8007 in-interface-list=WAN protocol=tcp to-addresses=192.168.3.8 to-ports=8007
add action=dst-nat chain=dstnat comment=Checkmk_2 dst-port=162 in-interface-list=WAN protocol=udp to-addresses=192.168.3.5 to-ports=162

I'm starting to think that maybe the Sophos is the culprit that NAT doesn't work...

Kind regards!!

mkx · Wed Jul 17, 2024 6:53 am

Hello!!
Please, could you explain me this again? Thank you very much!!

The NAT rules seem fine. The use from 192.168.2.0/24 subnet is to use (e.g. checkmk) 192.168.2.2:161 (so you don't reference any address inside 192.168.3.0/24 in your 192.168.2.0/24 subnet, it doesn't exist for those hosts).

A question instead: how are servers proxmox1 and proxmox-backup1 configured to communicate with checkmk server?

mrbyte · Fri Jul 19, 2024 5:41 pm

Hello!!
I have been thinking about how to solve this problem, I have realized that I have a very easy solution: Change from router with AP to switch with AP

This involves me migrating the machines from the 192.168.2.0/24 network to 192.168.3.0/24, and then eliminating it.

Another steps I must do are move the ether1 interface to the LAN bridge, enable DHCP client, disable DHCP server, and disable firewall. I think that should work

This new design of my network is already fine for me...

Kind regards!!

Problems with Checkmk and Proxmox Backup Server

Problems with Checkmk and Proxmox Backup Server

Re: Problems with Checkmk and Proxmox Backup Server

Re: Problems with Checkmk and Proxmox Backup Server

Re: Problems with Checkmk and Proxmox Backup Server

Re: Problems with Checkmk and Proxmox Backup Server

Re: Problems with Checkmk and Proxmox Backup Server

Re: Problems with Checkmk and Proxmox Backup Server

Re: Problems with Checkmk and Proxmox Backup Server

Re: Problems with Checkmk and Proxmox Backup Server

Re: Problems with Checkmk and Proxmox Backup Server

Re: Problems with Checkmk and Proxmox Backup Server

Who is online