I’m trying to figure out this problem I’m having with packets on my network.
I have two Proxmox servers running different VM machines and a Mikrotik router switch.
The issue is that when I (or any other computer on the network) tries to make a request to a VM the first request usually fails.
For example, here I’m trying to connect to a server called ca:
$ curl https://ca
curl: (7) Failed to connect to ca port 443 after 3 ms: Connection refused
The second request always succeeds.
I have traced this issue to if the Mikrotik Bridge knows about the MAC address or not. I the bridge knows about tha MAC address everything works, if the bridge doesn’t know, it fails (this is in the Hosts table of the Bridge).
The reason it fails (I think) is because when the bridge doesn’t know the MAC address it seems to send the SYN packet to all connected devices on the VLAN. And for some reason I get a reply from both of my Proxmox hosts. One reply says [RST, SYN] the other says [ACK, SYN]. My computer then sees the [RST, SYN] packet first and sends RST back to the server.
Anyone else had this issue? I assume it’s not really a Mikrotik problem, but I’m not 100% sure…
A way to prolong the period when it is working have been to lengthen the Ageing Time of the bridge. Then the MAC will be remembered for longer, but I’m not sure if this can cause other issues…
If anyone have any input for me that would be greatly appreciated!
It is normal for bridge (or switch) to send an unicast ethernet frame to all ports if FDB doesn’t contain appropriate entry (MAC+bridge port) - that table gets populated as bridge/switch receives unicast ethernet frames according to src-mac-address. Any device, receiving ethernet frame whose dst-nac-address doesn’t match its own MAC address is required to silently drop it. Any IP stack, receiving IP packet whose dst-address doesn’t match irs own IP address, should drop it (or forward it in case it’s a part of router setup).
Your description of the problem lacks necessary details to say where the problem lies. But it seems that you’ve got some misconfiguration in your proxmox config.
Thanks @mkx, I think you are right, it’s most likely a Proxmox issue. I will dig deeper and see if I can find the cause. I’ll try to write something here if I figure it out.
Ok, now I think I understand the problem much better.
It’s definitely not a MikroTik problem!
But I will write something here for other people in a similar situation or for my future self.
Turns out I have a VM with a default rule to REJECT all traffic not matching the firewall rules. This seemed like a good idea at the time, however, this not only applies to traffic to the VM itself, but also all incoming packets that are not intended to get to the VM. So since it’s Enternet bridges “all the way down” the interface the firewall works on sometimes receives packets not intended for that specific host. Because that’s how bridges/switches work… So when this VM specific firewall bridge interface gets this random packet it send back a RST message to tell the host the packet came from that it has been rejected. But what should have happened is that the firewall should have dropped the packet without doing anything since the packet was not meant for that specific VM.
The solution was to change REJECT to DROP (or remove the rule). So this was something set on a “random” VM that cause all other communication to work in a strange way.