Hello,
I have two identical servers that are announcing the same network via OSPF to a mikrotik router (model: CCR2004-1G-12S+2XS, current-firmware: 6.47.2).
The router sees two equal paths to that network, so ECMP should be enabled:
I would expect to see packets with the same source/destination pair to be routed to the same server, but it switches every few seconds.
Because of this ~1% of the TCP connections break, because the router starts sending packets to the other server in the middle of the connection.
Here’s an example of a TLS connection: https://imgur.com/a/Cze1Ke5
48:8f:5a:52:52:ae = mikrotik router
ca:0c:c0:a9:9e:33 = server 1
9e:d6:60:88:3d:5c = server 2
IETF-VRRP-VRID_0a (00:00:5e:00:01:0a) = VRRP address of mikrotik router
As you can see, the first few packets are exchanged between the router and server 1, but then the router decides to start sending packets to server 2, thus breaking the connection.
I’ve tried an almost identical setup using a router from a different vendor and it works as expected.
Any ideas of what might be the problem?
Thanks
I don’t think I’m misusing it, let me explain a bit better
The two servers act like routers in a certain way, they route every packet with destination 10.10.10.0/24 to themselves. In fact, OSPF considers this route external.
So, from the mikrotik’s point of view, the destination could be a single host that lies beyond the server 1 and 2, it doesn’t know that the servers re-route the packets to themselves.
When a packet with destination 10.10.10.0/24 gets in the mikrotik router, ECMP computes a hash based on Source Address, Destination Address, Protocol, Source Port, Destination Port, and that decides whether the packet is sent to gateway 10.20.20.2 or 10.20.20.3, right?
In the capture that I’ve linked two packets (211 and 217) with same Source Address, Destination Address, Protocol, Source Port, Destination Port are sent to two different gateways.
That’s what I find strange.
Not quite. According to this wiki page forwarding decisions are cached based on source address, destination address, source interface, routing mark and ToS.
I read that wiki page, but there’s also this post, which is what I was referring to.
There’s another document that says it’s based on source/destination IP pair, but it’s from 2007 so it’s surely outdated.
I couldn’t find a definitive answer.
Just to be sure I tried repeating the test and making sure that the source interface is always the same (I haven’t configured anything relative to ToS or routing marks, so I guess that those parameters should stay the same too in the default configuration), but no luck, problem’s still there.
But that’s the very point. The Mikrotik feels free to use any of the two gateways, because it cannot even dream about such kind of misuse. Normally, if you send a packet towards destination address A, that address A is on a single machine, and so it doesn’t matter through which network path (beginning with a gateway) you send them as all network path end, sooner on later, in the same point.
In your weird setup, the two servers advertise the same address (subnet), but the two apparently equivalent network paths are not equivalent at all.
ECMP can be only used this way if you don’t mind that you must have routing cache activated (be aware that routing cache is gone in ROS 7 because it is gone from linux kernel for security reasons) and if you don’t mind that it gets flushed now and then, and the next packet after the flush may choose the other gateway.
You can use a dstnat rule combined with a per-connection-classifier or other distribution rule (random) if you want to distribute the traffic from clients to two physical servers visible under the same IP address to their clients, but not ECMP.
Just curious, what application are these servers hosting? It seems like an application load balancer would be a better tool than ECMP in routing since you’re expecting TCP sessions from a specific source to stay with the server they hit.
Huh, very interesting. So, to summarize:
the first packet is routed randomly between the available gateways, then the result is cached (based on source address, destination address, source interface, routing mark and ToS), and all packets are routed to the same gateway until that cache entry is flushed (which happens every 300ms ~ 800ms in a standard linux kernel if I’m not mistaken, don’t know if the values are different in RouterOS).
Then the next packet is routed randomly (or round-robin, or something similar) again between the available gateways. If this happens in the middle of an ssl connection, and a different gateway is chosen, then the connection breaks.
This seems like a possible explanation.
If I disabled the routing cache the router would use round-robin to route all the packets between the two gateways, correct?
@IPANetEngineer I know that the setup look unusual, but there’s a reason for it. The servers provide a few different services, and they’re part of a bigger infrastructure. A load balancer would slightly complicate things, nothing terrible, but a couple rules like sindy suggested would be a much simpler solution in this case.
These rules are what load balancer is mostly.
And now meaning the “destination”.
The only thing that is lacking - taking the up/down status of the servers into account.
