Thu Feb 03, 2022 12:41 pm
Doing NAT (in fact, anything related to connection tracking) on another router addresses just one possible source of CPU load, but there are other sources too.
The real issue is that tearing down and establishing a PPPoE connetion is a CPU-intensive task, and that there is apparently no prioritization of processing of PPPoE connection control packets as compared to other traffic. So when "something big" happens, that affects processing of connection control packets so much that the connections are considered dead, the teardown process begins, loads the CPU even more, so even more connections are considered dead, so ultimately all connections go down and start re-establishing slowly. And if the CPU load is close to the edge, that "something big" may not be that big at all, it is just big enough to push the load over the edge and the self-locking effect takes care of the rest.
So moving "NAT" away from the PPPoE server just lowers the base load and leaves more space for some traffic spikes to be handled without hitting disaster threshold. However, if you move "NAT" away from the PPPoE server but keep connection tracking active on it, you gain nothing as the real load associated to NAT consists in matching every single packet to the full list of existing connections, regardless whether the connection to which the packet belongs is actually NATed or not - the rewriting of the packet source and/or destination address takes much less CPU than the search.
In specific environments, that "something big" may be the removal of masqueraded connections from connection tracking, but it only happens when the IP address, to which these connections are masqueraded, changes. So a teardown of a single PPPoE connection triggers that removal at the client, not at the server (unless the client acts as an uplink of the server but that doesn't happen in real world deployments). At the server side, it only happens in multi-WAN environments with masquerade rules, where a WAN interface going down triggers the removal of masqueraded connections, but that's a special case and should normally not happen unless your WAN IPs are dynamic and you thus cannot use a plain src-nat instead of masquerade.
Also, people sometimes forget it is necessary to add blackhole routes towards the subnets from which the PPPoE clients get their address assignments. So when a client disconnects, responses to connections open by this client keep coming, but as the client is down, the connected /32 route to its address doesn't exist any more, and the packets are sent down the default route, which means back to the upstream router. So depending on their TTL, their keep circulating between the PPPoE server and the upstream router, adding extra load to the CPU. A blackhole route to the pool subnet, with distance higher than 0, makes sure that this does not happen.
