Community discussions

MikroTik App
 
pe1chl
Forum Guru
Forum Guru
Topic Author
Posts: 10486
Joined: Mon Jun 08, 2015 12:09 pm

CAKE status display (and CAKE reordering problem)?

Thu Jul 25, 2024 2:25 pm

I am experimenting with CAKE and I experience a problem.
Is there any way to view the status and performance of a CAKE queue?
With queue tree I am used to viewing the rate at each priority, number of queued and dropped packets, etc.
But I cannot find such a display for CAKE queues....

Actual problem I experience: I have an RB5009 connected to a 5G gateway via ethernet, speed is about 125M/30M, and I enabled CAKE with mostly default settings on the ethernet port connected to the 5G gateway (traffic there is just standard IP over ethernet).
/queue type
add cake-autorate-ingress=yes cake-diffserv=diffserv4 cake-nat=yes \
    cake-overhead=18 kind=cake name=cake
Over this connection I run a L2TP/IPsec VPN to another site, and from that site I connect to the router (winbox, telnet).
A short time after selecting the CAKE queue the connection mostly hangs. I can still ping and telnet but when I issue a command that outputs considerable data, or when I connect winbox, it just hangs.
Looks suspiciously like an MTU problem. But without CAKE everything works fine.
Last edited by pe1chl on Fri Jul 26, 2024 4:54 pm, edited 1 time in total.
 
pe1chl
Forum Guru
Forum Guru
Topic Author
Posts: 10486
Joined: Mon Jun 08, 2015 12:09 pm

Re: CAKE status display?

Fri Jul 26, 2024 11:45 am

My theory on what is going wrong here....
When the L2TP clients sends large packets, they are fragmented by MLPPP (because I have set MRRU of 1504 to get a 1500-byte MTU on the L2TP interface).
So each packet larger than about 1400 bytes is split at the MLPPP level, i.e. before passing through IPsec.
Probably as usual, MLPPP does not split the packet in two about equal halves, but instead splits it in one "Max MTU" sized (1400 in my case) and the remaining smaller part.
These two are then passed through IPsec and sent as IPsec-ESP (we are not behind CG NAT). If we were, it would be UDP port 4500.
So at the interface, two fragments are seen: a large and a small one.

Could it be that CAKE decides to send the smaller packet first as that in other cases might result in a shorter latency, and that this particular protocol (L2TP on top of PPP on top of MLPPP on top of IPsec) does not tolerate that re-ordering?
It would at least explain why small packets are passing through without problem, and large ones do not.
 
pe1chl
Forum Guru
Forum Guru
Topic Author
Posts: 10486
Joined: Mon Jun 08, 2015 12:09 pm

Re: CAKE status display?

Fri Jul 26, 2024 3:25 pm

Indeed the re-ordering appears to be the problem...
I made a packet trace on the central router (where the L2TP server is) for the traffic from the client router that uses CAKE, and when I use tshark on the captured pcap file and grep for the sequence numbers, I get this:

ESP Sequence: 8218
ESP Sequence: 8219
ESP Sequence: 8222
ESP Sequence: 8223
ESP Sequence: 8220
ESP Sequence: 8225
ESP Sequence: 8226
ESP Sequence: 8221
ESP Sequence: 8227
ESP Sequence: 8228
ESP Sequence: 8229
ESP Sequence: 8224
ESP Sequence: 8230

I.e. the packets are heavily re-ordered, probably decided by size.
Is there any way within CAKE to force the ordering to remain the same for a session, or when that already is the case, to force this L2TP/IPsec session to be considered a single session that is not to be re-ordered?
 
dtaht
Member Candidate
Member Candidate
Posts: 215
Joined: Sat Aug 03, 2013 5:46 am

Re: CAKE status display (and CAKE reordering problem)?

Fri Jul 26, 2024 10:26 pm

I would love a packet capture of this behavior.

YES, if packets are fragmented, they will be delivered out of order. Ideally your tunnel should be signalling back that the mtu is too big.

Most IPSEC implementations I reviewed had a reasonably sized reorder buffer - at least 32 packets. Some were much larger. Wireguard, I think, is 8192 packets, however, which should be more than enough. So I thought (then), in most cases the underlying ipsec implementation would sort it out.
 
pe1chl
Forum Guru
Forum Guru
Topic Author
Posts: 10486
Joined: Mon Jun 08, 2015 12:09 pm

Re: CAKE status display (and CAKE reordering problem)?

Sat Jul 27, 2024 11:01 am

Ok tell me how I can send it to you, I still have two captures of behavior with and without CAKE. Is it OK when I e-mail it to your gmail address?

I don't know on which layer it is going wrong. It could be IPsec, but it could also be MLPPP.
The list of packet numbers shown below was as captured on the external interface, so before any handling by the central router.

In this case, I do want a 1500 byte MTU on the tunnel. Normally I do not enable MLPPP (MRRU) but in this case I did because I want to use this tunnel to provide the clients behind that other router with IPv6 which the 5G network does not offer.
And that works well. I prefer to give them a 1500 byte MTU to avoid any issues with "not or slow loading websites" and therefore I enabled the MLPPP which I normally don't do, I usually just have a 1400 byte MTU on the L2TP tunnel and TCP MSS clamping.

Is there some way to trick CAKE into not reordering the packets for one connection, e.g. by setting some connection mark, packet mark, DSCP or packet priority? Then I could use a mangle rule to set that on this ESP traffic.

The original question I asked in this topic also still stands: is there no way at all to display the behavior of a CAKE queue?
It would be nice to see some stats on what it is doing. Usually I use "queue tree" and it nicely shows what it puts where and what queues are being overloaded.
 
dtaht
Member Candidate
Member Candidate
Posts: 215
Joined: Sat Aug 03, 2013 5:46 am

Re: CAKE status display (and CAKE reordering problem)?

Tue Jul 30, 2024 5:58 am

A packet capture is likely to be too large to email. You can upload it to one of my servers if you send me a ssh key, or use dropbox?

I guess a question that fell out of this is what size is the re-order buffer set to on the tunnel?
 
dtaht
Member Candidate
Member Candidate
Posts: 215
Joined: Sat Aug 03, 2013 5:46 am

Re: CAKE status display (and CAKE reordering problem)?

Tue Jul 30, 2024 5:59 am

I had campaigned for mikrotik to be able to output the tc -s qdisc show
stats for cake 2 years ago. More folk should. At least drops and marks...
 
pe1chl
Forum Guru
Forum Guru
Topic Author
Posts: 10486
Joined: Mon Jun 08, 2015 12:09 pm

Re: CAKE status display (and CAKE reordering problem)?

Tue Jul 30, 2024 9:30 am

I sent you two traces as a tar.gz file, is only about half a megabyte so should not be a problem.
There is no setting (as far as I know) for the re-order buffer in RouterOS.
So at Linux level it probably is at default values.
I will ask MikroTik about this.
Thanks again for your attention!

Who is online

Users browsing this forum: pc327 and 26 guests