TZSP Sniffer: Missing out some (bigger?) UDP packets

When trying to sniff on UDP portrange 23301-23302 with both wireshark on device and TZSP sniff on my hex poe, I get different amount of packets: it seems that the sniffer does miss some.

It appears that packets with more than 219 bytes aren't recorded, even if the MTU is much higher (1500).

Looking at posts such as packet sniffer missing outgoing ESP datagrams , I do wonder on how to be able to sniff on bigger UDP packets... is there a way to increase the MTU only for TZSP traffic or something like this? Funny enough, the exported pcap file from RouterOS does also not contain any of the missing packets.

Even if we take the TZSP header into account, the amount of bytes should still be under the MTU, so I'm quite unsure on how to improve my situation.

If the pcap file generated by the Packet Sniffer also doesn't contain the packets, you can't blame it on the TZSP stream since it can't stream what wasn't captured.
Also, "TZSP Sniffer" = Firewall Mangle rule with action=sniff-tzsp or streaming-enabled=yes in the Packet Sniffer?
There is a bug currently with the TZSP Stream from the Packet Sniffer but that's not it.
I didn't encounter missing packets either.

Also, "TZSP Sniffer" = Firewall Mangle rule with action=sniff-tzsp or streaming-enabled=yes in the Packet Sniffer?

Tried with both, both didn't send all packets.

Sending all UDP packets to the sniff target is unrealistic because of the sheer amount of them, and the target is a raspi...

There is a bug currently with the TZSP Stream from the Packet Sniffer but that's not it.

What is it?

Are you sure that the packets are there to begin with? Try capturing them with some other method too, proving their existence.
Regarding the bug I've mentioned, some packets over a certain size (~2000 bytes) get truncated when sent over the TZSP stream generated by the Packet Sniffer, they don't go missing though, and it works fine with the sniff-tzsp mangle rule.
And they are properly saved in the pcap file by the Packet Sniffer.
It doesn't seem to be related in anyway to your problem though.

Are you sure that the packets are there to begin with? Try capturing them with some other method too, proving their existence.

As said in the OP, I had wireshark listening on the device (PC) and also getting the TZSP packets from the sniff. The former had much more packets to show.

Is there a way to generate those problematic packets so we can try to replicate the issue?

I try to migrate from wireshark-on-device to sniff-on-router for an application, the only thing I can provide, is this screenshot:

The left side: wireshark
The right side: TZSP.exe > the pcap file

ip.flags.df eq false does reduce the amount of packets on both sides, but there are some left.

The info shows the packet ids: as it can be seen, the first couple of packets have the right ids, then skipping happens, which reports completely different ones. Scrolling up to the end, I can see 0xf7a8 in some of the last packets on the left side. So no packet manipulation of any form happens between the capture on device and the router's sniffing.

The generation essentially happens by opening the application. Not much to say on that front.

Is this a known app?
You're not giving many details.

Ultimately, my project is to have reliquary-archiver/src/capture/pcap.rs at main · IceDynamix/reliquary-archiver · GitHub be outsourced onto tzsp and from the router instead of pcap.

That's not what I've asked.
Is there a way to replicate your issue?

Yes,

install game (as denoted by the repository) and run it according to the instructions (as to generate the UDP traffic on port=23301-23302).
create a mangle rule

/ip firewall mangle
add action=sniff-tzsp chain=forward comment="TZSP Traffic" log-prefix=HOYO: port=23301-23302 protocol=udp sniff-target=192.168.20.254 sniff-target-port=37008

run on your device:

the tzsp -> wireshark

tzsp | wireshark -i - -k # in one terminal (git-bash helps)

wireshark with capture filter

udp portrange 23301-23302

On my end, there is as discrepancy in UDP packets reported, as can be seen from the screenshot in my comment above.

tzsp | wireshark -i - -k

Not sure what tzsp is in this context - its an application that likely strips off the tzsp header and passes to STDIN for Wireshark? Where to get this? Anyway, I would simplify until it all works - look directly at the traffic on the interface, don't pass it through anything yet and the capture filter listed would have to be on the command line the way you are running this, and its not, so is it really a display filter? If a capture filter, suggest stop using for now until the problem is under control. Too much filtering and I always wonder what is happening to the L3 fragmented packets. What do you see from wireshark directly on the interface?

In the first post this is mentioned:

So I wouldn't blame it on TZSP.
Capture filters were not mentioned for the pcap generated on the router, though.

I have not had issues with TZSP.

Also not used nor familiar with tzsp | ....

IMO, you'd better of setting the BPF on the interface to udp port 37008 to filter TZSP before the packet filters and then set the dissector to use tzsp for the port. Or at least I know that works.

Not sure what tzsp is in this context - its an application that likely strips off the tzsp header and passes to STDIN for Wireshark? Where to get this?

Correct, it's https://crates.io/crates/tzsp/0.1.0

don't pass it through anything yet and the capture filter listed would have to be on the command line the way you are running this, and its not, so is it really a display filter?

What do you see from wireshark directly on the interface?

I have 2 wiresharks open:

  • one which has either capture filter or display filter (both show the same thing) to filter out port 23301-23302
  • accept the tzsp stream and show it (tzsp utility converts to pcap, so wireshark can read and display it)

the latter is either made using tzsp sniffer tool or a firewall mangle rule; in both cases filtering udp with the 23301-23302 ports.

Capture filters were not mentioned for the pcap generated on the router, though.

On the router, the sniffer tool had the same settings as my posted firewall mangle rule.

Also not used nor familiar with tzsp | ....

As explained, it's a utility to convert a tzsp stream into a pcap stream, so wireshark can show it. The conversion has no issues. The Sniffer tool (produce a pcap + streams tzsp) and firewall mangle /tzsp stream) are both giving the same "broken" amount of packets back.

Again: Sniffer (tool & firewall) doesn't provide the same output as using wireshark on-device.

If you say so. I tend agree with @Znevna here since I have not found issues where they unexpectedly diverge (since a direct listen is going to get OS packets too).

That does not mean there is not a bug, or config error, or "odd rule" related for fasttrack/fastpath/etc that's interfering. My only point be if you think there is a bug, there is not a lot of hope of fix...unless you can produce a test case that isn't "install this game" and "see screenshot".

I don't have much trouble with this feature though don't use it all that often - prefer to use the switch feature and mirror the port, but everyone has their preference and use cases. The firewall mangle rule has a udp selector, as someone pointed out. Is the same filter used with the sniffer tool? You don't provide a lot of detail with your pictures so I can't look for differences - type of L4 traffic, fragments, etc., but we understand it could be confidential. I still would consider the capture filters on the Mikrotik side as possibly blocking the traffic. Can you open them up?

So here's what I don't get.
You did mention in the first post, this topic that contains this post:

And yet your filter rule on the mikrotik does have a filter by port (and you've mentioned the same filter used for Packet Sniffer).
Did you try without any filters on the MikroTik or not?