Undecoded packets

When testing the upload to a Speedtest server I get undecoded packets in Mangle.

Underneath a capture in Wireshark of a ICMP type 3-4 packet in Mangle during the upload test of Speedtest. These packets are generated on that exact moment.
I can log them in RAW and Filter but not look in the packets on that moment because Sniff is not possible there. The packets in Filter are also 576 bytes so they have to be the same ones.
PMTUD.JPG
Half of the PMUTD works and that is when downloading and it finds the right size for the packets. When uploading the PMTUD can’t be determined and I have to do that manually by setting a MTU size in Mangle for TCP packets.

I am seeing this when I use a IKEv2 connection and before also with a L2TP/IPSEC connection.

How did you get the packet into Wireshark? It is 99% an ICMP packet, but the ethertype value 0x0800 is missing between the fake Ethernet header consisting of all 0s and the beginning of the IP header. So it’s either an issue of the sniffer (which adds fake ethernet headers to mere IP packets) or of the way you’ve imported the hex dump into Wireshark.

The packets are sent to Wireshark by the the snifferline in mangle over UDP port 37008. The line only catch ICMP 3/4 wich are not handled on that moment.

Can you sniff other packets the same way (action=sniff-tzsp in mangle) to check whether this issue (not adding the 0x800 ethertype word between the fake Ethernet header and the IP packet) is a systematic bug of the TZSP encapsulation or it only affects these ICMP packets?

Just did it myself (sniffing using a mangle rule on a GRE interface which is also IP-only), and the ethertype value is correct. In your case, it is actually not missing, it is just wrong (0x0 instead of 0x800, sorry, I’ve missed that when looking at it initially). So unless this mistake is a specialty of your architecture, it may be that it is not a sniffing bug but really a bug in creation of those packets.

In any case, if what you actually need is just to have the contents decoded, you can export the encapsulated data in Wireshark (starting from the 45 c0 onwards) into a hex dump and use “Import from Hex Dump”, indicating the Encapsulation type as “Raw IP”. If you need to understand whether the packet has been actually matched by an IPsec policy and delivered to the destination, I’m afraid you’ll have to set up your own IPsec peer to use instead of the public VPN you use.

EDIT: there is a much simpler way to get those packets decoded in Wireshark. While the packet with a wrong ethertype value is selected in the packet list, right-click it and choose Decode as… from the drop-down menu. Add a row to the table, and set its Field to Ethertype, Value to 0, and Current to IPv4.

So I disabled my MSS line again to generate the those packet. On upload they are generated and typing/browsing here in the Mikrotik forum new ones were coming in Wireshark. So I have now a second instance where those strange packets are generated.

My snifferline is catching postrouting/out packets, so I can’t put an IPSEC-in on it and it filters on ICMP and at the same time snifferline logs and source address it the local IKEv2 IP and the correct client computer running the speedtest or using the Mikrotik forum.
45c0.JPG
I made an error changing the sniffer line and activated the IPV4 Options field with option no record route by accident and the packets still came in. Changed the options to record route and the packets were not caught anymore.

This 100% reproduce on my side.

We have a long tradition of me not understanding what you actually mean by what you write. Whether this effect is bi-directional is unknown to me :slight_smile:

So in this case, are you saying that these packets appear as you browse the forum but nevertheless the browsing works even while the MSS-changing rule is disabled? Because normally, the Path MTU Detection works with a single ICMP feedback per each bottleneck - when a packet with Don’t Fragment bit set comes to a router where it doesn’t fit to the MTU of the outgoing route, that router sends back an ICMP type 3 code 4 message indicating the available MTU, and the sender network stack or application takes the necessary action (which is, in case of TCP stack, to send a smaller portion of the inupt buffer in a packet just fitting the MTU reported by the ICMP type 3 code 4 message, again with the DF flag set, in order to eventually identify an even narrower bottleneck downstream to the destination). In case of TCP, the whole goal is to slice the incoming payload stream into small enough TCP packets already at source instead of letting them be fragmented during transmission, because fragmentation during transmission uses more overhead data than sending smaller packets at source.

So I can imagine the sender to be hammering the destination with retransmissions of too-large-to-fit packets with DF set but never receivind the ICMP type 3 code 4 reports back, but I cannot imagine the resulting connection to ever be set up. That’s why I am double-checking whether I have understood you well this time.


And here we may be coming to the reason why the PMTUD doesn’t work well in this direction. I remember you used to have a very specific setup of two piggybacked routers, but this may have changed in the meantime. In any case, these ICMP type 3 code 4 messages do not directly match the tracked connection they are related to. So the connection tracking module of the firewall identifies them as related to an existing tracked connection, and labels them with connection-state=related, by checking the contents of their payload (where the IP and TCP header of the triggering packet are copied). Hence a filter rule action=accept connection-state=…,related,… must exist in order that these ICMP packets could pass through the firewall. Is this the case all the way from the router which generates these packets to the client PC?

These ICMP towards the client (i.e. triggered by too-large-to-fit packets sent in client->server direction) are generated by the router doing the IPsec encapsulation. The IPsec transport packet is bigger than the payload packet it transports by the size of the encapsulation headers, so if the resulting transport one doesn’t fit to the MTU of its out-interface, the router performing the encapsulation sends an ICMP type 3 code 4 packet back to the address of the client from its own address - that’s why you can see this packet in the postrouting and/or output chain on that router.

So I’d first of all sniff for these packets on the client-facing physical interface of the router generating them, using the “sniff to file” functionality (so that you could distinguish those caught by the mangle rule from those caught by sniffing on the interface if doing both simultaneously). If they are caught by the sniff-tzsp mangle rules, but aren’t visible in the sniff on the physical interface, they are generated or routed improperly.

Thanks for your patience Sindy.

I use still the piggyback config and as I see it the tunnel (4500) is routed through the outer (GW) router and only de content of the tunnel is decoded at the VPN provider side and on the inner router on my side where also those packet get created.

Writing is was not correct but I can 100% generate those packet when I have my MSS line disabled and click the button Preview in this forum. I had always troubles with this forum and the Antary.de site. I was locked out and could not login here some long duration so nice quiet from my side. :wink: This happened also when I was using L2TP/IPSEC before I switched to IKEv2.
This makes testing for me a lot easier because before I had to run a Speedtest and had to wait till it was ready with downloading the test traffic before it reached the upload test traffic.

I used the Sniffer to file with the filter on ICMP packets, nothing was written and the file stayed empty. I ran it with no interface and with the specific Ethernet port selected. So, yes those packet never reaches the client and I also tested that al few months ago when started to look into this.

Tested it with related disabled on the snifferline in Mangle and then no packets are sent to Wireshark so that is confirmed.

I don’t often use related in my lines so I have to check the complete way packets travel through the firewall. That will be later today and I will post here about it.

Update a quick check: related is active on slow traffic (established and related not fast-tracked) in filter and in other pages I have connection state not active.

“with no interface and with the specific Ethernet port selected” makes little sense to me. There is a lot of confusion regarding “interface” and “port” in RouterOS, because if IP configuration is attached directly to etherX, etherX is the L3 interface, whereas if etherX is a member port of a bridge, the L3 configuration should be attached to the bridge and thus the L3 interface is the bridge. But this is the perspective of the firewall; when setting up sniffer, everything is referred to as interfaces. So if you have told the sniffer to sniff at the Ethernet interface through which the packets towards the client PC should be routed, it doesn’t matter whether it was the actual L3 interface or just a member port of the bridge. So maybe you might sniff really without specifying any interface at all, and just specify ip-protocol=icmp, to see whether the type 3 code 4 packets are eventually routed somewhere else than expected.

And there is a good chance that policy routing (i.e. use of connection-mark and routing-mark) causes these locally originated ICMP type 3 code 4 packets to be misrouted, or that an IPsec policy matches on them and diverts them because a NAT rule matching on connection-mark NATs them to a source address which the IPsec policy matches on (because packets related to a connection inherit its connection-mark)


Basically if the first rule in forward chain accepts related traffic without any limiting conditions, it doesn’t matter what the subsequent rules do about it as it never reaches them. So the fastest test whether dropping of “related” traffic is the reason, just place a rule chain=forward action=accept connection-state=related as the very first one in chain=forward of any router between the one which generates the packets and the client. On the generating router, these packets may be dropped by firewall only if you had some specific rules in chain=output which is typically empty. So I rather vote for them being misrouted rather than dropped by firewall.


Regarding the OP - I’ve done some tests here and my conclusion is that the incorrect population of the ethertype field in the cooked ethernet header by 0x0 instead of proper 0x800 is a behaviour specific to action=sniff-tzsp in chain=output of /ip firewall mangle. It does not depend on IP protocol being carried by the packet being sniffed (TCP packets are affected as well), the less on the particular ICMP type. On the other hand, action=sniff-tzsp in chain=postrouting of /ip firewall mangle does not suffer from this issue, even for the ICMP type 3 code 4 packets generated by the router itself. So the ICMP type 3 code 4 packets themselves are not malformed (as sniffing using chain=postrouting will show you), it is solely an issue of the sniffing in chain=output of mangle.

It might be worth notifying support@mikrotik.com about this issue.

I tried it with no interface given and with interface given and either one wrote information in the file.

And there is a good chance that policy routing (i.e. use of > connection-mark > and > routing-mark> ) causes these locally originated ICMP type 3 code 4 packets to be misrouted, or > that an IPsec policy matches on them and diverts them because a NAT rule matching on connection-mark NATs them to a source address which the IPsec policy matches on (because packets related to a connection inherit its connection-mark)

Time for VTI

[quote=“, post:8, topic:134293”]
Update a quick check: related is active on slow traffic (established and related not fast-tracked) in filter and in other pages I have connection state not active.
[/quote]

Basically if the first rule in forward chain accepts > related > traffic without any limiting conditions, it doesn’t matter what the subsequent rules do about it as it never reaches them. So the fastest test whether dropping of “related” traffic is the reason, just place a rule > chain=forward action=accept connection-state=related > as the very first one in chain=forward of any router between the one which generates the packets and the client. On the generating router, these packets may be dropped by firewall only if you had some specific rules in > chain=output > which is typically empty. So I rather vote for them being misrouted rather than dropped by firewall.

I checked that forward/input/output with an accept and established/related are on top under the dummy line for fasttracking.

Regarding the OP - I’ve done some tests here and my conclusion is that the incorrect population of the > ethertype > field in the cooked ethernet header by 0x0 instead of proper 0x800 is a behaviour specific to > action=sniff-tzsp > in > chain=output > of > /ip firewall mangle> . It does not depend on IP protocol being carried by the packet being sniffed (TCP packets are affected as well), the less on the particular ICMP type. On the other hand, > action=sniff-tzsp > in > chain=postrouting > of > /ip firewall mangle > does not suffer from this issue, even for the ICMP type 3 code 4 packets generated by the router itself. So the ICMP type 3 code 4 packets themselves are not malformed (as sniffing using > chain=postrouting > will show you), it is solely an issue of the sniffing in > chain=output > of mangle.

Output or Postrouting do both create those strange ethernet headers my setup.

It might be worth notifying > support@mikrotik.com > about this issue.

BTW, the fallback to 576/536 did not work now so I really lost the connection when clicking Preview.

I agree with you, however the implementation date may be “never”. So we either need a solution without VTI or a demonstration that no such solution is possible and that only the VTI would be a solution.

So can you confirm you do use the connection-mark assignment by mode-config to let RouterOS dynamically create the rule in src-nat chain? And if you do, can you paste the output of /ip firewall nat print chain=srcnat while the IPsec connection is active (I’m actually only interested in the dynamic rule itself and whether it is really added as the first one in chain=srcnat as I suppose)?

Anyway, there should be a remedy - a static IPsec policy action=none src-address=0.0.0.0/0 dst-address=the.client’s.subnet placed before the policy template which is used to build the dynamic policy with the responder-provided IP address as src-address. This action=none policy will shadow the dynamically generated one so even though the ICMP code 3 type 4 packet will likely get src-nated by the dynamic src-nat rule, it will not reach the dynamic policy (which would divert it into the tunnel) so it will make it to the client. The client won’t care about the source address as it has no relevance for it, so it should adjust the size of the re-sent TCP packet and all the subsequent ones accordingly.


That’s an interesting bit of information - I’ve first placed the rule only to chain=output and seen the TZSP packet to come with the bug. Then, I’ve added the same rule also to chain=postrouting, and the TZSP packets started coming in pairs, the first one with the bug and the second one without. I’m running 6.45.7 on hAP ac², can you test the same (both rules in place) and state your sw version and hw model?


OK, it seems I finally understand how you test. So once you disable the change-mss rule, you press Preview, and see whether the establishment of the new HTTPS connection succeeds or not. So you really can see the retransmissions to hammer, not each keypress in a working connection to trigger one ICMP packet. Good.

I AM FLABBERGASTED, AFTERS MONTHS OF TRYING AND SEVERAL SUPPORT REQUESTS YOU SOLVED THIS NAGGING PROBLEM.
THANK YOU SO MUCH SINDY!!!

This line moved above the dynamic lines in /ip ipsec policy, and now my sniffer line gives Destination unreachable (Fragmentation needed). Never been so pleased to read te word unreachable before.

The line is and replace 192.168.88.0/24 by your own local IP range:

add action=none dst-address=192.168.88.0/24 src-address=0.0.0.0/0

The Dynamic line is added to the top in the NAT screen:

0  D ;;; ipsec mode-config
chain=srcnat action=src-nat to-addresses=xx.x.x.xx connection-mark=NordVPN

On TZSP packet, I have a longstanding habit of putting my hardware and RouterOS version in my signature underneath all my posts. :wink:

I am so pleased that it is finally not mystery anymore why the ICMP 3/4 packets did not arrive with the client. I will report this back to Mikrotik and maybe they will add this to the config for further IKEv2 connections like NordVPN or even dynamically generate it when on creation of the Dynamic Nat line(s) for IKEv2 if it is not already present. Then the local address range should be known and that might be a problem.

:slight_smile:

Glad to help. However, as the title and OP of this topic do not reveal the actual issue you were fighting, it would be fine if you could find your other topic which is more clearly related and post the solution there (maybe with a link to the detailed explanation here in this topic) - if anyone else falls into the same rabbit hole, they’re unlikely to find this topic (containing the actual solution) by keywords, let alone the title.

In most setups where you use VPN to anonymize your access to internet, you can add an action=none “shadowing” policy for each of the three RFC1918 ranges, 10.0.0.0/8, 172.16.0.0/12, and 192.168.0.0/16. I don’t think these should be added automatically.

Another possibility (similarly complex to set up as the one with action=none policies, though) is to set both the address-list and the connection-mark in the mode-config row, and add several address ranges, excluding the LAN address(es) of the router itself into that adress-list (like 0.0.0.1-192.168.88.0, 192.168.88.2-255.255.255.255 if router’s own address is 192.168.88.1). This way, the dynamically created NAT rule will not match on the packets sent by the router itself, so it won’t make them visible to the dynamically generated policy, and no shadowing policies will be needed.

I have send a reply back to Mikrotik support about what you found out today to solve this mystery. The nice thing is that other protocols are also handled by the IPSEC policy action none and so UDP packets are also handled what the Mangle MSS line could not do.

I also said that there will be a new support request on the strange packet when using TSZP on the out/postrouting stream. Let me know if you want to request that or that I shall do that?

The address ranges from clients can also be seen in /IP Address and then specific the Bridge master and if not present Ethernet ports

I will create a new topic with a meaningful title and content and I hope that I will have a reply from Mikrotik support by then.

Again many thanks for helping me in this and so also helping others seeing odd thing happening while other sites works normally.

I usually follow the principle “the one who’s affected opens the support ticket”. Gents will ask for supout.rif etc. and I have enough open tickets of my own :slight_smile:


Yes, sure, what I have in mind is that I don’t think Mikrotik should dynamically generate the exception rules based on this information, because there are more VPN scenarios than just anonymization of internet access, namely those “real” VPNs where you connect to your enterprise network or interconnect several sites of it, and in these cases the shadowing policies open this wide would be harmful. So the shadowing policies have to be tailored to the particular configuration, and this simply cannot be done automatically.


Ah, I thought you have already open one before and I’ve just missed it.

I got some monologue topics like this one: Still struggling with MSS/MTU IKEv2 but to me it be more a topic like a notice or if Mikrotik it pleases, be in the Wiki. These IKEv2 connections to an VPN provider should be easy and ‘fool prove’ and be easy to activate. It has to be transparent to user and all the work should be done in the background as it all like magic…good magic like it is as for me. :wink:

The NordVPN case of them throwing out methods of connecting urged Mikrotik to keep up with them and we got a solution that works and was expanded on request of the users with connection-marking. We have now at least IKEv2 to VPN providers which was a big improvement in the pre-RouterOS7 time-frame.

I forgot to add the one using source address:

 0  D ;;; ipsec mode-config
      chain=srcnat action=src-nat to-addresses=188.xx.xx.x src-address-list=VPN dst-address-list=!VPN

Directly underneath I have a catch for when the IKEv2 is still starting and the client already started to sent packets.

 1    ;;; Catches traffic when IKEV is inactive
      chain=srcnat action=src-nat to-addresses=127.0.0.1 src-address=192.168.88.xx log=no log-prefix="No escaping

For the connection marked I have a separate “no escaping” line.

It seems I was just lucky when packets copied using a postrouting rule weren’t affected by that issue on my hAP ac². In another capture scenario, some packets copied using the very same postrouting rule were affected by the bug while other ones were not. So I’ve edited one of my earlier posts here (which is still related to the title of this topic) with a more useful method of working that bug around.

The posting I made about the MTU not being sent to the client. Let me know if anything has be put differently?

Well, my only remark is that I don’t see it as a workaround but as a final solution because that’s how policies work.

And what surprises me is that the order of the policies is not relevant, because imagine these two hypothetical policies:

src-address=0.0.0.0/0 dst-address=192.168.1.1
src-address=192.168.2.3 dst-address=0.0.0.0/0

If order of policies was not important, none of these policies would be the “correct” one for a packet from 192.168.2.3 to 192.168.1.1, because both match it.

So either action=none policies always win over action=encrypt ones, regardless the order, or there is something rotten in policy matching.