Page 1 of 1

/tool sniffer Code: 3 (Port unreachable)

Posted: Thu Apr 18, 2019 6:41 am
by hendry
Hi, I'm trying to update a tutorial that uses Mikrotik hardware to capture packets and send to a machine running Wireshark for analysis.

The problem I have is noise. Lots of Code: 3 (Port unreachable) messages that don't make any sense.

Image

Even filtering on tcp only doesn't appear to help!

Here's a screencast: https://s.natalian.org/2019-04-18/mikrotik-sniffing.mp4

I've asked the Wireshark community, and they seem to think I have the Mikrotik misconfigured somehow:
https://ask.wireshark.org/question/8619 ... iff-noise/

Re: /tool sniffer Code: 3 (Port unreachable)

Posted: Thu Apr 18, 2019 11:36 pm
by Sob
That's sort of correct, because TZSP used by packet sniffer for streaming sends packets to udp port 37008. But on target computer, nothing listens on that port, Wireshark certainly doesn't. And if it's not blocked by firewall, your computer sending back icmp port unreachable is exactly what should happen.

I don't know what's the correct solution. Wireshark sniffing on interface is definitely not it. A clean one would be some helper program that would be actually listenening on udp 37008, receiving packets from RouterOS, removing TZSP wrapping and passing them to Wireshark. There's "UDP Listener remote capture" which sounds good, but it doesn't seem to be it. It listens on port 5555 (I didn't find a way how to change it from Wireshark) and when I forwarded TZSP packets through another router to change the port, the output is weird anyway.

Re: /tool sniffer Code: 3 (Port unreachable)

Posted: Fri Apr 19, 2019 2:52 am
by McSee
You may try to use "udp port 37008" as a capture filter in Wireshark, it works for me.

And it's also a good idea to filter stream on mikrotik's side at least down to a certain interface as poor Wireshark
gets confused by the same packets captured several times.

Re: /tool sniffer Code: 3 (Port unreachable)

Posted: Fri Apr 19, 2019 4:14 am
by Sob
Capture filter in Wireshark doesn't change the fact that RouterOS is sending packets to closed port. So if target computer sends back "port unreachable" and you're capturing traffic on same interface where it's connected, you will see it in captured data (if there's not a filter that excludes it).

But it seems that so far nobody cared about proper solution, i.e. something that would actually receive those packets and pass them to Wireshark. I don't know if I'm just using wrong keywords for searching, but I found nothing.

Re: /tool sniffer Code: 3 (Port unreachable)

Posted: Fri Apr 19, 2019 6:00 am
by McSee
I do not have these "port unreachable" icmp packets in my capture (see below).
Settings as follows:
only-headers: no
memory-limit: 1000KiB
memory-scroll: yes
file-name: 
file-limit: 30000KiB
streaming-enabled: yes
streaming-server: 192.168.10.101
filter-stream: yes
filter-interface: bridge
filter-mac-address: 
filter-mac-protocol: 
filter-ip-address: 
filter-ipv6-address: 
filter-ip-protocol: 
filter-port: !winbox
filter-cpu: 
filter-direction: any
filter-operator-between-entries: or
W1.PNG

Re: /tool sniffer Code: 3 (Port unreachable)

Posted: Fri Apr 19, 2019 6:24 am
by Zwe
Good job bro.

Re: /tool sniffer Code: 3 (Port unreachable)

Posted: Fri Apr 19, 2019 12:11 pm
by hendry
Spent some of my good Friday trying some things out between two computers:
  • "udp port 37008" is not a good solution since on both MacOS / Arch I get bytes missing in capture file when I follow the stream
  • filter-stream option seems to have no effect
  • macos's `brew cask install wireshark` does NOT have this Code: 3 (Post unreachable) problem :-? UPDATE: Actually I saw it on MacOS, so now I am confused.
  • think filter-interface: bridge1 works better than the previous interface I tried
Screencast: https://s.natalian.org/2019-04-19/sniffing.mp4

Re: /tool sniffer Code: 3 (Port unreachable)

Posted: Fri Apr 19, 2019 4:40 pm
by McSee
  • "udp port 37008" is not a good solution since on both MacOS / Arch I get bytes missing in capture file when I follow the stream
  • filter-stream option seems to have no effect
  • macos's `brew cask install wireshark` does NOT have this Code: 3 (Post unreachable) problem :-? UPDATE: Actually I saw it on MacOS, so now I am confused.
  • think filter-interface: bridge1 works better than the previous interface I tried
Following your list:

1. This is strange as Mikrotik's packet sniffer should send all captured packets to UDP/37008. And when you set "TZSP" view filter in Wireshark
you should see only UDP packets with dst port 37008 - the same packets that should be captured with "udp port 37008" capture filter.
It seems I get the same results with either method.

2. Filter-stream very much works for me as without it enabled I get ~170 Mbps of some (fragmented ?) UDP packets captured out of the blue, as on the screenshot below. And it's also cancels out those "port unreachable" packets when I've tested on macOS, but you of course still see them captured locally if do not set capture filter in Wireshark, and you can always tuck them away with "TZSP" view filter.
W2.PNG
3. Yes, I finally have been able to see those "port unreachable" on macOS, but none of them when testing on Windows 10, with any settings of Mikrotik's packet sniffer and Wireshark.

Re: /tool sniffer Code: 3 (Port unreachable)

Posted: Fri Apr 19, 2019 6:33 pm
by Sob
If closed udp port (= no process is listening on it) receives packet, system generates icmp port unreachable and sends it to client. That's normal behaviour. This does not happen when the port is firewalled and incoming packet is silently dropped before it can reach the closed port. I don't know about MacOS, but firewall in Windows by default blocks everything, so that could explain the difference.

Next part is RouterOS, it depends on where you capture data and where you send them. Let's say you have two interfaces on router (LAN and WAN) and your Wireshark machine is connected to LAN. If you capture packets on WAN, then even if Wireshark machine sends icmp port unreachable to router, it doesn't matter, because it's on LAN interface and sniffer doesn't see it. But if you capture packets on LAN, then sniffer sees incoming icmp port unreachable and will send them back to you, wrapped in TZSP.

The main problem is that capturing TZSP with Wireshark this way is simply wrong. What you want to see are packets from router, exactly as the router had them, but it's not what you're getting:
tzsp.png
The whole red part shouldn't be there, it's TZSP wrapping and you're not intersted in that. It's confusing for both user and sometimes also for Wireshark, it seems. But it doesn't look like anyone implemented proper solution yet (something actually receiving TZSP packets and unwrapping originals).

Re: /tool sniffer Code: 3 (Port unreachable)

Posted: Fri Apr 19, 2019 11:15 pm
by r00t
Yes, this is the problem with TZSP and UDP. Often some packets get lost, meaning IDS can't inspect all of them. Also wireshark doesn't like it and some filters apply to the UDP TZSP stream instead of sniffed packets.
It would be better to have an option to do port mirroring for bridges in ROS, where all traffic would be mirrored to a selected port. You could then set up a tunnel to get these packets to the IDS or receive them on PC on separate network interface (like OpenVPN).

Re: /tool sniffer Code: 3 (Port unreachable)

Posted: Sat Apr 20, 2019 2:24 am
by Sob
I played with it a little more and made a proof-of-concept of what I consider proper solution. Wireshark supports plugin interface called extcap. It's external executables or scripts that acquire data and feed them to Wireshark. And it's exactly what's needed here, something that receives TZSP packets on port 37008, unwraps original packets and passes them to Wireshark.

It's nowhere near to really be THE solution, it's minimalistic version without any error checks and with ugly shortcuts, but if you're not too unlucky, it works. It's made with PHP, which is probably quite unusual choice, but I really like PHP, especially for quick tests. It's tested with Wireshark 3.0.1 on Windows 7.

<Wireshark directory>\extcap\tzspdump.cmd:
@echo off
<path to php directory>\php.exe "<Wireshark directory>\tzspdump.php" %*
<Wireshark directory>\tzspdump.php:
<?php
  $options = getopt('', Array('extcap-interfaces', 'capture', 'extcap-interface:', 'fifo:', 'extcap-capture-filter:'));
  if(empty($options)) {
    echo 'Wireshark - tzspdump.php v0.0.1'.PHP_EOL;
  } elseif(isset($options['extcap-interfaces'])) {
    echo 'extcap {version=0.0.1}'.PHP_EOL;
    echo 'interface {value=tzspdump}{display=TZSP Listener remote capture}'.PHP_EOL;
  } elseif(isset($options['capture'])) {
    if(isset($options['fifo'])) {
      $pipe = str_replace('\\\\.\\pipe', '\\\\'.php_uname('n').'\\pipe', $options['fifo']);
      $ph = fopen($pipe, 'a+');
      fwrite($ph, "\xd4\xc3\xb2\xa1\x02\x00\x04\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x04\x00\x01\x00\x00\x00");
      $sock = socket_create(AF_INET, SOCK_DGRAM, 0);
      socket_bind($sock, '0.0.0.0' , 37008);
      while(true) {
        socket_recvfrom($sock, $buf, 10000, 0, $remote_ip, $remote_port);
        $size = strlen($buf);
        $t = explode(' ', microtime());
        $size -= 5;
        if(!fwrite($ph, pack('L', intval($t[1])).pack('L', intval(substr($t[0], 2))).pack('L', $size).pack('L', $size))) break;
        if(!fwrite($ph, substr($buf, 5))) break;
      }
      socket_close($sock);
    }
  }
?>
Once installed, it will add another interface to Wireshark:
tzsp1.png
If you select it and start capture, Wireshark will start the script and it will listen on udp/37008 (you may need to allow it in firewall) and will pass all received packets to Wireshark. Here is an example of router sending CDP packet (notice that it's the original packet, as was on the router, no TZSP wrapping):
tzsp2.png
And that's all folks. I don't plan to develop this further, because PHP script is a dead end, regular people don't have PHP installed and to get it only for this is unreasonable. Best solution would be to convince Wireshark guys to properly implement this. Or someone else to do it and send them a patch. Feel free to do any of that, I'm not very good with convincing someone and too lazy to write proper code. ;)

But that's not really all, there's one more thing. It seems that there might be a bug in RouterOS. The option to filter stream (to not capture TZSP packets sent by router) doesn't seem to always work. With small packets, everything is ok. But large packets don't fit in one TZSP packet and need to be sent fragmented. And then something like this happens, router captures own TZSP packets (it's captured using regular Wireshark capture on ethernet interface):
tzsp3.png

Re: /tool sniffer Code: 3 (Port unreachable)

Posted: Sat Apr 20, 2019 12:30 pm
by McSee
Sob,

have you seen Mikrotik's very own Trafr utility?
( download link -- http://www.mikrotik.com/download/trafr.tgz )
Which is supposed to make proper ".pcap" from TZSP.

Then there is also Tzsp2pcap ( https://github.com/thefloweringash/tzsp2pcap ).
And it might be possible to remove extra TZSP bits from captured packets with pcap capture file editor Bittwiste
( http://manpages.ubuntu.com/manpages/bio ... ste.1.html )

Re: /tool sniffer Code: 3 (Port unreachable)

Posted: Sat Apr 20, 2019 6:10 pm
by Sob
I vaguely remember trafr, but looking at it now, it's only Linux binary, so not useful for me on Windows. Tzsp2pcap looks like it can work under Cygwin, so that would be better. They both qualify as proper solution too. Bittwiste looks interesting, but probably not the right tool for this.

But to be honest, I currently prefer my experiment, regardless how ugly it's inside, because it's nicely integrated with Wireshark, just one click away. :)

Re: /tool sniffer Code: 3 (Port unreachable)

Posted: Mon Apr 22, 2019 10:47 am
by hendry
I did manage to create a follow up video: https://www.youtube.com/watch?v=xAJlpSbbIhk

The suggestions here seem to require introducing some intermediate steps to get a nice session in Wireshark. Which is far from ideal.

Same goes in the Wireshark community's comments. https://ask.wireshark.org/question/8619 ... st-id-8635

Oh noes. Was really hoping I could recommend Mikrotik for this sniffing task, but now I'm not so sure.

Re: /tool sniffer Code: 3 (Port unreachable)

Posted: Mon Apr 22, 2019 9:55 pm
by Sob
But it's not really MikroTik's fault (except that possible bug with fragmented packets, someone should look into that), it's more Wireshark not having the best possible default support for handling TZSP. And maybe it's ok too, because I'm not sure if TZSP is used that much.

Re: /tool sniffer Code: 3 (Port unreachable)

Posted: Mon Apr 22, 2019 10:23 pm
by sindy
I wonder whether you have tried to follow my advice from ask.wireshark.org - to exclude the interface via which the packets leave towards the wireshark machine from sniffing? I know that the way the /tool sniffer handles the filter is somehow weird, but I believe that even with filter-operator-between-entries set to and, you can set filter-interface to a list of interfaces rather than a single one (if you need it) and still get packets from all the interfaces on the list; however, in your case this should not even be necessary, as you are only interested in the traffic on the wireless interface, aren't you?

As for the rest - if some packets do not make it to the Wireshark capture while others do, the CPU of the Mikrotik is most likely overloaded while sniffing; if none makes it there (which is not your case), it may be some anti-virus software which is hooked closer to the wire than WinPcap/npcap.

Whether the PC sends back its icmp complaints about incoming TZSP packets may depend on a lot of things including firewall settings, so I wouldn't pay too much attention to the fact that your machine does that and some other ones don't.