Netflow (IPFIX) issue

RouterOS General
1

I am using Netflow (IPFIX) on two different routers that both use IPv4 and IPv6.
I configured the Netflow interface to be the internet interface.

On a CCR1009 it works OK, I get all netflow information.
On a RB750Gr3 I am seeing strange behaviour: I do get both the inbound and outbound flows for IPv4, but
for IPv6 I mostly get only the outbound flows, and an occasional inbound flow but most of them are missing.
(e.g. when a https connect is made I do get the flow with DstPort 443, but not the corresponding flow with
SrcPort 443 and the addresses swapped)
Maybe 1% of the flows are still reported, though. It is just that the majority is missing so bytes downloaded
per users is very small.

Anyone seen that before? It happened with 6.46.2 but that was when I configured the Netflow, and recently
I updated to 6.47 and the issue has remained the same (I sort of hoped the reboot would fix it)

2

Meanwhile I found this is not an issue in Netflow but there is something wrong with the IPv6 routing at that location.

3

Slightly offtopic but which Netflow platform do you use ? Something commercial or opensource ?
I’ve been playing some days ago with Netflow/IPFIX on my RB3011.

4

I use a Perl script that uses the Net::Flow library (it is mostly copied from an example provided with that library) to write the flow info to a tab-separated file rotated each day.
Then I wrote another Perl script that adds-up the byte counts grouped per user MAC to see where exceptionally high usage comes from when the line is overloaded.
It is mainly intended as a log in case of abuse or piracy, not to make nice graphs. Normally nobody even looks at those logs until there is some issue.

5

This is my script:

#!/usr/bin/perl -w
# decode NetFlow (IPFIX) info

use strict;
use warnings;

use POSIX 'strftime';
use Net::Flow qw(decode);
use Net::Flow::Constants qw(%informationElementsByName %informationElementsById);
use IO::Socket::INET;

my $receive_port = 4739;		# IPFIX port
my $packet;
my %TemplateArrayRefs;
my %PrintedTemplate;
my $sock = IO::Socket::INET->new(LocalPort => $receive_port,Proto=>'udp');

my $sender;

$SIG{INT} = sub { exit; };

while ($sender = $sock->recv($packet, 0xFFFF)) {
    my ($sender_port, $sender_addr) = unpack_sockaddr_in($sender);
    $sender_addr = inet_ntoa($sender_addr);

    my ($HeaderHashRef, $FlowArrayRef, $ErrorsArrayRef) = ();

    # template ids are per src, destination, and observation domain.
    # Ideally the module will handle this, but the current API doesn't
    # really allow for this.  For now you are on your own.
    my ($version, $observationDomainId, $sourceId) = unpack('nx10N2', $packet);
    my $stream_id;
    if ($version == 9) {
	$stream_id = "$sender_port,$sender_addr,$sourceId";
    } else {
	$stream_id = "$sender_port,$sender_addr,$observationDomainId";
    }
    $TemplateArrayRefs{$stream_id} ||= [];
    my $TemplateArrayRef = $TemplateArrayRefs{$stream_id};

    ($HeaderHashRef, $TemplateArrayRef, $FlowArrayRef, $ErrorsArrayRef) = Net::Flow::decode(\$packet, $TemplateArrayRef);

    $TemplateArrayRefs{$stream_id} = $TemplateArrayRef if defined $TemplateArrayRef;

    if (@{$ErrorsArrayRef}) {
	grep { print STDERR "$_\n" } @{$ErrorsArrayRef};
	next;
    }

    my $unixsecs = $HeaderHashRef->{'UnixSecs'};
    my $timestamp = strftime('%F %T',localtime($unixsecs));

    foreach my $FlowRef (@{$FlowArrayRef}) {
	if (!defined($PrintedTemplate{$FlowRef->{'SetId'}})) {
	    print "Template\t",$FlowRef->{'SetId'};

	    foreach my $Id (sort keys %{$FlowRef}) {
		next if ($Id eq 'SetId');
		print "\t";
		my $name = $informationElementsById{$Id}->{name} // "$Id";
		print $name;
	    }

	    print "\n";
	    $PrintedTemplate{$FlowRef->{'SetId'}} = 1;
	}

	print $timestamp,"\t",$FlowRef->{'SetId'};

	foreach my $Id (sort keys %{$FlowRef}) {
	    next if ($Id eq 'SetId');

	    print "\t";
#	      my $name = $informationElementsById{$Id}->{name} // "$Id";
#	      printf '%s=', $name;
	    my $type = $informationElementsById{$Id}->{dataType} // "";
	    my $value = $FlowRef->{$Id};

	    if (ref $value) {
		foreach my $val (@{ $value }) {
		    printf '%s,', unpack('H*', $val);
		}
	    } else {
		if ($type =~ /^unsigned/) {
		    printf '%u', hex(unpack('H*', $value));
		} elsif ($type =~ /^signed/) {
		    printf '%d', hex(unpack('H*', $value));
		} elsif ($type eq "ipv4Address") {
		    print join '.',unpack 'C4',$value;
		} elsif ($type eq "ipv6Address") {
		    my @unp = unpack 'n8',$value;
		    my $n = 0;
		    foreach my $word (@unp) {
			printf "%x", $word;
			print ":" if ($n++ < 7);
		    }
		} elsif ($type eq "macAddress") {
		    my @unp = unpack 'C8',$value;
		    my $n = 0;
		    foreach my $word (@unp) {
			printf "%02x", $word;
			print ":" if ($n++ < 5);
		    }
		} else {
		    printf '%s.%s', $type, unpack('H*', $value);
		}
	    }
	}
	print "\n";
    }
}

1;
6

Interesting! Thanks for sharing. I will give it a try here on my environment.

7

Ok it would be started from this sh script:

#!/bin/sh
# rotate the netflow log once a day and compress logfiles

base="router"
now=`date +%Y%m%d`

cd /var/log/netflow || exit 1

# remove logfiles older than 180 days

find . -mtime +180 -exec rm -f {} \;

# restart daemon writing to logfile for today

killall -s 2 netflow.pl
sleep 2
umask 077
/usr/local/bin/netflow.pl >>$base-$now 2>>netflow.out </dev/null &
sleep 2

# compress logfiles before today

for file in $base-????????
do
    if [ "$file" != "$base-$now" ]
    then
	bzip2 -9 $file
    fi
done

This is run at startup of the machine and at 00:00 every day. It writes in the directory /var/log/netflow that should be writable for the user that runs the script.
(does not need to be root)
It is normal that some errors “WARNING : NOT FOUND TEMPLATE=258” are written to netflow.out, this is because the netflow template is inserted every so many records, so records received before this template comes by cannot be processed. This is e.g. set to once every 20 records / 1800 seconds.