NAT logging with netflow9

RouterOS General
1

Hello guys!

Since of 6.33 release of ROS it’s possible to track full NAT translations history via Netflow protocol version 9. In most countries providers must keep track users access log to goverment agency. It was hard to do it with NAT usually.

Did someone setup configuration with NAT logging using opensource collector software?

My infestigation is the folowing:

  1. Mikrotik itself is providing nessesarry information according to Netflow V9 standart.

  2. I run some modern netflow collector code writen on node.js: http://deliantech.blogspot.com/2014/06/netflow-version-9-library-for-nodejs.html and got full information from Mikrotik flow. But it’s not usable in production because of text output and lack of administrative features.

  3. I run nfcapd/nfdump for years: http://nfdump.sourceforge.net/ This software has fine toolset, but unfortunately has lack of support of Netflow9. You can only compile some predifine fixed templates used by Cisco devices. I think code is not designed for flexible template support.

  4. I found very powerfool set of netflow tools called SiLK: https://tools.netsa.cert.org/silk/ but it’s very complecated. You need probably a lot of time only to install and configure it. May be someone already have tested it with Mikrotik?

Please share your experience!

2

By the way, here is example of the netflowjs output from mikrotik version9 flow:

{ header:
   { version: 9,
     count: 20,
     uptime: 387036340,
     seconds: 1448913629,
     sequence: 50976101,
     sourceId: 0 },
  flows:
   [ { last_switched: 387021220,
       first_switched: 387021220,
       in_pkts: 3,
       in_bytes: 156,
       input_snmp: 19,
       output_snmp: 18,
       ipv4_src_addr: '46.33.68.47',
       ipv4_dst_addr: '7.777.176.64',
       protocol: 6,
       src_tos: 0,
       l4_src_port: 80,
       l4_dst_port: 18477,
       ipv4_next_hop: '192.168.9.10',
       dst_mask: 0,
       src_mask: 0,
       tcp_flags: 17,
       in_dst_mac: '4c5e0c242dc5',
       out_src_mac: '4c5e0c242dc3',
       postNATSourceIPv4Address: '46.33.68.47',
       postNATDestinationIPv4Address: '172.16.26.78',
       postNAPTSourceTransportPort: 80,
       postNAPTDestinationTransportPort: 18477,
       fsId: 256 },
     { ...
3

I’ve heard of SiLK and currently use a great tool that is compatible with it: flow-viewer

Time to look into some SiLK :sunglasses:

4

I’m willing to share some NodeJs code I made today, it uses the library that you linked. This is my first ever Nodejs code..
It simply logs all NAT translations into a mysql table with schema:

CREATE TABLE `nat_translations` (
  `unixTime` int(10) unsigned NOT NULL,
  `lanSrcAddr` int(10) unsigned NOT NULL,
  `lanSrcPort` smallint(5) unsigned NOT NULL,
  `postNatSrcAddr` int(10) unsigned NOT NULL,
  `postNatSrcPort` smallint(5) unsigned NOT NULL,
  `dstAddr` int(10) unsigned NOT NULL,
  `dstPort` smallint(5) unsigned NOT NULL,
  UNIQUE KEY `7tuple` (`unixTime`,`lanSrcAddr`,`lanSrcPort`,`postNatSrcAddr`,`postNatSrcPort`,`dstAddr`,`dstPort`)
);

IP addresses are stored as ints to save space.

Made a simple web form to go along with this, and can now look up NAT translations. Time to NAT EVERYTHING!! jk.

Code: http://pastebin.com/ryfn4Msu

5

Hi, it seems that NFDUMP works as well. You just need to compile it with NSEL/NEL support (–enable-nsel configure option). Tested with the latest version 1.6.15 (https://github.com/phaag/nfdump/releases/tag/v1.6.15) and ROS 6.34.6:

nfdump -o nel ...
2016-10-21 08:04:28.500 INVALID TCP       10.0.247.x:57974 ->     52.85.x.x:443       85.207.x.x:57974 ->     52.85.x.x:443




nfdump -o raw ...
Flow Record: 
  Flags        =              0x06 FLOW, Unsampled
  export sysid =                 1
  size         =                76
  first        =        1477037068 [2016-10-21 08:04:28]
  last         =        1477037068 [2016-10-21 08:04:28]
  msec_first   =               500
  msec_last    =               500
  src addr     =      10.0.247.x
  dst addr     =      52.85.x.x
  src port     =             57974
  dst port     =               443
  fwd status   =                 0
  tcp flags    =              0x10 .A....
  proto        =                 6 TCP  
  (src)tos     =                 0
  (in)packets  =                 2
  (in)bytes    =                82
  input        =              2538
  output       =                 3
  src xlt port =             57974
  dst xlt port =               443
  src xlt ip   =      85.207.x.x
  dst xlt ip   =      52.85.x.x