Logging of all traffic - No Blocking

RouterOS General
1

Hi,

I wondered what the best option would be for logging and generating reports on all internet traffic not just web traffic from within an office. I do not need to block any traffic but is purely for reporting.

Some Information I would like to capture would be:-

  • HTTP / HTTPS domains visited and by who
  • Protocol usage
  • The ability to view Torrent traffic especially to see who and when

I’ve tried playing with ntopng using traffic-flow however it seems a little inaccurate.

What options are recommended? If external server is required for collecting information I would rather it Linux based than windows.

2

traffic-flow is the way to go. what is inaccurate about it?
of course you could also setup a port mirroring or packet sniff streaming and send all traffic to an external computer doing the work.

3

Ok, so Mikrotik is set up as follows:-

/ip traffic-flow
set active-flow-timeout=1m enabled=yes
/ip traffic-flow target
add dst-address=10.1.1.24

Ntopng is installed as a VM via KVM and nprobe is set up and ntopng run as follows:-

nprobe -i none -n none -3 2055 --zmq tcp://127.0.0.1:1234 &
ntopng --local-networks="10.1.1.0/24" -i tcp://127.0.0.1:1234 &

So lets say I start a download of a 512MB file with wget on a laptop:-

wget http://ipv4.download.thinkbroadband.com/512MB.zip
--2016-10-07 22:20:36--  http://ipv4.download.thinkbroadband.com/512MB.zip
Resolving ipv4.download.thinkbroadband.com... 80.249.99.148
Connecting to ipv4.download.thinkbroadband.com|80.249.99.148|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 536870912 (512M) [application/zip]
Saving to: ‘512MB.zip.1’

512MB.zip.1                                       100%[===========================================================================================================>] 512.00M  16.3MB/s    in 35s

2016-10-07 22:21:11 (14.8 MB/s) - ‘512MB.zip.1’ saved [536870912/536870912]

As you can see it did “16.3MB/s in 35s”. I then search for the host in ntopng, give it 10 minutes or so and the traffic counters for the host have barely registered anything, maybe a few MB. Also the speedo at the bottom that shows the overall speed fluctuates a lot but generally stays below 200 Kbps even when downloading at 16MB/s.

I must be missing something?

As for setting up a port mirror, the server I’m using for ntopng is a KVM VM so that would be quite hard.

4

Ok, so it appears that uploads from the client are working fine but downloads are not and the percentage between sent and received it like 98%/2% in favour of sent.

Could something like fasttrack be the issue? Could traffic incoming from the internet be skipping the traffic-flow capture?

5

In old versions, yes. Did you update RouterOS?

6

Yes on 6.37.1

7

Has anybody got any thought on why only one part of the traffic is not being sent to ntopng? Any parts of my configuration that would be useful?

8

Simple try to use netflow v5 in targets options.

9

That seems better. Some devices are showing more accurately. Some others I’m not so sure but I’ll keep an eye on it. Thanks