I’m collecting Netflow data using Fluentd and storing it as Parquet files on AWS S3. I’m then using Athena to query these files using SQL and Grafana to visualise the data.
I’ve tried using Netflow v5, v9 and IPFIX. When I use v5 I can get the timestamps easily:
SELECT flow_seq_num, from_iso8601_timestamp(first_switched) as startTime, from_iso8601_timestamp(last_switched) as endTime FROM "netflow"."netflow5_event";
Example output:
flow_seq_num: 783160
startTime: 2022-04-27 04:59:51.735 UTC
endTime: 2022-04-27 04:59:51.735 UTC
Netflow 9 and IPFIX don’t store timestamps - I’ve read that it shows the time in milliseconds since system boot, but the numbers don’t look right and systeminittimemilliseconds is 0 for most entries with IPFIX. I’ve also tried exporting all three at the same time, hoping I could get the timestamp from v5 and use the rest of the data from v9 or IPFIX, but the flows don’t match (I presume RouterOS is processing them separately, not just exporting 3 different outputs for the same data set).
v5 is useful today, but it doesn’t support IPv6, so I’d rather use v9 or IPFIX as we’re planning on implementing IPv6 soon. Is there any way to get the timing of the flows from v9 or IPFIX?
All devices will be running RouterOS v7, as I need the REST API for other aspects of this project.