Community discussions

MikroTik App
 
anserk
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 51
Joined: Wed Mar 02, 2022 5:08 pm

Bandwidth usage per IP

Fri May 13, 2022 1:14 am

Recently I started to get close to reaching my ISP data cap, so I wanted to track monthly bandwidth usage per each device behind my home router hAP ac2. I searched the forum and saw several ideas, but they weren't quite what I was looking for. So I wrote my own script that I would like to share with the community.

Key features
  • Everything is local on the router. You don't need any external servers or software to collect the data. You only need a computer to view the report.
  • The script uses kid-control functionality in RouterOS, which has a nice bonus - tracking per IP bandwidth.
  • I'm using it with a USB flash drive where I don't worry about write cycles. If your device doesn't support external storage, you could schedule the script to run less frequently (at the expense of more collected data loss due to unexpected reboots).

Limitations
  • Tested with RouterOS 7 only.
  • Kid-control counts all inter-VLAN traffic, not just WAN. Potentially you could re-write the script and use something like viewtopic.php?t=168427 instead of kid-control.
  • The HTML report (placed in the root of flash drive) has .txt extension - this is RouterOS limitation. You need to download the report from WinBox Files window and then remove .txt so that only .html is left. You could set up SFTP or FTP and use the fetch tool to rename it or upload/rename to another location on your network. There are some discussions about using the branding package for serving custom static web pages directly from the router. I haven't looked into that.
  • RouterOS supports only integers, so report accuracy is +/-1GB. If you need more precision, you can modify the script to report in megabytes. Note, all calculation and saving is done in bytes, so there are no repeated rounding losses. The rounding (or rather truncating) happens only at the last step when the report is generated.

How to use
  1. Enable bandwidth tracking in kid-control. With configuration below kid-control doesn't limit anything but starts tracking traffic.
    /ip kid-control add name=Monitor mon=0s-1d tue=0s-1d wed=0s-1d thu=0s-1d fri=0s-1d sat=0s-1d sun=0s-1d
  2. Install and format USB drive.
  3. Create a directory called "bandwidth" on it (the easiest way is to create a folder on your PC and drag into Winbox Files window). You can change the paths as needed by changing the script variables.
  4. Schedule the script to run every hour (or however often you like) at x:59:59. On the last day of the month at 23:59:59 the script will generate the final report for the month and won't touch it again. The next scheduled run will use the new month name for data files and the report.

Here is how the report looks like:

report.png
You do not have the required permissions to view the files attached to this post.
Last edited by anserk on Fri Aug 12, 2022 10:56 pm, edited 3 times in total.
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 2989
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: Bandwidth usage per IP

Fri May 13, 2022 4:11 am

very interesting, thank you for sharing :idea:
 
User avatar
Jotne
Forum Guru
Forum Guru
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Bandwidth usage per IP

Mon May 16, 2022 8:22 am

5. How to read report?
can you do? http://your.router.ip/xxxx
Last edited by Jotne on Tue Jan 03, 2023 7:29 pm, edited 1 time in total.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Bandwidth usage per IP

Mon May 16, 2022 10:36 am

I have already seen that line on Splunk...
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18959
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Bandwidth usage per IP

Mon May 16, 2022 3:00 pm

Should be built into the router as a selectable function!!
 
anserk
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 51
Joined: Wed Mar 02, 2022 5:08 pm

Re: Bandwidth usage per IP

Mon May 16, 2022 6:23 pm

You read the report by downloading it to your computer and removing .txt, so that the resulting file is report-2022may.html (for May). Then you can open it in any browser. It's a basic HTML with JavaScript leveraging Google Charts.

You are correct, the script doesn't do any magic and fully relies on kid-control to provide the data. The main point is storing the data persistently without anything external. And also presenting the data in a friendly format. It is especially useful for users who don't have any servers/Raspberry PI's/NAS/etc running somewhere to collect the data. It's very simple to set up, just copy and paste the script.

Ideally, if RouterOS had the ability to serve custom static web pages, then it would be possible. Like I mentioned above, the branding package might do this but I haven't looked into that.
 
holvoetn
Forum Guru
Forum Guru
Posts: 5317
Joined: Tue Apr 13, 2021 2:14 am
Location: Belgium

Re: Bandwidth usage per IP

Sat Jul 23, 2022 11:52 am

@anserk,
just a little shout out that I am using your script on an SXT device used for internet access in a vacation home setup.

MUCH prettier then simply mailing output from kid-control devices print stats :lol:
 
User avatar
bpwl
Forum Guru
Forum Guru
Posts: 2978
Joined: Mon Apr 08, 2019 1:16 am

Re: Bandwidth usage per IP

Sat Jul 23, 2022 2:39 pm

VERY interesting feature of kid-control. Replacing "IP accounting" ?
TXS for the script!

HOW to get the authenticated user (RADIUS) into this Kid-Control device list ?

(Wifi AP authenticated via WPA2/Enterprise, Username value can be copied into Radius "MT Comment" attribute with User Manager 5, and remains visible in the Registration table, as long as the device is connected. DUDE collects in "RouterOS Info" all registration tables, including the comment field)

The username/password is stored in the client device, and is never entered again for the same SSID. So this should be acting as automated RSSO (Radius single signon).
But how to make Kid-control or Hotspot pick up this signon ?
 
anserk
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 51
Joined: Wed Mar 02, 2022 5:08 pm

Re: Bandwidth usage per IP

Sat Jul 23, 2022 9:45 pm

If you downloaded the script before July 23 2022, please update line 35 from:
  :local ip [get $device ip-address]
# To
  :local ip [:tostr [get $device ip-address]]

I recently ran into issue with traffic counter overflow on the report, messing up all statistics. It turns out when a device has multiple IP addresses, the script was grabbing wrong fields for traffic counters. The issue is resolved by forcing the whole IP field to a string.

@bpwl I've never used RADIUS - wouldn't that authenticated user device get an IP from DHCP server? I don't know exactly how kid-control works under the hood, it seems to simply pick up devices by MAC address. Presumably anything that goes through the firewall, which is why it also counts traffic from one local VLAN to another. So as long as that user sends some traffic, it should be on the list. The difficult part is identifying who is who. If the device doesn't register a name with DHCP server, then only MAC and IP are shown. And with many phones (like Apple) periodically changing their MAC address, it becomes impossible to identify all devices accurately. So kid-control is quite a limited feature, but works well in simple configurations with mostly fixed number of devices.
 
User avatar
bpwl
Forum Guru
Forum Guru
Posts: 2978
Joined: Mon Apr 08, 2019 1:16 am

Re: Bandwidth usage per IP

Sun Jul 24, 2022 12:29 am

The difficult part is identifying who is who.

That's why I use RADIUS with EAP/PEAP wifi authentication (= username/password).
The data is stored once in the device (until you 'forget" the wifi network in the known wifi network list.)

Limits are then per user. Hotspot login does the same but with many APP as primary use, the browser popup for logging in is sometimes missed. And Hotspot cookies are just a means for avoiding repeated login actions for the same MAC.

Whatever they do with the private MAC addresses, the user identification with EAP/PEAP is dependent on the stored login credentials only.
RADIUS accounting would collect the usage per user, but that is mostly all (wifi) traffic, and the interesting part is the WAN usage only.
RADIUS could give a specific range of IP addresses to one user. But when you have multiple network segments (eg VLAN) then it becomes more difficult.
RADIUS could also force a specific VLAN for a user.
RADIUS can be used for VPN, DHCP, Hotspot, wifi, and other services. But afaik only the wifi is persistent. (The device connects with that user without any interaction of the user, even when not around for more than a year, and when it comes with a different private MAC address.)
 
holvoetn
Forum Guru
Forum Guru
Posts: 5317
Joined: Tue Apr 13, 2021 2:14 am
Location: Belgium

Re: Bandwidth usage per IP

Sun Jul 24, 2022 12:32 am

I would think if the same sort of info can be retrieved from the Radius part (incl. user name), the concept of the script can still be used.
 
User avatar
bpwl
Forum Guru
Forum Guru
Posts: 2978
Joined: Mon Apr 08, 2019 1:16 am

Re: Bandwidth usage per IP

Mon Jul 25, 2022 11:53 am

RADIUS itself is a bid special in this. Is some device authenticated to what user or IP address? Is not easy to tell.

RADIUS authentication is an event, that in the best case gets logged, username and MAC address of the AP. (not the client), in my case.
De-authentication in de most cases does not involve the RADIUS server. (There is CoA, but mostlly the wifi disconnects, "reason=3" by AP, "reason=8" by roaming client, extensive data loss (client is gone or out of range?)) So the RADIUS authenticator cannot tell if a user is still authenticated or not, and does not have the IP address of the client.
The AP maintains the registration (on MAC address) but does not register the user used for the authentication.

The way RADIUS knows about the disconnect is through the RADIUS accounting. (User name , IP address, MAC address, quota consumed). But RADIUS misses sometimes an accounting record.
Actually I'm using the old MT User Manager for collecting RADIUS accounting, even with another RADIUS server doing the authentication.
Reason is that MT limits the number of accounted sessions to 19, for a L4 license. Accounting updates is set to someting like every 10 minutes.
RADIUS authentication is quite central, for multiple AP, so the number of active sessions is likely above 19. Even L5 only allows for 50 sessions.
I have 10.000 RADIUS authentications per day, caused by 40 named users, 100 client devices on 34 MT AP. There is a lot of roaming in that large 1 km2 area, also between 2.4 and 5GHz on the same AP. (Playing kids, indoor/outdoor swap, ...)
Roaming can be every few seconds, and takes a split second. But steady clients their connections remain active for multiple days.
DHCP lease in this does not interfere (roaming does not renew the lease). Lease time is 24h, most client devices keep the same IP for the whole holiday stay period.

All data is there, but not correlated. DUDE collects the logs from (non-MT) RADIUS server, (non-MT) DHCP server, and MT AP's.
MT User manager only can follow 19 Active sessions with license level L4, via RADIUS accounting.
 
anserk
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 51
Joined: Wed Mar 02, 2022 5:08 pm

Re: Bandwidth usage per IP

Fri Aug 12, 2022 11:00 pm

I encountered another bug with the script. One device that showed up in kid-control didn't have an IP address. That empty field was screwing up the report. I added the fix to the attached script above. It's just one new line.
  :if ([:len $name] = 0) do={ :set $name "unknown" }
  :local ip [:tostr [get $device ip-address]]
  :if ([:len $ip] = 0) do={ :set $ip "no-ip" }   <== new line
  :local down [get $device bytes-down]
  :local up [get $device bytes-up]
 
Dt17
just joined
Posts: 4
Joined: Thu Oct 10, 2019 10:00 am

Re: Bandwidth usage per IP

Tue Jan 03, 2023 8:54 am

Hi, your script works great and i really appreciate this tool that you provided

I seemed to have messed up some of the older reports by moving them before the last day of the month

Is there anyway to generate the report with all the older raw data? (in Txt format and also still viewable in Kid Control)

i am not familiar with Google Charts
 
anserk
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 51
Joined: Wed Mar 02, 2022 5:08 pm

Re: Bandwidth usage per IP

Wed Jan 04, 2023 10:48 pm

Is there anyway to generate the report with all the older raw data? (in Txt format and also still viewable in Kid Control)
The script uses the current date to produce the txt filenames, which are then used for generating the report. You can make a copy of the script and edit the line:
:local sysdate [/system/clock/get date]
Change it to a static date for older month. The format is jan/04/2023.

As far as Kid Control goes, that data gets reset every time the script runs. So what you see there is only new data since the last run.
 
Dt17
just joined
Posts: 4
Joined: Thu Oct 10, 2019 10:00 am

Re: Bandwidth usage per IP

Thu Jan 05, 2023 9:11 am

Thank you, Appreciate your expertise and assistance!
 
Rox169
Member
Member
Posts: 432
Joined: Sat Sep 04, 2021 1:47 am

Re: Bandwidth usage per IP

Thu Jan 05, 2023 10:11 am

MIKROTIK PLEASE PLEASE INTEGTRATE THIS!!!!!!!! OR GIVE AS BACK IP ACCOUNTING!!!!!!! PLEASE PLEASE
 
Dt17
just joined
Posts: 4
Joined: Thu Oct 10, 2019 10:00 am

Re: Bandwidth usage per IP

Mon Feb 06, 2023 9:46 am

So this works for me out of 3 of 4 Mikrotik(2 x RB3011 and 2 x RB1100AHx4) devices that i currently run them

The 4th one(RB1100AHx4) keeps giving me this error for the last 3 months

The only result generated in the html is below

Does anyone has any idea what is the error? All the boxes were running ROS 7.6 and now ROS 7.7
<html><head>
<script type='text/javascript' src='https://www.gstatic.com/charts/loader.js'></script>
<script type='text/javascript'>
google.charts.load('current', {packages: ['corechart', 'bar']});
google.charts.setOnLoadCallback(drawMultSeries);

function drawMultSeries() {
var data = google.visualization.arrayToDataTable([
['Device', 'Download', 'Upload', 'Total'],
 
failure: new contents too long
 
anserk
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 51
Joined: Wed Mar 02, 2022 5:08 pm

Re: Bandwidth usage per IP

Thu Feb 16, 2023 8:51 pm

That error
failure: new contents too long
seems to imply you are running out of space somewhere.
 
sebus46
newbie
Posts: 31
Joined: Sat Jun 17, 2023 4:59 pm

Re: Bandwidth usage per IP

Thu Jun 22, 2023 8:27 pm

Pity RB4011 does not have USB...
 
User avatar
Jotne
Forum Guru
Forum Guru
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Bandwidth usage per IP

Thu Jun 22, 2023 9:33 pm

Without USB you can send information to an external server using Syslog
 
bingtonewali
just joined
Posts: 1
Joined: Thu Jul 27, 2023 10:44 pm

Re: Bandwidth usage per IP

Fri Jul 28, 2023 9:40 am

i tried to run on SXTsq Lite5 AC [6.48.6 (long-term) ]. i tried but i failed to setup this. I don't where i am getting stuck. No report generated. Can you please provide a video tutorial how to set it up?
1st i do turn on kid control through you give the code [ /ip kid-control add name=Monitor mon=0s-1d tue=0s-1d wed=0s-1d thu=0s-1d fri=0s-1d sat=0s-1d sun=0s-1d ]
and then
2nd i go to script and then paste the whole code and run interval of 00:59:59 .

nothing getting output. i change the directory according to me. i don't have external storage so i create a folder on Mikrotik and then i choose the path.


Troubleshoot by me:
I tried to run in scheduler not in script nothing happen
i tried to paste the whole code in script and then run it through schulder.
i tried to change the interval runtime to 10 sec to 1 min and 5min nothing results.


Please make a video tutorial and the installation. Please. I need this very badly to track my own internet ussage without any external thing.
 
User avatar
Jotne
Forum Guru
Forum Guru
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Bandwidth usage per IP

Fri Jul 28, 2023 12:25 pm

You can follow the steps to setup monitoring here:

viewtopic.php?t=179960
 
cmassey
just joined
Posts: 20
Joined: Fri Nov 08, 2019 8:06 am

Re: Bandwidth usage per IP

Mon Aug 07, 2023 9:26 pm

Kid-control doesn't seem to give accurate bandwidth usage information. I'm using 7.10 on an hap ac2 as a test. When I run a speedtest on my computer, I get about 100Mbps down and about 45Mbps up. Kid control is showing about 30Mbps down and kbps up.

Is anyone else having the same issue? Does it seem to work with applications other than speedtest? It might just be showing one of the speedtest streams which would be a little unlucky.

I am currently using torch which gives accurate information but it would be nice to use kidcontrol instead so I don't need to add up all the connections for the same IPs.

Who is online

Users browsing this forum: No registered users and 67 guests