Community discussions

MikroTik App
 
rajeshrouthu
newbie
Topic Author
Posts: 46
Joined: Sun Jun 22, 2008 10:30 am

MIkrotik Syslog New Format

Wed Jun 30, 2021 5:06 pm

Hi Guys,

Mikrotik default user syslog format i am getting as below it was fine but our telecom department request different format

Current Mikrotik Syslog

"src-mac c8:1f:66:16:eb:99, proto TCP (SYN), 192.168.88.90:61496->202.133.XX.YY:80, len 52"

Telecom Department Requested Format

"src=172.16.189.56 dst=34.117.60.153 sport=58712 dport=80 src=34.117.60.153 dst=175.101.1XY.2Z sport=80 dport=58712 [ASSURED] delta-time=41 [start=Fri Apr 30 20:07:16 2021] [stop=Fri Apr 30 20:07:57 2021]"


how can we get the session start time and stop time , any body can please help in this issue.

Thanks in Advance

@
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11982
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: MIkrotik Syslog New Format

Wed Jun 30, 2021 6:06 pm

2 time src/dst/sport/dport?
src=172.16.189.56 dst=34.117.60.153 sport=58712 dport=80
src=34.117.60.153 dst=175.101.1XY.2Z sport=80 dport=58712
[ASSURED] delta-time=41 [start=Fri Apr 30 20:07:16 2021] [stop=Fri Apr 30 20:07:57 2021]

You also like delta-time with milk or not?


How can RouterBOARD say when is stopped?
If connection is mantained, for example, 1 hour, how can go back to the past to correct the log entry,
or how can predict the future in the past?


You MIX syslog with sessions, like your "Telecom Department" as usual he doesn't know what he's talking about,
just someone who gets up in the morning and asks 'I want it like this', without thinking about what he is asking.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 19125
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: MIkrotik Syslog New Format

Wed Jun 30, 2021 6:23 pm

Well that may be, obviously they are not getting up in the morning and having a proper cup of Italian cafe.
On the other hand, the TELECOM portion of the company should state what the requirement is in clear terms that tney require information and for what purposes.
One cannot change router software but if its too populate a telecom excel spreadsheet, then we know who has to make the changes to accommodate!!!
 
rajeshrouthu
newbie
Topic Author
Posts: 46
Joined: Sun Jun 22, 2008 10:30 am

Re: MIkrotik Syslog New Format

Thu Jul 01, 2021 2:17 pm

Your 100 % Correct, but they don't listen our version. they need only the below format .

Image


2 time src/dst/sport/dport?
src=172.16.189.56 dst=34.117.60.153 sport=58712 dport=80
src=34.117.60.153 dst=175.101.1XY.2Z sport=80 dport=58712
[ASSURED] delta-time=41 [start=Fri Apr 30 20:07:16 2021] [stop=Fri Apr 30 20:07:57 2021]

You also like delta-time with milk or not?


How can RouterBOARD say when is stopped?
If connection is mantained, for example, 1 hour, how can go back to the past to correct the log entry,
or how can predict the future in the past?


You MIX syslog with sessions, like your "Telecom Department" as usual he doesn't know what he's talking about,
just someone who gets up in the morning and asks 'I want it like this', without thinking about what he is asking.

Image
You do not have the required permissions to view the files attached to this post.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11982
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: MIkrotik Syslog New Format

Thu Jul 01, 2021 2:40 pm

Is clear who write the guide do not know anyting about how the things works,
the line "mm:dd:yyyy" confirm that.


Your example also differ on what requested, where is [ASSURED] and "delta-time"?

Following the guide the log must be like:
Jul:01:202113:43:22,Jul:01:202113:43:23,1.2.3.4,5678,11.22.33.44,5678,159.148.147.239,443
Last edited by rextended on Thu Jul 01, 2021 2:45 pm, edited 3 times in total.
 
User avatar
Paternot
Forum Veteran
Forum Veteran
Posts: 953
Joined: Thu Jun 02, 2016 4:01 am
Location: Niterói / Brazil

Re: MIkrotik Syslog New Format

Thu Jul 01, 2021 2:43 pm

They may want it as much as they like: it can't be done properly.

1) TCP connections may be closed by the client, the server or by timeout. Some bad written software don't close the connection at the end - it just expires. The best that could be done would be to log the timeout - but it wouldn't tell them the duration of the event.
2) UDP connections doesn't exist: UDP doesn't do connections. Sure, the software using UDP may implement the connection layer - quite common with VPNs - but to the firewall/router... nope.

They are thinking about traditional telephony, where a point to point connection is made before the telephone rings. Network doesn't work this way. What they want doesn't make sense.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11982
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: MIkrotik Syslog New Format

Thu Jul 01, 2021 2:46 pm

...They are thinking about traditional telephony... Network doesn't work this way.... What they want doesn't make sense...
again: who write the guide do not know anyting about how the things works
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11982
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: MIkrotik Syslog New Format

Thu Jul 01, 2021 2:49 pm

Simple solution, stop doing NAT and provide qualty service to your client assing one Public IP for eachone.
The log is simply: each client has his own IP.
 
rajeshrouthu
newbie
Topic Author
Posts: 46
Joined: Sun Jun 22, 2008 10:30 am

Re: MIkrotik Syslog New Format

Thu Jul 01, 2021 3:28 pm

Simple solution, stop doing NAT and provide qualty service to your client assing one Public IP for eachone.
The log is simply: each client has his own IP.
that should be possible only when we implement IPV6.:-)
 
rajeshrouthu
newbie
Topic Author
Posts: 46
Joined: Sun Jun 22, 2008 10:30 am

Re: MIkrotik Syslog New Format

Thu Jul 01, 2021 3:31 pm

Is clear who write the guide do not know anyting about how the things works,
the line "mm:dd:yyyy" confirm that.


Your example also differ on what requested, where is [ASSURED] and "delta-time"?

Following the guide the log must be like:
Jul:01:202113:43:22,Jul:01:202113:43:23,1.2.3.4,5678,11.22.33.44,5678,159.148.147.239,443
if i get the log below format it will be fine
Jul:01:202113:43:22,Jul:01:202113:43:23,1.2.3.4,5678,11.22.33.44,5678,159.148.147.239,443
The delta time is negligible , those logs are taken from different server not in mikroitk.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11982
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: MIkrotik Syslog New Format

Thu Jul 01, 2021 3:44 pm

MikroTik do not change "log" format for two reason:
1) what asked require more CPU and RAM (also suggested from other users) for memorize "session time",
2) no reason to change format.

You can't always ask to change things for the needs of one, unless you get a custom version of RouterOS (ask for a quote at sales@mikrotik.com).

Make your own database of syslog (https://www.graylog.org/ for example),
elaborate separately the data,
then send on required format.

And I suggest https://www.graylog.org because I use it.
Last edited by rextended on Thu Jul 01, 2021 3:51 pm, edited 1 time in total.
 
andriys
Forum Guru
Forum Guru
Posts: 1526
Joined: Thu Nov 24, 2011 1:59 pm
Location: Kharkiv, Ukraine

Re: MIkrotik Syslog New Format

Thu Jul 01, 2021 3:48 pm

What you need is a NetFlow collector. You setup the collector, then configure you router to export the traffic flow information (see the Traffic Flow manual page). Once your traffic data is collected you can export it in whatever format you want.

However, please note that:
(a) you cannot run a NetFlow collector on the router itself, it will have to be installed on a separate computer or VM; and
(b) a decent/suitable NetFlow collector will cost some money, or require some amount of coding/programming, or both.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11982
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: MIkrotik Syslog New Format

Thu Jul 01, 2021 3:56 pm

Still remain the problem of "session" time, netflow do not elaborate it directly on one line of log, but must be calculated on remote server.
 
pe1chl
Forum Guru
Forum Guru
Posts: 10197
Joined: Mon Jun 08, 2015 12:09 pm

Re: MIkrotik Syslog New Format

Thu Jul 01, 2021 4:24 pm

What you need is a NetFlow collector. You setup the collector, then configure you router to export the traffic flow information (see the Traffic Flow manual page). Once your traffic data is collected you can export it in whatever format you want.

However, please note that:
(a) you cannot run a NetFlow collector on the router itself, it will have to be installed on a separate computer or VM; and
(b) a decent/suitable NetFlow collector will cost some money, or require some amount of coding/programming, or both.
For this purpose I wrote a very simple netflow collector in Perl using the Net::Flow library available from CPAN.
It is merely receiving the netflow data and writing it in a tab-separated file (good enough for me), started from the example provided with Net::Flow. I posted it on the forum some time ago.
However it would be easy to write in any format and even to syslog using this method.

But you are right, it cannot be done on the router itself, it requires a separate system.
(if only we had the possibility to install user programs running on a restricted user but able to have network sockets, similar to what MetaROUTER used to do but not an entire VM, just a user process...)
 
mada3k
Long time Member
Long time Member
Posts: 687
Joined: Mon Jul 13, 2015 10:53 am
Location: Sweden

Re: MIkrotik Syslog New Format

Thu Jul 01, 2021 5:59 pm

The stop/start is nonsense. An packet gets forwarded when it does, it's not a phone call. However, I can agree that I would also prefer a key=value like format for firewalls logs.

Anyways, this re-formattning can be done with logstash. Just parse the default messages from the Mikrotik, and re-format as a key=value message, and re-send as syslog.
https://www.elastic.co/guide/en/logstas ... -grok.html
https://www.elastic.co/guide/en/logstas ... utate.html
https://www.elastic.co/guide/en/logstas ... s-udp.html
 
pe1chl
Forum Guru
Forum Guru
Posts: 10197
Joined: Mon Jun 08, 2015 12:09 pm

Re: MIkrotik Syslog New Format

Thu Jul 01, 2021 6:15 pm

Anyways, this re-formattning can be done with logstash. Just parse the default messages from the Mikrotik, and re-format as a key=value message, and re-send as syslog.
... but the default messages lack a lot of information. they would have to be expanded and when that is done indeed it should be done in a key=value style to allow further expansion without affecting all processing.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11982
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: MIkrotik Syslog New Format

Thu Jul 01, 2021 6:53 pm

This is the requred format from "Telecom" Office:
Jul:01:202113:43:22,Jul:01:202113:43:23,1.2.3.4,5678,11.22.33.44,5678,159.148.147.239,443
I do not see any key=value
 
mada3k
Long time Member
Long time Member
Posts: 687
Joined: Mon Jul 13, 2015 10:53 am
Location: Sweden

Re: MIkrotik Syslog New Format

Thu Jul 01, 2021 7:01 pm

As i understood the first post showed an example of the intended format. I would just set the start= and stop= to the whatever the event time is and call it a day.
 
User avatar
Jotne
Forum Guru
Forum Guru
Posts: 3292
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: MIkrotik Syslog New Format

Fri Jul 02, 2021 11:21 am

Problem with syslog is that there are no defined format for the Message field. Its up to each to create their own.

From Wikipedia
Since each process, application, and operating system was written independently, there is little uniformity to the payload of the log message. For this reason, no assumption is made about its formatting or contents. A syslog message is formatted (RFC 5424 gives the Augmented Backus–Naur form (ABNF) definition), but its MSG field is not.
If MT changes their Syslog format, I do have to rewrite my MikroTik for Splunk app (see signature).

If you have all fields from MikroTik, but in work format, there is no problem send Syslog to a Linux server, rewrite the log, then send it out as Sylog data to FT.
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: MIkrotik Syslog New Format

Fri Jul 02, 2021 11:33 am

Perhaps you should just look at an external solution. Use "rsyslog" with "mmfields" then you can (in realtime) rework the messages to some format you want.
So from Mikrotik just forward everything to the RSYSLOG-host and from there do what you want with it.
The same could be achieved with things like ELK-stack (elasticsearch/logstash/kibana)

I agree this case is not so easy to have it consistent in this format. Some fields will not be there etc.
 
pe1chl
Forum Guru
Forum Guru
Posts: 10197
Joined: Mon Jun 08, 2015 12:09 pm

Re: MIkrotik Syslog New Format

Fri Jul 02, 2021 12:14 pm

If MT changes their Syslog format, I do have to rewrite my MikroTik for Splunk app (see signature).
That is why in principle it is a good idea to, when the format would change, change it to something that a good parser could analyse even when new fields are added.
What we have now is just a random concatenation of fields, some with "name:" tag, some with "name value", some space separated, some comma separated.
That could be improved a lot. When everything has the format name=value or name:value or whatever, and is always consistently separated by space or comma or whatever, your job would be much easier.
 
pe1chl
Forum Guru
Forum Guru
Posts: 10197
Joined: Mon Jun 08, 2015 12:09 pm

Re: MIkrotik Syslog New Format

Fri Jul 02, 2021 12:20 pm

As i understood the first post showed an example of the intended format. I would just set the start= and stop= to the whatever the event time is and call it a day.
The intention of these requirements (by authorities) is to have a log of sessions made by users behind a (CG)NAT router.
So when they have some record of a TCP connect from 1.2.3.4:12345 to 4.5.6.7:443 at time X, they want to know which user behind the (CG)NAT router 1.2.3.4 made that connection.
For that, it is required to have the start and end time of that TCP session as seen by the router.

Contrary to what some people seem to think, these start and end times are actually well defined and known by the NAT subsystem of the router, at least for sessions that are successful and have been closed.
Where it goes wrong is the use of the logging feature of NAT rules. This triggers the log at the beginning of the session, and there is no corresponding log at the end.
That would be an additional feature: have some way for NAT rules to log at the time they get deleted. At that time, log the info that was requested including the start time (which would have to be recorded in the NAT entry all the time) and the end time (now).

RouterOS does not provide this. But RouterOS provides an alternative: Netflow. It has problems, e.g. it uses 32-bit values where 64-bit is required, but it does provide all the info that the authorities need.
But, as already written, you need an external system that receives the Netflow info and converts it to the format required.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11982
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: MIkrotik Syslog New Format

Fri Jul 02, 2021 12:29 pm

...Contrary to what some people seem to think, these start and end times are actually well defined and known by the NAT subsystem of the router,...
...This triggers the log at the beginning of the session, and there is no corresponding log at the end....
 
User avatar
Jotne
Forum Guru
Forum Guru
Posts: 3292
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: MIkrotik Syslog New Format

Fri Jul 02, 2021 12:32 pm

That is why in principle it is a good idea to, when the format would change, change it to something that a good parser could analyse even when new fields are added.
That is why I like Key=Value peer.
+ Easy to automatic decode.
+ New fields would be easy recognized.
- Larger logs du to keys for all fields all the time.

XMl would also work fine, but its even larger than Key=Value peer.

Who is online

Users browsing this forum: jaclaz, pajapatak and 43 guests