Community discussions

MikroTik App
 
aglabs
newbie
Topic Author
Posts: 39
Joined: Mon Dec 28, 2020 1:05 am

7.1rc1+ Traffic Flow Reports non-Existing IP's as SRC in Flow reports for packets that never exist on the wire.

Fri Sep 03, 2021 5:52 pm

Noticed after upgrading to 7.1rc2 flow exporting is reporting src ip address in flows for IP addresses that do not exist on the network.
Using a CCR2004
/ip/traffic-flow export:
/ip traffic-flow
set cache-entries=256k enabled=yes
/ip traffic-flow target
add dst-address=172.16.16.11 port=4739 src-address=172.16.96.1 version=ipfix
If i review wireshark capture of the IPFIX traffic, I can see clearly there is a flow defined with srcaddr: 172.16.202.186, which is a IP not in use, does not reply to ping, or any other traffic.

steps to reproduce:
1) Configure flow export (not sure if it matters but config I am using is above)
2) start a packet capture to capture IPFIX traffic (i am using a packet capture on device 172.16.16.11 to capture all traffic on port 4739 (my configured port)
3) attempt to establish tcp session with an IP that does not exist by either opening a web browser and navigating to http://somefakeip or nmap -p 80 -Pn notarealhost
4) I am using the following filter in wireshark : cflow.srcaddr == 172.16.202.186. I would expect nothing to show up, however I see flows reported for anytime something attempts a tcp handshake

Worth noting ICMP traffic does not appear to trigger this, from what Ive seen so far its only tcp traffic that triggers it.

Edit: Additional Findings
Was also able to reproduce this on demand using a 2nd CCR2004 as well as a RB4011, so it appears the behavior is consistently producible.
Last edited by aglabs on Wed Dec 01, 2021 2:44 am, edited 1 time in total.
 
aglabs
newbie
Topic Author
Posts: 39
Joined: Mon Dec 28, 2020 1:05 am

Re: 7.1rc2 traffic flow reporting non existent IP's as src in flows for packets that never existed.

Tue Sep 21, 2021 6:23 pm

Update, just tested rc4, issue still exists. Flow reporting is basically broken because its not reliable in its current state.
 
aglabs
newbie
Topic Author
Posts: 39
Joined: Mon Dec 28, 2020 1:05 am

Re: 7.1rc2 traffic flow reporting non existent IP's as src in flows for packets that never existed.

Mon Nov 08, 2021 1:43 am

question to the broader community looking for whether folks have seen or not seen this that are using netflow reporting. Has anyone leveraging flow reporting seen this behavior? Ive been able to reproduce this across the different models I have access too, supprised no one else has noted it.

I verified 7.1rc5 observes the same problem.

The root of the issue is mikrotik devices report via netflow a flow that never existed on the wire. With flow reporting turned on, wget http://some-ip-that-does-not-exist-on-network-but-defined-in-network-subnet will generate a netflow for the packet going out (TCP SYN) (expected) but a return packet of ACK/FIN from the IP that does not exist. packet sniffer/torch do not show any such return packet existing on the network. it seems its purely fabricated within flow reporting on mikrotik, nowhere else.

The outcome of this is nmap -p 80 -Pn 192.168.0.0/16 or 10.0.0.0/8 or 172.16.0.0/12 will destroy any flow reporting tool creating endpoints in them that does not exist, in my testing I verified against ntop-ng.

Ive also verified vyos, pfsense, cisco, juniper, ruckus, mikrotik prior to 7.1 RC series devices using netflow do not behave this way. There is no return packet reported (as expected)
 
aglabs
newbie
Topic Author
Posts: 39
Joined: Mon Dec 28, 2020 1:05 am

Re: 7.1rc2 traffic flow reporting non existent IP's as src in flows for packets that never existed.

Wed Dec 01, 2021 2:43 am

verified 7.1rc7 still has the issue. Also verified for the cases side, SUP-60021 if anyone interested, not that the case has had any movement in the last month or so support side as they say this is expected?!? 7.1 beta6 and below are NOT affected as they do not send flow reports for traffic that never existed.

Still hoping for some attention on Mikrotik side so this can be resolved in a future release.

Also further testing, it appears out of UDP, ICMP, TCP, GRE, IPIP, L2TP... TCP is still the only protocol I've found that triggers these 'ghost' flow reports.
 
aglabs
newbie
Topic Author
Posts: 39
Joined: Mon Dec 28, 2020 1:05 am

Re: 7.1rc1+ Traffic Flow Reports non-Existing IP's as SRC in Flow reports for packets that never exist on the wire.

Fri Dec 03, 2021 6:39 pm

7.1 TESTING still shows the problematic behavior.
 
User avatar
Jotne
Forum Guru
Forum Guru
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 7.1rc1+ Traffic Flow Reports non-Existing IP's as SRC in Flow reports for packets that never exist on the wire.

Sat Dec 04, 2021 9:59 am

You have send an email to support@mikrotik.com?
If not, I do suggest you do that.
 
aglabs
newbie
Topic Author
Posts: 39
Joined: Mon Dec 28, 2020 1:05 am

Re: 7.1rc1+ Traffic Flow Reports non-Existing IP's as SRC in Flow reports for packets that never exist on the wire.

Sat Dec 04, 2021 7:27 pm

You have send an email to support@mikrotik.com?
If not, I do suggest you do that.
I have. Case number is in a previous post. Have not heard anything on case back in over 45 days but been providing updates to case as well as this post.
 
aglabs
newbie
Topic Author
Posts: 39
Joined: Mon Dec 28, 2020 1:05 am

Re: 7.1rc1+ Traffic Flow Reports non-Existing IP's as SRC in Flow reports for packets that never exist on the wire.

Wed Dec 08, 2021 6:33 am

--- documenting incase anyone else in the community encounters the same situation ---

An update regarding the case, last response I've had from mikrotik was 10/4 regardless of the updates I've provided to the case as requested. My last effort before including this in my decision come network refresh in January. Simply looking for some kind of acknowledgement either way at this point whether this will be addressed, unfortunately my use cases require netflow to work properly.

sent to support regarding my case:
I wanted to follow up on SUP-60021. I have not had any response in over 2 months now.

Now that v7 is marked 'stable' the last and lack of response to the case has me concerned.

I was last told this is probably intended behavior. However in the last two months I have proven (attachments on the case) the behavior did not exist in 7.1beta6 and prior (including 6.x) and exists 7.1rc1 and later.

Can someone please reply so I know this issue is being investigated? Or at least let me know if this isn't going to be addressed until more people who rely on netflow complain, so I can start looking at a network manufacturer that abides by RFC's.
There is need to leverage what v7.1 has to offer in my production network, however netflow being inoperable makes the upgrade impossible.

TLDR; netflow in its current implementation will deliver flow report when tcp session is attempted to a non existent ip, the non existent ip is sent in a flow report as a src of traffic triggering flow analyzers to believe the non existent ip is actually a real endpoint, skewing reporting and making netflow an unreliable source of information. In 7.1BETA6 and below, this does not occur.
 
User avatar
sergejs
MikroTik Support
MikroTik Support
Posts: 6694
Joined: Thu Mar 31, 2005 3:33 pm
Location: Riga, Latvia
Contact:

Re: 7.1rc1+ Traffic Flow Reports non-Existing IP's as SRC in Flow reports for packets that never exist on the wire.

Fri Oct 07, 2022 3:49 pm

After researching your report further, at least we were able to get the same behaviour on v6.49.6, after brief looking into whitepaper, I guess it is done by netflow to close flows properly.
 
aglabs
newbie
Topic Author
Posts: 39
Joined: Mon Dec 28, 2020 1:05 am

Re: 7.1rc1+ Traffic Flow Reports non-Existing IP's as SRC in Flow reports for packets that never exist on the wire.

Tue Oct 11, 2022 4:42 pm

After researching your report further, at least we were able to get the same behaviour on v6.49.6, after brief looking into whitepaper, I guess it is done by netflow to close flows properly.
Thanks for researching. I appreciate the effort.

However that doesn't explain why it's new behavior. I've verified within my support case this behavior did infact change. Additionally it only happens to tcp and additionally only happens sometimes. If this was necessary behavior to close out a session I'd expect it to be consistent which it is not. Also mikrotik is the only vendor I tested that does this.


For what it's worth. It's not an issue in my network any longer as we tech refreshed routers where proper netflow functionality is required with a different vendor.
 
aglabs
newbie
Topic Author
Posts: 39
Joined: Mon Dec 28, 2020 1:05 am

Re: 7.1rc1+ Traffic Flow Reports non-Existing IP's as SRC in Flow reports for packets that never exist on the wire.

Wed Nov 02, 2022 6:03 pm

Also one additional detail/question to add to my previous comment, if the explanation is this is necessary to close out a session, what session? There was never one established due to the fact nothing exists on that IP, so a session shouldn't even exist to begin with.

Who is online

Users browsing this forum: No registered users and 24 guests