netflow monitoring on mirrored port

RouterOS Beginner Basics
1

Hi

I have arranged for a port mirror on a switch so that all data passing through the switch is pumped to ‘ether2’ on my mikrotik. I also configured /ip traffic-flow to monitor ether2 and export netflow data to my collector.

I expected to see netflows of all traffic on the collector, but I only see netflows from broadcast traffic. I have played with various approaches, such as creating a bridge from ether2 and ether3 (which is not connected) but to no avail.

I made sure that the data is on the wire, I can see all (non-broadcast) traffic if I connect the wire to a PC and tcpdump, and I can also see the traffic in /tool sniffer.

I also found this wiki page http://wiki.mikrotik.com/wiki/Manual:IP/Traffic_Flow that says

For example, you set up mirror port on switch, connect mirror port to router and set traffic flow to count mirrored packets. Unfortunately such setup will not work, because mirrored packets are dropped before they reach input chain.

… but it is not very informative about how to MAKE IT WORK.

I have router OS 4.6 on a mikrotik 800 routerboard.

Any ideas?

2

You can’t make such setup work in RouterOS.

If you need all packets counted by traffic flow then you have to forward all traffic through the router.

3

Is it possible to create a dummy interface of some sort, where the traffic would get forwarded? (On Linux I solve this by using such a dummy interface.)

4

Or perhaps a dummy switch connected to the other port? ANYTHING?

5

Other postings have mentioned that if you use a bridge you need to set the bridge to use-ip-firewall=yes for the netflow to work. I didn’t get that to work but didn’t do extensive testing of that setup.

I do have a setup that works for me but am seeing the netflow sequence error mentioned in other posts. My next step is to update to 4.9.

An issue for me was that the mirrored traffic is all packets with ethernet destination addresses that don’t match the MikroTik router (RB750 in my case). So when a router port receives an ethernet packet that with an ethernet destination address that doesn’t match the ethernet address of the router port the packet is dropped. That is why bridging is a possibility, a bridge is supposed to flood packets with an unknown ethernet destination address to all ports in the bridge.

What I did is make the incoming interface the only interface in a bridge with the web interface. It added:

/interface bridge
add admin-mac=00:00:00:00:00:00 ageing-time=5m arp=enabled auto-mac=yes
comment=“” disabled=no forward-delay=15s l2mtu=1524 max-message-age=20s
mtu=1500 name=bridge-interface priority=0x8000 protocol-mode=none
transmit-hold-count=6

/interface bridge port
add bridge=bridge-interface comment=“” disabled=no edge=auto external-fdb=
auto horizon=none interface=ether2-local-master path-cost=10
point-to-point=auto priority=0x80

Looking at the packet flow diagram I was going to try to rewrite the destination MAC address with dstnat. That didn’t seem to work but looking at the manual page for bridge nat I saw the redirect action and tried that, and that worked for me.

So I have the incoming interface as the only port in a bridge and I set

/interface bridge nat
add action=redirect chain=dstnat comment=“” disabled=no

(If I had more than one bridge interface I would need to modify that action to match on the correct interface.)

What this does is cause the router to accept all packets just as if they were sent to the MAC address of the router.

I have the router set with a default route out another port that connects to another switch. All of the mirrored traffic is routed through the MikroTik router and sent to that switch so I have an ACL on that switch that drops everything unless it has a source IP address of the MikroTik router. (The netflow packets will have a source address of the MikroTik router).

One other thing that puzzled me at first was that when I put the incoming interface in the bridge the netflow traffic stopped. (That was before I did the bridge nat redirect, I had gotten it working by changing the MAC address of the ethernet port on the MikroTik to be the same as the destination address in the mirrored packets.) I later realized that I hadn’t added the bridge-interface in the list of interfaces to watch for netflow.

/ip traffic-flow
set active-flow-timeout=5m cache-entries=512k enabled=yes
inactive-flow-timeout=15s interfaces=ether2-local-master,bridge-interface

When I was searching with google the question of generating netflow from mirrored packets seemed fairly common. It might be a good idea to put the info above in a manual page or FAQ.

6

Thanks… I’ll try these suggestions. Unfortunately I can’t afford to pump any mirrored traffic out of the Mikrotik – I can’t bother people owning the switch with setting up an ACL just because my router has f—ed up setup. So I’ll try to enable such an ACL on the router. If that fails I have some other ideas, plans B and C, so let’s see how that works out.

When I was searching with google the question of generating netflow from mirrored packets seemed fairly common. It might be a good idea to put the info above in a manual page or FAQ.

Or how about just having it working right out of the box?

7

I haven’t tested this yet but in theory you should be able to put our outbound port in a separate bridge (not the same bridge as the incoming port) and then apply a drop action in the srcnat chain ( you will want an accept action for packets with a source IP of your MikroTik router).

8

Things did not go well. The little bastard is so persistent in not doing what I want.

When you create a bridge between 2 ports on the same switch the netflow monitoring works but soon creates network storm due to the loop and grinds the switch to halt.

When you add a firewall rule to drop forwarded packets on the bridge the network storms stop but so does netflow accounting.

When you create a dumb switch that’s connected only to the router then the router does not forward anything to that switch possibly because it knows that none of the mac addresses involved is on that switch.

AFAIK there is no null interface on the router to forward packets into.

So is there a way to make this scenario work or not?

9

what about creating a blackhole route and forward everything to it? You probably also want to remove the TX pins on that ethernet port so it won’t reply to anything on accident. In the end, netflow is broken on RouterOS due to the flows being sent out of order.

10

dummy interface on RouterOS - empty bridge without ports should do.

11

I finally banged out a solution for this. If anyone has anything better, please post it! Here’s what i did - created a MetaRouter, created a virtual eth for the MR, and joined that to the bridge that has the connection to the mirror port. Add the ip-firewall to the bridge, then add a dest-nat to the bridge with an action of dst-nat. Now under Interface list, all of the RX traffic coming from the mirror is seen as TX on the ViF port. more important, the stuff finally gets exported as flows which I can see in my collector. I don’t care what the MR does with it (probably drops it I would think), I just want to see the flow across the bridge.

12

Netflow monitoring on the mirrored port (tested on my RB 750GL FW 5.24):

  1. Connect port 1 to the mirrored port
  2. Use port 2 for administration purposes
  3. Set master port to none on ports 4 and 5
  4. Connect port 4 to the port 5 with cable
  5. Create bridge, add ports 1 and 4 to the bridge
  6. Enable IP firewall bridge option and set Protocol Mode to none
  7. Configure port 1 bridge options: Edge:no, Point-To-Point: no, External FDB: yes
  8. Configure port 4 bridge options: Edge:no, Point-To-Point: no, External FDB: yes
  9. Enable and configure NetFlow options for the bridge
  10. Disable connections tracking to reduce CPU usage