I have an infected computer somewhere in the VM machines by look

I am getting lots of these packets at the router
output: in:(unknown 0) out:ether1, connection-state:invalid proto UDP, 0.0.0.0:9001->10.0.2.152:9001, len 1492

I can see it's attacking 10.0.2.152 but how do I work out the source machine .. I really don't want to crawl thru 200 VM machines

Use Torch to find the source MAC address. If it hasn’t been spoofed (as that 0.0.0.0 source IP has) it’ll guide you to the matching VM configuration.

If the malware is smart enough and deeply enough dug into your systems that it can spoof the MAC, too, then it sucks to be you today. Sorry.

As usual, like the others, you don't write what you're talking about...
What is the version of RouterOS?

What does it take to understand that at least the hardware (in this case virtual machine used) and the RouterOS version must be specified?

The system hosting the virtual machines could also be infected...

output: in:(unknown 0)

in unknown???

If RouterOS is unable to define on what input the packet is coming from, probably is the RouterOS itself than generate the traffic...

If the NIC is used from more than one VM or the OS, can be leaked packet between os...

Sorry any omission in details is because we are newbie and don't know any better

The mikrotik is a CCR-2004-16G-2S+ with O/S 7.6
To mix that up I put a CCR1036-8G-2S+ with OS 6.49.6

No difference both routers showed same behaviour

There are 6 VM's all different hardware but all running VMWARE eXSI 6.5.0

I was about to say that we should focus on the fact it is in "output" chain and not in "forward" but then I tested it in lab and noticed that forwarded traffic with src-address 0.0.0.0 actually goes through firewall in the output chain! What the heck?

Right now I can't remember which service is the one using UDP port 9001 with IP 0.0.0.0, something related to IPv6 tunnels?
But as pointed out by @vecernik87, they are packets generated by RouterOS and sent,
they don't arrive from outside the machine where they are seen (or at most they leave in response of something).
If no RouterOS default services use them, the internal network is probably compromised.

One /export can generate some light.

This document says that UDP port 9001 is often used by tor ...

But as pointed out by @vecernik87, they are packets generated by RouterOS and sent,
they don't arrive from outside the machine where they are seen (or at most they leave in response of something).

mate, thats exact opposite of my post I originally also thought they are from the router, but I wanted to be sure so I tested it in lab, generated packets from a PC with 0.0.0.0 as src-address and sent them to 1.1.1.1 through the router. In ROS I created filter and mangle rules with action=log in each available chain.
It blew my mind that forwarded packet from LAN to WAN appeared in the output chain and not in the forward chain. For mangle (which has few more chains) it was prerouting, output, postrouting.. It did not hit the "forward rules" at all, despite the fact it was being forwarded.

On the other hand, I also had to disable my RP filter, otherwise these packets did not get to the firewall at all.

EDIT: somehow, I can't get the same result today. I promise I was not drunk last afternoon and I was certain that I got the result right....

But your packets had an "in-interface" in the logs, his packets list the interface as unknown, mate.

Vecernik87 is correct I can only filter the packets on the output stream .. the initial log was on the receiving tik

so you pick them up at source tik and drop them via
chain = output interface = the_interface src_address = 0.0.0.0/8

Now you get this log if you log it
output: in:(unknown 0) out:ether1, connection-state:invalid proto UDP, 0.0.0.0:9001->10.0.2.152:9001, len 1492

That drops CPU load by about a quarter on both TIKS .. which made me very happy

There is also possibly something in what Vecernik87 said that something triggers the behaviour because initially it doesn't happen on site
After a while depending how many people are on network it starts
I thought it just took a while for the attacker to find the new tik which actually doesn't make sense when I think about it because its the main gateway router.

Now a new piece to puzzle ... If I exchange the TIK for a ubiquiti ER12 the behaviour hasn't been seen in 2 days.
Unfortunately the ER12 is running at route limits and I can't put queues on the ER12 or it totally bogs down so I can't use it as a solution.

I am sort of at a loss and currently having to see if I can refurb an old cisco router because I need a solution.
If no-one else has seen the problem it must be my setup but WTF it is I am completely stumped.

Found it by accident it's dropbox client that triggers the problem not sure if it's malicious or just my setup

I have an IP4 only network and these are tiks in the middle of the network
The clients have IPV6 ethernet cards and the dropbox client can tunnel directly thru my IPV4 network it ignores all IPV4 queues limits, firewalls and everything
Those packets are max size because they are a tunnel to dropbox

I am guessing the log is sort of a IPv4 reporting an IPv6 packet bug ... they just magically appear on the output filter and seem to report like that.

I have never stressed over the IPV6 firewalls especially on routers away from edge not really considering they were in play ... wrong

I haven't looked why the ubiquiti edgerouter doesn't have the problem but I am guessing they have a default IPV6 firewall that drops all.

So anyone able to help with what I should have on the IPV6 firewalls to drop everything

From my post #6

[…] something related to IPv6 tunnels? […]

The clients have IPV6 ethernet cards

That's highly unlikely. In the vast majority of Internet-connected hosts, IPv6 is part of the OS kernel. The primary exception is if you have high-end network interfaces with TCP offloading. Otherwise, IPv6 is well above the level of the "Ethernet card," which is most often just an IC on the motherboard these days, not a separate add-in card at all.

I'm not being merely pedantic: thinking unclearly about where IPv6 lives will lead you into misdiagnoses.

what I should have on the IPV6 firewalls to drop everything

The fact that your local hosts all use IPv4-only addressing doesn't mean you don't need IPv6. You don't control the Internet, which does increasingly use IPv6. Also, some LAN-bound applications will use IPv6 internally, amongst themselves, if allowed.

This is because, for about the last 15 years, client OSes have shipped with IPv6 enabled by default. This has a number of implications.

For one, it means that when an application program does a DNS request through certain standard APIs, it's implicitly asking for any address, not just IPv4. It's perfectly legal to have an IPv4-only LAN but for IPv6 network connections to go out through the gateway, fetch data from an IPv6 server, and deliver it into your IPv4-only LAN.

The only way to keep your head in the sand on this is to go around to every Internet client — even including things like IoT devices — and disable IPv6 to prevent them from even trying. Blocking IPv6 at the router won't stop the clients from trying, leading to annoying timeouts: with IPv6 still enabled, they may try it first, then have to wait until they give up on it working before they fall back to IPv4.

Even if you do go and disable IPv6 on every LAN host, there's an increasing chance that something will then mysteriously break, and then you'll be back here with another bogus diagnosis about the reason why, having forgotten that you broke it yourself with your blinkered IT policies.

It's perfectly fine to have an IPv4-only LAN config, but it's getting increasingly hard to ignore the fact that applications and services will use IPv6 unless forcibly prevented.

Time to get on the train.

Sorry tangent your answer ignores reality

>>>> I don't OWN any IP range in IPV6 how would I even know how to route it to where and why? <<<<

What I do own and control is IPv4 C class licenses and so I must knock down ANY and ALL IPV6 traffic.
At the end of the day I am at the internet edge not in the middle of it.
Whatever advantages IPv6 has is mute on me because I don't own any range.

I get your bit about the equipment will turn up more and more so I am going to have to learn how to knock it down for now.
I will investigate what is the go with getting IPv6 range licenses but not a huge priority for a WISP.

@rextended yes you were on the money and I wish I had thought about that a bit more would have saved some time.

So we can all agree that your setup sucks.

In your world you can't do anything on the internet without IPv6 and IPv4 licenses and most small last mile operations shouldn't be there

or are you saying

Newbie Mikrotik users shouldn't be allowed because Mikrotiks don't start with everything locked off like other vendors?

It's slightly more complicated than that. In modern IT era one can pretty well live without IPv6 ... but since many devices now expect to have it, they try to use it. So either you have to try real hard to block/disable it or live with some nuances or embrace it (to certain extent). And the last option is the only future-proof option.

Yes I get that but as I said above the TIKs start in open to world so newbies to them will make mistakes and struggle with them.
Having used Ciscos, Junipers and Ubi Edgerouters for years it feels a bit like a crash test dummy with your first TIK.

If the supply chain strains of current never happened we would probably never have ever purchased a Tik and now we have around 40 of them
They have some nice features but you do fall down some holes as well

>>>> I don't OWN any IP range in IPV6

It's currently free to get a /40 or smaller from ARIN if you already have a v4 block from them. If you're elsewhere in the world, governed by a different addressing authority, there's likely a similar policy.

You might not even have to go through that bit of bureaucracy. It is likely that your upstream ISP offers DHCPv6 and will give you a /64 or /56 for free. Ask. You might be surprised. This happens even for home ISP customers these days.

It's the huge IPv6 space that makes this possible. An IPv4 "class C" (properly called a /24 now) is difficult to get and expensive because they're both rare and economically valuable, creating a marketplace for them, whereas IPv6 is plentiful to the point of having zero market value.

There's nothing saying you can't have both IPv4 and IPv6. It's called dual-stack, with lots of fun options atop that.

What I do own and control is IPv4 C class licenses and so I must knock down ANY and ALL IPV6 traffic.

That's like saying "I own a BMW, so I must destroy all Chevies." IPv4 and IPv6 coexist.

Thanks that is useful information and I will follow those up.

Yes what I was explaining is commercial reality the IPv4 /24 blocks are extremely expensive because there is huge demand at the edge.

Even if I could get IPV6 space many of my RF and Fibre Links can't carry it because the equipment doesn't support it.
Many of us at the edge would love IPv6 on our management networks because our public /30 links with tunnels consumes those expensive IPv4.
You run privates where you can but eventually you have to come out onto the internet to get back to NOC.
That is why it's worth following up that I might be able to get IPv6 space.

Commercially unless all the end mile equipment is supporting IPv6 then the market will basically stay IPv4 based.
The is some of the newer stuff with IPv6 but remember we have networks with lots of legacy gear so commercially you just buy the IPv4 and be done with as it's a cost you can factor.
Upgrading an entire network section would have so many unpredictable deployment costs and risks
For that reason I would start with our monitoring network because losing it for a day or two doesn't affect customers.

Even if I could get IPV6 space

There's no "if" about it. You can. You just haven't, yet.

many of my RF and Fibre Links can't carry it because the equipment doesn't support it.

Seriously? Name and shame, please. IPv6 is now literally decades old. What equipment are you using that is that far out of date?

Furthermore, "links" are generally agnostic about the traffic going over them. They care about things like Ethernet frames, not about IPv4 vs IPv6.

I fear we have another confusion about "IPv6 Ethernet cards" coming…

This is getting sidetracked into a waste of time about old legacy equipment which has it's own routing, bridges, spanning trees and management networks.
As a simple example most CPE have a PPPOE client in the firmware it's strictly IPv4 how would I get it play with an IPv6 feed?
Don't suggest an exterior modem that is another component and another power POE or DC and sets up issues at every site.
There are no firmware upgrades and even the companies like Alvarion, Ceragon, Morola Canopy etc no longer exist or got sold to a new parent.
Probably someone of your ilk might be able to research and craft ipv6 to pass thru them but commercially there is no point we go back to theme

>>>> Upgrading an entire network section would have so many unpredictable deployment costs and risks <<<<

You agreeing with me or not is not going to change my evaluation because it would take monumental research and lab testing.

I will take your advice and consider it on new sections and with new equipment but on the legacy network sorry it's a flat no on commercial grounds.

Anyhow thanks for input I will mark this as solved as I just installed a IPv6 firewall that drops everything and problem gone.

old legacy equipment which has it's own routing, bridges, spanning trees and management networks.

Bridges pass Ethernet frames. They don't care about IPv4 vs IPv6. (Or IPX, or DECnet, or…) Some bridging implementations allow assigning an IP for management purposes, including RouterOS's, but having an IPv4 address on an interface doesn't stop it from passing IPv6. Furthermore, RouterOS allows you to have multiple IPs on an interface, including that for a bridge.

STP operates at layer 2, below the IP level. Adding IPv6 to your network will not affect it.

Management networks can remain IPv4-only, as I pointed out above. All I'm suggesting is that you allow IPv6 as well, not that you re-IP the world.

The only thing on your list that touches IP is routing, and I covered that above: you go dual-stack, keeping your existing IPv4 routing rules (plus firewalling, etc.) and simply add IPv6 rules.

If the problem was as dire as you suggest, how did the world get to the current level of adoption in the first place?

most CPE have a PPPOE client in the firmware

If by "CPE" you refer to that which your customers have, presumably provided by you, then why are you bringing them into it? The thing that started this thread is that one of your WISP's internal service VMs was trying to use IPv6. Allowing that doesn't change what your customers do except insofar as fixes made to allow it probably allow your customers to switch over to IPv6 as and when they're able.

If you mean some bit of CPE provided by your own upstream ISP to you, we can't be talking about a huge capital expenditure to replace it if it truly is as bad as you suggest.

In both cases, even if we accept your claim that the CPE has a hard-coded PPPoE client that can't be upgraded to support IPv6, that still doesn't affect whether you can support IPv6. If you actually dig into the dual-stack links, you will find that there are plentiful options for getting around that, which is one of the answers behind my question about Google's IPv6 adoption data.

Alvarion

A WiFi company that went out of business in 2013, when 802.11n was still the hotness. It's probably time to upgrade anyway.

Also, WiFi — being a transport medium — is layer 2, below both IPv6 and IPv4. The radios doubtless have management IPs, but that can remain IPv4.

The only way this claim about them being old and out of support matters is if they have routing rules of their own and refuse to pass IPv6. That sounds like a consumer-grade all-in-one design, not a WISP backbone, where the paths are more or less fixed, to get traffic back and forth to the local center of operations.

If your Alvarion WiFi radios are merely data transports, then we're back to the "it doesn't matter" case.

Ceragon

…is still operating according to Wikipedia. Its web site is still working, and they claim to be providing wireless networking equipment still.

Yes, they got sold. So what? As long as the equipment is still getting updates, what does it matter?

Also, being a wireless networking company, I'd bet we're back into "dumb transport" territory anyway, where IPv4 vs IPv6 doesn't matter.

Morola Canopy

The Motorola Canopy system does at least appear to be out of production, but I found a manual, and it is as I claim above. Quoting it: "Switched Layer Transport with support for all common Ethernet protocols including IPV6, NetBIOS, DHCP, IPX, etc."

IT DOESN'T CARE.

Probably someone of your ilk might be able to research and craft ipv6 to pass thru them…

What irritates me is that as a WISP operator, you are presumed to be of my "ilk." If you truly are incapable of doing this, I'm tempted to ask what ISP you run so I can be sure to not ever use it. Not only do I use IPv6 services, today, I have to wonder what else you're dragging your feet on.

I mean, even frickin' Comcast has got their ducks in a row on IPv6 these days! You're behind them!

commercially there is no point

Customer satisfaction and retention are pointless?

unpredictable deployment costs and risks

First, there are experts who can predict this. You may be able to hire one, if only as a consultant for a time.

Second, if you must do this yourself, with existing staff, that's what testing labs are for.

Third, being nearly three decades into this project, a lot of this is written down, recorded in video form, and available for in-person training. It's not like you're blazing new ground here.

monumental research and lab testing

And now we get down to the nub: motivated reasoning to avoid doing work.

I just installed a IPv6 firewall that drops everything and problem gone.

And now you have a new problem.

I appreciate your technical background and comments but you are running about -100 in the commercial stakes.

The basic fact you are ignoring is who pays for all this extra playing around with a legacy network?
The clients don't get anything extra that they are "willing to pay for" from having IPv6.
I don't get any extra income from upgrading IPv6 onto that old network
All I do is waste money because you think it should be like that.

I get you think it's a problem but it's really not
1.) I knock the IPv6 down from any client ... if they want to upgrade to IPv6 I will upgrade them at there cost.
2.) I knock down all IPv6 entering from my upstream ... go to point 1 if customer wants it.

Eventually the old legacy network will die and I will have to replace it and at that point I can put up a spiffy
new IPv6 one you would approve of. So at that point I will take your advice.

Honestly I think you would be a customer no ISP would ever want because it would never be cutting edge
enough. Let me give you a Newsflash not everyone dreams of bleeding edge Technology like you most
just want it to work enough for what they need. You are passionate about technology I admire that but man
you are always going to be broke if you owned a company
.
Challenge: find me something the customer will pay for and I will upgrade the network otherwise it stays as
is until I have got full amortization. I know you don't approve you made your point but I am never going to
do what you want and it's my decision not yours.

I don't get any extra income from upgrading IPv6 onto that old network

Do your customers agree that their ongoing subscription costs go to pay for badly-outdated technology?

if they want to upgrade to IPv6 I will upgrade them at there cost.

Why would one customer have to bear the entire cost of upgrading your network rather than you aggregating a tiny slice of all your customers' fees for the purpose?

Eventually the old legacy network will die and I will have to replace it

You aren't going to upgrade until the entire network dies? Don't ever let your customers find this thread.

I think you would be a customer no ISP would ever want because it would never be cutting edge enough.

IPv6 is twenty-seven years old.

you are always going to be broke if you owned a company

I'm no businessman, but I think I can predict the same for you if you run all your customers off by refusing to upgrade decades-old equipment merely because it hasn't fallen over dead yet.

the network otherwise it stays as is until I have got full amortization.

You speak of WiFi equipment that can't be any newer than 2013 and still you haven't amortized it yet?

I hesitate to suggest a proper schedule on this, having never run a WISP, but I'd think you should have everything on a 3-5 year schedule, max.

it's my decision not yours.

Of course, but didn't you come here for advice? Now you have mine.

FYI so you get how commercially naive you are ... an average effective life, given the specified
telecommunications assets within TAX rulings predominately have an effective life of 10 years,
and protection systems typically have a 15-year effective life.

No-one in the Telco industry would turn over equipment on 5year unless there is a market reason
to do so because it costs you.... well except Tangent Telco Inc which is headed for bankrupcy

This has now gone full crazy and yes I got your advice .. My advice don't ever go start your own business.

I'm not talking about taxable asset depreciation, and I didn't tell you you had to replace everything on the 3-5 year business amortization schedule. I'm saying that you should structure your business to have the capital equipment paid off in that time, so that if you do have to replace it, you aren't still underwater on it.

The fact that your local tax authority won't let you fully depreciate it for 10-15 years is a completely separate matter.

@LdB, I don't know which State you live in,
but certainly if you keep the same CPE for 10 years, not only is the device already obsolete, but also is obsolete the way of thinking about the customers...

When the "5n" came out I replaced ALL my "5a" devices with "5n" devices in about 2 years,
when the "5ac" came out I replaced in about 3/4 years (I had much many customers as before) ALL of my "5n" devices with "5ac" devices.
Obviously all the infrastructure, not just the CPEs.
When I first started shipping IPv6, I didn't need to buy anything, just a few "clicks" here and there in the configuration...

I have never increased the fees, only the maximum speed available for the same price...
And IPv6... FREE!!!

I don't see all these problems in making customers happy, they are the ones who pay you for live...

What did you do with the money from the last 10 years (or more) if you run the network with the equipment you started the network in the first place?
Equipment that is now outdated, EOL, has security holes etc.
That's not a business, that's a scam.