MikroTik

maznu

Doesn't seem to do very much, at least not when added to a BGP Connection. Bird2 is showing this for a peer with ROSv7.6 with add-path-out=all enabled: Local capabilities Multiprotocol AF announced: ipv4 Route refresh Graceful restart Restart time: 120 AF supported: ipv4 AF preserved: 4-octet AS num...

maznu

I saw a similar issue on 7.6, opened a ticket with MikroTik support, and was advised this is fixed in 7.7rc. CCR1036-12G-4S no longer crashing with 7.7rc2.

maznu

10 minutes on 6.49beta22 so far, no issues to report. This device only has 10G links in the SFP+ ports, though, so your mileage may vary with SFP or SFP28. So ROS 6.49.22 version solve the packet loss problem? I don't think it does, at least not if you have many thousands of routes with nexthops on...

maznu

We've had the same, and will be deploying it on pre-production routers tonight. Maznu let know us if it works thx Giuseppe We'll know if it doesn't immediate crash and burn later tonight… but from experience it might be a few weeks before we know if it's stable or not. Will keep this thread updated...

maznu

We've had the same, and will be deploying it on pre-production routers tonight. Maznu let know us if it works thx Giuseppe We'll know if it doesn't immediate crash and burn later tonight… but from experience it might be a few weeks before we know if it's stable or not. Will keep this thread updated...

maznu

There is a new related item in the latest beta: *) switch - improved resource allocation on 98PX1012 switch chip for CCR2004-1G-12S+2XS device; Just got confirmation The latest RouterOS v6.49beta22 has improved resource allocation which has been extensively tested before the release and addresses C...

maznu

For us the 1Gs report 10G. Same for you? Hard down, will not establish link. Light is received, but port does not negotiate or come up when set to 1G. Is anybody getting any feedback on support tickets from Mikrotik? I was told that they are working on it but not idea if any fix seems imminent or i...

maznu

For us the 1Gs report 10G. Same for you?

Hard down, will not establish link. Light is received, but port does not negotiate or come up when set to 1G.

maznu

Same here (though what we were sent didn't look like a full command). We're having packet loss when using only two 10G interfaces, but only when routing production-like traffic (i.e. lots of src and dst addresses). A single iperf runs fine. Support ticket to MikroTik updated with full details. We'v...

maznu

6.49beta11 did not fix packetloss but interface instability. BUT beware - all ports negotiate in full 10G even if SFP is 1G. Works kind of until it becomes unstable causing ospf to flap. I got this same message from tech support to try this exact same thing. It actually made the packet loss worse f...

maznu

This tool exists http://manpages.ubuntu.com/manpages/disco/en/man1/aggregate.1.html Also you can use https://github.com/snar/bgpq3 , for example: bgpq3 -A -4 -j AS-FACEBOOK Here I'm using JSON output format so that I can have access to a prefix-length range, but you could equally use bgpq3 -A -4 AS...

maznu

I have done several tests with GNS3 using CHR 6.44.2 (stable) and as long as the router has enough memory, it doesn't crash. In my tests, the attack 'steals' around 180 MiB. Using a CHR with 256 MB, system resources shows a total memory of 224 MiB and free-memory of 197 MiB before attack. During th...

maznu

It is an upgrade problem because of no free space on the router, not related to this thread at all.

I have 6.43.14 installed on a hAP ac lite (64Mb RAM), and it is still vulnerable. Ticket#2019040222005195 and Ticket#2019032922005182

maznu

This is far from over.

Please refer to ticket 2019040422005244 and advise.

I'm hearing reports that this isn't fixed on routers with 64Mb or less of RAM. Is your ticket about this, eben? Or something else? :-|

maznu

What I don't understand: why is it not possible to firewall against it. When you limit the addresses that are routed, e.g. by dropping traffic in the raw prerouting table, does it still create entries for the dropped traffic in the route cache or neighbor table? Why? If you DROP in PREROUTING then ...

maznu

Just wondering what will happen / be the effect when "under attack" and hitting memory limit? * on neighbour mem limit * on routing cache limit Router will survive, but what with the legit connections? The tests I did on 6.45beta23 suggested different levels of memory usage would be used ...

maznu

Fixel also in long-term - 6.43.14
and Current - 6.44.2

Attacked both, and both releases fix CVE-2018-19299. Fantastic news — but now the hard work for all us network operators begins:

1. 🔬 test

2. 🧠 plan

3. 🔨 deploy

4. 🔍 monitor

5. 🍺🍻🎉

6. 😴 🛌

maznu

With extreme fragmentation, it can result in no contiguous memory that satisfies the malloc() or realloc() and you either segfault in userland or (I'd imagine) panic in the kernel, hence the reboot even with memory theoretically available. The data structure that the Linux kernel used in RouterOS v...

maznu

More testing has yielded more data. This has not been properly replicated by anyone else that I know of, so take it as plausible hypothesis. I think I found more fallout from the ipv6 flaw: boxes that have their ND cache or their ipv6 route cache run up but not to the point of OOM reload experience...

maznu

@maznu - beta23 fixes both vulnerability? Did you test? I emailed MikroTik yesterday, tweeted, and posted about this on the 6.45beta thread - yes! MikroTik has said that another beta is expected to make the settings on the affected components more "optimal" for devices with low RAM. I hop...

maznu

Also, send @maznu a present/gift/bounty/4011. He sure as hell earned it.

That's very kind, but after we've all got the patch in longterm and stable, I want to know how I can mail order a crate of beer to MikroTik's offices to say thank you for getting this fixed.

maznu

maznu - can you contact me via Twitter? I sent you a tweet already.

My timeline exploded a bit, as you might imagine. I'm @maznu on Twitter, DMs are open :)

maznu

where the reporter didn't report it as a security concern and left it for 6 months till he was able to get a CVE The full timeline will be available next week. But when I reported this in April 2018, my request to MikroTik was to plead with support to treat this as a serious security vulnerability,...

maznu

Congratulations! I have tested this beta and I confirm that with 300 Mb RAM the router's memory doesn't fill. A CHR with 300 Mb of RAM with OSPF-v3 has 237 Mb of free-memory and during the attack it keeps on around 200 Mb. Hopefully this fix will be in long-term and current branches soon. I concur....

maznu

FastNetMon may work if the netflow is being generated by an intermediate device in the path (like off of a tap), it's very fast and can potentially mitigate assuming null routing is performed before cache write. EDIT: with only: * a route back to the attacker * and only a default null route in my v...

maznu

Seems that one of these was considered as CVE and another one was not. Since author of these CVEs still has a problem, seems that actually #1 was not included in this CVE. However, this "problem" actually is not much of an issue. RouterOS IPv6 route cache max size by default is 1 million....

maznu

Seems that one of these was considered as CVE and another one was not. Since author of these CVEs still has a problem, seems that actually #1 was not included in this CVE. However, this "problem" actually is not much of an issue. RouterOS IPv6 route cache max size by default is 1 million....

maznu

Sorry @maznu but I don't get the same md5sum you expected. Maybe mine is a different but correlated attack It is possible we are using different tools to trigger the same issue — there is more than one way to make some IPv6 packets. ;-) If you're happy to discuss in private anyway, please drop me a...

maznu

I have been spreading the word around in other forums. If it's of any interest / help I am happy to act as a remote test case providing no harm is done. At this stage, my best advice would be that people monitor the memory usage on their routers and graph it. If your memory usage is stable for many...

maznu

As a consequence, I am now assuming the exploit is out there in the wild and is being used. Thanks for this information, @MichaelHallager. I've saw something similar several times in the first two weeks of March this year, and advised MikroTik on 2019-03-15 about this, asking for urgent action. At ...

maznu

When I launch the attack, the chr reboots but the other routers are not affected by the attack. Firewall rules seems not to be effective. But if I increase the chr memory from about 300 MiB to 3000 MiB the router seems to be ok: the free memory goes between 2200 and 2400. As my lab is made in gns3 ...

maznu

now that we know what command to run @IPANetEngineer: do you want to compare notes now that we are probably on the same page? Prompted by something MikroTik told me last thing on Friday about the nature of the underlying problem, and following my own research last night, I've got some good news to ...

maznu

If you haven't already, I would strongly encourage those of you who discovered and reverse engineered these bugs to compare notes and check that they are in fact the same methods - the last thing we need is for MikroTik to release a fix for the original issue, and then find that those who reverse e...

maznu

despite the author has being less than helpful about providing details. I have provided MikroTik with every detail at every step of the way. I cannot provide anyone else with any more detail at all as this would literally give them the means to carry out the attack. I have not shared any mitigation...

maznu

Normis...i'm pretty confident we have replicated the conditions of one of the CVEs from doing some digging on our own for this issue. Without the rules, the router crashed. When we added the rules the router stayed online. Meanwhile CVE-2018-19299 still needs fixing, because even with those perform...

maznu

Maznu, do the following: ip service disable [find] Verify that even with all Mikrotik access media services the problem occurs? Yes, that is still vulnerable (my test lab has no services enabled because it has no Internet connectivity - only console access). These IPv6 handling problems are not abo...

maznu

Maznu, thank you for showing that you are seeking a solution for the whole community. Could you inform me if disabling SSH and Winbox service also works the exploit? Using RoMon only can be a "temporary" solution? How you access the router isn't the major factor here… I'm not sure I under...

maznu

Normis...i'm pretty confident we have replicated the conditions of one of the CVEs from doing some digging on our own for this issue. Without the rules, the router crashed. When we added the rules the router stayed online. May I please add "discovered independently by a third party" to th...

maznu

For CVE-2018-19299, Are systems that do not have IPv6 connection tracking enabled affected?

Yes.

maznu

No. He did not send proof of concept for all issues, just a generic report about a crash. When he now said that CVE number such and such is not fixed, It was not clear, since we don't know what he will publish in that CVE. There is not a single issue, there are multiple issues, we fixed most, now h...

maznu

This version fixes: 1) Soft lockup when IPv6 router is forwarding IPv6 packets; 2) Soft lockup when the router is forwarding packets to a local network (directly connected) due to large IPv6 Neighbor table. We are still working on improvements for IPv6 Neighbor table processing in userspace which c...

maznu

We fixed the crashes that were reported to us. You said, we have not fixed "The CVE". I don't know what you will publish in the CVE. You have only provided a video that doesn't help at all. The CVE, CVE-2018-19299, was communicated to you in October 2018. It is literally just the number t...

maznu

For everyone here, I wanted to clarify, that to my best knowledge, the author of the CVE has not contacted MikroTik and we are in the dark as to what he plans to publish. There has been plenty of communications on this matter, normis. The most recent, specifically about what I plan to publish, was ...

maznu

For those who won't notice it otherwise: MT just announced ROS 6.45 beta version which includes fix for these two issues. Hopefully fix will land in other (stable and long term) branches shortly. CVE-2018-19299 is not fixed in 6.45beta22, I am afraid. Please clarify https://www.youtube.com/watch?v=...

maznu

will it be backported to versions 6.40.x and 6.43.x? Version 6.45beta22 has been released. !) ipv6 - fixed soft lockup when forwarding IPv6 packets (CVE-2018-19299); !) ipv6 - fixed soft lockup when processing large IPv6 Neighbor table (CVE-2018-19298); Sorry, but CVE-2018-19299 is not fixed in 6.4...

maznu

For those who won't notice it otherwise: MT just announced ROS 6.45 beta version which includes fix for these two issues.

Hopefully fix will land in other (stable and long term) branches shortly.

CVE-2018-19299 is not fixed in 6.45beta22, I am afraid.

maznu

So all this "close to year" shouting is overestimation. So i suggest to keep calm and wait for release, as MikroTik admitted 2nd CVE as vulnerability.

Second "bug" was acknowledged by MikroTik on 2018-04-20.

maznu

I believe I saw you comment that this can't be mitigated in MIkroTik at Layer3. What about using a MikroTik router at Layer 2 (or a non-MikroTik) inline in bridge mode before the Internet connection and using the firewall to filter out whatever is in the crafted packet that creates the issue? I'm a...

maznu

We aim to fix the issue before the mentioned publication date.

That is very welcome news, normis.

If you or your developers wish to contact me privately for any further information, you've got my email address.

Good luck!

maznu

If you just have the package enabled and absolutely no configuration from an IPv6 perspective are you okay?
I also would like to know this.

If you cannot route IPv6 packets, you should be safe.

maznu

Would somebody please post some additional information about this. I need to understand what is the problem, the potential impact and what vulnerabilities are possible. Where can I find information to read/learn about this? MikroTik acknowledged this issue on 2018-04-20. To learn more about it: I a...

maznu

why r u being so disruptive and trying to break mikrotik? Multiple MikroTik staff have repeatedly and continuously called this a "bug" and not a "vulnerability". If reporting "bugs" is now deemed disruptive then could someone please stop the world, because I would like...

maznu

The common practice to go public with a vulnerability is to do it in coordination with affected vendor, and their release of a fix. To do otherwise is irresponsible and unprofessional. I have been asking MikroTik for exactly this approach for nearly a year. They will not commit to a date, or even t...

maznu

Thankfully I'm in the position to do the above (and just have on my edge routers, in fact). I am nothing short of apoplectic that I've had to, however. Secretly hoping that either 6.44.1 was a fix for this or that it's a complete hoax. Either is better than what appears to be reality. Edit: It real...

maznu

Hi, I'm Marek Isalski. I've been trying desperately to get MikroTik to resolve this issue since they acknowledged it on 2018-04-20. I know for a fact other people have figured this vulnerability out, and I believe I've seen exploitation of it in the wild in the last 2-4 weeks. MikroTik's response to...

maznu

How many routes do you have? We have not experienced this issue at all with OSPFv2 with 450 OSPFv2 routes and 130 routers. It's been completely stable. 760000 in BGPv4, 60000 in BGPv6 And was several hundred in OSPFv2 (/32 per customer device), and several hundred in OSPFv3 (/48, /56, /64 per custo...

maznu

The log will repeat this when it breaks until I flap the instance: 08:28:37 route,ospf,info OSPFv2 neighbor 10.255.0.3: state change from Full to 2-Way 08:29:18 route,ospf,info Database Description packet has different master status flag 08:29:18 route,ospf,info new master flag=false 08:29:18 route...

maznu

We're using one for OEO of a 10G wave: https://twitter.com/NetworkMoose/status ... 4025182209

Thumbs up for this device so far!

maznu

I have never seen increasing memory usage due to IPv6 forwarding. But apparently your use case or configuration is different. This is an out-of-the-box configuration, plus IPv6, NOTRACK, and some static routes. MikroTik confirmed to me back in March that they have reproduced this issue. I'm just ho...

maznu

ND is like ARP. It is used to find the hardware address corresponding to the IPv6 address. Transit routers to not use it. (but they could use tracking) To refer you back to my post, and why ND is not to blame (despite using an "ND exhaustion tool"): RaspberryPi ---- hAP ac2 ---- hEX If I ...

maznu

So it is ND (also indicated by the name of the tool). No, you're doing exactly the same thing MikroTik support did — that is, not reading the addresses that are being targetted. Despite using a tool for ND crashing, it is not ND which is causing the problem — it's just an easy to find tool which wi...

maznu

As Normis already wrote, these are not really bugs but you are merely exhausting the capacity of the router, either for IPv6 ND or for IPv6 connection tracking.

Happens with IPv6 set to NOTRACK. It's not tracking causing this.

maznu

Here is the original message I sent to support on 2018-04-16: I have just run a trial with two MikroTik devices, all running latest release candidate. RaspberryPi ---- hAP ac2 ---- hEX On the raspberry pi, eth0 = 2a01:9e02:0:4242:xxxx:xxxx:xxxx:xxxx/64 (autoconf address, doesn't matter) On the hAPac...

maznu

Can it be prevented with firewall? It can be firewalled by not routing any IPv6. But if you have a RouterOS device anywhere in the path between one subnet and another subnet, even if not directly connected to that router, and it is forwarding IPv6 packets, it is vulnerable to being crashed. Maybe y...

maznu

Yes, it is exactly that. Denial of service from some type of IPv6 packet flood, where router runs out of resources. It was answered, that we accept this as a bug, but we would not call it a vulnerability, because there are many ways how to exhaust resources of any device. If I send IPv6 packets at ...

maznu

FYI https://threatpost.com/poc-attack-escalates-mikrotik-router-bug-to-as-bad-as-it-gets/138076/ Thankfully those CVEs appear to be fixed in 6.40.9 and 6.42.7. Good to see that MikroTik is taking RouterOS security seriously with those CVEs. Meanwhile, I'm still waiting for MikroTik to confirm when ...

maznu

Relief.. written and spellchunked! :D

Literally hours to spare! Time for a celebratory drink… ;-)

maznu

but as soon as I hook up more than 175 IP's the network does not accept more IP and crashes everything until I remove the devices. What do you mean by "hook up more than 175 IPs"? Do you mean you've got 175 devices connecting to the 2.4GHz WLAN? Or you've got more than 175 devices connect...

maznu

Usually I would either: 1) redistribute connected in /routing bgp instance (so that routes that your BGP-speaking router is connected to are "announced" via BGP) and/or 2) add a "network" in /routing bgp network which you want to announce ("synchronised" if you don't wa...

maznu

definitely there Monday

Glad to hear it, coz you're one of the speakers ;-)

(look forward to hearing yours — it's very much in everyone's minds with the worms currently floating around the Internet — well done for volunteering to speak at the MUM!)

maznu

Ok, who is going? Is anyone up for dinner/drinks the evening before…?

maznu

I want a bunch of these! * tiny 10G "ring" device (with options for 1G and 10G customer connections) * 10G inline mirror for IPS/etc * not-quite-transparent 10G transponder * OEO for wavelength regeneration (again, not quite transparent because it won't pass LACP/etc) I can see plenty of u...

maznu

I can't install updates because I'm interrupted by the message "ERROR: could not save package". This problem only exists on one of my MikroTiks. Does anyone recognize this symptom? Any help would be much appreciated. Check the /file or "File" menu — it might be you've got a big ...

maznu

At least disable the local users if AAA is configured and reachable. TACACS would be nice, but the current radius is functional, just doesnt disable local accounts. Why not just set your one local admin account to have an impossible IP address restriction, and then you've still got console-level ac...

maznu

Are both ends of the link set to autonegotiate? Have they both agreed on 100-full or 1000-full? Or did was one end set to 100-full and the other has "negotiated" 100-half?

maznu

Full tables? That's all you can do, unfortunately. The BGP implementation isn't amazing (constant scanning of tables), single-core, and the CCR's individual cores are not amazing for computation. Alternatively look at this excellent EU MUM talk for how to use CHR on 64-bit Intel for higher convergen...

maznu

For those of you using https://prometheus.io for monitoring, there are two options:

* https://gitea.faelix.net/FAELIX/ros2prom (requires Python, pipenv)
* https://github.com/nshttpd/mikrotik-exporter (which does not calculate bits-per-second on ports, for example)

maznu

Will we ever see some improvement here? MikroTik teased a new BGP implementation some years ago, part of the mythological "RouterOS v7". More recently we've been hearing, "almost everything we were going to put in v7 is now in v6"… except things like not crashing when pushing lo...

maznu

Our "solution" was to take a BGP feed into a routeserver on Bird, and point https://github.com/sileht/bird-lg at it.

maznu

Unfortunately problem is not resolved yet. I also can not give you any ETA for such fixes.
When problem will be resolved, then RouterOS release notes will include such fix description.

I guess we keep on waiting, and hoping...

maznu

PayPal announced this change two years ago…

https://www.paypal.com/au/webapps/mpp/tls-http-upgrade

https://www.paypalobjects.com/digitalas ... _paper.pdf

maznu

danielduffin, blackwp - Please contact support@mikrotik.com and send supout file. Are you sure that downgrade to 6.42.4 fixes the problem? Ther are no UM related changes in this version The problem might be related to PayPal deprecating old versions of TLS on 26th June: https://www.paypal.com/au/we...

maznu

is anyone else getting this error with userman and who do i resolve it PayPal - ssl connection error: handshake failed: error 14077410 (6) Thank you Yes we can confirm this error also. No Paypal payments being accepted. We tried adding an ssl cert to the RB but that didn't work either. "Mikrot...

maznu

A common reason for this is that the address families within the BGP session do not match. For example, I've always configured bird6 to talk to RouterOS BGP with only IPv6 address-family enabled.

maznu

I think all the emails back and forth were testing Maris' patience, but I am so incredibly grateful that they have listened and we have understood each other.

"We will test this scenario."

Excellent news - and good luck, MikroTik team!

maznu

Response from Maris: You can exhaust resources of any device by simply sending large amount of data. Set up ipv6 firewall to protect your router. Best way is to limit amount of accepted icmpv6 packets in IPv6 RAW firewall. I have asked them to reconsider, as this isn't traffic destined for the route...

maznu

we may see it in v7 whenever that comes out. :-)

Can't tell if those are the words of a man who has had a sneak peek of something…

…or words that are heavily laden in sarcasm! ;-)

maznu

[Ticket#2018041622003823] opened for the ICMPv6 transit crash.

maznu

Speaking theoretical if you redistribute your full views into ospf, then yes. But i have not heared any success stories on doing that. I would strongly advise against redistributing 700k BGP routes into OSPF. Probably better to have your loopbacks in OSPF and carry your full tables in BGP. …now if ...

maznu

i called iparchitechs and the sales person quoted me $2,500.00 - $5,000.00 to configure 5 routers! I think that is a little steep. Your hardware is probably going to be a similar sort of expenditure. I'm not sure how much you're paying for your fibre ring, but I imagine there are ongoing costs for ...

maznu

If you are receiving full tables (almost 700k routes now) from both your upstreams, then by default RouterOS will use AS path length as one of the criteria for BGP route selection. Assuming you've not set higher local preference or metrics on the routes (by using "in" filters), then you wi...

maznu

Maybe it is connection tracking. I added a "notrack" rule to ipv6 raw prerouting, and RouterOS still crashes. Of course pumping a gigabit of ICMP probes like those friendly programs do will kill the router. I could understand if the router dropped packets because it does not have the CPU ...

maznu

This doesn't seem to affect Linux itself, wonder what crazy stuff Mikrotik are doing with IPv6 to introduce a vulnerability like this? Maybe this is just a case of bad values for some IPv6 sysctl parameters — e.g. memory exhaustion by the kernel because of a value set too large. It will be interest...

maznu

In addition to everything else this is what I'm seeing. Just transiting a high volume of ICMPv6 traffic is enough to cause the router to reboot. I updated to 6.42rc56 this morning and no stealth fixes were contained in that release. It's trivial to reboot spam any MikroTik router w/ndpexhaust26 on ...

maznu

In the UK there were various government and non-government initiatives (usually in the form of grants to businesses) to reduce material waste, resource usage in manufacturing, etc. I would not be surprised if the PR team at MikroTik has thought about "Corporate Social Responsibility". It's...

maznu

I'm a bit confused about what is your problem. When it is in the ND it should only affect reachable networks. Are you getting a static prefix from your ISP? (e.g. a /48) Did you make sure to have an "unreachable" route for your entire prefix in the route table? (with only /64 routes to yo...

maznu

So ... Grab a Linux box, put it on your LAN and prepare to LULz to the point of tears. At this time, this is now another IPv6 deficiency that will prevent me from recommending MikroTik products for anything other than home routing. The fact the router can be REBOOTED simply by an end-user running n...

maznu

Another one for our IPv6 address space scanning shitlist: add address=2607:f140:4800::/48 list=shitpit It's likely being done by one of the authors of this paper (or someone working with them) at Berkeley University: https://conferences.sigcomm.org/imc/2017/papers/imc17-final245.pdf I've been conta...

maznu

Start laughing or crying at that point because your HEX is rebooting. Yes, rebooting. Target a Cisco 1841 w/15.0 code and default settings ... nothing happens. Confirmed. It'll OOM a CCR with 2GB RAM and 256k IPv6 neighbour entries. It doesn't even need to be to a subnet which is directly connected...

maznu

So, any news?

https://mum.mikrotik.com/presentations/ ... 914661.pdf

maznu

Start laughing or crying at that point because your HEX is rebooting. Yes, rebooting. Target a Cisco 1841 w/15.0 code and default settings ... nothing happens.

For what it's worth, I've raised this with MikroTik support, Ticket#2018040822000592.

maznu

Not sure if it's from the host failing because of an OOM state Oh, yes, and the other reason I don't think it's an OOM: Increasing the IPv6 neighbour cache size (e.g. doubling it) is an instant fix. At least till the tables have filled up again. But that bought me enough time to figure out where th...

maznu

Now, in Linux I wonder what entries the garbage collector purges. The op has stated he loses connectivity. Not sure if it's from the host failing because of an OOM state or because the router loses the neighbor table data for downstream devices while being scanned. If it's the latter I'd suspect th...

maznu

In the time it's taken me to email their network abuse contact, and shitpost about them on Twitter, they've probed ~150k addresses. Fun times. I received a reply just now: Hi Marek, We've added your prefix […] to our blacklist. We spread probes as evenly as possible across routed prefixes, and shuf...

maznu

They're doing a sort of enumerative scan of IPv6 address space, depth-first, which results in about 100pps of IPv6 traffic from them. That soon fills up a neighbour cache on a smaller device (even if you've set it to 100k+ entries), and pretty soon afterwards your little device loses its connectivi...

maznu

Another one for our IPv6 address space scanning shitlist: add address=2607:f140:4800::/48 list=shitpit It's likely being done by one of the authors of this paper (or someone working with them) at Berkeley University: https://conferences.sigcomm.org/imc/2017/papers/imc17-final245.pdf They're doing a ...

maznu

I finally got my hands on an infected device, spent some time with it, and can confirm that this appears to be Hajime I can confirm that upgrading to 6.40.6 removes the /flash/etc/rc.d directory tree, which of course deletes the startup script and thus renders the Hajime binaries in /flash/bin iner...

maznu

checks if port 80 is available, then uses the vulnerability that was fixed in 6.38.5 to copy itself into that system Does it require valid username and password to do that, or is it sufficient to have access to the webserver? If it's a botnet using ChimayRed, as suggested earlier in the thread, the...

maznu

360 Netlab are tweeting: "So the old Hajime botnet is coming back with a new exploit which was published only about 13 days ago ( https://www.exploit-db.com/exploits/44284/ ), it also looks for some old exploits like tr-064 but nothing exciting there." — https://twitter.com/360Netlab/statu...

maznu

Unfortunately the answer is in the subject. The designation of the product as 1Splus means it has 1-SFP-plus port. The "1C" (combo) port is either RJ45 or SFP - i.e. will only work at 1Gbit/sec. You'll need a 2S+ router if you want 10G uplink and a separate 10G connection to a switch.

maznu

Thanks for the answer Zerobyte. What about resources/capacity, would it take a full route and 100Mbps from each ISP ? Easily routes 100Mbit/sec - but be careful how lots of rules in /ip firewall filter etc might impact performance. Full routing tables: on CCR it takes maybe 5 minutes to converge. Y...

maznu

The CRS326 Link Aggregation without CPU utilization in RouterOS is planned in near future.

Hoping this will be coming to the CRS317 (and others) at the same time…?

maznu

That type of scanning technique is indeed pointless or dimwitted, however it may have served it's purpose... That might be the real context. It was still >2million more attempts (many hours) before this "attack" ceased. Maybe neighbour cache exhaustion was the purpose… never heard back fr...

maznu

If by FICON you mean https://en.wikipedia.org/wiki/FICON, then bad news: FICON isn't IP over ethernet. MikroTik devices don't speak Fibre Channel, and they don't speak FICON.

maznu

And kudos for using the doc prefix in your example. :) Thanks. Currently I'm unsure whether this scan (coming from a university network) is "legitimate research" or "pwned box" — as yet, no response from their abuse contact — otherwise my example might have named-and-shamed the ...

maznu

A few days ago one of our routers was hitting IPv6 neighbor cache exhaustion. The symptoms were occasional unreachability via IPv6. I pulled up Torch and found someone was actually scanning our network, probing consecutive addresses in a /64 to see if anything responded! Dropping this traffic is eas...

maznu

Would Switzerland be good enough (latency), or is Germany required (sovereignty)?

maznu

Having some OVPN problems having done an upgrade on clients from 6.38.5 to 6.40.1. Server is 6.37.5 still. No VPN session stays up more than a minute: 11:53:40 ovpn,info,account XXXX logged in, A.B.C.D 11:53:40 ovpn,info <ovpn-XXXX>: connected 11:54:20 ovpn,info <ovpn-XXXX>: terminating... - nothing...

maznu

Maybe not yet in stock. So just email a few of them to find out.

Normis: is there any news on the super special MPLS features on this chipset that you hinted about in January…?

maznu

Already available! Suggested price is $399 and yes, it can forward 160Gb/s wire speed. Search google for the part number and you can find that some distributors have it on their web page, if not, ask them.

Stealth release :-)

maznu

100km? I have never heard that the DWDM SFP+ can reach up to 100 km. How is it possible?

Mostly they're just "good" 80km optics with some extra Forward Error Correction thrown in. :-)

maznu

We're in the market for some of these… would love to know if we're buying 'Tik or might have to go elsewhere.

maznu

Excellent thread. I would like to add: IPv6 route rules and VRF The ability to do /ipv6 route rule routing-mark="foo" ... (and corresponding /ipv6 route routing-mark="foo" ... ) would be fantastic. Even older Linux kernels support this already (3.2.0 test box seems to have it), s...

maznu

About the WireGuard idea, are you a time traveller writing to us from future? :)

Spoiler alert: Trump gets impeached!

…but I'm not going to reveal which one is released first: WireGuard v1.0 and RouterOS v7.0 :)

maznu

You know how everyone's always saying "we want UDP support in OpenVPN" and "we want LZO"? And MikroTik say that their OVPN implementation is really nasty code that's hard to work on? How about instead we look to the future: WireGuard https://www.wireguard.io Clients for every maj...

maznu

user-name radius attribute is equal to clients local-identity, IOS by default puts its ip address as local-identity. Eap username and password for authentication is inside eap message. What a lovely information leak... Thanks for the info, mrz! Now to build the FreeRADIUS configuration from hell :-)

maznu

6.39rc12 and 6.38.1 are both still sending through an IP address (the road warrior's local IP) as the "Username" attribute in RADIUS. Isn't this meant to be the Username I specified in the iOS IKEv2 client? 185.134.196.4.36540 > 185.134.196.12.1812: [udp sum ok] RADIUS, length: 111 Access ...

maznu

iPhone client (IKEv2, User Authentication, with username and password), talking to v6.39rc12 with FreeRADIUS. The RADIUS packet received has the Username set to the iPhone's IP address - not the username specified in the "Authentication" section of iOS. Is this expected behaviour? Shouldn'...

maznu

I'm trying to set up eap-radius with Windows NPS, but i keep getting these errors on my windows radius server: An Access-Request message was received from RADIUS client 192.168.xx.xx with an Extensible Authentication Protocol (EAP) message but no Message-Authenticator attribute. Anyone know how to ...

maznu

Next RC will include message-authenticator attribute

…if this forum had a "like" button, I would press it :-)

Thank you!

maznu

...and same on 6.38rc52 :-)

maznu

We've had some success getting IKEv2 and RADIUS and EAP talking to each other... but we've hit an interesting stumbling block. We're using CHR 6.38 rc 51. The RADIUS packets generated by IKEv2 authentication attempts do not have a Message-Authenticator attribute: 19:43:57.474594 IP (tos 0x0, ttl 63,...

maznu

I asked this in the 6.38rc thread, but maybe here is better. I will admit that I've not kept up with how quickly the IKEv2 support has moved in these RCs. Well done to MikroTik's developers for doing this so fast! My question is whether or not it is possible to create an IKEv2 configuration on Route...

maznu

This DFS and radar detect is killing me. It takes forever to detect radar before an access point starts running. since 6.37 It would appear there's no way to disable it so I had to downgrade everything back to 6.36. Please mikrotik help!! You might want to watch the talk about DFS from the MikroTik...

maznu

*) ipsec - various additional work in IKEv2 support; I will admit that I've not kept up with how quickly the IKEv2 support has moved in these RCs. Well done to MikroTik's developers for doing this so fast! My question is whether or not it is possible to create an IKEv2 configuration on RouterOS whi...

maznu

VLAN's and not really an option, because for each VLAN I will loose some public IP's. You can use /32 addressing and not lose any IP addresses. On the router (which will be the default gateway, 192.168.12.1 in this example, for all your VLANs): /ip address interface=vlan45 address=192.168.12.1/32 n...

maznu

I've had no problems getting CHR to run under Ganeti with Xen as the hypervisor. That's not necessarily the same thing as "Xen Server", but it will run under a Xen dom0. I used a HVM domU, but specified paravirtual NICs and IOEMU for the HDD. I've not yet found a way to get it to run as a ...

maznu

Do you have a "strict" reverse path filter in /ip settings? Have both your upstream providers made sure that their upstreams in turn have sorted out BGP prefix filters, ACLs, etc? A clue of AS number and/or the prefixes you're announcing would help a bit - or you could use a tool like BGPl...

maznu

in the switching market, they have a very strong proposition on features/performance for a 1U switch now that loop prevention protocols are coming out. I'm looking forward to an announcement of 10GE switches by MikroTik. The CCR1072 was a strong hint that this could happen. Of course, the user inte...

maznu

Who do you see as competitors to MikroTik that are are currently beating them in innovation at the same price point? Right now, I don't. I thought that I'd been clear in all my posts in this thread. I see that there are bugs, but I've had bugs on every platform I've ever used - just as you've said ...

maznu

While relevant, lets not get distracted too much from the larger theme with this routing filter issue I agree. The route filter issue... I think Alex Hart meant we shouldn't get distracted from the larger theme: this thread is originally a discussion about whether we think RouterOS is moving forwar...

maznu

That would be a completely different bug, and I cannot confirm it.
Maybe it is a fasttrack or route cache bug. That could be. I don't use those features.

+1, cannot confirm. Haven't had any phonecalls about reachability problems.

We don't have fasttrack, but we do have routecache.

maznu

With RP Filter set to loose it should.

Ahhhh, that's the bit I didn't grok. I'd only ever encountered reverse path filtering in a strict sense as part of a poor-man's BCP38. Ok, that's interesting.

Thanks - I learned something today.

maznu

I think the last question I have about this for Murmaider is: Will your RP-based blackhole approach work where there are multiple valid routes (learned by BGP) to external addresses, but only one is "active" in the RouterOS routing table? Because we see traffic come in on interfaces which ...

maznu

Can you please share the code for this source based blackhole? From above I can see that I have to packet mark so filter is still involved?! Two different people debating different approaches. Murmaider's using a loose RP filter and route injection to trick the router into dropping packets. I'm usi...

maznu

You can still use these, just have them trigger a script that speaks to the mikrotik API, for example: # /usr/bin/blackhole.php <attacker ip> <time> It calls the mikrotik api and stores this in a db or flat file. then have a cron that runs every minute that looks in the db / flat file for items whi...

maznu

means no matter if its 1072 or supermicro with x86, during full routes propagation, it will still use just 1 core ? All the cores are used for routing packets between interfaces, so if you've got many Gbps of traffic you will see CPU usage on all the cores. But only one core is running the BGP proc...

maznu

2 & 3 - We do this using Wanguard. It detects the hacking / attack traffic and informs the filter (honeypot in your case) to inject a route into the mikrotik to redirect this traffic to itself where it can be captured for analysis. This is also filtered and the clean traffic can then be injecte...

maznu

I read somewhere Microtik CCR1072-1G-8S+ cannot handle multiple cores processing even its 72 cores ? Is it true ? The BGP implementation on RouterOS 6 will only use one core for doing the BGP update processing. If you're planning on having this router take a full transit table, that means that one ...

maznu

Can we have more details sent to support? Do you modify routing filters frequently? We've seen this problem several times on CCR with very recent versions of RouterOS. Day-to-day, I kept thinking I was going mad because I am sure I set up filters correctly. Disable the peer, enable it, same problem...

maznu

Are there really that much WISPs that need Linux commandline (bash or whatever), and DON'T need performance and features of CCR ??? This concern looks *really* far-fetched to me.. MikroTik obviously recognise there is a future in "Software Defined Network" because they have an experimenta...

maznu

Do you not lose fastpath by doing this though? Yes, but in our use case that isn't something I'm worried about. I need to use an address-list (so lose fastpath) so that I can redirect abusive traffic: 1) to display a "stop typing your password wrong!" message to customers failing to log t...

maznu

I don't believe any of the main UK distributors for MikroTik equipment will sell anything which is not permitted for use in the UK (e.g. I don't believe LinITX or MSDist sell any of the 900MHz gear). We've done a couple of "not quite line of sight" installs with MikroTik SXT devices - e.g....

maznu

Thanks to everyone at MikroTik for putting on this event. And thanks to the other speakers for their presentations - good to meet so many of you and talk about firewalling, network security, DDoS attacks, abusive traffic mitigation...

Looking forward to the next one :-)

maznu

Firewalling can be bad mmmkay Firewalling is great and everyone should use it, but try to do this as close to the edge (as close to the customer / server) of your network as possible. Do not firewall on your border routers (your routers between your network and the internet) unless you absolutely h...

maznu

BGP flow-spec. While routers in the DFZ can use a public IP2ASN database, for anybody with a complex peering network it'd be amazing to have ASN numbers within NetFlow/IPFIX data.

maznu

One of the things that struck me from the UK MUM yesterday was the desire for better scripting within RouterOS. There's a real danger of losing out to Ubiquiti's Edge Router, which has a more feature-filled full Linux shell for its command-line. I already see plenty of WISPs consider Ubiquiti for th...

maznu

I've seen similar behavior on my 3011 unit as well. I believe it's just erroneous data returned by routeros. It spikes for a second or two and then returns to normal values. Ok, at least it doesn't sound like I have a faulty unit - though the three units I do have all spike by different amounts, bu...

maznu

I could not determine if that router has a fan

No fan, completely silent operation. Just very strange spikes! :)

maznu

A picture says a thousand words. http://fs.maz.nu/rb3011-temp.png These two routers are new, in an air-conditioned data-centre, with minimal CPU load (3%). The temperatures come from SNMP, as reported by the MIB iso.3.6.1.4.1.14988.1.1.3.10.0 Does anybody else see this behaviour? Is it incorrect tem...

maznu

Similarly, it'd be great if we could use RADIUS to specify the profile to use for a PPP/VPN connection. There's already a Mikrotik-Group attribute which would be meaningless on a PPP session. Could MikroTik re-use that for profile name on PPP?

maznu

[quote="bcsteeve"]But are there any queues going on there?[/quote]

No queues, but plenty of firewall rules, and BGP... oh, and it is a VPN endpoint for their roadwarriors.

maznu

In the last couple of days, DNS over TLS was published as an RFC: https://datatracker.ietf.org/doc/rfc7858/ This addition to DNS - nicknamed "dprive" - would deprive intelligence agencies of their ability to sniff DNS traffic. I think that would be an amazing thing for MikroTik's RouterOS ...

maznu

We have a customer with a 100/100Mbit link, and their RB2011 doesn't get above 25% CPU load.

maznu

There's been a really useful thread about this on some other hardware (but related to the switch chip, probably) here: http://forum.mikrotik.com/viewtopic.php?f=1&t=97895

maznu

[quote="madmucho"]Please let us know[/quote]

I will report back soon, madmucho. The plan is to move the switches on Friday night this week...

maznu

v6.33 firmware 3.22 CRS226-24G-2S+

Uptime 30d 15:28:47
Link Downs 0
Last Link Up Time Nov/07/2015 19:26:14

I guess it's time to go to production...

maznu

v6.33: 16 days, still going strong.

But I notice this:

[quote]What's new in 6.33.1 (2015-Nov-17 09:55):
*) CRS2xx - fixed occasional switchip resets (broken in 6.33);
[/quote]

I guess that means the good developers at MikroTik are looking at this thread? ;-)

maznu

v6.33: 10 days in lab, no flaps or crashes (just 70Mbit/s test traffic continuous). That's much better than we ever had with the CRS226 before. But we will test for another 5-7 days more.

maznu

[quote="madmucho"]I spend very long time to debug that with Mikrotik support. Is something wierd in on low level on this model.[/quote] Thanks for your information, madmucho. We've been testing a CRS226 running v6.33 in our lab for five days now (only a few MAC addresses, only ~70Mbit/sec ...

maznu

[quote="madmucho"]6.32.3 still problem.[/quote]

I see that 6.33 has landed. I have two CRS226 switches which we pulled from production. I shall get these running in our lab, send several hundred Mbit/sec across them, and see what happens over the next few days.

maznu

[quote="madmucho"]problems with CRS226 still presist[/quote]

Are you joking?! The CRS226 will still flap all its ports and reset randomly, and MikroTik haven't found a solution yet?

maznu

[quote="madmucho"]Any news regarding described problem? [/quote]

We replaced our CRS226 with a CRS125, and it has been stable for a few weeks.

We have not yet tried 6.31 on CRS226, so please do not interpret my silence as "problem fixed" — I do not yet know.

maznu

[quote="kgninfos"]as the last time (leap second issue) my CCR did not responded interface led ware constant[/quote]

kgninfos: we had one of our CCRs crash at exactly 2015-08-01 00:00 UTC. It was still running 6.27, and had crashed at the leapsecond too. What version were you running?

maznu

[quote="normis"]We already have implemented it, have you tried "tool traceroute 8.8.8.8" in the recent months? [/quote] Normis: I think the OP suggests that it be possible to have a traceroute or ping to an IP address running in the background for a long period of time, and that ...

maznu

From Janis today:

"We have reproduced this problem, we have a setup in lab where ports are flapping on CRS226. The developer may fix it any day, it should be soon."

Fingers crossed!!

maznu

Having switched to the 125, with identical software and router configuration, it's been rock solid stable for almost 10 days.

uptime: 1w2d19h37m11s

Definitely a problem with the CRS226.

maznu

[quote="105547111"]I'd compare resources both and play around. Ive got to pack up mine now, but Ive got two more sipouts for support showing link downs with nothing connected! [/quote] Glad to hear your 125 is working ok. Similarly our third 125 has replaced the 226 (for now). The only cha...

maznu

[quote="105547111"]As I've still got a several days left to return the 226 for a full refund, I ordered a 125-RM[/quote] Funnily enough, our 125 arrives tomorrow morning. We'll swap out the 226 for the 125 (there are two other 125s on our network, been running fine for 60 days). And then m...

maznu

[quote="maznu"]Trying 6.30rc29 now... hasn't flapped in a whole 1 hour so far! [/quote] 6.30rc29: kernel crash and switch reboot after under 10 hours. I can't keep on testing firmware that is this unreliable, so am downgrading to v6.29.1 which at least will run for five days before all the...

maznu

Tried 6.30rc28 for ten minutes. Ports were flapping so much (every couple of minutes) that the switch was unusable.

Trying 6.30rc29 now... hasn't flapped in a whole 1 hour so far!

maznu

[quote="maznu"]No flaps yet, but random reboots isn't better. [/quote] Well, we just had the flaps: 14:18:34 interface,info ether02-mmr link down 14:18:34 interface,info ether23-wap1 link down 14:18:34 interface,info ether01-mmr link down 14:18:34 interface,info ether06-mmr-metronet link d...

maznu

Random reboot after about 18-24 hours running v6.30rc22:

14:04:16 system,error,critical router was rebooted without proper shutdown

No flaps yet, but random reboots isn't better.

maznu

[quote="IPANetEngineer"]Just curious....are the flapping ports connected to other MikroTik equipment or another vendor? If it's another vendor, which one is it? [/quote] For us: CCR1009 (x2), MikroTik mAP 2n (x1), and two connections to other suppliers who use a mix of Cisco and another ve...

maznu

[quote="pchott"]Any response that in 6.30 would be fixed?[/quote] All I know is what Janis from MT Support sent in my exchange on the ticket: Please upgrade to RouterOS release candidate version 6.30rc22. If the changes in the latest version do not solve this problem completely, supout fil...

maznu

[quote="becs"]But the CRS226 interface resets seem to appear due to insufficient CPU resources in routing or bridging applications. We are working to address this issue in upcoming RouterOS version.[/quote] Thanks, becs. Glad to hear you're looking at this. The CRS226 we have had problems ...

maznu

[quote="normis"]4) synchronization to server that have proper Leap Second implementation, not just time adjustment on next synchronization [/quote] This is interesting. Of the two identical CCRs we have, the one that crashed was synchronised to our stratum 1 DCF time server. Our DCF time s...

maznu

[quote="pukkita"]Seems history is repeating with CRS226, sending supout.rif and opening a ticket is the course of action to take. [/quote] We opened the ticket many weeks ago now. Each time it happens, we tell MikroTik and send a supout, they say nothing. It happens again, they tell us to ...

maznu

[quote="pukkita"]Have not yet used CRS226 but per your comments it looks not enough stable to put it in production. [/quote] Very few people seem to be reporting this problem. Also, it is strange that the person who started this thread has the same problem with the CCR1009. We also have tw...

maznu

Two CCR1009-8G-1S-1S+

Both v6.27 with v3.22 firmware.
Both with "ntp" package installed and enabled.
Both with "NTP Server" enabled.
Both with "NTP Client" enabled and syncing to (different) NTP servers.

One crashed at 23:59:60.
One running ok.

maznu

[quote="pukkita"]A general preventive measure with routers behaving weird, specially due to previous bugs, is a reset to no defaults (exporting the configuration to a rsc file before), then reload the configuration to make sure all gets initialized correctly. [/quote] Have done that. Have ...

maznu

Sorry, we update the firmware at the same time as updating the software.

All on 3.22, which is the most up-to-date for CRS226.

maznu

[quote="pukkita"]Make sure firmware is updated.[/quote]

That's not helping, pukkita. 6.27, 6.28, 6.29, 6.29.1, and 6.30rc22 all seem to have the same problem.

maznu

[quote="105547111"]It included the supout that shows a clean reboot then 9 hours later all the ethers in use cycled :-) [/quote] Oh dear. I guess I will be saying the same thing in a few days' time, then. I hope our supout files help, at least! Our ticket is #2015052766000629 (opened almos...

maznu

[quote="105547111"]I'll keep you posted it will NEVER go 24h stay tuned!
[/quote]

Thanks - and good luck! :)

maznu

We replaced the CRS226. After five days: the same problems.

We have been told by MikroTik support to try 6.30rc22. We will install this tomorrow, and test for another six days.

maznu

We have had the same problems with the CRS226: http://forum.mikrotik.com/viewtopic.php?p=485373#p485373 We even tried a different CRS226, but after 5 days and 16 hours, the same problems: all ports go down for a second, then come back up. Another 20 minutes later, they all go down for a few seconds,...

maznu

…and today, after about six days of uptime, a kernel failure.

16:50:22 system,error,critical System rebooted because of kernel failure
16:50:22 system,error,critical router was rebooted without proper shutdown

maznu

After 5 days, 17 hours, v6.29.1 has had the same problem: down up down up. 15:39:15 interface,info ether01-mmr link down 15:39:15 interface,info ether02-mmr link down 15:39:15 interface,info ether06-mmr-metronet link down 15:39:15 interface,info ether21-mmr-exa link down 15:39:15 interface,info ethe...

maznu

No joy with v6.29 for us on the CRS226. Trying v6.29.1 now...

Search found 207 matches