For all of you who has x86 router and want to upgrade to v3.12, please take a note that the upgrade will crash the router if you have a simple queue rules on it. If you have installed fresh v3.12 then after adding one single simple queue rule the router will crash and you will need to reinstall.
For now use v3.11 on x86 boxes.
This applies only to x86 architecture.

Who are you?
Member of the mikrotik developer team?
Or just another mikrotik user?

Don't you think this announcement is a little bit late?
I loose all my trust to mikrotik. All - every bit of it.
And today I will be installing vyatta on my router machine right after my customers goes to sleep.

Thank you mikrotik! For the worst day of my life..

Who are you?
Member of the mikrotik developer team?
Or just another mikrotik user?

Don't you think this announcement is a little bit late?
I loose all my trust to mikrotik. All - every bit of it.
And today I will be installing vyatta on my router machine right after my customers goes to sleep.

Thank you mikrotik! For the worst day of my life..

Ever thought it might be a smart idea to test a new release if you plan on deploying it in a production environment?? .. mikrotik aren't perfect you know.. everyone makes mistakes.

Don't blame mikrotik for your bad practices.

RouterOS 3.12 is less than 2 days old, and you've gone and installed it on a production system.
You should have done tests.

Sure, I am guilty for my trust.
You are right.
It's all my fault to behave in such a noob way.
Anyway. Lesson learned. One more time

It also seems to crash when I enabled hotspot with transparent proxy with smp enabled. I rebooted it and it crashes even on login prompt. So now i have a dead IDE flash on my hands ?

I've upgraded a test x86 system from 3.11 to 3.12 by advise from Mikrotik support. And it also crashes as soon as pppoe users begin to connect - perhaps when simple queues start to work.

--
Konstantin

Hi, i updated 3 "secondary" routers with simple queues in the settings and no problem, all is fine (until now). To static-x: as my grandfather would say: "living and learning" and "better late than never". Congratulations to mikrotik for this excellent and democratic forum.

I have tried to install fresh version, but it is not going to boot from hard drive, and it is all done, system rebooting
and so one,
I am glad, because I have done just tests
I am waiting for newer version, because I have problem with my clients according to PPPOE and PPTP.

thank god the automatic upgrading doesnt work
if there is a big disaster with this version why they produce it ????
we prefer to wait for months for a good , reliable and bugs-free version ..

Congratulations to mikrotik for this excellent and democratic forum.

is it the election day today ??

the interesting this is it is STILL available for download on main page!

Hey Mikrotik PEOPLE wake up. 3.12 is BUGGY. Tens of people announced it here. Some like me who trust, loose valuable time. Please remove it from downloads or at least put a disclaimer! Not everybody visits forums everyday..

And about me: yes I learned it once more as I said. I know have 1 working + 2 spare routers on my shelf + 5 different copies of backups in different locations

I know have 1 working + 2 spare routers on my shelf + 5 different copies of backups in different locations

I have to say, it serves you right for putting this onto production boxes.

You should always test a new release from ANY vendor on spare hardware in a non-production environment. It's called "best practice"

IMO this is in no way Mikrotik's fault, it is your own.

I have to say, it serves you right for putting this onto production boxes.

You should always test a new release from ANY vendor on spare hardware in a non-production environment. It's called "best practice"

IMO this is in no way Mikrotik's fault, it is your own.

While it is best practice to do so it says a lot when you release a full version (not alpha, beta, etc) and it doesn't even work (at all?) on a certain architecture that your product is sold on.

Just food for thought.

I have seen this many many times from other vendors.

e.g. from the two leading security vendors:

FortiOS 3.00 MR2 and MR5 from Fortinet with the IPSEC issues, and MR6 with SSL issues, and 3.00 erasing configs or corrupting BGP configuration

SceenOS 5.4 from Juniper with the RIP mem dump issues and 6.0 with the IPSEC mem dumps and OSPF issues

It's nothing new, don't test on production!

If 3.12 is so buggy, what can we expect of 3.13?

I think I will wait for 3.14

OK .. its our fault in ONE condition : if Mikrotik team wrote in a big red letters above the download links :
install it on your own risk

I don't feel anyone should use this forum to debate if it was a wrongful action to upgrade without warning or not. We need a resolution.

Everyone can agree that releasing this update causing such significant damage (mind you there is no way they even installed it to test; 15 minutes into using it at most would show it is not release ready) but all it does is damage the MikroTik name. I feel we can also all agree MikroTik has done all of us more then enough good to make up for this.

Lesson learned to those who have not learned already:

DO NOT UPGRADE PRODUCTION OR DEPLOYED REMOTELY. PERIOD.

Also don't update to a two day old release.

Haha you guys are obviously not aware of the VMWare issues in the last day. Basically they released a patch that had a build expiry on the 12th of August and it caused VMotion and all VM startups to stop on ALL boxes running the latest version of VMWare it was only fixed this morning. So even the BIG boys have major patching issues.

Stop bagging on Mikrotik, they make an excellent product at an excellent price.

Haha you guys are obviously not aware of the VMWare issues in the last day. Basically they released a patch that had a build expiry on the 12th of August and it caused VMotion and all VM startups to stop on ALL boxes running the latest version of VMWare it was only fixed this morning. So even the BIG boys have major patching issues.

Stop bagging on Mikrotik, they make an excellent product at an excellent price.

Oh , i feel fine now , thanks , cause Vmware makes mistakes too !!!
because Mikrotik did exellent jobs , we are not expecting such a disaster !!!
beside (for example ) we still suffering from the web-proxy blocking problem , and no solution ..

please we are not supposed to be beta testers ..what should we say to 100's of our clients ? forgive us we were testing ?

Can I give a small suggestion?

I think the problem here is the enterprise culture. Obviously Latvians are a brave, corageous people but I think maybe too much =). Maybe MikroTik folks should do some sort of Release Candidate for each point release too. I think 1-2 weeks should be fine. It seems to me that they are coming out too hot to call them "releases". Let them bake a little before tagging them as a "release". Yes, I understand and agree with the best practices but I also think that a release is something you should trust. The examples of other companies cited are true but if any of them had a *snafu* rate of one every 3 or 4 releases like MikroTik their image wouldn't be so good. Everytime this happens it chips away a little piece of the reliability image that I am shure MikroTik wants to build.

Obviously this is not a request, only a suggestion but I think that a more mature release process would help a lot to introduce MikroTik in bigger enterprises. I am not talking about anything like BSD releases but Firefox release process seems pretty much a good balance between speed and responsibility.

Thanks!

Oh, and another quick suggestion:

As a Sys Admin like all of you I don't have time to check this forum so often. The release of 3.12 was announced to me via e-mail, so I think this should be the way to warn users about problems like these. Release notices via e-mail and complete crash notices only trough forums seems a little like sweeping dirt under the carpet to me...

Be more transparent. It takes courage but pays off in the long term.

Well - I think I found more details about this problem - it is very specific,

it only persist if [/b]
Even if you have single core CPU.

AFAIK with multi-cpu=yes you just enable other Linux Kernel with SMP support - and it looks like it have problems with the changes in the simple queues

SO if you had multi-cpu=no there shouldn't be any problems with the update.

I have to say, it serves you right for putting this onto production boxes.

You should always test a new release from ANY vendor on spare hardware in a non-production environment. It's called "best practice"

IMO this is in no way Mikrotik's fault, it is your own.

Strongly disagree to that.
It's a loosy way of thinking.
Period.
You do test, it's self understood, but you are supposed to trust what you get. Not to _everytime_ not trust it, and hope it will be good.

Oh, and another quick suggestion:

As a Sys Admin like all of you I don't have time to check this forum so often. The release of 3.12 was announced to me via e-mail, so I think this should be the way to warn users about problems like these. Release notices via e-mail and complete crash notices only trough forums seems a little like sweeping dirt under the carpet to me...

Be more transparent. It takes courage but pays off in the long term.

+10 to the mail quote.

I recieved the mail with the _release available_ TODAY (14.08.2008). With the release, knowing it was buggy. No announcement ref. the x86 thing.
I in no way criticize the product or the stuff, but in my opinion, after all the storms with this specific release, and other things relative to the wireless part, and some of the routing (OSPF and RIP) in the late releases, i think a longer release cycle with an accent on thorough testing and stability would be far more benefic for the MT than rapid release cycles, with ..... some of the releases having "hidden" "features" ( ex. the 2.9.45 release, wich "fixed" the 5 ghz band in some regulatory domains, and made me climb towers and travel unexpected)

I mean, i don't update unless i need to. I still have routers with 2.9.43, and 3.7, 3.5. They work just as they are. If i don't need a new specific issue, it's no use to do it. Some of my boxes have 2.9.23, and are phisically untouched for tens of weeks. I like it that way. Makes me sleep well at night. Keep it that way.
It's an alarm sign for me that in the changelog has from 3.x releases, more and more lines appear to be starting with "fixed" than starting with "added".

Best regards.

It's an alarm sign for me that in the changelog has from 3.x releases, more and more lines appear to be starting with "fixed" than starting with "added".

You know there are old saying: "Only God can create perfect things!" in this case "Only God can add a feature without future fixes required!"

I don't know how it is there, but here this "simple queue issue" was not a big deal (needed to replace one of my core routers with the backup from the office) - such things might happen and not only because of software.

BTW: all routers in my network except multi-core core routers are running 3.12 - and I am having a good night sleep. Cheers!

It's an alarm sign for me that in the changelog has from 3.x releases, more and more lines appear to be starting with "fixed" than starting with "added".

You know there are old saying: "Only God can create perfect things!" in this case "Only God can add a feature without future fixes required!"
..............
BTW: all routers in my network except multi-core core routers are running 3.12 - and I am having a good night sleep. Cheers!

How did you managed to find that only multi-core with smp enabled are having this problem ?
Guess i'll find out.

"I'm going in."
(and I still think about that alarm sign.)

About God, I don't know......... It made us..... and we're sure thing NOT perfect.

How did you managed to find that only multi-core with smp enabled are having this problem ?
Guess i'll find out.

"I'm going in."
(and I still think about that alarm sign.)

Careful!
Not "multi-core with smp enabled", but "any x86 with smp enabled"

Hot I did find out? I was messing around with my dead core router - trying to revive it and find what exactly was causing the problem, then I noticed that MT guys wrote that simpe queues was the problematic place - but i at that moment was able to add simple queues without any problems - my multi-cpu was no

Careful!
Not "multi-core with smp enabled", but "any x86 with smp enabled"

Hot I did find out? I was messing around with my dead core router - trying to revive it and find what exactly was causing the problem, then I noticed that MT guys wrote that simpe queues was the problematic place - but i at that moment was able to add simple queues without any problems - my multi-cpu was no

Good work. And also for notification. I'm trying to reproduce it on an intel celeron 733, with 3.12, right now.
If it's so, then smp might not be the way to go right now, for numerous other issues, with no fault from MT at all.

It's an alarm sign for me that in the changelog has from 3.x releases, more and more lines appear to be starting with "fixed" than starting with "added".

I trust releases with most "fixed" lines in changelog. Currently, latest trusted (so-so:) release for me is 3.10, because I realised there is no need for me to upgrade every production router to 3.11.

I'm not gonna upgrade even my test routers to .12 untill v.13 comes out and fixes few major issues I need to be fixed.

Best regards,
Che

wew, it won't even work installing fresh from ISO to my CF, it finished installing all the stuff it needs then reboot, i only see

Loading system with initrd
Uncompressing Linux... Ok, booting the kernel

that's it.
Doing the same after reinstalling.

the bug is getting bigger since 3.11
need a better bug killer ...

Couple of points:

"doveryai, no proveryai" (Russian: Доверяй, но проверяй) - Trust, but Verify"

Anyone who upgrades their production systems to a new release without checking the forum first is asking for trouble.

Anyone who upgrades their production systems to a 2-day-old release is asking for trouble.

I lived through both Cisco and Lucent problems taking down two national Frame Relay networks in the US in the late 90s largely due to buggy releases. It happens to the best test department, even with a bazillion dollars to throw at testing.

...It happens to the best test department, even with a bazillion dollars to throw at testing.

If 3.12 went through any test department I believe someone needs fired.

What makes you think someone isn't getting fired?

MT is a lot more open and responsive to customer criticism these days, this forum reflects that very clearly.

I've finally got most things upgraded to 3.11 or 2.9.51 as they have been out for a while without problems that affect my use.

When a new version comes out, I download it for the architectures I need, and sit on it for a few days.

2.9.51 and 3.11 didn't have this problem, but often enough, there is are a mix of issues like we've seen here that pop up in the day following release.

If there are not a bunch of issues that come up after a few days, I install it on a few machines close by.

As times allows, I then have it installed on other machines, wireless links both ends at the same time of course. Routerboards first because I figure MT has easier testing on them, and PCs last since there is so much variety of hardware, it could take a while for bugs to be diagnosed.

Unless I am getting the new version to fix a diagnosed problem, there really is no huge rush for me to interrupt service to customers by rebooting routers everywhere.

I think it is good idea to create beta (release candidate) testers team for such issues. MT team can give us free licenses for such work =)

Why all this shouting? This was not such a big bug after all - it affected only those who have simple queues on X86 with multi-cpu=yes (in other words - core routers, and we all know that for those usually you must have a backup box).

BTW - v3.13 is out already.

v3.13 is Ok with optoin multicpu=yes.

I *will* not upgrade
Lesson learned

just check if everything you need is working and you are good to go

Every thing is ok, except queues,


No Simple queue is working,
no traffic is seen on queues,

has anybody tested queues?

Ever thought it might be a smart idea to test a new release if you plan on deploying it in a production environment?? .. mikrotik aren't perfect you know.. everyone makes mistakes.

Regardless of best practice something as trivial as a SIMPLE QUEUE causing a crash on x86 that requires a complete reinstall of the OS points out how recklessly inadequate the QA is on "releases".

I completely agree with the suggestion that was made to call new releases "Release Candidate" and let it stew in the wild for two weeks to see what reports come back off people installing it on their dev systems. This simple testing method would vastly increase the stability and quality of actual RELEASE versions, without requiring Mikrotik to spend a lot of money on QA testing. Take for example what happened with 3.12... it wasn't even out for 3 days before 3.13 replaced it. If this had been 3.12rc1 the bug would have been found and 3.12rc2 released with the fix, then when that qualified it would have become 3.12 - instead of having a whole release branded as "OMG DO NOT USE THIS IT HAS A ROUTER-CRASHING BUG!". This would go a LONG ways towards bolstering the perceived and actual reliability of ROS releases.

I've been complaining about this for years, as have many others - but it is a small price for the users to pay for such a LOW licensing fee for the software.

Basically it comes back to what all of the other posters have said. Don't put it on production so fast, test it on dev boxes, and make sure to check the forums for reported problems.

Mikrotik is a great product at a great price, while the QA on releases does leave something to be desired, it's not a deal-breaker for me.

I agree with Release Candidate process.