Hello everyone,
I am trying to understand if there is any testing and QA happening before releasing new stable software, like few days ago, we got 7.15, today I see 7.15.1, which has 12 fixes that were introduced on 7.15. This is not the only stable version I see this kind of issues, it is essentially on all of the previous versions.
Like most versions, there are beta’s.
In the case of 7.15betas were out around 2 months.
Feedback was a plenty in the forums, let alone private submitted bugs to support.
Not bad for a free release !
Of course there is.
Have a look at the testing channel and you’ll see 7.16Beta1. Look at the history of 7.15 and you’ll see all the Betas and RC releases. There are Alpha versions too but usually they’re not public.
Not everyone will be willing to use Betas and Release Candidates in production. Some bugs still slip through. New features may be added in Betas but bug fixes only in RCs
All software is the same be it Windows, Linux, Firefox, Chrome - all software will have bugs and it’s impossible to test absolutely every possible scenario in which that software might be used.
On the plus side and to their credit, Mikrotik are quite quick to fix any major bugs that do appear hence the quick release of 7.15.1
I understand what you guys mean, however it seems that Mikrotik software development teams themselves dont have very good testing suite or QA, and from your sayings it seems that they depend on the user feedback that much?
I mean, it is quite dangerous to have anything in production if you are “security conscious”, and you do all the updates as they appear, you can easily and repeatedly get affected by those. Sort of trying to guess what is going to break in every update.
I mean the hardware and it’s pricing is great and stuff, but the software, is very tricky in my experience (I am using mikrotik for the last 15 years), it wasn’t like this in the past. Once 7 and AX routerboards appeared, quality dropped a lot.
What makes you think that? Like above posters said, we have a long internal alpha stage, then we have beta stage. You the users are only a part of the full set of testing that is done internally.
You really think MikroTik just blindly builds betas and dumps them in the forum?
Gee thanks
Thank you for your response @normis,
As I said above, this is my observation from my experience using the products at home these days and in business in the past, I am not a guru or something, just a plain “user”, I had very good experience with the older versions but devices with RouterOS 7 and AX are very buggy,
My setup is the simplest, like bridge all interfaces, APs with the same SSID, all on the same network, all Mikrotiks, no fancy configuration, everything default, some port forwarding, oh I tried also wireguard which very fast realised it wasn’t worth the time I spent as it never worked as I wanted, on my raspberry it took me like 30 minutes to figure out how to configure and make it work exactly like I want, but this is a separate discussion, someone could argue that I dont know the “mikrotik way”, which is true, I am just putting my experience here.
I undertand, the platform is new, but I had too many issues so I stopped using the AX and I replaced it with my old AC, which works very well without issues. Now these days I am about to replace the network to AX, and I am quite sceptical if I should invest in Mikrotik or not. Hence opened the thread to get input from others in the community.
I dont think that everything is dropped blindly, but I have my concerns on the actual quality of the testing there, there is a repeatitive theme on the last few releases that a release is introducing many bugs, some serious, some unnoticable.
RouterOS is a whole operating system, it’s not some smartphone app. There are millions of potential configurations and therefore, conflicting configurations. We already have insane amounts of tests, but sometimes bugs do appear.
Don’t you think it’s about time to charge extra premium to ROS I know a lot of people is willing to shell out money us included and the money you gather on this can fund ROS even further to improve not only the software stack but to improved the product overall?, I hate to admit that sometimes i slam you guys out of frustration because it seems like you don’t know your priorities and your target market or audience, believed me your strength at least in Asia where I came from is not only on SOHO markets but also in Service Provider and datacenter.
Please allow this community to help you improved your product by shelling out money we don’t necessarily needs to be involved on your decision making where your product development is heading into, we know you don’t want the community to be involved period we learned that in a hard way, I personally believed you have a great potential and really a good alternative to the big guys in the network industry your hardware is good you just need to focus and align yourself what you really want to be.
This is just an honest perspective from an outsider to your company, best of luck to both of us our company was lock into your ecosystem already which is not a bad thing at all
Don’t you think it’s about time to charge extra premium to ROS
Kidding right? It’s not a software developing company. You can’t sell a licensed hardware with buggy os inside and ransom money for fixing under premium thing.
Am i kidding hell no! we are not a hobby shop and we need real stability on ROS I know money is hard to come by these days that’s why we are lock into their ecosystem if we can motivate them by giving some premium or extra why not? a lot of small and medium size business depends on ROS whether we like it or not at least from our world, a lot of business will surely want peace of mind we can’t wait/afford for MT to fix their stuff on their own pace, I know there’s a lot of alternatives but the alternatives cost fortune and legs we want a middle ground as much as possible
You know that ROS is a operating system that you can buy license only as well.
And for the buggy ransom thingy: that’s what “enterprise” manufacturers normally do: they sell you licenses, nowadays subscriptions for their buggy software, even some require paid subscriptions to even use their hardware and let you pay extra for everything, put enterprise hardware EOL only a few years of launch and ransom you into cloud.
Its funny, I work with a large Cisco environment( and have a lot of MT gear too 
).
But Cisco’s software is also buggy as, let alone full of CVE’s( funny when you compare against MT ! ).
I’v experienced expensive and ‘compatible’ cisco sfps’ that wont work in there hardware with various os releases, is insane, let alone when they bugger up IPSEC on a release, and that’s with a paid licensing.
So its a pretty sweet deal with MT I rekon.
I’ve offered many times to pay an extra fee (in thousands of USD per year) for things like premium support with <1 day response times.
I’ve begged many times to return the concept of a long term release that doesn’t get new features until they have been baked into the regular releases for at least a year
I’ve cried every time they introduce a config breaking change in a build called “stable”
I die a little inside every time something breaks that was only recently fixed for the first time in ages.
Please Normis & all the staff at MikroTik, understand that we love the product and everything, but as a service provider that deploys thousands of devices it’s the stability that we care about and are spending money for, NOT new features like wireguard or rose storage or containers or any number of other things that have caused various breaks bugs and updates. If we actually need those non-core features we’ll happily load a different build or the package into the device that’s needed for, but for the thousands of devices that we deploy that don’t need anything but the core of the OS, can we PLEASE go back to having some form of a long term (or service provider?) build tree again where new stuff simply doesn’t show up until it’s been tested and used by others for a year or two?
Of course we have no way to know (from outside) the amount of tests you make before releasing RoS versions.
All we can say is what it seems to be done: NOTHING (or very, very little).
The releases are so frequent that it is unlikely that each one has been properly tested and the number (at EACH release) of corrections to bugs introduced in the previous version is so huge that it seems that there is not an effective testing process.
The way the releases are tagged/named is one of two steps ahead what they really are (I know that a rose, by any other name, would smell as sweet, but your naming is really optimistic) I use, personally, a translation table, as follows:
Mikrotik Real Comment
Beta Pre-alpha Don’t even think of using this
RC Alpha Mixed up with fixes and new features (undocumented) and riddled with evident bugs
Stable Early Beta[1] Most bugs in previous versions fixed, but several new ones introduced, most changes, good or bad, undocumented or poorly documented
[1] a more suitable code could be “Delta Tau Chi” which reminds us of National Lampoon’s Animal House …
A part of the issues clearly comes out because (some) users do have “crazy” configurations, the question is why do they have these crazy configurations?
Possible answers:
It is clear that Mikrotik thinks that the right answers are #1 and #2, and does nothing to attempt solving or mitigating the problems related to #3.
At each new release of the RoS, the thread on the forum is full of replies of the kind (let’s say they represent 70% of the replies):
Hi, I had a configuration on a device that worked just fine with (insert here a previous release number) BUT since I upgraded to (insert here latest release) it stopped working.
Now, while it is obvious that the people having issues and reporting them on the ROS release thread are a minority of the users, I cannot believe that all of them have these “crazy” configurations, are stupid or lazy, so the possibility that #3 above is one of the causes should cross the minds of Mikrotik’s developers.
As well, a number of the posts on the release thread are not by “common” users, but by professional ISP’s or network administrators or by long time experts on the forum, and it is IMHO improbable that they have “crazy” (and fundamentally “wrong”) configurations.
I fully agree with that.
But I’d like to add: these users from point “#1” most likely use Winbox which magically introduces config they are not aware of (polarizing opinion), or they just loose the overview because of all the windows/tabs. They visit this forum to ask for help for their issue and for the first time ever seen their “full config” as textual representation just because forum members request the “/export file=any anynameyouwish”.
Yup, its about time they started to learn!!
I would prefer that they are taught to ensure their first post contains coherent information so taht we dont have to hunt and peck for information EVERY time.
However you are straying from the gist of the thread which is testing etc…
Kudos to MT to having a dynamic and consistent work force of dev and test attached to RoS. I just wish they could have more bodies.
Dont bother the folks in the trenches doing the work, talk to the owners since they decide how much of the profit goes into their pockets and how much is reinvested in the company.
If you dont like it, buy elsewhere or start your own company
Stable releases 7.14 > 7.15 are not released daily or often. Like one of the above posters said, you should just ignore beta and RC releases if you are not interested to be part of the development process. This is a hardcore enthusiast forum and we are releasing these … basically internal nightly builds for those people, that want to live on the bleeding edge. Those releases are not meant to be used in real and important network setups. If you are looking for stability, stay on “long term” release tree.
On Mikrotik Youtube channel your colleagues very often use non-stable releases for demonstration. As a viewer I would get the feel that using beta is quite safe “because MT staff uses it on their videos”
https://youtu.be/sQPlwDSd5LM?t=205
https://youtu.be/pzWXErH4cIA?t=141
https://youtu.be/37aff6d14Xk?t=133
https://youtu.be/EdwcHcWQju0?t=46 (using alpha netinstall to install rc version of ROS)
I could go on. beta/alpha in how-to videos for public demonstration. What do you think is the impression viewer get?
And “long term” release tree does not (yet?) exist for v7. 
@Normis
Sure.
And I wish to renew my personal appreciation for these hardcore enthusiasts that have the guts to risk their home lab setups (and sometimes even some non-lab setups) with the Beta’s, the RC’s and the Stable Delta Tau Chi versions and report their findings, thus contributing to the bettering of RoS.
Thank you all. 
For what is worth, a not-so-much-unrelated recent poll:
http://forum.mikrotik.com/t/long-term-release-or-new-functions/176637/1
Wait there. We are not publicly releasing nightly builds (cough cough anymore). Even Beta versions go through thousands of automated tests before public release.
The problem is that RouterOS supports too many devices on too many platforms for too long, and the number of use-case permutations aims toward infinity. I’d release RouterOS v8 with no legacy support. Unfortunately (but luckily for ancient device users), my opinion is minor here.