Community discussions

MikroTik App
 
User avatar
Larsa
Forum Guru
Forum Guru
Topic Author
Posts: 1025
Joined: Sat Aug 29, 2015 7:40 pm
Location: The North Pole, Santa's Workshop

Software life cycle and release management (re: auto upgrade)

Wed Jun 08, 2022 5:55 pm

This is a breakout topic continued from "Re: v7.3 [stable] is released!"

@pe1chl wrote: Indeed one should never have some scripted daily auto-upgrade tied to the "stable" channel! it was again demonstrated that a new version that had 2 release candidates in "testing" is suddenly promoted to "stable" with known issues and even some last-minute changes. That is just not a good thing to do, a "stable" version should have been a release candidate at least for some time to get reports like this, and then it should just be promoted with no changes whatsoever. When there are changes, first make a new release candidate.

Now MikroTik argues that for this auto-upgrade one could use "long-term". However, the release policy of long-term is exactly the same: there are long-term versions that are not copies of the last stable, but have some additional quick fixes that could result in the same situation.

So ideally when you run an auto-upgrade script is should only trigger after the posted version is at least some days old (like 14 days or so), but that is probably difficult to do.


Yeah, question is if MT ever will acknowledge (or get to understand) that neither the "stable" nor "long-term" channel is suitable for any kind of "auto upgrade" as the current process is constructed?

In my opinion, MT need to rethink the whole software development life cycle strategy in terms of priorities and release management. This also include online services like ip-cloud, help docs and similar, like for example how to communicate with the public if a service stops working.

As an outside observer it might be hard (or impossible) to grasp the underlying mechanisms why a crippled release management process seems to be a forever ongoing treadmill that MT for some reason is unable to jump off. They certainly proceed with business as usual like it was a nonexistent problem.

Regardless reason, MT doesn't seem to be much concerned and rejects any active discussion to fix this issue. Instead, they seems more interested on focusing on and having an active dialog regarding a new "Controller" whatever that means.

When it comes to odd priorities of what features/fixes should be included in new releases of v7, it might be possible the marketing department to some extent controls the content when looking for new or emerging market positions. But that doesn't explain the lack of basic features from v6 like bgp and so forth.

Bottom line, pretty remarkable priorities if you ask me but hopefully they will be better organized (soon enough).
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 3169
Joined: Sun May 01, 2016 7:12 pm
Location: California

Re: Software life cycle and release management (re: auto upgrade)

Wed Jun 08, 2022 6:34 pm

100% agree, well stated.
See Normis' Auto Upgrade video - https://www.youtube.com/watch?v=WPW3mHlEzn4
If MT is going to promote auto-upgrade & control the OS: why make your users cut-and-paste a script they don't understand & require a aggressive/error-prone ":delay 3s" in their code... And claiming "20 seconds" outage for an upgrade – perhaps if you're lucky and using ethernet/fiber, but Normis' claim in video is likely easily disproven.

Nor even in 2.5 minute video about auto-upgrade, no mention that maybe they'd might want to set "long-term"? The lack of explanation of the channels in the video highlights the importance they place on managing the various release trains – if viewer's router were on "testing" and followed the auto upgrade video, easy to imagine the "Help!!!" forum post (or, for MT, more support cases).
 
pe1chl
Forum Guru
Forum Guru
Posts: 10186
Joined: Mon Jun 08, 2015 12:09 pm

Re: Software life cycle and release management (re: auto upgrade)

Wed Jun 08, 2022 10:27 pm

Yeah, question is if MT ever will acknowledge (or get to understand) that neither the "stable" nor "long-term" channel is suitable for any kind of "auto upgrade" as the current process is constructed?
I think they don't. Or at least Normis does not understand it.
Everytime I start this discussion he comes back with the same naive remarks and statements indicating he does not understand at all what it is about.
When it comes to odd priorities of what features/fixes should be included in new releases of v7, it might be possible the marketing department to some extent controls the content when looking for new or emerging market positions. But that doesn't explain the lack of basic features from v6 like bgp and so forth.
I have indicated several time that the first thing we need now is feature completeness of v7 (feature parity with v6).
Normis does not understand at all. At least that is what is indicated by his replies.
I don't know if there are other employees who understand what I mean. At least it seems that all of the outsiders do.
 
User avatar
Larsa
Forum Guru
Forum Guru
Topic Author
Posts: 1025
Joined: Sat Aug 29, 2015 7:40 pm
Location: The North Pole, Santa's Workshop

Re: Software life cycle and release management (re: auto upgrade)

Wed Jun 08, 2022 11:09 pm

I think they don't. Or at least Normis does not understand it. Everytime I start this discussion he comes back with the same naive remarks and statements indicating he does not understand at all what it is about

Well, If the threads below and similar ones don't provide satisfactory evidence, it's probably impossible to continue with any further argumentation as the recipient likely has a personal or hidden agenda that prevent a public acknowledgement regarding this subject.


In regards of "feature completeness" of v7 AFAIK there hasn't been a single attempt to explain the road map and the corresponding priorities.
 
User avatar
normis
MikroTik Support
MikroTik Support
Posts: 26291
Joined: Fri May 28, 2004 11:04 am
Location: Riga, Latvia

Re: Software life cycle and release management (re: auto upgrade)

Thu Jun 09, 2022 1:17 pm

You are advanced users with specific configurations, discussing them on a technical forum. The userbase in this forum is like 0.001% of real users. For those people, v7 works beautifully.
Don't get me wrong, we need your complaints and edge cases, they help to fix bugs, but believe me, when I say, there are tens of thousands of people using v7 and having no idea what issues you are talking about. They are rare. And thanks for helping us find them.
 
pe1chl
Forum Guru
Forum Guru
Posts: 10186
Joined: Mon Jun 08, 2015 12:09 pm

Re: Software life cycle and release management (re: auto upgrade)

Thu Jun 09, 2022 1:49 pm

But you should also understand "we" are poking in a black box. We get new versions with a list of changes, but they present bugs in other areas.
Maybe seldomly used areas (like using an SFP) but still quite dramatic. I would not want to be responsible for releasing untested software as "stable" just after having presented an easy way to upgrade to the latest version automatically...
Sure I am also running v7 on my hAP ac2 used as access point, and it works without issue at every upgrade.
 
User avatar
normis
MikroTik Support
MikroTik Support
Posts: 26291
Joined: Fri May 28, 2004 11:04 am
Location: Riga, Latvia

Re: Software life cycle and release management (re: auto upgrade)

Thu Jun 09, 2022 1:55 pm

Yes, I know. That's a topic for another thread and we do see many ways how we could improve.
My comment was only about the perceived instability of v7, but this forum is not a good measure. Most people are fine using auto-upgrade scripts.
 
pe1chl
Forum Guru
Forum Guru
Posts: 10186
Joined: Mon Jun 08, 2015 12:09 pm

Re: Software life cycle and release management (re: auto upgrade)

Thu Jun 09, 2022 2:22 pm

Fortunately, I do not experience v7 instability. My problem is incompleteness, we use v6 features in our network that are still to be finished in v7.
 
User avatar
Larsa
Forum Guru
Forum Guru
Topic Author
Posts: 1025
Joined: Sat Aug 29, 2015 7:40 pm
Location: The North Pole, Santa's Workshop

Re: Software life cycle and release management (re: auto upgrade)

Thu Jun 09, 2022 10:19 pm

You are advanced users with specific configurations, discussing them on a technical forum. The userbase in this forum is like 0.001% of real users. For those people, v7 works beautifully.
Don't get me wrong, we need your complaints and edge cases, they help to fix bugs, but believe me, when I say, there are tens of thousands of people using v7 and having no idea what issues you are talking about. They are rare. And thanks for helping us find them.

There are probably plenty of working devices after an upgrade from the [stable] and [long term] release channels, but this is beside the point. The topic is about when things do NOT work.

If MT truly cares about the consumer market and takes the complaints seriously in the links I provided, let me go back to the main topic which was about when new releases do NOT work and how the procedures can be strengthened to mitigate any future downtime.

Since it's a proven fact that the release channels [stable] and [long term] are not quality tested enough to be suitable for automatic updates or installed immediately. This is due to new severe bugs that almost always shows up after volunteers performed their own tests. This applies to both v6 and v7.

When it comes to professionals and semi-professionals (ie advanced SOHO users) most folks are by now painfully aware of theses shortcomings and waits for a distinct time to catch up any new potential bugs. However, as for the consumers this is a total different matter which also should be the first and main concerns of MT to remediate.

It's pretty clear that someone at MT is unaware of or has chosen to ignore this very obvious problem. However as MT completely depends on voluntary labor for its final quality test (and please notice: this is during or after a [stable] or [long term] is released), I think MT should make a more serious effort to try to do something about it.

Question: given this, is it feasible that MT might be willing to consider any (long-term) changes regarding the release management in cooperation with the people on this forum?

IMHO, if the release management is sorted out once for all, this will if anything, be a key factor in securing a smooth path for larger volumes in the consumer markets.
 
User avatar
Larsa
Forum Guru
Forum Guru
Topic Author
Posts: 1025
Joined: Sat Aug 29, 2015 7:40 pm
Location: The North Pole, Santa's Workshop

Re: Software life cycle and release management (re: auto upgrade)

Fri Jun 10, 2022 12:25 pm

Furthermore a well stated summary from @pe1chl in this reply: Well, I think most of the confusion comes from use of the word "stable"...
 
OlofL
Member Candidate
Member Candidate
Posts: 113
Joined: Mon Oct 12, 2015 2:37 pm

Re: Software life cycle and release management (re: auto upgrade)

Fri Jun 10, 2022 5:39 pm

In fact, there does not seem to be any channels - there are only different snapshots of the SAME channel of RouterOS releases. This is obvious and already claimed by Mikrotik staff in forum when 7.3 was released.

It would be more optimal to actually have two or more development channels where fixes and features are cherry picked into a more tested, and conservative release.


There have been so many breaking changes in recent times, and I don'd understand how normis and others at mikrotik still can claim "its normal for most users" etc. Mikrotik already have a bad reputation of bugs, and its not getting any better. They need to admit something has to change, and communicate a roadmap of solutions to their users IMO.
 
Sob
Forum Guru
Forum Guru
Posts: 9119
Joined: Mon Apr 20, 2009 9:11 pm

Re: Software life cycle and release management (re: auto upgrade)

Fri Jun 10, 2022 6:12 pm

But it's also true that it may not be so easy. Something suitable for automatic (security) updates would ideally be frozen and only changes would be those security updates and nothing else, no new features or anything. That would be as safe as possible. But it doesn't go well with current unlimited updates "forever" (which is killer feature, so don't even think about touching that). Such security-only branch would also need to be updated forever. But they'd need to keep adding new ones to support new hardware, so eventually they'd end up with tens of active branches (not too active, but still...).
 
User avatar
Larsa
Forum Guru
Forum Guru
Topic Author
Posts: 1025
Joined: Sat Aug 29, 2015 7:40 pm
Location: The North Pole, Santa's Workshop

Re: Software life cycle and release management (re: auto upgrade)

Fri Jun 10, 2022 6:43 pm

Yeah, that might be a possible outcome if you go to extremes in the opposite direction.

However, you are also able to achieve some kind of improvement just by leaving a release out for "general" testing (just like today) for a certain period of time and when the worst deviations have been corrected it can be released in a [long term tested] channel or equivalent.
 
User avatar
mrz
MikroTik Support
MikroTik Support
Posts: 7038
Joined: Wed Feb 07, 2007 12:45 pm
Location: Latvia
Contact:

Re: Software life cycle and release management (re: auto upgrade)

Fri Jun 10, 2022 6:50 pm

That is mostly what we are doing. When 7.3 or 7.4 will become stable enough it will move to long-term where only security and important fixes will be backported.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Software life cycle and release management (re: auto upgrade)

Fri Jun 10, 2022 6:52 pm

I hope also for v6.... no more new features, ok, but bugfixed.... at least security issues...
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 3169
Joined: Sun May 01, 2016 7:12 pm
Location: California

Re: Software life cycle and release management (re: auto upgrade)

Fri Jun 10, 2022 7:07 pm

Overall @pe1chl's description matches my outsider view of their release management process. I think the biggest issue is many customers may not be aware they need to follow an Internet forum if they want to learn about Mikrotik's roadmap or see if a bug is a "known issues". This should all be on some more official place. I mean https://mikrotik.com/thedude includes the phrase:
The Dude network monitor is a new application by MikroTik
show how much care they put into documenting/update things when they release software, which you'd think include the marketing materials on the website. And, on this singular example, how was the status of Dude communicated in the two-year push to get V7 out?

How they managed the v6 to v7 transition overall made two big mistakes:
1. not communicating enough with users via their website/official channels on what was state it's in (e.g. what's left to be done for "feature completeness")
2. bundling all needed "major improvements" into one thing (e.g. not breaking the new kernel driver model , cache-less routing engine, MBIM/LTE/IoT stuff, and BGP implementation as separate point-releases along the way to a v7).

That doesn't mean that V7 isn't working well – we use it production all the time. But I'm hopeful as the last bugs get fixed, more thought/care/time, and testing, can be done before just through a build over the wall.

Now on the specifics...
But it's also true that it may not be so easy. Something suitable for automatic (security) updates would ideally be frozen and only changes would be those security updates and nothing else, no new features or anything.
And that the missing choice. IMO, this is largely solved by treating "long-term" as an important "release management" decision, and one carefully considered. I don't think they put much care into what's "long-term" decision today – and they should.

Maybe some "Mikrotik recommended" version/channel to use, to simplify the decision making process for new users & with wide deployment easy to know what works/doesn't work (e.g. you'd only need "testing"/"stable"/etc if you needed a new feature, or something in your network changed require one)?
 
DarkNate
Forum Veteran
Forum Veteran
Posts: 998
Joined: Fri Jun 26, 2020 4:37 pm

Re: Software life cycle and release management (re: auto upgrade)

Thu Jun 16, 2022 12:12 am

Yes, I know. That's a topic for another thread and we do see many ways how we could improve.
My comment was only about the perceived instability of v7, but this forum is not a good measure. Most people are fine using auto-upgrade scripts.
At the very least @normis provide us with detailed changelogs for the firmware/routerboot/bootloader upgrades. We can avoid unnecessary upgrades.
 
User avatar
Larsa
Forum Guru
Forum Guru
Topic Author
Posts: 1025
Joined: Sat Aug 29, 2015 7:40 pm
Location: The North Pole, Santa's Workshop

Re: Software life cycle and release management (re: auto upgrade)

Tue Jul 05, 2022 8:06 pm

Regarding "v7.4rc1 is released!"

I have to admit that I really don't understand how the current release management process works and the underlying mechanisms behind the scene for this release.

Can someone with some insight please explain to me how MT reasons when they choose to push a test version as a new "Release Candidate" where there are 1) new functions added, 2) plenty of new fixes and 3) old flaws needed to be corrected, instead of just continuing to iterating in "Test channel" where it it belongs IMHO?

Note 1 to @MT: Please do not quick drop RC into Stable this time.

Note 2 to @MT: Please respect dedicated forum users who work completely free of charge to discuss bugs and fixes to test Ros thus move rather than delete posts you feel are off topic, is a sensitive subject or questions you want to avoid in general.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Software life cycle and release management (re: auto upgrade)

Tue Jul 05, 2022 8:51 pm

Larsa please stop create useless topic, with "redirect" inside, that point to another topic...

Who is online

Users browsing this forum: No registered users and 16 guests