Release cadence #5223
Replies: 4 comments 3 replies
-
|
The release cadence is for me not a fixed goal, but more like a guideline to help us decide when to release something. My stating minor versions 4 times a year (which in reality have been more like 2 times a year for the last 3-4 years), is more like saying two things:
We have over time reduced the backlog, I believe we started at around 500 issues, and now we are down to around 250. So we're actually eating them slowly, but consistently - which is good. A major version to me is :
Other new features can simply go into a minor release, together with non-critical bugs. Critical bugs goes into a patch release, but we try to avoid that. Now, if it all was this easy, - it is not ------ read on: Bugs are of two categories:
One important addition is that the boundary between the two is not always clear. What looks like an implementation bug can reveal an ambiguity or gap in the specification. In that sense, many actual bugs are partly both: the code made a choice where the spec was unclear. Although most of our bugs is category 1, we do have several of category 2. A category 2 bug may break the user, but the user is actually having a faulty test. They may have worked around it, or adjusted their surroundings to match the perceived functionality we have. To us, we need to figure out: Is this a breaking change that require us to add it to a major release? Or should we state that this is a specification/functionality bug, and we simply fix it as a minor version change? The last one I noted of this case is the one @manfred-brands raised as #5221 . We allow "useless combinations" as described there, and when we fix this (in a minor version update), the user who have done such stuff will have their code breaking. (Currently the analyzer will catch a lot of these, so people have been warned - in this case). Imho, this is a case where I believe we should announce the fix, but keep it in a minor version - if doable. Mostly....... Users depends on behavior, not intent, so if the analyzer haven't warned them, and the design is non-obvious, then we might have to make it into a breaking change, thus a major version. |
Beta Was this translation helpful? Give feedback.
-
|
Interesting. If bugs in category 1 last long enough, they tend to become category 2 and fixing them may be seen as a breaking change in the eyes of users. There's no way to solve that problem completely except to fix them as fast as possible but I think you can mitigate it by being very clear about what you guarantee, i.e. something less than full binary compatibility with consistent semantics. I've tried to do that for the engine and especially for my GUI. In the GUI, for example, we reserve the right to change how the UI is used so long as the same functionality is available. That is, a dropdown menu may become a dialog. This is harder to do for the console runner because of user scripts but I've made some attempt anyway. Of course restricting what you guarantee doesn't mean you won't at least try to do more than that for compatibility but I feel like it takes a bit of the pressure off. With respect to time versus number of changes, I'm in full agreement. Rather than four times a year, I'd say we expect to release two to four times. If we got to six months with only 15 changes, maybe we would release anyway, for example. For the major releases, it could be every year or two depending on how technical debt builds up. In the case of the engine/console I'm currently trying to avoid 3.x releases and focus on 4.0. But once 4.0 is out, I think this would be a good framework, possibly with some of the numbers changed since. One question... are we still generally agreed that dropping a platform that's no longer supported by Microsoft is "non-breaking" from our point of view? |
Beta Was this translation helpful? Give feedback.
-
|
At our work we have 100s of repositories that all use semver ranges for referenced nuget packages, like "[3.14.0, 4.0.0)". As upgrading to a new major version is a lot of work we need to be convinced that moving to a new version is worth it. Having said that, I'm against updating a major version for non-breaking changes. So if we can introduce features or "correct" the API a minor version increase is preferred. Those kind 2 changes won't affect
When do we need a new major version.
Back to where this started. I'm happy to not ever need major versions if we can introduce features without breaking things. |
Beta Was this translation helpful? Give feedback.
-
For the "useless" variations I noticed that the analyzer picks up several of them. If we regard that as a warning turned to error - we should be "allowed" to keep the change in same major. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Based on comments in #4824 , copied below, we should clarify how we want our release cadence to be going forward, and if/how we're going to announce what we plan to do.
The discussion started here #4824 (comment) and evolved.
Copied the relevant comments here:
OsirisTerje
14 hours ago
Member
@manfred-brands You're gold ! ❤️
That attribute is a life-saver. I was not at all aware of it, not a clue! Awesome!
Then we just proceed.
I think marking TestDelegate Obsolete is good, because we can then use that to point people on how to resolve it.
It also means we can move them gradually over to the new functionality, and perhaps we can then even implment this in the 4.X series, and use that as a way forward, and in 5.X remove the TestDelegate - or we could let it be for a while longer and remove it in 6.X, as long as it doesn't disturb us.
manfred-brands
manfred-brands commented 14 hours ago
manfred-brands
14 hours ago
Member
I wonder what would be the benefit of retaining TestDelegate at all if you're talking about a major release where breaking changes are to be expected.
@CharliePoole There is expected with warning and out of the blue.
We marking it obsolete in 5.X and remove it in 6.X.
We could mark it obsolete in 4.6 and remove it in 5.X except for the one incompatible changes as mentioned in the PR comments.
stevenaw
stevenaw commented 12 hours ago
stevenaw
12 hours ago
Member
Author
Nice thinking about OverloadResolutionPriorityAttribute @manfred-brands
I notice that it was introduced in .NET 9. Would that also work for .NET Framework consumers?
manfred-brands
manfred-brands commented 10 hours ago
manfred-brands
10 hours ago
Member
I notice that it was introduced in .NET 9. Would that also work for .NET Framework consumers?
It is purely a compiler feature. As long as they use an SDK for .net 9 or later. But as we already use C#14 extension methods we require the .NET 10 SDK anyhow
CharliePoole
CharliePoole commented 6 hours ago
CharliePoole
6 hours ago · edited by CharliePoole
Member
@manfred-brands OK, if this is the general view of the team about major releases then that makes sense.
One reservation...
IMO, such a conservative approach is more appropriate for a project with frequent major releases - i.e. every year or two or less. NUnit is (has been) a project that does major releases very infrequently. Version 4.0 was released 23 years after version 1.0. So maybe it's a good idea to discuss when we expect to see 5.0, 6.0, etc. in general terms.
I think it would be OK to change, but it should be talked about. Some projects do an annual major release or the equivalent of one if they are not using Semver. For a project that works that way and also stops doing maintenance fixes on the previous year's release, then I agree that a very conservative approach to breaking is needed.
FWIW in the 4.0 Engine, we started out with a view that many breaking changes should be gotten out of the way as quickly as possible. We've been very open about that but even so I'm sure some users will stick with 3.x for years after 4.0 is released - exactly as happened in moving from 2.0 to 4.0.
So bottom line... are we thinking of 5.0 this year, next year or when? Same question for 6.0...
UPDATE: Obviously I'm broadening the discussion way beyond this one issue. Maybe we should discuss elsewhere. :-)
OsirisTerje
OsirisTerje commented 5 hours ago
OsirisTerje
5 hours ago
Member
We have started to try to start a release when we have more than 20 closed issues. Normally that has been minor releases. Also, that we should try to release approx 4 times a year. Version 4.5 had 40 issues, 4.4 had 26, 4.3 had 29, all with approx 6 months in between.
I think we can go with the same with the major releases, except not that cadence, perhaps once a year, but it should also be above a certain level of issues resolved, or some major changes we find we need to push out.
With todays pace in development, I think we should try to keep ut.
And, it's fun!
CharliePoole
CharliePoole commented 4 hours ago
CharliePoole
4 hours ago
Member
@OsirisTerje That's a reasonable decision to make but I think you have to address how to make a transition from one major release every 8 to 10 years to a more frequent cycle.
If you were to do 5.0 this year, that's three years since 4.0, so shorter than previous but not as short as annual. So maybe 5.0 needs slightly different criteria from 6.0, which would hypothetically come in 2027. Maybe we announce this new approach at the same time as you do the 5.0 release or even with the 4.6 release.
If we apply the same approach to the engine and console, 4.0 will be coming out this year (I hope), 11 years after 3.0. However, all the breaking changes have been announced widely and some (but not all) have been deprecated in 3.X. So for that project a different approach may be called for... maybe a final 3.X release with added deprecation of features missing in 4.0.
The kind of announcement I'm talking about might be something like...
"In addition to this new major release of the NUnit (framework | console and engine | adapter), we wanted to let you know that we plan to increase the pace of new major releases so as to introduce new features more rapidly. The next major release of the (framework | engine | etc.), version X.X is anticipated in MONTH, YEAR."
It could be less specific if we don't know when it's expected, of course.
Beta Was this translation helpful? Give feedback.
All reactions