You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
vNext: Make the abi-breaking-changes branch available on GitHub #169
I suggest you consider releasing the ABI breaking changes in a way that those who know what they are doing and want something that breaks ABI can get them. (but where anyone who doesn't explicitly download the ABI-breaking-changes branch wont have to worry) Or if its not possible to publish this stuff (either because there are also ABI changes in vcruntime, vcstartup, concrt or other non-open bits or because vcruntime, vcstartup, concrt or other non-open bits need recompiling with the new ABI-breaking code to pick up the new ABI) then at least consider publishing a comprenensive accurate list of the things that have/can be fixed but only by breaking ABI (so people know where the STL has bugs (or differences with the C++ standards and why they can't be fixed at this time)
Reacted by David Hunter, Rick V, Milan, Arkady Shapkin, MikeGitb, Kenny Kerr, Federico Ceccatto, Marcel Raad, Ky Waegel, Martin Hořeňovský and 10 more
I think the best thing (and I believe they already plan to do it) would be to put the code into a separate branch here on github, but not to ship it in any official form.
The worst possible outcome would be both version being shipped and supported for quite some time in parallel and people to start #ifdef ing their code based on which version the code is built against and incompatible binaries floating around. Imo that would be a disservice to the eco system.
All that aside: I guess one of the problems may be that those patches no longer apply cleanly to the current master version of the code.
Releasing it as a separate branch is the answer for sure. People who want to ship it can grab that branch and get a set of binaries out of it knowing full well those dlls are ABI-breaking compared to the official binaries.
Of course it may not be possible to publish the code at all if the ABI-breaking branch would require changes to the compiler or changes or recompilation of other non-open binaries like vcruntime, vcstartup or concrt (that said, it looks like concrt is the only runtime library component shipped with the compiler that actually links to and pulls in the STL so its entirely possible the answer to any problems with an ABI incompatible STL solved by making concrt open source, something that would be nice to see anyway given the circular dependency between concrt and STL)
ABI breaking STL could be released on vcpkg. To avoid conflicts with the built-in STL, the headers could be located under another folder, like so: #include <stl/string>
We do plan to get the binary-breaking branch integrated here eventually; we effectively must do so to achieve our goal of primarily developing the STL out of this repository. I would in no way refer to those plans as "releasing" so much as "developing", we still have quite a few ABI-breaking changes that need to be implemented on that branch.
MikeGitb (@MikeGitb)'s "I guess one of the problems may be that those patches no longer apply cleanly to the current master version of the code" could not be more spot on. We switched from TFS to Git a couple of years ago, and the ABI-breaking branch lies largely abandoned on our old TFS instance. There will be quite a bit of hand-porting involved, with the likely exception of the threading bits that were completely rewritten. We did pull in the <atomic> rewrite a few months ago, for example, and that wasn't too bad except for a few days spent to maintain the old ABI.
Are there any cases of abi-breaking changes that need to be made to vcruntime, vcstartup. concrt or other closed modules or any cases where abi-breaking changes require closed modules to be recompiled to work?
changed the title [-]Consider releasing the abi-breaking-changes branch[/-][+]vNext: Make the abi-breaking-changes branch available on GitHub[/+]on Oct 13, 2019
I recall some changes being made to vcruntime, in addition to a new "vcplatform" layer being added. I don't recall any ConcRT changes; we'll be able to drop the ConcRT link along with our XP targeting support. We'll need to rethink vcplatform as we reapply the changes to GitHub. Among other things, we clang-formatted the STL's sources in the meantime and changed other conventions, so all of the changes will need to be handled line-by-line.
It did drop XP targeting support (in the IDE, etc.) but unfortunately we can’t remove it from our DLLs, as that would damage any XP machines if the redist were to be installed there.
Would it make sense to create an entirely separate branch here on github that is just a copy-paste version of you internal TFS branch just to give it exposure to the community (there are tools that would preserve history, so you can actually see the diffs that where applied relative to the old master and not just compared to the current one). The actual integration work could then either be done in yet another branch, or by adding the changes directly into master but hidden behind appropriate #ifdefs.
Creating a separate branch also creates many process challenges. You'd probably only want breaking changes to go into this STL Next branch and all other changes to go to master. For every change to master, you need to cherry-pick the changes and merge them into the STL Next branch which can be problematic. Just before master is released you probably want to lock it down and only accept high priority bug fixes to it as final testing is done. At this point, the STL Next branch can take non-breaking changes. When master is released you now have a problem of merging STL Next back into master. There are many ways to deal with the myriad problems with managing multiple branches but all of them have many pain points. And not to mention the issue associated with managing separate CI systems for multiple branches.
It's not just a matter of creating an STL Next branch and going wild on it. That's a recipe for disaster.
To clarify: What I had in mind was not to backport anything to old stable. Rather to implement new features in old stable and merge them forward to vNext and only do abi breaking work on vNext. Also, old stable would stay the only official supported version and vNext would be hidden behind a /experimenral:abi2 flag or some such until it is "done" (and maybe the next version of VS is released).
But of course: If you can take the proper time and exclusively work on vNext that is even better.
That aaide, I know I'm repeating myself, but I want to point out again that I (and I guess many others) very much appreciate the fact that you people share your thoughts on these matters here in the open, even if they are speculative and things turn out differently.
Stephan T. Lavavej (@StephanTLavavej) My apologies, I guess my last post sounded really ungrateful, underestimating all the work you've all done. I've been with the MS compiler since VS 6.0(skipping VS 2002) and saw all the good and the bad. Since project rejuvenation stared long time ago I really saw how with each year not only the products, but the company transformed from disregarding(and even hating) open-source to loving it.
The design work behind the current stable ABI was vary brave(breaking the runtime to many satellite DLLs - api-ms-win-*.dll) in order to add new functionality and keep compatible ABI, although this ended up not really great and the idea behind it was to satisfy both regular users and corporate customers and I do appreciate all of that. I also saw how quickly you've reacted to my bug report about std::function capture in lambda and fix was implemented just a few days after the reporting. So by NO means I would want to suggest that you(all) are not addressing tech depts or not hearing the community.
Now back on the main topic a bit. :) As far as I understand, keeping ABI between major releases is to allow corporate customers to make more timely migrations to the new versions and that you have tremendous positive feedback about that. However I would also like to share my years of experience in large corporations regarding upgrades. All of the ones that I've been working for are still migrating slow. For example in my current company(40k+ people) are using VS 2019 v16.9.x, because it is still supported and they are now looking forward to the next Long-Term supported version in order to avoid updating as much as they can and while the upgrade will be easier, because of the ABI compatibility(they are not going to recompile some of the libs), this does not mean that if there were an ABI break, they would reconsider. That's because of ISO policies which require only supported software to be used with or without ABI compatibility. So while in case of ABI break, the work would've been a bit more, it was not going to be avoided.
I guess my point is that freezing an ABI for 10+ years in general would lead to positive testimonials(because they've saved some time recompiling, but the time for testing the product is still the same), it is not worth it. :) I know you don't have much saying regarding the matter and if it's up to you, you will break the ABI on every major release, but perhaps you can push for the company to make a bit different surveys regarding corporations upgrading to newer version(for example breakdown of the time spent on compilation and testing). My guts tell me that you may get surprised.
To clarify: What I had in mind was not to backport anything to old stable. Rather to implement new features in old stable and merge them forward to vNext and only do abi breaking work on vNext.
I see. That's effectively the same amount of work, though - we'd still have to ensure that features can work in both "universes". Some would be easy (those with few dependencies on existing code), some would be hard, but the amount of extra work would be basically independent of where the feature was originally developed.
Essentially the reason I'm so allergic to the idea of adding features to two diverging branches simultaneously is that it takes all of our effort and skill just to keep a single branch up to date with the Standard and always shipping at production quality. We have what I believe is the largest single team of Standard Library maintainers at a single company, working alongside an awesome group of contributors here, but our capacity is still finite and we have to be careful what we spend time on.
I (and I guess many others) very much appreciate the fact that you people share your thoughts on these matters here in the open, even if they are speculative and things turn out differently.
Since project rejuvenation stared long time ago I really saw how with each year not only the products, but the company transformed from disregarding(and even hating) open-source to loving it.
😻 😸
I also saw how quickly you've reacted to my bug report about std::function capture in lambda and fix was implemented just a few days after the reporting.
Yeah, that one was real interesting - JonCaves explained to me that it had always been a compiler bug, where the lambda's "secret" constructors (that the compiler uses to initially construct the object, but which should never be visible afterwards) were still visible to SFINAE, and the new changes in std::function constraints revealed this issue which had lurked for 10+ years!
For example in my current company(40k+ people) are using VS 2019 v16.9.x, because it is still supported and they are now looking forward to the next Long-Term supported version in order to avoid updating as much as they can and while the upgrade will be easier, because of the ABI compatibility(they are not going to recompile some of the libs), this does not mean that if there were an ABI break, they would reconsider.
This is very useful information, thanks for explaining. It's things like this that might help convince our bosses and boss-like entities!
perhaps you can push for the company to make a bit different surveys regarding corporations upgrading to newer version(for example breakdown of the time spent on compilation and testing). My guts tell me that you may get surprised.
Yeah, that's a good idea - we do such research from time to time, we could ask questions along those lines.
Reacted by Nikolay Baklicharov, Evgeny Vrublevsky, A. Jiang and Yan
Last 10 years we had a new major Visual Studio update roughly each 2 years. VS2022 is soon 2 years. Any news or rumors on the next major update? Is an announcement right after the corner? =)
Evgeny Vrublevsky (@vrubleg) We try to be very open about our plans for the STL in this repo, but there are a few things we can't ever comment on - unannounced releases and release dates are major examples.
Yeah, I understand it, it was rather a joke question while we are looking forward to official announcements. Thanks for your work and openness regarding to STL =)
Hello and sorry to bother you again Stephan T. Lavavej (@StephanTLavavej) :)
Hope you are doing well. I see that STL is in top shape for C++23. Thanks to you and your teammates. Keep up the good work!
Given all that, I'm wondering. C++26 is far away, C++23(at least on the STL side) is completed(the compiler teams seems to be lacking behind a lot this time, hopefully they will catch up soon).
So, do you have any plans to pause for a bit the feature work and open the doors on vNext? In general, do you have anything interesting to share at this moment? :)
Mostly out of curiosity: The STL release 17.10 contains an ABI break due to the constexpr mutex constructor (#4338). As far as I understood it, the new STL headers combined with old STL-DLLs cause crashes. But old headers with new DLLs work. So apparently (and according to the documentation) this kind of ABI break is accepted by Microsoft.
My question: How does vNext relate to this? Why was #4338 not a vNext issue? Do all issues tagged with vNext break the compatibility of old headers with newer DLLs (i.e. the other way around compared to #4338)?
"ABI breaking change" means a change that will cause existing compiled code that uses the STL (that is, compiled object code that needs to be linked with libcp* or msvcp* libraries) to either fail to link or to fail to run properly after its linked with the libcp* or msvcp* library version that contains the "ABI breaking change"
Yes, our binary compatibility is explicitly documented as one-way, allowing applications to use prebuilt third-party libraries, but not the other way around. We regularly exercise the requirement that the final link be performed by the newest toolset. constexpr mutex construction exercised the requirement that the VCRedist be from the newest toolset - we really ought to exercise that requirement more frequently, and I've made a note that we should do that for vNext.
v14 => vNext will be a total ABI break (like VS 2008 => 2010 => 2012 => 2013 => 2015), including renaming all of the versioned DLLs. No OBJ/LIB mixing will be possible, and DLL/EXE mixing will work only if the DLL interfaces are ABI-stable (e.g. COM, or completely extern "C" with no trace of STL types, etc.). We're going to change the representations of tons of types, and remove/change a ton of STL DLL exports. vNext is how C++ development is ordinarily done - it's preserving binary compatibility that is extremely difficult and requires unusual development practices that very few people outside our maintainer/contributor team are familiar with.
Reacted by h-vetinari, Sedenion, Antoine Pitrou, Suraj and Mathias Berchtold
Activity
MikeGitb commentedon Oct 11, 2019
I think the best thing (and I believe they already plan to do it) would be to put the code into a separate branch here on github, but not to ship it in any official form.
The worst possible outcome would be both version being shipped and supported for quite some time in parallel and people to start #ifdef ing their code based on which version the code is built against and incompatible binaries floating around. Imo that would be a disservice to the eco system.
All that aside: I guess one of the problems may be that those patches no longer apply cleanly to the current master version of the code.
jonwil commentedon Oct 11, 2019
Releasing it as a separate branch is the answer for sure. People who want to ship it can grab that branch and get a set of binaries out of it knowing full well those dlls are ABI-breaking compared to the official binaries.
Of course it may not be possible to publish the code at all if the ABI-breaking branch would require changes to the compiler or changes or recompilation of other non-open binaries like vcruntime, vcstartup or concrt (that said, it looks like concrt is the only runtime library component shipped with the compiler that actually links to and pulls in the STL so its entirely possible the answer to any problems with an ABI incompatible STL solved by making concrt open source, something that would be nice to see anyway given the circular dependency between concrt and STL)
sylveon commentedon Oct 11, 2019
ABI breaking STL could be released on vcpkg. To avoid conflicts with the built-in STL, the headers could be located under another folder, like so:
#include <stl/string>CaseyCarter commentedon Oct 11, 2019
We do plan to get the binary-breaking branch integrated here eventually; we effectively must do so to achieve our goal of primarily developing the STL out of this repository. I would in no way refer to those plans as "releasing" so much as "developing", we still have quite a few ABI-breaking changes that need to be implemented on that branch.
MikeGitb (@MikeGitb)'s "I guess one of the problems may be that those patches no longer apply cleanly to the current master version of the code" could not be more spot on. We switched from TFS to Git a couple of years ago, and the ABI-breaking branch lies largely abandoned on our old TFS instance. There will be quite a bit of hand-porting involved, with the likely exception of the threading bits that were completely rewritten. We did pull in the
<atomic>rewrite a few months ago, for example, and that wasn't too bad except for a few days spent to maintain the old ABI.jonwil commentedon Oct 11, 2019
Are there any cases of abi-breaking changes that need to be made to vcruntime, vcstartup. concrt or other closed modules or any cases where abi-breaking changes require closed modules to be recompiled to work?
[-]Consider releasing the abi-breaking-changes branch[/-][+]vNext: Make the abi-breaking-changes branch available on GitHub[/+]StephanTLavavej commentedon Oct 13, 2019
I recall some changes being made to vcruntime, in addition to a new "vcplatform" layer being added. I don't recall any ConcRT changes; we'll be able to drop the ConcRT link along with our XP targeting support. We'll need to rethink vcplatform as we reapply the changes to GitHub. Among other things, we clang-formatted the STL's sources in the meantime and changed other conventions, so all of the changes will need to be handled line-by-line.
jonwil commentedon Oct 13, 2019
I thought VS 2019 dropped XP support (or are there enough users out there that still need XP support that bringing it back to VS 2019 is necessary?)
sylveon commentedon Oct 13, 2019
Probably just would break the ABI to drop all the XP support bits
StephanTLavavej commentedon Oct 13, 2019
It did drop XP targeting support (in the IDE, etc.) but unfortunately we can’t remove it from our DLLs, as that would damage any XP machines if the redist were to be installed there.
jonwil commentedon Oct 14, 2019
I forgot that the 2019 dlls are also the dlls for versions of VS where the XP targeting support is still actively supported.
MikeGitb commentedon Mar 3, 2020
Would it make sense to create an entirely separate branch here on github that is just a copy-paste version of you internal TFS branch just to give it exposure to the community (there are tools that would preserve history, so you can actually see the diffs that where applied relative to the old master and not just compared to the current one). The actual integration work could then either be done in yet another branch, or by adding the changes directly into master but hidden behind appropriate
#ifdefs.SunnyWar commentedon Mar 3, 2020
Creating a separate branch also creates many process challenges. You'd probably only want breaking changes to go into this STL Next branch and all other changes to go to master. For every change to master, you need to cherry-pick the changes and merge them into the STL Next branch which can be problematic. Just before master is released you probably want to lock it down and only accept high priority bug fixes to it as final testing is done. At this point, the STL Next branch can take non-breaking changes. When master is released you now have a problem of merging STL Next back into master. There are many ways to deal with the myriad problems with managing multiple branches but all of them have many pain points. And not to mention the issue associated with managing separate CI systems for multiple branches.
It's not just a matter of creating an STL Next branch and going wild on it. That's a recipe for disaster.
33 remaining items
MikeGitb commentedon May 17, 2022
To clarify: What I had in mind was not to backport anything to old stable. Rather to implement new features in old stable and merge them forward to vNext and only do abi breaking work on vNext. Also, old stable would stay the only official supported version and vNext would be hidden behind a /experimenral:abi2 flag or some such until it is "done" (and maybe the next version of VS is released).
But of course: If you can take the proper time and exclusively work on vNext that is even better.
That aaide, I know I'm repeating myself, but I want to point out again that I (and I guess many others) very much appreciate the fact that you people share your thoughts on these matters here in the open, even if they are speculative and things turn out differently.
TheStormN commentedon May 17, 2022
Stephan T. Lavavej (@StephanTLavavej) My apologies, I guess my last post sounded really ungrateful, underestimating all the work you've all done. I've been with the MS compiler since VS 6.0(skipping VS 2002) and saw all the good and the bad. Since project rejuvenation stared long time ago I really saw how with each year not only the products, but the company transformed from disregarding(and even hating) open-source to loving it.
The design work behind the current stable ABI was vary brave(breaking the runtime to many satellite DLLs - api-ms-win-*.dll) in order to add new functionality and keep compatible ABI, although this ended up not really great and the idea behind it was to satisfy both regular users and corporate customers and I do appreciate all of that. I also saw how quickly you've reacted to my bug report about
std::functioncapture in lambda and fix was implemented just a few days after the reporting. So by NO means I would want to suggest that you(all) are not addressing tech depts or not hearing the community.Now back on the main topic a bit. :) As far as I understand, keeping ABI between major releases is to allow corporate customers to make more timely migrations to the new versions and that you have tremendous positive feedback about that. However I would also like to share my years of experience in large corporations regarding upgrades. All of the ones that I've been working for are still migrating slow. For example in my current company(40k+ people) are using VS 2019 v16.9.x, because it is still supported and they are now looking forward to the next Long-Term supported version in order to avoid updating as much as they can and while the upgrade will be easier, because of the ABI compatibility(they are not going to recompile some of the libs), this does not mean that if there were an ABI break, they would reconsider. That's because of ISO policies which require only supported software to be used with or without ABI compatibility. So while in case of ABI break, the work would've been a bit more, it was not going to be avoided.
I guess my point is that freezing an ABI for 10+ years in general would lead to positive testimonials(because they've saved some time recompiling, but the time for testing the product is still the same), it is not worth it. :) I know you don't have much saying regarding the matter and if it's up to you, you will break the ABI on every major release, but perhaps you can push for the company to make a bit different surveys regarding corporations upgrading to newer version(for example breakdown of the time spent on compilation and testing). My guts tell me that you may get surprised.
StephanTLavavej commentedon May 18, 2022
MikeGitb (@MikeGitb)
I see. That's effectively the same amount of work, though - we'd still have to ensure that features can work in both "universes". Some would be easy (those with few dependencies on existing code), some would be hard, but the amount of extra work would be basically independent of where the feature was originally developed.
Essentially the reason I'm so allergic to the idea of adding features to two diverging branches simultaneously is that it takes all of our effort and skill just to keep a single branch up to date with the Standard and always shipping at production quality. We have what I believe is the largest single team of Standard Library maintainers at a single company, working alongside an awesome group of contributors here, but our capacity is still finite and we have to be careful what we spend time on.
😻
Nikolay Baklicharov (@TheStormN)
😻 😸
Yeah, that one was real interesting - JonCaves explained to me that it had always been a compiler bug, where the lambda's "secret" constructors (that the compiler uses to initially construct the object, but which should never be visible afterwards) were still visible to SFINAE, and the new changes in
std::functionconstraints revealed this issue which had lurked for 10+ years!This is very useful information, thanks for explaining. It's things like this that might help convince our bosses and boss-like entities!
Yeah, that's a good idea - we do such research from time to time, we could ask questions along those lines.
vrubleg commentedon Aug 30, 2023
Last 10 years we had a new major Visual Studio update roughly each 2 years. VS2022 is soon 2 years. Any news or rumors on the next major update? Is an announcement right after the corner? =)
StephanTLavavej commentedon Sep 2, 2023
Evgeny Vrublevsky (@vrubleg) We try to be very open about our plans for the STL in this repo, but there are a few things we can't ever comment on - unannounced releases and release dates are major examples.
vrubleg commentedon Sep 3, 2023
Yeah, I understand it, it was rather a joke question while we are looking forward to official announcements. Thanks for your work and openness regarding to STL =)
TheStormN commentedon Jan 28, 2024
Hello and sorry to bother you again Stephan T. Lavavej (@StephanTLavavej) :)
Hope you are doing well. I see that STL is in top shape for C++23. Thanks to you and your teammates. Keep up the good work!
Given all that, I'm wondering. C++26 is far away, C++23(at least on the STL side) is completed(the compiler teams seems to be lacking behind a lot this time, hopefully they will catch up soon).
So, do you have any plans to pause for a bit the feature work and open the doors on vNext? In general, do you have anything interesting to share at this moment? :)
cpplearner commentedon Jan 28, 2024
Please upvote https://developercommunity.visualstudio.com/t/Support-for-breaking-the-STL-ABI/1473763
StephanTLavavej commentedon Jan 29, 2024
We're trying to get approval, but nothing to share at the moment.
Sedeniono commentedon Aug 6, 2024
Mostly out of curiosity: The STL release 17.10 contains an ABI break due to the constexpr mutex constructor (#4338). As far as I understood it, the new STL headers combined with old STL-DLLs cause crashes. But old headers with new DLLs work. So apparently (and according to the documentation) this kind of ABI break is accepted by Microsoft.
My question: How does vNext relate to this? Why was #4338 not a vNext issue? Do all issues tagged with vNext break the compatibility of old headers with newer DLLs (i.e. the other way around compared to #4338)?
TheStormN commentedon Aug 6, 2024
vNext breaks it in such a way that already compiled binaries and static libraries will become unusable. Nothing more, nothing less.
jonwil commentedon Aug 6, 2024
"ABI breaking change" means a change that will cause existing compiled code that uses the STL (that is, compiled object code that needs to be linked with libcp* or msvcp* libraries) to either fail to link or to fail to run properly after its linked with the libcp* or msvcp* library version that contains the "ABI breaking change"
StephanTLavavej commentedon Aug 6, 2024
Yes, our binary compatibility is explicitly documented as one-way, allowing applications to use prebuilt third-party libraries, but not the other way around. We regularly exercise the requirement that the final link be performed by the newest toolset.
constexpr mutexconstruction exercised the requirement that the VCRedist be from the newest toolset - we really ought to exercise that requirement more frequently, and I've made a note that we should do that for vNext.v14 => vNext will be a total ABI break (like VS 2008 => 2010 => 2012 => 2013 => 2015), including renaming all of the versioned DLLs. No OBJ/LIB mixing will be possible, and DLL/EXE mixing will work only if the DLL interfaces are ABI-stable (e.g. COM, or completely
extern "C"with no trace of STL types, etc.). We're going to change the representations of tons of types, and remove/change a ton of STL DLL exports. vNext is how C++ development is ordinarily done - it's preserving binary compatibility that is extremely difficult and requires unusual development practices that very few people outside our maintainer/contributor team are familiar with.Address review feedback on IDL invariants comment
Scope the IDL invariants to the levels where they hold