Repository navigation
Closed
Labels
area/release-engIssues or PRs related to the Release Engineering subprojectIssues or PRs related to the Release Engineering subprojectarea/securitykind/cleanupCategorizes issue or PR as related to cleaning up code, process, or technical debt.Categorizes issue or PR as related to cleaning up code, process, or technical debt.priority/important-soonMust be staffed and worked on either currently, or very soon, ideally in time for the next release.Must be staffed and worked on either currently, or very soon, ideally in time for the next release.sig/releaseCategorizes an issue or PR as relevant to SIG Release.Categorizes an issue or PR as relevant to SIG Release.sig/testingCategorizes an issue or PR as relevant to SIG Testing.Categorizes an issue or PR as relevant to SIG Testing.
Description
- it's tough to maintain 2 things in parallel and keep them in sync
- with the switch to go modules/cache, most of the reasons why bazel was used (speed up compilations) are gone
- gets in the way in multiple ways (example, update of golang versions)
- etc...
Reacted by Mo Khan, Tomasz Klosinski, Bart Smykla, Stephen Augustus, Sascha Grunert, Y, Manuel Alejandro de Brito Fontes, Carlos Tadeu Panato Junior, Rob K, pablochacin and 138 more
Reacted by Jens Fosgerau, Taahir Ahmed, janluk, Pablo Moncada, Bradley Weston, mishudark, Marek Siarkowicz, Michael Galati, Edward McFarlane, Alex Kabakaev and 4 more
Reacted by Nikolaos Moraitis, Verónica López, Tim Meusel, Tomáš Nožička, Christian Richter, Claudio Fontana, scuzhanglei, Joyce Kung, Maël Valais, LBJ and 3 more
Reacted by Ross Georgiev, Joel Speed, Damien Mathieu, Goutham Veeramachaneni, Peeyush Gupta, Nikolaos Moraitis, Verónica López, Dor Kleiman, Patrik Cyvoct, Angus Lees and 13 more
Reacted by Sascha Grunert, Edward Vielmetti, Damien Mathieu, Nikolaos Moraitis, Verónica López, Jintao Zhang, Allison D, Tim Meusel, Tomáš Nožička, Christian Richter and 6 more
Reacted by Vijay Dhama, Adel Zaalouk, Jeff Grafton, Pieter Lange, mishudark, Ahmad Nurus S., Elana Hashman, Ilya Dmitrichenko, William Zhang, zhlhahaha and 4 more
Metadata
Metadata
Assignees
Labels
area/release-engIssues or PRs related to the Release Engineering subprojectIssues or PRs related to the Release Engineering subprojectarea/securitykind/cleanupCategorizes issue or PR as related to cleaning up code, process, or technical debt.Categorizes issue or PR as related to cleaning up code, process, or technical debt.priority/important-soonMust be staffed and worked on either currently, or very soon, ideally in time for the next release.Must be staffed and worked on either currently, or very soon, ideally in time for the next release.sig/releaseCategorizes an issue or PR as relevant to SIG Release.Categorizes an issue or PR as relevant to SIG Release.sig/testingCategorizes an issue or PR as relevant to SIG Testing.Categorizes an issue or PR as relevant to SIG Testing.
Activity
cblecker commentedon Feb 25, 2020
/sig release
+1 from me
justaugustus commentedon Feb 25, 2020
area: everyone
/area build-infra release-eng
/sig release testing
cc: @kubernetes/release-engineering
k8s-ci-robot commentedon Feb 25, 2020
@justaugustus: The label(s)
area/build-infracannot be applied, because the repository doesn't have themDetails
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
liggitt commentedon Feb 26, 2020
Bazel-related/specific issues:
justaugustus commentedon Feb 26, 2020
+1000000000
We've had a bunch of Slack threads about this, so thank you @dims for actually turning this into an issue.
Here's the latest thread: https://kubernetes.slack.com/archives/C2C40FMNF/p1582673341098700?thread_ts=1582668769.091500&cid=C2C40FMNF
The #bazel channel is filled with questions like, "How do I do this thing that should be moderately simple to do if it wasn't wrapped in bazel?"
This is a huge maintainability burden, with our limited set of bazel experts being some of the most bandwidth-constrained contributors in the project.
@kubernetes/release-engineering is on board to help tear this down.
Just let us know which threads we need to start pulling on to make this happen.
SataQiu commentedon Feb 26, 2020
+1 👍
bartsmykla commentedon Feb 26, 2020
Big +1 from me
saschagrunert commentedon Feb 26, 2020
👍 from me as well, happy to help with the transition
66 remaining items
saschagrunert commentedon Sep 23, 2020
/remove-lifecycle stale
MonzElmasry commentedon Oct 25, 2020
Hey 👋 Bug Triage here. Wanted to follow up on the status of this issue as we're approaching code freeze on 12.11.2020. This issue is tagged for 1.20, is it still planned for this release?
justaugustus commentedon Nov 12, 2020
This is unfunded for v1.20.
I'm dropping the milestone, but let's circle up and see what can be done for 2021.
/milestone clear
q3k commentedon Jan 19, 2021
-1 on this from us (Monogon). We use Bazel internally within our codebase to build Kubernetes (as libraries), and being able to just pull in an external repository (with only some minor patches to fix Starlark machinery paths) is crucial for us. I can't imagine working with Kubernetes as a dependency if it required running a bunch of impure scripts before calling
go build.If pulling in Kubernetes via Gazelle continues to function we can work with that, otherwise we'll have to maintain our own overlay of Bazel BUILDfiles.
I understand the various pain points working with Bazel sometimes - but for large scale projects this still seems to be the best build system we have. No other system provides this level of granular caching (for incremental builds) and hermeticity (for reproducible builds) simultaneously.
BenTheElder commentedon Feb 4, 2021
After much discussion, attempting to move forward on this. Please see the KEP kubernetes/enhancements#2421, for the reasons outlined there.
/assign
The Kubernetes project does not support importing this repo directly as a dependency, bazel or otherwise. If you wish to import a staging repo like "client-go" the supported version is at github.com/kubernetes/client-go, which does not and has never included BUILD files or required any particular build system, just golang sources consumable from your preferred build solution as a normal go package.
Other packages are not supported as dependencies.
so:
It will function as much as it is supported today, and in fact because we check in generated code etc. so that people can build without our make / bazel etc., it will be trivial to wrap. None of our libraries require using any of our build scripts, only our binary artifacts / docker images.
If you're depending on Kubernetes/Kubernetes to get release artifacts, I highly suggest switching to consume our official release builds instead of sources.
rules_k8sfor example consumeskubectlbinaries, not kubernetes/kubernetes to source build it.FWIW, we have ~equally reproducible builds under make (which is getting pretty trivial for go binaries, essentially in modern go you just need to set
-trimpathand pin dependencies withgo.sum/ GOPROXY, orvendor(we use both)).We also see that
go build/go testhas pretty excellent caching now, and that we need to exclude our large docker images (which go would not cache for us, but perhaps buildx which we are migrating to for other reasons) from the bazel cache to avoid net performance losses (these rarely cache well when developing kubernetes, binary source changes will invalidate them anyhow, and they're quite large).You can see more on this in kubernetes/enhancements#2421, maintaining both systems is not free, and bazel does not map particularly well to this project
q3k commentedon Feb 4, 2021
We can't, as we need a fully static build of Kubernetes. We also heavily patch it to remove unnecessary (to us) functionality that's not gated behind build tags, the subset of which is specific enough for our usecase that attempting to upstream such buildtags would be at least seen as uncourteous. Notably, we remove a sizeable amount of in-tree volume providers (that we know aren't compatible with our product, anyway) and nearly all in-tree legacy cloud providers. We also always carry a number of fixes pending upstreaming.
We know it's unsupported and we are more than ready to carry the burden of occasionally unbreaking things and running our own QA. Additionally, with Bazel, the line between building from our proejct and building from the k8s.io/kubernetes source itself is quite blurred - it's
bazel build @io_k8s_kubernetes//cmd/kubeletin both cases. That's why we appreciate the Bazel build system in Kubernetes so much, as we can plainly pull in all of Kubernetes into our main build graph, without having to have intermediary release processes or patched source trees. The same build graph in which we have all of our dependencies, including non-Go code, complex code generation, even the build toolchains themselves.We also need to be able to build from source that is not within the supported client libraries, eg. to build a hyperkube-like binary containing core control plane services, our own wrapped kubelet, etc.
Yes, I will review the KEP.
BenTheElder commentedon Feb 4, 2021
At this point it sounds like you're maintaining a fork. Your fork can maintain a build with bazel then as well, without us carrying this upstream, just like these other patches, and other upstream developed-with-
go builddependencies?And in fact this way you can customize the build as much as you want without colliding with what we do upstream. You can just ignore the makefiles etc.
Similarly, we stopped maintaining an upstream build for hyperkube, because the costs outweighed the limited usage and the ability to do this downstream for anyone that wants it / lack of these consumers participating in the ongoing costs.
q3k commentedon Feb 4, 2021
Not a true fork - we continue to track upstream as the source of truth, rebasing our changes on top at buildtime. We've also successfully consistently decreased the amount of patches we apply as we continue to upstream our changes. In an ideal world, we would just follow upstream, we have no practical interest in maintaining a fork other than the technical reasons to do that.
If we have to maintain a Bazel compatibility patchset, we will. But I know we're not the only organization that imports Kubernetes into their buildsystem this way, so it's a shame that this will now basically have to fall onto the community, or will have to be repeatedly done by multiple organizations independently.
Delete Kubernetes description from users.md
Delete Kubernetes description from users.md