Skip to content

Please remove bazel based build infrastructure #88553

Closed

Description

@dims
  • it's tough to maintain 2 things in parallel and keep them in sync
  • with the switch to go modules/cache, most of the reasons why bazel was used (speed up compilations) are gone
  • gets in the way in multiple ways (example, update of golang versions)
  • etc...

Activity

  1. added
    kind/bugCategorizes issue or PR as related to a bug.
    on Feb 25, 2020
  2. added
    needs-sigIndicates an issue or PR lacks a `sig/foo` label and requires one.
    on Feb 25, 2020
  3. cblecker commented on Feb 25, 2020

    @cblecker
    Member

    /sig release

    +1 from me

  4. added
    sig/releaseCategorizes an issue or PR as relevant to SIG Release.
    and removed
    needs-sigIndicates an issue or PR lacks a `sig/foo` label and requires one.
    on Feb 25, 2020
  5. justaugustus commented on Feb 25, 2020

    @justaugustus
    Member

    area: everyone
    /area build-infra release-eng
    /sig release testing
    cc: @kubernetes/release-engineering

  6. k8s-ci-robot commented on Feb 25, 2020

    @k8s-ci-robot
    Contributor

    @justaugustus: The label(s) area/build-infra cannot be applied, because the repository doesn't have them

    Details

    In response to this:

    area: everyone
    /area build-infra release-eng
    /sig release testing
    cc: @kubernetes/release-engineering

    Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

  7. added this to the v1.19 milestone on Feb 25, 2020
  8. justaugustus commented on Feb 26, 2020

    @justaugustus
    Member

    +1000000000
    We've had a bunch of Slack threads about this, so thank you @dims for actually turning this into an issue.

    Here's the latest thread: https://kubernetes.slack.com/archives/C2C40FMNF/p1582673341098700?thread_ts=1582668769.091500&cid=C2C40FMNF

    The #bazel channel is filled with questions like, "How do I do this thing that should be moderately simple to do if it wasn't wrapped in bazel?"

    This is a huge maintainability burden, with our limited set of bazel experts being some of the most bandwidth-constrained contributors in the project.

    @kubernetes/release-engineering is on board to help tear this down.
    Just let us know which threads we need to start pulling on to make this happen.

  9. SataQiu commented on Feb 26, 2020

    @SataQiu
    Member

    +1 👍

  10. bartsmykla commented on Feb 26, 2020

    @bartsmykla

    Big +1 from me

  11. saschagrunert commented on Feb 26, 2020

    @saschagrunert
    Member

    👍 from me as well, happy to help with the transition

  12. 66 remaining items

  13. added
    lifecycle/staleDenotes an issue or PR has remained open with no activity and has become stale.
    on Sep 22, 2020
  14. saschagrunert commented on Sep 23, 2020

    @saschagrunert
    Member

    /remove-lifecycle stale

  15. removed
    lifecycle/staleDenotes an issue or PR has remained open with no activity and has become stale.
    on Sep 23, 2020
  16. MonzElmasry commented on Oct 25, 2020

    @MonzElmasry

    Hey 👋 Bug Triage here. Wanted to follow up on the status of this issue as we're approaching code freeze on 12.11.2020. This issue is tagged for 1.20, is it still planned for this release?

  17. justaugustus commented on Nov 12, 2020

    @justaugustus
    Member

    This is unfunded for v1.20.
    I'm dropping the milestone, but let's circle up and see what can be done for 2021.

    /milestone clear

  18. removed this from the v1.20 milestone on Nov 12, 2020
  19. q3k commented on Jan 19, 2021

    @q3k

    -1 on this from us (Monogon). We use Bazel internally within our codebase to build Kubernetes (as libraries), and being able to just pull in an external repository (with only some minor patches to fix Starlark machinery paths) is crucial for us. I can't imagine working with Kubernetes as a dependency if it required running a bunch of impure scripts before calling go build.

    If pulling in Kubernetes via Gazelle continues to function we can work with that, otherwise we'll have to maintain our own overlay of Bazel BUILDfiles.

    I understand the various pain points working with Bazel sometimes - but for large scale projects this still seems to be the best build system we have. No other system provides this level of granular caching (for incremental builds) and hermeticity (for reproducible builds) simultaneously.

  20. BenTheElder commented on Feb 4, 2021

    @BenTheElder
    Member

    After much discussion, attempting to move forward on this. Please see the KEP kubernetes/enhancements#2421, for the reasons outlined there.

    /assign


    -1 on this from us (Monogon). We use Bazel internally within our codebase to build Kubernetes (as libraries), and being able to just pull in an external repository (with only some minor patches to fix Starlark machinery paths) is crucial for us. I can't imagine working with Kubernetes as a dependency if it required running a bunch of impure scripts before calling go build.

    The Kubernetes project does not support importing this repo directly as a dependency, bazel or otherwise. If you wish to import a staging repo like "client-go" the supported version is at github.com/kubernetes/client-go, which does not and has never included BUILD files or required any particular build system, just golang sources consumable from your preferred build solution as a normal go package.

    Other packages are not supported as dependencies.

    so:

    If pulling in Kubernetes via Gazelle continues to function we can work with that, otherwise we'll have to maintain our own overlay of Bazel BUILDfiles.

    It will function as much as it is supported today, and in fact because we check in generated code etc. so that people can build without our make / bazel etc., it will be trivial to wrap. None of our libraries require using any of our build scripts, only our binary artifacts / docker images.

    If you're depending on Kubernetes/Kubernetes to get release artifacts, I highly suggest switching to consume our official release builds instead of sources. rules_k8s for example consumes kubectl binaries, not kubernetes/kubernetes to source build it.

    I understand the various pain points working with Bazel sometimes - but for large scale projects this still seems to be the best build system we have. No other system provides this level of granular caching (for incremental builds) and hermeticity (for reproducible builds) simultaneously.

    FWIW, we have ~equally reproducible builds under make (which is getting pretty trivial for go binaries, essentially in modern go you just need to set -trimpath and pin dependencies with go.sum / GOPROXY, or vendor (we use both)).

    We also see that go build / go test has pretty excellent caching now, and that we need to exclude our large docker images (which go would not cache for us, but perhaps buildx which we are migrating to for other reasons) from the bazel cache to avoid net performance losses (these rarely cache well when developing kubernetes, binary source changes will invalidate them anyhow, and they're quite large).

    You can see more on this in kubernetes/enhancements#2421, maintaining both systems is not free, and bazel does not map particularly well to this project

  21. q3k commented on Feb 4, 2021

    @q3k

    If you're depending on Kubernetes/Kubernetes to get release artifacts, I highly suggest switching to consume our official release builds instead of sources.

    We can't, as we need a fully static build of Kubernetes. We also heavily patch it to remove unnecessary (to us) functionality that's not gated behind build tags, the subset of which is specific enough for our usecase that attempting to upstream such buildtags would be at least seen as uncourteous. Notably, we remove a sizeable amount of in-tree volume providers (that we know aren't compatible with our product, anyway) and nearly all in-tree legacy cloud providers. We also always carry a number of fixes pending upstreaming.

    The Kubernetes project does not support importing this repo directly as a dependency, bazel or otherwise.

    We know it's unsupported and we are more than ready to carry the burden of occasionally unbreaking things and running our own QA. Additionally, with Bazel, the line between building from our proejct and building from the k8s.io/kubernetes source itself is quite blurred - it's bazel build @io_k8s_kubernetes//cmd/kubelet in both cases. That's why we appreciate the Bazel build system in Kubernetes so much, as we can plainly pull in all of Kubernetes into our main build graph, without having to have intermediary release processes or patched source trees. The same build graph in which we have all of our dependencies, including non-Go code, complex code generation, even the build toolchains themselves.

    We also need to be able to build from source that is not within the supported client libraries, eg. to build a hyperkube-like binary containing core control plane services, our own wrapped kubelet, etc.

    You can see more on this in kubernetes/enhancements#2421

    Yes, I will review the KEP.

  22. BenTheElder commented on Feb 4, 2021

    @BenTheElder
    Member

    We also heavily patch it to remove unnecessary (to us) functionality that's not gated behind build tags, the subset of which is specific enough for our usecase that attempting to upstream such buildtags would be at least seen as uncourteous. >

    At this point it sounds like you're maintaining a fork. Your fork can maintain a build with bazel then as well, without us carrying this upstream, just like these other patches, and other upstream developed-with-go build dependencies?

    And in fact this way you can customize the build as much as you want without colliding with what we do upstream. You can just ignore the makefiles etc.

    We also need to be able to build from source that is not within the supported client libraries, eg. to build a hyperkube-like binary containing core control plane services, our own wrapped kubelet, etc.

    Similarly, we stopped maintaining an upstream build for hyperkube, because the costs outweighed the limited usage and the ability to do this downstream for anyone that wants it / lack of these consumers participating in the ongoing costs.

  23. q3k commented on Feb 4, 2021

    @q3k

    At this point it sounds like you're maintaining a fork.

    Not a true fork - we continue to track upstream as the source of truth, rebasing our changes on top at buildtime. We've also successfully consistently decreased the amount of patches we apply as we continue to upstream our changes. In an ideal world, we would just follow upstream, we have no practical interest in maintaining a fork other than the technical reasons to do that.

    If we have to maintain a Bazel compatibility patchset, we will. But I know we're not the only organization that imports Kubernetes into their buildsystem this way, so it's a shame that this will now basically have to fall onto the community, or will have to be repeatedly done by multiple organizations independently.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area/release-engIssues or PRs related to the Release Engineering subprojectarea/securitykind/cleanupCategorizes issue or PR as related to cleaning up code, process, or technical debt.priority/important-soonMust be staffed and worked on either currently, or very soon, ideally in time for the next release.sig/releaseCategorizes an issue or PR as relevant to SIG Release.sig/testingCategorizes an issue or PR as relevant to SIG Testing.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions