Skip to content

Reducing Kubernetes Build Maintenance #2420

Closed

Description

@BenTheElder

Enhancement Description

Additional detailed PR listing below #2420 (comment)

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

/assign
/sig testing
/sig release

Activity

  1. added
    sig/testingCategorizes an issue or PR as relevant to SIG Testing.
    sig/releaseCategorizes an issue or PR as relevant to SIG Release.
    on Feb 4, 2021
  2. changed the title [-]Reducing Build Maintenance[/-] [+]Reducing Kubernetes Build Maintenance[/+] on Feb 4, 2021
  3. added this to the v1.21 milestone on Feb 4, 2021
  4. added
    tracked/yesDenotes an enhancement issue is actively being tracked by the Release Team
    on Feb 5, 2021
  5. spiffxp commented on Feb 8, 2021

    @spiffxp
    Contributor

    Initial draft PR: #2421

  6. BenTheElder commented on Feb 8, 2021

    @BenTheElder
    MemberAuthor

    followup for implementable #2469

  7. BenTheElder commented on Feb 9, 2021

    @BenTheElder
    MemberAuthor

    Discussed today in SIG Testing meeting, no concerns there. (this has also been circulated in SIG Testing and Release previously).

  8. BenTheElder commented on Feb 25, 2021

    @BenTheElder
    MemberAuthor

    UPDATE 02/24/2021

    We moved pull-kubernetes-e2e-kind-ipv6 to build without bazel, so far so good. kubernetes/test-infra#21009

    We're now running pull-kubernetes-unit (non blocking) which runs ~make test with the race detector enabled as an approximate equivilant to pull-kubernetes-bazel-test (our heaviest bazel dependency in CI). This is working well. kubernetes/test-infra#21008

    Nominally speaking we are on track to not depend on this build in CI, opening the path to notify and remove.


    However there are a handful of test targets in pull-kubernetes-bazel-test that are not golang unit tests and therefore will not be covered by make test.
    I produced the following list by inspecting the junit output and ignoring :go_default_test targets (go unit tests under bazel).

    • 1. //cluster:common_test

    Per BUILD:

    # These tests just verify that bash can interpret the file.

    This one I think we can ignore. This doesn't appear to function as intended and we don't normally do this.
    EDIT: kubernetes/kubernetes#99486 to remove. See details there.

    • 2. //cluster:kube-util_test

    Per BUILD:

    # These tests just verify that bash can interpret the file.

    This one I think we can ignore. This doesn't appear to function as intended and we don't normally do this.
    EDIT: kubernetes/kubernetes#99486 to remove. See details there.

    • 3. //hack/boilerplate:boilerplate_test IMPORTANT

    Python script that ensures the boilerplate headers are in place.

    This one is already covered by pull-kubernetes-verify / make verify so we don't need to do anything.

    • 4. //pkg/util:verify-util-pkg

    This is a sh_test that basically ensures that ./pkg/util/ only contains code in subdirectories, not ./pkg/util/foo.go

    I'm not sure why we couldn't just let approvers handle this, it seems a bit heavyweight, but if we still need it we can move it to pull-kubernetes-verify / make verify.
    EDIT: Another approach we could take here is to use prow's blockade plugin to block ^pkg/util/[^/]*\.go by automatically adding do-not-merge/blocked-paths label and posting a configurable explanation comment.
    EDIT: Fixed by kubernetes/test-infra#21037 (blockade)
    EDIT: kubernetes/kubernetes#99486 to remove.

    See: kubernetes/kubernetes#15634

    • 5. //test/conformance:conformance_test IMPORTANT

    This is a sh_test that ensures the generated conformance.yaml is up to date.

    We should probably rework this to run under make verify / pull-kubernetes-verify.
    EDIT: fixed in kubernetes/kubernetes#99549 to work under make verify and hack/update-conformance-yaml.sh

  9. added
    lifecycle/activeIndicates that an issue or PR is actively being worked on by a contributor.
    on Feb 25, 2021
  10. dims commented on Feb 25, 2021

    @dims
    Member
  11. BenTheElder commented on Feb 26, 2021

    @BenTheElder
    MemberAuthor

    Update 02/25/2021

    1. pull-kubernetes-unit is looking pretty good https://prow.k8s.io/?job=pull-kubernetes-unit
      pkg/controlplane sometimes times out at the current 240s limit we're setting for make test, we could raise it, but we should fix that test Umbrella issue: slow/timing-out unit tests kubernetes#98486
    === FAIL: pkg/controlplane  (0.00s)
    WARNING: Package "github.com/golang/protobuf/protoc-gen-go/generator" is deprecated.
    	A future release of golang/protobuf will delete this package,
    	which has long been excluded from the compatibility promise.
    FAIL	k8s.io/kubernetes/pkg/controlplane	240.803s
    
    1. pull-kubernetes-e2e-kind-ipv6 is working fine building with make with ~30m total runtime https://prow.k8s.io/?job=pull-kubernetes-e2e-kind-ipv6. Writing a PR to switch more jobs currently.

    2. Filed add blockade to kubernetes for pkg/util test-infra#21037 for 4) on the list above Reducing Kubernetes Build Maintenance #2420 (comment), checking that one off.

    3. Filed Cleanup old test targets kubernetes#99486 to cleanup 1) 2) 4) from above.

  12. aojea commented on Feb 27, 2021

    @aojea
    Member

    /cc

  13. BenTheElder commented on Feb 28, 2021

    @BenTheElder
    MemberAuthor

    Update 02/28/2021

    1. I finished migrating the e2e jobs for the Kubernetes/Kubernetes master branch to make AFAICT.

    2. I've opened conformance.yaml: generate with make kubernetes#99549 to address migrating conformance.yaml / 5) above (Reducing Kubernetes Build Maintenance #2420 (comment))

    Once 2) is done, we can flip pull-kubernetes-bazel-test -> pull-kubernetes-unit as blocking (when conformance.yaml is covered in pull-kubernetes-verify).
    EDIT: 2) is done. The first follow-up is in kubernetes/test-infra#21070 (block PRs on pull-kubernetes-unit)

    After that I'll have a PR out to rewrite hack/update-bazel.sh to drop all BUILD and cleanup any remaining references. We will hold that PR until we notify kubernetes-dev@ etc.

    When that is done we can drop the -bazel-build Kubernetes jobs.

    Finally we can let the remaining release branches age out.

  14. 40 remaining items

  15. BenTheElder commented on Sep 23, 2021

    @BenTheElder
    MemberAuthor

    n.b.: #2420 (comment) is a note from the bi-weekly SIG testing meeting, so I agree with my own comment ;-)

    just waiting for 1.19 turndown from SIG release

  16. BenTheElder commented on Nov 4, 2021

    @BenTheElder
    MemberAuthor

    More cleanup PRs:

    We are on-track to stable with the turn down of 1.19 related Kubernetes project resources. There's a bit more to do but we're nearly done.

    Next up: We run a reasonably large custom bazel cache (greenhouse) instance in each CI cluster primarily aimed at Kubernetes/Kubernetes, we should either turn this down or at least reduce the capacity / operational costs significantly.

  17. kevindelgado commented on Nov 8, 2021

    @kevindelgado
    Contributor

    Thanks for the update @BenTheElder! Still gonna give you the copy/pasta with all the deets:

    Hi @BenTheElder 👋

    Checking in once more as we approach 1.23 code freeze at 6:00 pm PST on Tuesday, November 16.

    Please ensure the following items are completed:

    • All PRs to the Kubernetes repo that are related to your enhancement are linked in the above issue description (for tracking purposes).
    • All PRs are fully merged by the code freeze deadline.
    • Have a documentation placeholder PR open by Thursday, November 18.

    As always, we are here to help should questions come up.

    Thanks!!

  18. spiffxp commented on Nov 24, 2021

    @spiffxp
    Contributor

    We're past code freeze and test freeze at this point, I'm hoping / assuming nothing remains to be done in kubernetes/kubernetes itself. What tasks do we need to complete to call this done?

  19. BenTheElder commented on Nov 24, 2021

    @BenTheElder
    MemberAuthor

    There's some pretty minor stuff but we could probably close it out now that we've done the rest.

    • turn down greenhouse test-infra#24247
    • there's still a few errant references in kubernetes/kubernetes, off the top of my head the publishing bot is still find-and-removing bazel related files still, because we took longer to turn down the 1.19 branch (and also missed turning up 1.23 ...)
  20. BenTheElder commented on Nov 24, 2021

    @BenTheElder
    MemberAuthor

    none of those references are in production release binary code or release / build scripts though, just in tests, publishing bot config, that sort of thing.

  21. gracenng commented on Jan 9, 2022

    @gracenng
    Member

    Hi @BenTheElder ,

    1.24 Enhancements Lead here. Could you please update this enhancements's KEP status to implemented, then close this issue?

    Thanks :)

  22. added
    tracked/noDenotes an enhancement issue is NOT actively being tracked by the Release Team
    and removed
    tracked/yesDenotes an enhancement issue is actively being tracked by the Release Team
    on Jan 9, 2022
  23. removed this from the v1.23 milestone on Jan 9, 2022
  24. BenTheElder commented on Feb 4, 2022

    @BenTheElder
    MemberAuthor

    filed #3209, we can follow up with any additional minor cleanup.

  25. BenTheElder commented on Feb 4, 2022

    @BenTheElder
    MemberAuthor

    @gracenng sorry for the delay, this is done!

  26. removed
    tracked/noDenotes an enhancement issue is NOT actively being tracked by the Release Team
    on Jan 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area/release-engIssues or PRs related to the Release Engineering subprojectkind/kepCategorizes KEP tracking issues and PRs modifying the KEP directorylifecycle/activeIndicates that an issue or PR is actively being worked on by a contributor.priority/important-longtermImportant over the long term, but may not be staffed and/or may need multiple releases to complete.sig/releaseCategorizes an issue or PR as relevant to SIG Release.sig/testingCategorizes an issue or PR as relevant to SIG Testing.stage/stableDenotes an issue tracking an enhancement targeted for Stable/GA status

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions