Skip to content

[Question] Kustomize URL formats #4454

Closed as not planned
Closed as not planned

Description

@thatsmydoing

kustomize's support for URL formats is not well-specified and provides shortcuts for sources like github that do not extend to other repositories and can be cause for confusion. For kustomization v1, I think it would be a good idea to formalize it since it will be a breaking change for other fields as well.

The problems with kustomize's url formats that I see are:

git url extraction

Typically, kustomize manifests live in subdirectories of a repository so a typical url looks like

https://github.com/kubernetes-sigs/kustomize/examples/helloWorld

where the repository to clone is https://github.com/kubernetes-sigs/kustomize and the subdirectory to build is examples/helloWorld. And kustomize just assumes that the repository is always 2 subdirectories deep. This leads to problems when this isn't the case and more special cases were added that say look one after the _git directory and look for the directory that ends in .git. So we support cloning the following repositories correctly:

but https://bitbucket.example.com/scm/project/repository will be treated as clone https://bitbucket.example.com/scm/project and build the subdirectory repository.

I think the better answer here is to just use // to delimit the repository url and the directory (this is already supported but is still subjected to the above). In that case, there is no need for special cases as we're not guessing anymore. So that https://bitbucket.example.com/scm/project/repository will mean just clone the repository repository and build the root. While https://bitbucket.example.com/scm/project//repository will mean clone the project repository and build repository

go-getter legacy and url magic

kustomize states that it supports "go-getter" format and previously actually did use go-getter (but not always for git cloning). For specifying a repository, all of the following are currently valid for HTTP clones

and the following are valid for SSH clones

and probably a lot more. The logic lives in https://github.com/kubernetes-sigs/kustomize/blob/master/api/internal/git/repospec.go which greatly needs an overhaul.

What's more, most of these only apply to github.com urls, so you can say github.com/kubernetes-sigs/kustomize/examples/helloWorld but bitbucket.com/kubernetes-sigs/kustomize/examples/helloWorld does not. There's also some weird logic with .git being appended to the repository but not if it's an AWS or Azure code host.

I think we should simply restrict the urls to one's accepted by git clone. Many of the above examples are not valid or wrong. Obviously gh: and git:: prefixes are not clonable.

  • git clone github.com/kubernetes-sigs/kustomize tries to clone the local directory github.com/kubernetes-sigs/kustomize
  • git clone github.com:kubernetes-sigs/kustomize will do an SSH clone as your current user but kustomize does an HTTPS one
  • git clone [email protected]/kubernetes-sigs/kustomize is not valid as you must use : to delimit host and directory
  • git clone ssh://[email protected]:kubernetes-sigs/kustomize is not valid as you must use / to delimit the host and directory

remote resource overlap

kustomize currently tries to load a resource as a "file" first and then as a base. kustomize also supports loading remote file resources via HTTPS so this means it's possible a resource that has https:// can be interpreted as either a file or a base and you'd have to try both to know which is which. This does mean for bases that begin with https://, kustomize will try to load it as a file first, then fail, then clone the repository and succeed. This is fine for local files and bases but has some performance penalty for remote files and bases.

I'm not really sure what can be done here so long as the formats can overlap. We could say that all remote bases must have a //. So urls without // are treated as files but those with // are treated as bases. This might be acceptable since I don't believe it's common to have kustomization.yaml at the root of a repository and for those cases we can just say add // or //. to the end. This is a completely breaking change though and can't be eased into like the other suggestions.

Activity

  1. added
    needs-kindIndicates a PR lacks a `kind/foo` label and requires one.
    on Feb 6, 2022
  2. k8s-ci-robot commented on Feb 6, 2022

    @k8s-ci-robot
    Contributor

    @thatsmydoing: This issue is currently awaiting triage.

    SIG CLI takes a lead on issue triage for this repo, but any Kubernetes member can accept issues by applying the triage/accepted label.

    The triage/accepted label can be added by org members by writing /triage accepted in a comment.

    Details

    Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

  3. added
    needs-triageIndicates an issue or PR lacks a `triage/foo` label and requires one.
    on Feb 6, 2022
  4. added
    kind/api-changeCategorizes issue or PR as related to adding, removing, or otherwise changing an API
    triage/acceptedIndicates an issue or PR is ready to be actively worked on.
    and removed
    needs-kindIndicates a PR lacks a `kind/foo` label and requires one.
    needs-triageIndicates an issue or PR lacks a `triage/foo` label and requires one.
    on Feb 7, 2022
  5. thatsmydoing commented on Feb 10, 2022

    @thatsmydoing
    ContributorAuthor

    In #4453, it was suggested that remote git bases should just be prefixed with a git:: to disambiguate between remote file resources.

  6. k8s-triage-robot commented on May 11, 2022

    @k8s-triage-robot

    The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

    This bot triages issues and PRs according to the following rules:

    • After 90d of inactivity, lifecycle/stale is applied
    • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
    • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

    You can:

    • Mark this issue or PR as fresh with /remove-lifecycle stale
    • Mark this issue or PR as rotten with /lifecycle rotten
    • Close this issue or PR with /close
    • Offer to help out with Issue Triage

    Please send feedback to sig-contributor-experience at kubernetes/community.

    /lifecycle stale

  7. added
    lifecycle/staleDenotes an issue or PR has remained open with no activity and has become stale.
    on May 11, 2022
  8. thatsmydoing commented on May 11, 2022

    @thatsmydoing
    ContributorAuthor

    /remove-lifecycle stale

  9. thatsmydoing commented on Jun 9, 2022

    @thatsmydoing
    ContributorAuthor

    It seems tests have been added in #4615

  10. k8s-triage-robot commented on Jul 9, 2022

    @k8s-triage-robot

    The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

    This bot triages issues and PRs according to the following rules:

    • After 90d of inactivity, lifecycle/stale is applied
    • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
    • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

    You can:

    • Mark this issue or PR as fresh with /remove-lifecycle rotten
    • Close this issue or PR with /close
    • Offer to help out with Issue Triage

    Please send feedback to sig-contributor-experience at kubernetes/community.

    /lifecycle rotten

  11. added
    lifecycle/rottenDenotes an issue or PR that has aged beyond stale and will be auto-closed.
    and removed
    lifecycle/staleDenotes an issue or PR has remained open with no activity and has become stale.
    on Jul 9, 2022
  12. 7 remaining items

  13. added
    lifecycle/staleDenotes an issue or PR has remained open with no activity and has become stale.
    on Jan 5, 2023
  14. thatsmydoing commented on Jan 5, 2023

    @thatsmydoing
    ContributorAuthor

    /remove-lifecycle stale

  15. removed
    lifecycle/staleDenotes an issue or PR has remained open with no activity and has become stale.
    on Jan 5, 2023
  16. thatsmydoing commented on Jul 9, 2023

    @thatsmydoing
    ContributorAuthor

    It looks like gh: and git:: are now deprecated as of #4954

  17. k8s-triage-robot commented on Jul 8, 2024

    @k8s-triage-robot

    This issue has not been updated in over 1 year, and should be re-triaged.

    You can:

    • Confirm that this issue is still relevant with /triage accepted (org members only)
    • Close this issue with /close

    For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/

    /remove-triage accepted

  18. added
    needs-triageIndicates an issue or PR lacks a `triage/foo` label and requires one.
    and removed
    triage/acceptedIndicates an issue or PR is ready to be actively worked on.
    on Jul 8, 2024
  19. k8s-triage-robot commented on Oct 6, 2024

    @k8s-triage-robot

    The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

    This bot triages un-triaged issues according to the following rules:

    • After 90d of inactivity, lifecycle/stale is applied
    • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
    • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

    You can:

    • Mark this issue as fresh with /remove-lifecycle stale
    • Close this issue with /close
    • Offer to help out with Issue Triage

    Please send feedback to sig-contributor-experience at kubernetes/community.

    /lifecycle stale

  20. added
    lifecycle/staleDenotes an issue or PR has remained open with no activity and has become stale.
    on Oct 6, 2024
  21. k8s-triage-robot commented on Nov 5, 2024

    @k8s-triage-robot

    The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

    This bot triages un-triaged issues according to the following rules:

    • After 90d of inactivity, lifecycle/stale is applied
    • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
    • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

    You can:

    • Mark this issue as fresh with /remove-lifecycle rotten
    • Close this issue with /close
    • Offer to help out with Issue Triage

    Please send feedback to sig-contributor-experience at kubernetes/community.

    /lifecycle rotten

  22. added
    lifecycle/rottenDenotes an issue or PR that has aged beyond stale and will be auto-closed.
    and removed
    lifecycle/staleDenotes an issue or PR has remained open with no activity and has become stale.
    on Nov 5, 2024
  23. k8s-triage-robot commented on Dec 5, 2024

    @k8s-triage-robot

    The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

    This bot triages issues according to the following rules:

    • After 90d of inactivity, lifecycle/stale is applied
    • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
    • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

    You can:

    • Reopen this issue with /reopen
    • Mark this issue as fresh with /remove-lifecycle rotten
    • Offer to help out with Issue Triage

    Please send feedback to sig-contributor-experience at kubernetes/community.

    /close not-planned

  24. k8s-ci-robot commented on Dec 5, 2024

    @k8s-ci-robot
    Contributor

    @k8s-triage-robot: Closing this issue, marking it as "Not Planned".

    Details

    In response to this:

    The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

    This bot triages issues according to the following rules:

    • After 90d of inactivity, lifecycle/stale is applied
    • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
    • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

    You can:

    • Reopen this issue with /reopen
    • Mark this issue as fresh with /remove-lifecycle rotten
    • Offer to help out with Issue Triage

    Please send feedback to sig-contributor-experience at kubernetes/community.

    /close not-planned

    Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/api-changeCategorizes issue or PR as related to adding, removing, or otherwise changing an APIlifecycle/rottenDenotes an issue or PR that has aged beyond stale and will be auto-closed.needs-triageIndicates an issue or PR lacks a `triage/foo` label and requires one.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions