Repository navigation
Description
kustomize's support for URL formats is not well-specified and provides shortcuts for sources like github that do not extend to other repositories and can be cause for confusion. For kustomization v1, I think it would be a good idea to formalize it since it will be a breaking change for other fields as well.
The problems with kustomize's url formats that I see are:
git url extraction
Typically, kustomize manifests live in subdirectories of a repository so a typical url looks like
https://github.com/kubernetes-sigs/kustomize/examples/helloWorld
where the repository to clone is https://github.com/kubernetes-sigs/kustomize and the subdirectory to build is examples/helloWorld. And kustomize just assumes that the repository is always 2 subdirectories deep. This leads to problems when this isn't the case and more special cases were added that say look one after the _git directory and look for the directory that ends in .git. So we support cloning the following repositories correctly:
- https://itfs.mycompany.com/collection/project/_git/somerepos
- https://bitbucket.example.com/scm/project/repository.git
but https://bitbucket.example.com/scm/project/repository will be treated as clone https://bitbucket.example.com/scm/project and build the subdirectory repository.
I think the better answer here is to just use // to delimit the repository url and the directory (this is already supported but is still subjected to the above). In that case, there is no need for special cases as we're not guessing anymore. So that https://bitbucket.example.com/scm/project/repository will mean just clone the repository repository and build the root. While https://bitbucket.example.com/scm/project//repository will mean clone the project repository and build repository
go-getter legacy and url magic
kustomize states that it supports "go-getter" format and previously actually did use go-getter (but not always for git cloning). For specifying a repository, all of the following are currently valid for HTTP clones
- github.com/kubernetes-sigs/kustomize/examples/helloWorld
gh:github.com/kubernetes-sigs/kustomize/examples/helloWorld- https://github.com/kubernetes-sigs/kustomize/examples/helloWorld
git::https://github.com/kubernetes-sigs/kustomize/examples/helloWorld- github.com:kubernetes-sigs/kustomize/examples/helloWorld
and the following are valid for SSH clones
- [email protected]:kubernetes-sigs/kustomize/examples/helloWorld
- [email protected]/kubernetes-sigs/kustomize/examples/helloWorld
git::[email protected]:kubernetes-sigs/kustomize/examples/helloWorld- ssh://[email protected]:kubernetes-sigs/kustomize/examples/helloWorld
- ssh://[email protected]/kubernetes-sigs/kustomize/examples/helloWorld
and probably a lot more. The logic lives in https://github.com/kubernetes-sigs/kustomize/blob/master/api/internal/git/repospec.go which greatly needs an overhaul.
What's more, most of these only apply to github.com urls, so you can say github.com/kubernetes-sigs/kustomize/examples/helloWorld but bitbucket.com/kubernetes-sigs/kustomize/examples/helloWorld does not. There's also some weird logic with .git being appended to the repository but not if it's an AWS or Azure code host.
I think we should simply restrict the urls to one's accepted by git clone. Many of the above examples are not valid or wrong. Obviously gh: and git:: prefixes are not clonable.
git clone github.com/kubernetes-sigs/kustomizetries to clone the local directorygithub.com/kubernetes-sigs/kustomizegit clone github.com:kubernetes-sigs/kustomizewill do an SSH clone as your current user but kustomize does an HTTPS onegit clone [email protected]/kubernetes-sigs/kustomizeis not valid as you must use:to delimit host and directorygit clone ssh://[email protected]:kubernetes-sigs/kustomizeis not valid as you must use/to delimit the host and directory
remote resource overlap
kustomize currently tries to load a resource as a "file" first and then as a base. kustomize also supports loading remote file resources via HTTPS so this means it's possible a resource that has https:// can be interpreted as either a file or a base and you'd have to try both to know which is which. This does mean for bases that begin with https://, kustomize will try to load it as a file first, then fail, then clone the repository and succeed. This is fine for local files and bases but has some performance penalty for remote files and bases.
I'm not really sure what can be done here so long as the formats can overlap. We could say that all remote bases must have a //. So urls without // are treated as files but those with // are treated as bases. This might be acceptable since I don't believe it's common to have kustomization.yaml at the root of a repository and for those cases we can just say add // or //. to the end. This is a completely breaking change though and can't be eased into like the other suggestions.
Activity
k8s-ci-robot commentedon Feb 6, 2022
@thatsmydoing: This issue is currently awaiting triage.
SIG CLI takes a lead on issue triage for this repo, but any Kubernetes member can accept issues by applying the
triage/acceptedlabel.The
triage/acceptedlabel can be added by org members by writing/triage acceptedin a comment.Details
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
thatsmydoing commentedon Feb 10, 2022
In #4453, it was suggested that remote git bases should just be prefixed with a
git::to disambiguate between remote file resources.k8s-triage-robot commentedon May 11, 2022
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules:
lifecycle/staleis appliedlifecycle/stalewas applied,lifecycle/rottenis appliedlifecycle/rottenwas applied, the issue is closedYou can:
/remove-lifecycle stale/lifecycle rotten/closePlease send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
thatsmydoing commentedon May 11, 2022
/remove-lifecycle stale
thatsmydoing commentedon Jun 9, 2022
It seems tests have been added in #4615
k8s-triage-robot commentedon Jul 9, 2022
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules:
lifecycle/staleis appliedlifecycle/stalewas applied,lifecycle/rottenis appliedlifecycle/rottenwas applied, the issue is closedYou can:
/remove-lifecycle rotten/closePlease send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten
7 remaining items
thatsmydoing commentedon Jan 5, 2023
/remove-lifecycle stale
thatsmydoing commentedon Jul 9, 2023
It looks like
gh:andgit::are now deprecated as of #4954k8s-triage-robot commentedon Jul 8, 2024
This issue has not been updated in over 1 year, and should be re-triaged.
You can:
/triage accepted(org members only)/closeFor more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/
/remove-triage accepted
k8s-triage-robot commentedon Oct 6, 2024
The Kubernetes project currently lacks enough contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
lifecycle/staleis appliedlifecycle/stalewas applied,lifecycle/rottenis appliedlifecycle/rottenwas applied, the issue is closedYou can:
/remove-lifecycle stale/closePlease send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
k8s-triage-robot commentedon Nov 5, 2024
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
lifecycle/staleis appliedlifecycle/stalewas applied,lifecycle/rottenis appliedlifecycle/rottenwas applied, the issue is closedYou can:
/remove-lifecycle rotten/closePlease send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten
k8s-triage-robot commentedon Dec 5, 2024
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.
This bot triages issues according to the following rules:
lifecycle/staleis appliedlifecycle/stalewas applied,lifecycle/rottenis appliedlifecycle/rottenwas applied, the issue is closedYou can:
/reopen/remove-lifecycle rottenPlease send feedback to sig-contributor-experience at kubernetes/community.
/close not-planned
k8s-ci-robot commentedon Dec 5, 2024
@k8s-triage-robot: Closing this issue, marking it as "Not Planned".
Details
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.