Skip to content

K8s: elasticluster setup fails, when a K8s cluster is already installed #678

Open

Description

@g-braeunlich

When running elasticluster setup <cluster> on a incomplete installation, I get:

TASK [kubernetes-common : Enable and start kubeadm services] *******************
ok: [master001] => (item=docker)
ok: [master001] => (item=kubelet)

TASK [kubernetes-master : Check current Kubernetes master status] **************
fatal: [master001]: FAILED! => {"changed": true, "cmd": ["kubeadm", "token", "list"], "delta": "0:00:00.057358", "end": "2020-05-11 06:40:04.062525", "msg": "non-zero return code", "rc": 1, "start": "2020-05-11 06:40:04.005167", "stderr": "failed to list bootstrap tokens: Get https://129.132.22.85:6443/api/v1/namespaces/kube-system/secrets?fieldSelector=type%3Dbootstrap.kubernetes.io%2Ftoken: Forbidden\nTo see the stack trace of this error execute with --v=5 or higher", "stderr_lines": ["failed to list bootstrap tokens: Get https://129.132.22.85:6443/api/v1/namespaces/kube-system/secrets?fieldSelector=type%3Dbootstrap.kubernetes.io%2Ftoken: Forbidden", "To see the stack trace of this error execute with --v=5 or higher"], "stdout": "", "stdout_lines": []}
...ignoring

TASK [kubernetes-master : Initialize Kubernetes master] ************************
fatal: [master001]: FAILED! => {"changed": true, "cmd": ["kubeadm", "init", "--pod-network-cidr=10.244.0.0/16", "--service-cidr=10.96.0.0/12"], "delta": "0:00:00.916074", "end": "2020-05-11 06:40:05.473674", "msg": "non-zero return code", "rc": 1, "start": "2020-05-11 06:40:04.557600", "stderr": "W0511 06:40:04.977920   17835 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]\n\t[WARNING HTTPProxy]: Connection to \"https://129.132.22.85\" uses proxy \"http://proxy.ethz.ch:3128/\". If that is not intended, adjust your proxy settings\n\t[WARNING IsDockerSystemdCheck]: detected \"cgroupfs\" as the Docker cgroup driver. The recommended driver is \"systemd\". Please follow the guide at https://kubernetes.io/docs/setup/cri/\nerror execution phase preflight: [preflight] Some fatal errors occurred:\n\t[ERROR Port-6443]: Port 6443 is in use\n\t[ERROR Port-10259]: Port 10259 is in use\n\t[ERROR Port-10257]: Port 10257 is in use\n\t[ERROR FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml]: /etc/kubernetes/manifests/kube-apiserver.yaml already exists\n\t[ERROR FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml]: /etc/kubernetes/manifests/kube-controller-manager.yaml already exists\n\t[ERROR FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml]: /etc/kubernetes/manifests/kube-scheduler.yaml already exists\n\t[ERROR FileAvailable--etc-kubernetes-manifests-etcd.yaml]: /etc/kubernetes/manifests/etcd.yaml already exists\n\t[ERROR Port-10250]: Port 10250 is in use\n\t[ERROR Port-2379]: Port 2379 is in use\n\t[ERROR Port-2380]: Port 2380 is in use\n\t[ERROR DirAvailable--var-lib-etcd]: /var/lib/etcd is not empty\n[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`\nTo see the stack trace of this error execute with --v=5 or higher", "stderr_lines": ["W0511 06:40:04.977920   17835 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]", "\t[WARNING HTTPProxy]: Connection to \"https://129.132.22.85\" uses proxy \"http://proxy.ethz.ch:3128/\". If that is not intended, adjust your proxy settings", "\t[WARNING IsDockerSystemdCheck]: detected \"cgroupfs\" as the Docker cgroup driver. The recommended driver is \"systemd\". Please follow the guide at https://kubernetes.io/docs/setup/cri/", "error execution phase preflight: [preflight] Some fatal errors occurred:", "\t[ERROR Port-6443]: Port 6443 is in use", "\t[ERROR Port-10259]: Port 10259 is in use", "\t[ERROR Port-10257]: Port 10257 is in use", "\t[ERROR FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml]: /etc/kubernetes/manifests/kube-apiserver.yaml already exists", "\t[ERROR FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml]: /etc/kubernetes/manifests/kube-controller-manager.yaml already exists", "\t[ERROR FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml]: /etc/kubernetes/manifests/kube-scheduler.yaml already exists", "\t[ERROR FileAvailable--etc-kubernetes-manifests-etcd.yaml]: /etc/kubernetes/manifests/etcd.yaml already exists", "\t[ERROR Port-10250]: Port 10250 is in use", "\t[ERROR Port-2379]: Port 2379 is in use", "\t[ERROR Port-2380]: Port 2380 is in use", "\t[ERROR DirAvailable--var-lib-etcd]: /var/lib/etcd is not empty", "[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`", "To see the stack trace of this error execute with --v=5 or higher"], "stdout": "[init] Using Kubernetes version: v1.18.2\n[preflight] Running pre-flight checks", "stdout_lines": ["[init] Using Kubernetes version: v1.18.2", "[preflight] Running pre-flight checks"]}

NO MORE HOSTS LEFT *************************************************************

Activity

  1. changed the title [-]elasticluster setup fails, when a cluster is already installed[/-] [+]K8s: elasticluster setup fails, when a K8s cluster is already installed[/+] on May 11, 2020
  2. shock0572 commented on May 13, 2020

    @shock0572
    Contributor

    as a note: i used / rebuild / reloaded setup'd again different clusters for #679
    Could it be a flannel bug? By using calico after, as a networking plugin, never got to experience this error on following elasticluster setup attempts

  3. riccardomurri commented on May 14, 2020

    @riccardomurri
    Contributor

    as a note: i used / rebuild / reloaded setup'd again different clusters for #679

    I think the bug here happens when you run elasticluster setup (or elasticluster resize) on a existing cluster. When you set up a new/different cluster afresh, everything will be smooth.

  4. shock0572 commented on May 14, 2020

    @shock0572
    Contributor

    Even in the case of elasticluster setup, during at least 10 tests, it's shown that using calico as a networking plugin doesn't get you through any errors of the like above, since it handles properly a pod network change (that's supposed to happen exactly when you run again elasticluster setup). Probably the scarce support of flannel hasn't got examples / hasn't ever experienced this exact situation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions