Describe the bug
Deploy a greenfield and pod/istio-security-post-install-1.1.0-snapshot.4 crashes.
From the pod log, it complains some role binding is missing
kubectl log -f -n istio-system istio-security-post-install-1.1.0-snapshot.4-8f6fn
log is DEPRECATED and will be removed in a future version. Use logs instead.
+ '[' 1 -ne 1 ']'
+ pathToResourceYAML=/tmp/security/custom-resources.yaml
+ kubectl get validatingwebhookconfiguration istio-galley
NAME CREATED AT
istio-galley 2019-01-10T17:54:04Z
+ '[' 0 -eq 0 ']'
+ echo 'istio-galley validatingwebhookconfiguration found - waiting for istio-galley deployment to be ready'
+ true
+ kubectl -n istio-system get deployment istio-galley
istio-galley validatingwebhookconfiguration found - waiting for istio-galley deployment to be ready
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
istio-galley 1 1 1 1 1m
+ '[' 0 -eq 0 ']'
+ break
+ kubectl -n istio-system rollout status deployment istio-galley
Error from server (Forbidden): deployments.apps "istio-galley" is forbidden: User "system:serviceaccount:istio-system:istio-security-post-install-account" cannot get deployments.apps in the namespace "istio-system"
istio-galley deployment rollout status check failed
+ '[' 1 -ne 0 ']'
+ echo 'istio-galley deployment rollout status check failed'
+ exit 1
But the service account and the binding is there tho:
kubectl get sa -n istio-system | grep istio-security-post-install-account
istio-security-post-install-account 1 1h
ccpuser@ci-ccp-kahoulei-pr1317-red26-tlca-cluster-upgr-masterec703268de:~/istio/templates$ kubectl get sa -n istio-system
NAME SECRETS AGE
default 1 1h
istio-citadel-service-account 1 1h
istio-egressgateway-service-account 1 1h
istio-galley-service-account 1 1h
istio-grafana-post-install-account 1 1h
istio-ingressgateway-service-account 1 1h
istio-mixer-service-account 1 1h
istio-pilot-service-account 1 1h
istio-security-post-install-account 1 1h
istio-sidecar-injector-service-account 1 1h
prometheus 1 1h
ccpuser@ci-ccp-kahoulei-pr1317-red26-tlca-cluster-upgr-masterec703268de:~/istio/templates$ kubectl get clusterrole -n istio-system
NAME AGE
admin 1h
calico-node 1h
ccp-monitor-grafana-clusterrole 1h
ccp-monitor-prometheus-kube-state-metrics 1h
ccp-monitor-prometheus-server 1h
cluster-admin 1h
edit 1h
elasticsearch-logging 1h
fluentd-es 1h
istio-citadel-istio-system 1h
istio-egressgateway-istio-system 1h
istio-galley-istio-system 1h
istio-grafana-post-install-istio-system 1h
istio-ingressgateway-istio-system 1h
istio-mixer-istio-system 1h
istio-pilot-istio-system 1h
istio-security-post-install-istio-system 1h
istio-sidecar-injector-istio-system 1h
metallb:controller 1h
metallb:speaker 1h
nginx-ingress 1h
prometheus-istio-system 1h
system:aggregate-to-admin 1h
system:aggregate-to-edit 1h
system:aggregate-to-view 1h
system:auth-delegator 1h
system:aws-cloud-provider 1h
system:basic-user 1h
system:certificates.k8s.io:certificatesigningrequests:nodeclient 1h
system:certificates.k8s.io:certificatesigningrequests:selfnodeclient 1h
system:controller:attachdetach-controller 1h
system:controller:certificate-controller 1h
system:controller:clusterrole-aggregation-controller 1h
system:controller:cronjob-controller 1h
system:controller:daemon-set-controller 1h
system:controller:deployment-controller 1h
system:controller:disruption-controller 1h
system:controller:endpoint-controller 1h
system:controller:expand-controller 1h
system:controller:generic-garbage-collector 1h
system:controller:horizontal-pod-autoscaler 1h
system:controller:job-controller 1h
system:controller:namespace-controller 1h
system:controller:node-controller 1h
system:controller:persistent-volume-binder 1h
system:controller:pod-garbage-collector 1h
system:controller:pv-protection-controller 1h
system:controller:pvc-protection-controller 1h
system:controller:replicaset-controller 1h
system:controller:replication-controller 1h
system:controller:resourcequota-controller 1h
system:controller:route-controller 1h
system:controller:service-account-controller 1h
system:controller:service-controller 1h
system:controller:statefulset-controller 1h
system:controller:ttl-controller 1h
system:coredns 1h
system:csi-external-attacher 1h
system:csi-external-provisioner 1h
system:discovery 1h
system:heapster 1h
system:kube-aggregator 1h
system:kube-controller-manager 1h
system:kube-dns 1h
system:kube-scheduler 1h
system:kubelet-api-admin 1h
system:node 1h
system:node-bootstrapper 1h
system:node-problem-detector 1h
system:node-proxier 1h
system:persistent-volume-provisioner 1h
system:volume-scheduler 1h
view 1h
vsphere-cloud-provider 1h
ccpuser@ci-ccp-kahoulei-pr1317-red26-tlca-cluster-upgr-masterec703268de:~/istio/templates$ kubectl get clusterrolebinding -n istio-system
NAME AGE
add-on-cluster-admin 1h
calico-node 1h
ccp-dashboard 1h
ccp-monitor-grafana-clusterrolebinding 1h
ccp-monitor-prometheus-kube-state-metrics 1h
ccp-monitor-prometheus-server 1h
cluster-admin 1h
fluentd-es 1h
hxprovisioner 1h
istio-citadel-istio-system 1h
istio-egressgateway-istio-system 1h
istio-galley-admin-role-binding-istio-system 1h
istio-grafana-post-install-role-binding-istio-system 1h
istio-ingressgateway-istio-system 1h
istio-mixer-admin-role-binding-istio-system 1h
istio-pilot-istio-system 1h
istio-security-post-install-role-binding-istio-system 1h
istio-sidecar-injector-admin-role-binding-istio-system 1h
kubeadm:kubelet-bootstrap 1h
kubeadm:node-autoapprove-bootstrap 1h
kubeadm:node-autoapprove-certificate-rotation 1h
kubeadm:node-proxier 1h
metallb:controller 1h
metallb:speaker 1h
nginx-ingress 1h
prometheus-istio-system 1h
system:aws-cloud-provider 1h
system:basic-user 1h
system:controller:attachdetach-controller 1h
system:controller:certificate-controller 1h
system:controller:clusterrole-aggregation-controller 1h
system:controller:cronjob-controller 1h
system:controller:daemon-set-controller 1h
system:controller:deployment-controller 1h
system:controller:disruption-controller 1h
system:controller:endpoint-controller 1h
system:controller:expand-controller 1h
system:controller:generic-garbage-collector 1h
system:controller:horizontal-pod-autoscaler 1h
system:controller:job-controller 1h
system:controller:namespace-controller 1h
system:controller:node-controller 1h
system:controller:persistent-volume-binder 1h
system:controller:pod-garbage-collector 1h
system:controller:pv-protection-controller 1h
system:controller:pvc-protection-controller 1h
system:controller:replicaset-controller 1h
system:controller:replication-controller 1h
system:controller:resourcequota-controller 1h
system:controller:route-controller 1h
system:controller:service-account-controller 1h
system:controller:service-controller 1h
system:controller:statefulset-controller 1h
system:controller:ttl-controller 1h
system:coredns 1h
system:discovery 1h
system:kube-controller-manager 1h
system:kube-dns 1h
system:kube-scheduler 1h
system:node 1h
system:node-proxier 1h
system:volume-scheduler 1h
vsphere-cloud-provider 1h
One thing i notice is there is no role and rolebinding:
ccpuser@ci-ccp-kahoulei-pr1317-red26-tlca-cluster-upgr-masterec703268de:~/istio/templates$ kubectl get rolebinding -n istio-system
No resources found.
ccpuser@ci-ccp-kahoulei-pr1317-red26-tlca-cluster-upgr-masterec703268de:~/istio/templates$ kubectl get role -n istio-system
No resources found.
Note: this problem only happens intermittently. I cannot always reproduce it but it is unpredictible.
Expected behavior
pod/istio-security-post-install-1.1.0-snapshot.4 should be running
Steps to reproduce the bug
Run kubeadm to spin up a cluster, install istio
Version
1.1 snapshot.4
Installation
{{ Please describe how Istio was installed }}
Environment
ubuntu
Cluster state
{{ If you're running on Kubernetes, consider following the
instructions
to generate "istio-dump.tar.gz", then attach it here by dragging and dropping
the file onto this issue. }}
Describe the bug
Deploy a greenfield and pod/istio-security-post-install-1.1.0-snapshot.4 crashes.
From the pod log, it complains some role binding is missing
But the service account and the binding is there tho:
One thing i notice is there is no role and rolebinding:
Note: this problem only happens intermittently. I cannot always reproduce it but it is unpredictible.
Expected behavior
pod/istio-security-post-install-1.1.0-snapshot.4 should be running
Steps to reproduce the bug
Run kubeadm to spin up a cluster, install istio
Version
1.1 snapshot.4
Installation
{{ Please describe how Istio was installed }}
Environment
ubuntu
Cluster state
{{ If you're running on Kubernetes, consider following the
instructions
to generate "istio-dump.tar.gz", then attach it here by dragging and dropping
the file onto this issue. }}