kOps clusters will not start with ci/latest version of Kubernetes

Me, @olemarkus and @rifelpet tried to find the root cause, but couldn't:
https://testgrid.k8s.io/kops-misc#kops-aws-misc-amd64-conformance

/kind bug