Skip to content

Sidecar doesn't start since certificates are missing on the sidecar pod. #9659

@axel7born

Description

@axel7born

Describe the bug
We have a scenario, where the egress gateway originates TLS with certificates that are mounted to the path /etc/istio/egressgateway-certs/.
This path is automatically mounted, if a secret with the name istio-egressgateway-certs exists.

We configure the the TLS origination by a destination rule:

piVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: egressgateway-pinger
spec:
  host: pinger.services.acme.com
  subsets:
  - name: pinger
    trafficPolicy:
      portLevelSettings:
      - port:
          number: 9000
        tls:
          caCertificates: /etc/istio/egressgateway-certs/ca.crt
          clientCertificate: /etc/istio/egressgateway-certs/client.crt
          mode: MUTUAL
          privateKey: /etc/istio/egressgateway-certs/client.key
          sni: pinger.services.acme.com

Our pods have sidecars, that do not start since the sidecar expects to find certificates in
/etc/istio/egressgateway-certs/

I extracted parts of the sidecar envoy log:

2018-10-30T15:33:14.237072Z	info	Monitored certs: []envoy.CertSource{envoy.CertSource{Directory:"/etc/certs/", Files:[]string{"cert-chain.pem", "key.pem", "root-cert.pem"}}}
2018-10-30T15:33:14.237222Z	info	Opening status port 15020

2018-10-30T15:33:14.237236Z	info	Starting proxy agent
2018-10-30T15:33:14.237487Z	info	Received new config, resetting budget
2018-10-30T15:33:14.237497Z	info	Reconciling retry (budget 10)
2018-10-30T15:33:14.237525Z	info	Epoch 0 starting
2018-10-30T15:33:14.238236Z	info	Envoy command: [-c /etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster client.default --service-node sidecar~100.96.1.17~client-845b478b6d-lkwgd.default~default.svc.cluster.local --max-obj-name-len 189 --allow-unknown-fields -l warning --v2-config-only]
[2018-10-30 15:33:14.270][16][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:243] gRPC config stream closed: 14, no healthy upstream
[2018-10-30 15:33:14.271][16][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:43] Unable to establish new stream
[2018-10-30 15:33:14.284][16][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:243] gRPC config stream closed: 14, no healthy upstream
[2018-10-30 15:33:14.284][16][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:43] Unable to establish new stream
2018-10-30T15:33:15.929821Z	info	Envoy proxy is NOT ready: 2 errors occurred:

* envoy stat missing: cluster_manager.cds.update_success
* envoy stat missing: listener_manager.lds.update_success
2018-10-30T15:33:17.929777Z	info	Envoy proxy is NOT ready: 2 errors occurred:

* envoy stat missing: cluster_manager.cds.update_success
* envoy stat missing: listener_manager.lds.update_success
[2018-10-30 15:33:19.366][16][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:226] gRPC config for type.googleapis.com/envoy.api.v2.Cluster update rejected: unable to read file: /etc/istio/egressgateway-certs/client.crt
[2018-10-30 15:33:19.366][16][warning][config] bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_mux_subscription_lib/common/config/grpc_mux_subscription_impl.h:70] gRPC config for type.googleapis.com/envoy.api.v2.Cluster rejected: unable to read file: /etc/istio/egressgateway-certs/client.crt
[2018-10-30 15:33:19.459][16][warning][config] src/envoy/utils/mixer_control.cc:171] ExtractInfo  metadata missing:
[2018-10-30 15:33:19.461][16][warning][config] src/envoy/utils/mixer_control.cc:171] ExtractInfo  metadata missing:
...
2018-10-30 15:33:19.529][25][warning][config] src/envoy/utils/mixer_control.cc:171] ExtractInfo  metadata missing:
[2018-10-30 15:33:19.529][25][warning][config] src/envoy/utils/mixer_control.cc:171] ExtractInfo  metadata missing:
[2018-10-30 15:33:19.532][25][warning][config] src/envoy/utils/mixer_control.cc:171] ExtractInfo  metadata missing:
[2018-10-30 15:33:19.532][25][warning][config] src/envoy/utils/mixer_control.cc:171] ExtractInfo  metadata missing:
[2018-10-30 15:33:19.532][25][warning][config] src/envoy/utils/mixer_control.cc:171] ExtractInfo  metadata missing:
[2018-10-30 15:33:19.532][25][warning][config] src/envoy/utils/mixer_control.cc:171] ExtractInfo  metadata missing:
...
2018-10-30T15:33:19.935145Z	info	Envoy proxy is NOT ready: 1 error occurred:

* envoy stat missing: cluster_manager.cds.update_success
2018-10-30T15:33:21.935445Z	info	Envoy proxy is NOT ready: 1 error occurred:

* envoy stat missing: cluster_manager.cds.update_success
2018-10-30T15:33:23.935120Z	info	Envoy proxy is NOT ready: 1 error occurred:

* envoy stat missing: cluster_manager.cds.update_success
2018-10-30T15:33:25.934052Z	info	Envoy proxy is NOT ready: 1 error occurred:
...

Expected behavior
We would expect, that the sidecar ignores the missing certificates, as it doesn't need them.
This seems to be the behaviour in 1.0.
Or some method to specify the gateway, where the destination rule should be active.

Steps to reproduce the bug

  • Create a secret with the required name for egress certificates.
  • Deploy a Destination rule like in the example above.
  • Start a pod with a sidecar.

Version
Istio 1.1.0-snapshot.2 - October 2018 Release
and the current master.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions