Skip to content

Handle innocuous error messages during control plane startup #2168

@grampelberg

Description

@grampelberg

What problem are you trying to solve?

When the controller containers start up, they log a lot of connection refused lines. This is a timing issue between the proxy and the controller container. As the proxy is not yet ready to receive traffic, the connections get retried.

Unfortunately, this scares users and can provide misleading information around debugging. In the past, it was not possible to opt-out of these lines because of glog.

How should the problem be solved?

The underlying problem here is that a Sync is kicked off before the proxy is ready to forward traffic. This feels like a general linkerd problem, and possibly something that could help our users out.

I would like to see a way for sidecars to know when they can start sending traffic out. It feels like this could be a readiness probe for the proxy. Then, the controller code could wait for the probe to pass and continue starting.

Alternatives considered

  • Mute klog - there are valid messages that will just make debugging more difficult if klog is completely muted.
  • Toggle klog around sync - feels pretty magical and fragile.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions