The reason we can't just block for a second every time data comes in, is that eventually the agents "write" call on the socket will start timing out. This sort of timeout can also happen for other reasons e.g. APM Server isn't working properly, or there's a faulty proxy in between. In other words, we can't just say that whenever a write in the agent times out, it means the queue in APM Server is full.

When APM Server queue is full we could simply write "Write error: timeout" to the agent logs, but it would be a much better user experience if agents are able to write a more meaningful message to their logs.

After discussions with @urso, we can't get rid of the 1s timeout without making significant changes to libbeat. For the first iteration of v2, i suggest we should do the simple thing and keep the 1s timeout in the APM Server, but return a ERR_FULL_QUEUE response to the agent immediately (see #1240) if we've been unable to deliver the events to the internal beats queue within the 1s timeout.

WDYT @elastic/apm-agent-devs ?

v2: Investigate "full queue" behavior #1298

Description

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions