What exactly will happen to the clients if we block and don't respond within a reasonable time?
Previously, we answered UNAVAILABLE, which seemed more native to the gRPC/HTTP environment.

Also, what was the trigger of this change?

If a request timeout is not set on the client side, they will wait indefinitely.
If timeout is set, the client will get a DEADLINE_EXCEEDED result, which should be handled as a temporary error (we do so in the otel dest).

https://grpc.io/docs/guides/deadlines/#deadlines-on-the-client

We have seen that syslog-ng "gives up" too quickly if it cannot forward the message. If we have multiple otel source workers, and they receive requests at the same exact time, the destination, which grabs those messages might not be able to grab them quickly enough. So even if we used a devnull destination, which can process 2 million messages per sec, sometimes it can put backpressure on the otel source, if there is a high transient load coming from it. We can raise log-iw-size() to buffer more messages and flatten these transients, but it raises the memory footprint considerably.

We could have a timeout for this, like trying blocking post, but surrendering after x seconds and returning unavailable. But I believe the current implementation is completely functional, too, at least on the server side. It would be nice to have options for request timeout on the client side (otel dest).

Sounds reasonable. My only concern is that the original otel-source implementation was not necessarily a "thread-per-client" implementation. By blocking, I think we make it so.

True.

However, concurrent-requests() makes us able to scale to more clients than workers(), which should be sufficient. With it the workers() option detaches from limiting the number of clients (gRPC will handle them for us if there are enough ServiceCalls registered), and just configures the processing parallelization.