The use of a buffer hides some of the type complexity of the inner
service (all of the type complexity, after #586). However, Linkerd
currently only uses buffers at the `Service` level, not at the
`MakeService` level. Since `MakeService`/`NewService`s tend to be
generic over both inner `MakeService` types _and_ over the produced
`Service`s (as well as potentially two future types), they probably have
much longer concrete types than `Service`s in most cases.

As @olix0r suggested, we can replace the `Lock` that's currently used to
ensure exclusive access to `Cache`s (which are at the `MakeService`)s
with `Buffer`s. The `Lock` is currently essentially doing something
quite similar to adding a `Buffer` anyway. Introducing buffers around
all caches erases the inner type for everything layered around the
cache, which should make overall type length much shorter.

This seems to have a fairly noticeable impact on build time and memory
use (see linkerd/linkerd2#4676). On my machine, running `make docker` on
#586 gets SIGKILLed (presumably by the OOM killer) after 7m53s. After
this change, `make docker` completes successfully after 1m44s.

Also, the `linkerd2-lock` crate can be removed, as it was used only by 
`Cache`.

To work around increased tail latencies when the buffer fills up, this
branch also sets the default buffer capacity to be equal to the 
default concurrency limit, rather than 10. This is fine, since the buffer
capacity isn't _actually_ what enforces a bound on proxy memory use.
The spawned tasks waiting on a full buffer are still sitting on the Tokio
executor, and it's actually the in flight limit that stops us from accepting
any more requests when we have too many in flight.

Depends on #586.
Fixes linkerd/linkerd2#4676.

Signed-off-by: Eliza Weisman <[email protected]>

This release increases the default buffer size to match the proxy's
in-flight request limit. This reduces contention in overload--especially
high-concurrency--situations, substantially reducing tail latency.

---

* update test-support clients and servers to be natively async (linkerd/linkerd2-proxy#580)
* Print build diagnostics in docker (linkerd/linkerd2-proxy#583)
* update test controllers to std::future/Tonic; remove threads (linkerd/linkerd2-proxy#585)
* buffer: Box the inner service's reponse future (linkerd/linkerd2-proxy#586)
* Eliminate Bind & Listen traits (linkerd/linkerd2-proxy#584)
* cache: replace Lock with Buffer (linkerd/linkerd2-proxy#587)

This release increases the default buffer size to match the proxy's
in-flight request limit. This reduces contention in overload--especially
high-concurrency--situations, substantially reducing tail latency.

---

* update test-support clients and servers to be natively async (linkerd/linkerd2-proxy#580)
* Print build diagnostics in docker (linkerd/linkerd2-proxy#583)
* update test controllers to std::future/Tonic; remove threads (linkerd/linkerd2-proxy#585)
* buffer: Box the inner service's reponse future (linkerd/linkerd2-proxy#586)
* Eliminate Bind & Listen traits (linkerd/linkerd2-proxy#584)
* cache: replace Lock with Buffer (linkerd/linkerd2-proxy#587)