The above results are using perf_crimson_msgr (client mode) to pressurize perf_crimson_msgr (server mode) or perf_async_msgr to understand apple-to-apple performance by CPU cores used.

Initially, both crimson and async msgr have similar performance curve, but further analysis shows they have different bottlenecks:

• Async msgr is mostly blocked by its locks at 16 CPUs;
• Crimson msgr is mostly blocked by some mempool counters in raw buffer at 16 CPUs;

Async msgr should share the same infrastructure about the buffer and mempool, but its immediate bottleneck is not there, which implies that its scaling issue might be synthetic.

For crimson msgr, if comment out the mempool counters in this PR, the CPU scaling problem is relaxed greatly, see the curve of crimson-poc.

In short:

• Both crimson and async msgr implements similar features, and their single-core performance looks similar when there is no racing;
• When goes to multiple cores, a single point of shared resource in the hot path can make crimson scale bad, with the end-to-end result approaches classic;

Contribution Guidelines

• To sign and title your commits, please refer to Submitting Patches to Ceph.

• If you are submitting a fix for a stable branch (e.g. "pacific"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.

Checklist

• Tracker (select at least one)
  • References tracker ticket
  • Very recent bug; references commit where it was introduced
  • New feature (ticket optional)
  • Doc update (no ticket needed)
  • Code cleanup (no ticket needed)
• Component impact
  • Affects Dashboard, opened tracker ticket
  • Affects Orchestrator, opened tracker ticket
  • No impact that needs to be tracked
• Documentation (select at least one)
  • Updates relevant documentation
  • No doc update is appropriate
• Tests (select at least one)
  • Includes unit test(s)
  • Includes integration test(s)
  • Includes bug reproducer
  • No tests