…oroutine

Signed-off-by: Samuel Just <[email protected]>

WaitSubop, WaitTrimTimer, and WaitRepop are pipeline stages local to
the operation.  As such they don't actually provide any ordering
guarrantees as only one operation will ever enter them.  Rather, the
intent is to hook into the event system to expose information to an
administrator.

This poses a problem for OrderedConcurrentPhase as it is currently
implemented.  PipelineHandle::exit() is invoked prior to the op being
destructed.  PipelineHandle::exit() does:

  void exit() {
    barrier.reset();
  }

For OrderedConcurrentPhase, ~ExitBarrier() invokes ExitBarrier::exit():

    void exit() final {
      if (barrier) {
        assert(phase);
        assert(phase->core == seastar::this_shard_id());
        std::ignore = std::move(*barrier
        ).then([phase=this->phase] {
          phase->mutex.unlock();
        });
        barrier = std::nullopt;
        phase = nullptr;
      } else if (phase) {
        assert(phase->core == seastar::this_shard_id());
        phase->mutex.unlock();
        phase = nullptr;
      }
    }

The problem comes in not waiting for the phase->mutex.unlock() to occur.
For SnapTrimEvent, phase is actually in the operation itself.  It's
possible for that continuation

        ).then([phase=this->phase] {
          phase->mutex.unlock();

to occur after the last finally() in ShardServices::start_operation
completes and releases the final reference to SnapTrimEvent.  This is
harmless normally provided that the PG or connection outlives it,
but it's a problem for these stages.

For now, let's just remove these stages.  We can reintroduce another
mechanism later to set these event flags without an actual pipeline
stage.

This is likely a bug even with pipelines not embedded in an operation,
but we can fix it later -- https://tracker.ceph.com/issues/64545.

Fixes: https://tracker.ceph.com/issues/63647
Signed-off-by: Samuel Just <[email protected]>

PG must already be active+clean.

Signed-off-by: Samuel Just <[email protected]>

…tages

SnapTrimEvent doesn't actually do or block on GetOBC or Process --
remove those stages entirely.  Entering Process, in particular, causes
problems unless we immediately leave it as SnapTrimObjSubEvent needs to
enter and leave it to complete.  Entering one of the stages removed in
a prior commit had a side effect of exiting Process -- without that
exit SnapTrimEvent and SnapTrimObjSubEvent mutually block preventing
snap trim or client io from making progress.

This leaves no actual pipeline stages on SnapTrimEvent, which makes
sense as only SnapTrimObjSubEvent actually does IO.

Signed-off-by: Samuel Just <[email protected]>

Otherwise, it parks on Process until the repop completes blocking any
other repops, including client IO.  Since we don't actually care about
ordering, simply calling handle.complete() would also be viable, but
this is a valid usage of the stage and does provide information to an
operator.

Signed-off-by: Samuel Just <[email protected]>