self-review: We don't need to store timeoutMs anymore. We can pass deadline directly to MailboxReceiveOperator.

i think releaseReceivingMailbox should only invalidate the mailbox cached inside the MailboxService.

MailboxOperator will have

1. info regarding the actual mailbox
2. the mailbox service that caches the mailboxID -> mailbox object mapping entry

we need to make the ownership relation a bit more clear. to me with the current model, cancel/close needs to be idempotent and states needs to be kept and even though it is only done 1 once per mailbox. each call into these state transition will have to be guarded by locks and it is not ideal.

IMO.

1. MailboxService ALWAYS owns the MailboxContentStreamObserver (and thus the GRPCReceivingMailbox)
   • Throughout the interaction of the GRPC layer, both the mailbox grpc stub (GRPCMailboxServer.open) and the MailboxReceiveOperator will access MailboxService.getReceivingMailbox and thus we cannot transfer the ownership of the receiving mailbox to the mailbox operator (e.g. delete it from the cache)
2. MailboxReceiveOperator DOES NOT OWN the observer. thus it should ONLY call method to interact with mailbox via mailboxService. This means:
   • it should not be acquiring any Receiving mailbox object, rather it should only use the APIs to get contents out of the receiving mailbox.

on the contrary, sending side, mailboxservice NEVER owns the GRPCSendingMailbox, b/c it is always constructed and initialized by the MailboxSendObserver, so mailbox send observer holds the reference to the StatusStreamObserver and thus responsible for calling close / cancel on it.

the relationship looks like this

 * ----------------------------------------------------------------------------
 * (Operator Layer)
 *            MailboxSendOperator ---------> MailboxReceiveOperator
 *                   |                                  |
 *                   |                                  |
 * ------------------|----------------------------------|----------------------
 *                   |                                  |
 * (MailboxService)  |                                  |  ( WAIT ON INIT )
 *                  \_/                                \_/                   
 *           SendingMailbox                      ReceivingMailbox
 * ------------------|---------------------------------/^\---------------------
 * (Physical Layer)  |                                  |
 * (e.g. GRPC)       |                                  |  ( INITIALIZE )
 *                   |                                  |
 *                  \_/                                 |
 *           StreamObserver -------------------> StreamObserver
 * ----------------------------------------------------------------------------

and the only place we require MailboxService to own the closable object is ReceivingMailbox.

This means

1. we need to make sure that MailboxReceiveOperator gets the mailbox but not store it
   • possibly changing the getReceivingMailbox API to TransferrableBlock receiveMail(MailboxID)
2. SendingMailbox can be change into a util-based initialization and doesn't need to go through MailboxServcie (but for the simplicity we can still keep the current way)

let's search for a better way to deal with already closed issue.
I think i read some article regarding close/half-close conditions. but we can follow up later.

This is not related to gRPC but our business logic. There can be concurrent cancellations/terminations of the stream so this can always happen.

i dont think this API is needed.

mailbox should only expose.
send, complete, cancel, close. whether it is closed should be an internal state of mailbox.

what's the reason for this? and how is this determined?
since it is controlled by the ReceiveOperator to move on to the next mailbox if the current one doesn'thave data. shouldn't we not put any wait here and simply return a null so that ReceiveOperator can decided what to do?

This is the timeout for the queue offer and not poll (it should be renamed). The receive operator will return null if there's no data in the receivingBuffer.

This should ideally be tied to the query deadline but we don't have it in MailboxContentStreamObserver right now. At present I kept it to a reasonably large number. We can consider either increasing or decreasing this.