Describe your environment

This bug is not related to any specific environment
Thankfully no SDK version with this bug has been released yet

What happened?

PR 4863 added this change to measurement consumer which uses weakref.WeakSet under a non re-entrant lock to take a snapshot of existing reader storages. This works fine under normal circumstances but if python GC is instrumented with metrics, the act of creating weakref.ref objects makes it very likely to trigger a GC event which then tries to create more measurements that ends up causing a dead-lock because the lock used is non re-entrant.

Steps to Reproduce

1. Create a GC callback function that records GC duration in histogram metrics (unfortunately I am unable to provide exact code snippet at this point)
2. Run an application that causes frequent GC collections (it can be as simple as a dummy loop calling gc.collect)
3. The application is extremely likely (though not guaranteed) to dead lock and hang

Expected Result

The application should continue running as expected and emit GC metrics when instrumented

Actual Result

Applications using GC instrumentation are very likely to hang and dead lock forever

Additional context

changing the lock to RLock and using a list / tuple snapshot that makes it less likely to trigger GC (creating one new object instead of N weakref.ref objects) solves the issue. As far as I can tell this is not problematic as adding a measurement to a storage that is going be to garbage collected is safe (missed data points aren't a real risk here as there was no reference to the storage if it was being garbage collected anyways).

Would you like to implement a fix?

Yes

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}