[12.9.x] Manual backport of #3308: lock the global callback map - #3376

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

/ok to test 24eeaf5

Hi Mike, could you review this 12.9.x backport of #3308 (global callback map lock)?

Did you generate this from cybind? Generated files should always be handled that way to avoid things getting out of sync. It doesn't look like it updated the hash at the top of the file.

Did you generate this from cybind? Generated files should always be handled that way to avoid things getting out of sync. It doesn't look like it updated the hash at the top of the file.

@mdboom It's not generated, for two reasons: (1) 12.9.x uses a Tempita template (runtime.pyx.in), and I'm not sure how cybind can still produce that; (2) regenerating changes far more lines than this PR does, which would make it hard to review.

I updated the hash with grep -v 'CYTHON-BINDINGS-GENERATED' runtime.pyx.in | sha256sum and committed it. But please let me know if you'd still prefer going through cybind. If so, could you point me to how to generate the Tempita (.pyx.in) files with it?

/ok to test 86320d7

Ok. It looks like the recent runtime changes (which don't have .pyx.in anymore) have not been backported to 12.9.x yet, and we are also waiting on #3410, which is waiting on #3322, to catch up to cybind main.

I suggest we get things on 12.9.x modernized with #3410 and #3322, then update runtime so it uses the new generator, then come back to this.

@lijinf2 : Now that both main and 12.9.x branches have been updated, you should be able to generate directly from cybind for these changes.

/ok to test 45e4918

@lijinf2 : Now that both main and 12.9.x branches have been updated, you should be able to generate directly from cybind for these changes.

Thanks @mdboom. After merging the latest 12.9.x and regenerating with cybind, driver.pyx and runtime.pyx match 12.9.x, because #3429 already brought in the #3308 fix. So this PR no longer touches them.

It now only adds the regression test and a release note.

Ready for review.

This looks good. However, (as of this morning), no longer need to put these things on the 12.9.x branch. It should go in the cuda_bindings_12 subdirectory on main now. I would just close this PR and open a new one.

Thanks @mdboom, that makes sense. I opened #3462 against main with the test and release note moved to cuda_bindings_12, so I'm closing this one.