@apaszke is likely going to reject this PR :)
Regardless, this is a good POC.
I think the right approach is to do it somewhere deep inside the autograd engine.

(added unit test)

(ah, our comments crossed en-route, or rather, I just wrote and pasted without noticing there were new comments :-) )

alright, fair enough. Can we keep this open until someone implement/proposes a better solution (no need to merge, but at least its a place-holder to remind us to find a better way to do something similar?)

(or ... could we merge for now, and then replace the user API method, ie retain_grad() with a better implementation, at some indefinite point in the future?)

I'll chat with @apaszke today and discuss, we'll keep this open for now.

@apaszke 's main reservations with this PR is that the PR in it's current state creates a reference cycle (which means higher memory usage until the GC kicks in). instead of holding onto self, you can create a weakref on self instead.

Also, the method can insert the same hook multiple times, which is unnecessary

closing in favor of #2199
Hugh's attribution has been preserved.

Awesome, thanks! :-)