🐛 Describe the bug

A scheduler node that has been removed from the graph (added to V.graph.removed_operations) currently will not be removed from the users variable of a SchedulerBuffer. This means that any consumers of the users variable will first have to check whether that user has been removed. This is not done in decide_inplace_update or get_read_write_buffers_sizes. Some sort of pruning step after dead_node_elimination is probably the best solution IMO.

Error logs

No response

Minified repro

No response

Versions

all

cc @ezyang @chauhang @penguinwu @voznesenskym @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @ColinPeppler @amjames @desertfire @aakhundov