You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Running connected components leaves dataframes cached in the spark runtime which can take up a substantial amount of memory when it is run many times. Specifically, the dfs in lastRoundPersistedDFs do not get cleaned up after convergence == true
Since the cached dataframe is not the one returned by the run method, the caller is unable to unpersist them without clearing all persisted dataframes.