Repository navigation
Resume RND exploration schedules from PPO checkpoints - #240
Merged
ClemensSchwarke merged 3 commits intoOct 8, 2026
Merged
Conversation
Collaborator
|
Hi @Afloat16, |
ClemensSchwarke
approved these changes
Oct 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PPO checkpoints restore RND parameters and Adam state but omit the Python
update_counter. Loading a checkpoint restarts step and linear exploration schedules even when every model and optimizer has been restored. This changes the next intrinsic reward and the subsequent policy objective after resuming.Persist
rnd_update_counteras plain integer PPO checkpoint metadata and restore it when the RND state is selected for loading. Older checkpoints default to the historical initial count of zero. Selective loads that exclude RND retain the current counter. The RND model state remains tensor-only for the existing distributed broadcast path, and the training step gains no new tensor conversion or synchronization.Six new cases construct actual PPO, MLPModel, RolloutStorage and RND components, perform on-disk weights-only round trips, and compare the next reward with independent step/linear formulas. They also cover legacy checkpoints, selective loads, no-RND checkpoints and the broadcast-state constraint. The complete original PPO and RND test modules are included unchanged in the matched before/after runs.
Genuine CPU validation with Torch 2.7.1, NumPy 1.26.4 and the complete package source: the same 31 collected cases change from 3 failures and 28 passes on the original source to 31 passes with the fix, including all 25 tests in the affected original modules unchanged. No tests were skipped. The final source passes the pinned Ruff 0.14.0 checks and the repository's complete
pre-commit run --all-fileshooks.CUDA, simulators, distributed execution and ONNX export are outside this validation scope.