chore(trainer): add data and model initializers guide#414
Conversation
|
🎉 Welcome to the Kubeflow SDK! 🎉 Thanks for opening your first PR! We're happy to have you as part of our community 🚀 Here's what happens next:
Join the community:
Feel free to ask questions in the comments if you need any help or clarification! |
There was a problem hiding this comment.
Pull request overview
Adds end-user documentation for Kubeflow Trainer “initializers” (dataset/model prefetch steps) and links it into the training docs and API reference to address #413.
Changes:
- Adds a new “Data and Model Initializers” guide page with examples and container-backend configuration/debugging.
- Extends the Trainer API reference to include initializer classes.
- Links the new guide from the training landing page and the main docs toctree.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| docs/source/train/initializers.rst | New guide explaining how to use dataset/model initializers, config knobs, and log debugging. |
| docs/source/train/index.rst | Adds a grid card linking to the new initializers guide. |
| docs/source/train/api.rst | Adds autoclass entries for initializer-related types in the API reference. |
| docs/source/index.rst | Adds the new initializers page to the main Trainer toctree. |
1Ayush-Petwal
force-pushed
the
docs/initializers-guide
branch
from
49f82b1 to
0abd70e
Compare
1Ayush-Petwal
changed the title
1Ayush-Petwal
force-pushed
the
docs/initializers-guide
branch
2 times, most recently
from
921027e to
f82ac9b
Compare
|
/ok-to-test |
|
/retest |
Fiona-Waters
left a comment
Fiona-Waters
left a comment
There was a problem hiding this comment.
Thanks @1Ayush-Petwal just one small question, otherwise lgtm
| from kubeflow.trainer.backends.container.types import ContainerBackendConfig | ||
|
|
||
| client = TrainerClient(backend_config=ContainerBackendConfig( | ||
| dataset_initializer_image="ghcr.io/kubeflow/trainer/dataset-initializer:v0.4.0", |
There was a problem hiding this comment.
We could use latest here for both images, what do you think?
| dataset_initializer_image="ghcr.io/kubeflow/trainer/dataset-initializer:v0.4.0", | |
| dataset_initializer_image="ghcr.io/kubeflow/trainer/dataset-initializer:latest", |
There was a problem hiding this comment.
Yes, that is better for the docs. I also updated the model_initializer_image to use latest.
Add docs/source/train/initializers.rst covering dataset and model initializers for the container backend (added in kubeflow#188, parallelised in kubeflow#313). Includes per-type code examples, combined usage, ContainerBackendConfig options, and debugging via get_job_logs(). Signed-off-by: Ayush Petwal <[email protected]>
- Fix model output path from /workspace/model-weights to /workspace/model to match the MODEL_PATH constant in constants.py - Clarify DataCacheInitializer is Kubernetes-only in the backend note and annotate the Available Initializers table row accordingly - Add DataCacheInitializer usage example with required fields (storage_uri, metadata_loc, num_data_nodes) and backend constraint note Signed-off-by: 1Ayush-Petwal <[email protected]>
Signed-off-by: 1Ayush-Petwal <[email protected]>
1Ayush-Petwal
force-pushed
the
docs/initializers-guide
branch
from
f82ac9b to
a6e9430
Compare
andreyvelich
left a comment
andreyvelich
left a comment
There was a problem hiding this comment.
Thanks @1Ayush-Petwal!
/lgtm
/approve
/hold in case @Fiona-Waters has additional comments.
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: andreyvelich The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
Nothing additional from my side. Thanks @1Ayush-Petwal |
What this PR does / why we need it:
docs/source/train/initializers.rst— new guide covering all 5 initializertypes (
HuggingFaceDatasetInitializer,S3DatasetInitializer,DataCacheInitializer,HuggingFaceModelInitializer,S3ModelInitializer),combined usage,
ContainerBackendConfigimage/timeout options, and debuggingvia
get_job_logs()autoclassentries for all 6 exported initializer types indocs/source/train/api.rstdocs/source/train/index.rsttrain/initializersto the Trainer toctree indocs/source/index.rstWhich issue(s) this PR fixes (optional, in
Fixes #<issue number>, #<issue number>, ...format, will close the issue(s) when PR gets merged):Fixes #413
Checklist: