[mt] Basic support for regularization.#11914
Merged
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds basic regularization support for multi-target trees in XGBoost, enabling l1/l2 regularization, min_split_loss (gamma), and max_delta_step parameters. It also makes split gradients optional for tree objectives and includes several code cleanups.
Key Changes:
- Implemented regularization parameters (lambda, alpha, gamma, max_delta_step) for multi-target tree training across CPU and GPU updaters
- Modified Python API to allow tree objectives to optionally return None for split gradients, falling back to value gradients
- Added comprehensive test coverage for regularization features with both single and multi-target scenarios
Reviewed changes
Copilot reviewed 23 out of 23 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/cpp/tree/test_tree_stat.cc | Added new test classes and cases for lambda, alpha, and max_delta_step regularization with multi-target support; refactored test helper functions |
| tests/cpp/tree/test_tree_policy.cc | Added multi_strategy parameter configuration for multi-target tests; removed redundant test cases |
| tests/cpp/tree/test_constraints.cu | Updated CompareFeatureList function signature to use const span and thrust constant iterators |
| src/tree/updater_quantile_hist.cc | Changed CHECK macros to LOG(FATAL) for constraint validation in multi-target mode |
| src/tree/updater_gpu_hist.cuh | Moved constraint checks to constructor; updated to use const feature set spans; removed debug synchronize check |
| src/tree/updater_gpu_hist.cu | Updated feature set queries to use const spans; removed redundant SetDevice calls; added gradient shape validation |
| src/tree/param.h | Minor code formatting improvement in CalcGain function |
| src/tree/multi_target_tree_model.cc | Added CHECK to ensure weight vector is not empty in SetRoot |
| src/tree/gpu_hist/multi_evaluate_splits.cuh | Changed buffer type from device_vector to DeviceUVector; updated type from uint32_t to bst_node_t |
| src/tree/gpu_hist/multi_evaluate_splits.cu | Improved variable naming; removed unused parent_gains buffer; updated invalid split handling |
| src/tree/gpu_hist/expand_entry.cuh | Added braces for single-line if statement; updated comment about invalid Hessian handling |
| src/tree/gpu_hist/evaluator.cu | Updated SortHistogram to use Context parameter; switched from caching allocator to CTP for thrust operations |
| src/tree/gpu_hist/evaluate_splits.cuh | Added Context parameter to SortHistogram function signature |
| src/tree/gpu_hist/evaluate_splits.cu | Updated SortHistogram call to pass Context parameter |
| src/tree/constraints.cuh | Updated Query function to use const spans; improved documentation formatting |
| src/tree/constraints.cu | Updated Query function implementation to accept const spans |
| src/learner.cc | Removed ConfigureTargets method (cleanup) |
| src/common/random.h | Added SetDevice calls to ensure feature sets are on correct device; removed unused includes |
| python-package/xgboost/objective.py | Changed TreeObjective.split_grad return type to Optional and default return to None |
| python-package/xgboost/core.py | Added logic to handle None return from split_grad and swap gradient roles accordingly |
| doc/parameter.rst | Enhanced parameter documentation with references to model tutorial and improved categorical feature note |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 23 out of 23 changed files in this pull request and generated 6 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Member
Author
|
cc @rongou |
rongou
approved these changes
Jan 7, 2026
Member
Author
|
Line 240 in f5ff41e Both conditions are evaluated to true, then the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ref #9043
The PR removes the debug flag requirement to ease testing. But the feature is still working in progress.