Skip to content

Preserve ranking-loss gradients with weighted log-sigmoid - #4116

Merged
tomaarsen merged 3 commits into
huggingface:mainfrom
Afloat16:fix-ranking-loss-logspace
Oct 12, 2026
Merged

tomaarsen merged 3 commits into
huggingface:mainfrom
Afloat16:fix-ranking-loss-logspace

Conversation

@Afloat16

@Afloat16 Afloat16 commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

LambdaLoss computes a weighted sigmoid in probability space, then clamps it before taking the logarithm. For strongly misordered documents, this caps the loss at -log(eps) and gives the model zero ranking gradients. RankNetLoss inherits the same behavior. With scores [0, 1000] and relevance [1, 0], the natural-log RankNet loss is about 23 instead of 1000.

Compute the weighted log-probability directly with logsigmoid, and convert to binary logarithms afterward. This preserves the pair selection, ranking weights, and mean reduction while retaining gradients for badly misranked documents.

Validation: CPU tests import the complete native source package and run the production losses with a deterministic trainable score model at the model-input boundary. The 50 new cases compare losses and gradients against independent float64 binary cross-entropy references, covering both losses, both logarithm bases, multiple sigma values, large positive/negative score gaps, and variable-length padded batches. Before the change: 16 failed, 34 passed. After: 50 passed. The final targeted run included the existing ListMLE numerical regression cases: 62 passed, 16 deselected. Ruff lint and formatting checks pass for both changed files. No pretrained-model or CUDA training run was performed.

@tomaarsen

Copy link
Copy Markdown
Member

Hello!

The fix makes sense. The probability clamps can zero out gradients for misordered pairs even at fairly modest score differences. Using logsigmoid fixes that while preserving ordinary loss values and gradients within floating-point precision.

I've pushed a follow-up to move the tests into test_misc.py and reuse the existing helper. All 62 focused tests pass locally. Looks good to me now.

  • Tom Aarsen

@tomaarsen
tomaarsen merged commit abd266f into huggingface:main Oct 12, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants