Studio: fine-tune Laya and Cloudflare Clef decision models and serve them - #12585

Studio can serve Laya decision models through the Decision API, but there is no way to train one on your own decisions, and Cloudflare's Clef and Clef-flash decision models are not supported at all. Picking convaiinnovations/laya or Cloudflare/clef-flash on the Train page treats it as a chat model, and FastDecisionModel.from_pretrained("Cloudflare/clef-flash") does not exist.

This PR adds decision model fine-tuning for both families, in Studio and in Python, and serves the results through the Decision API.

What you get

• Laya (convaiinnovations/laya): pick a checkpoint (Multilingual, English or Typed decisions) and LoRA or a full fine-tune on the Train page.
• Clef and Clef-flash (Cloudflare/clef, Cloudflare/clef-flash): QLoRA (default), LoRA or a full fine-tune. Clef-flash is also a stock Decision API model now, next to Laya.
• Datasets use the same format the Decision API takes: state, questions and gold columns. Decisions that cannot be used are skipped and counted.
• Up to 400 decisions are held out to report accuracy before and after training and to calibrate confidence, so a run ends with a line like Held-out accuracy 0.68 -> 0.78, calibration error 0.03.
• A finished run has a Use in Decision API button, and Settings > API lists your fine-tunes next to the stock models. Other accounts only see the fine-tune the owner picked.
• The same training is available in Python:

from unsloth import FastDecisionModel, DecisionTrainer

model, tokenizer = FastDecisionModel.from_pretrained("Cloudflare/clef-flash", load_in_4bit = True)
model = FastDecisionModel.get_peft_model(model, r = 16, lora_alpha = 16)
items, report = FastDecisionModel.build_dataset(rows, tokenizer, model)
train, held = FastDecisionModel.split_holdout(items)
DecisionTrainer(model = model, args = args, train_dataset = train, processing_class = tokenizer).train()
FastDecisionModel.calibrate(model, tokenizer, held)
model.save_pretrained_merged("clef_flash_finetune", tokenizer)

A saved Clef fine-tune keeps Cloudflare's layout, so it also loads with load_release_model from the Clef repo, with the calibrated temperature folded into the head.

Clef details

• Cloudflare's joint_schema_model.py (Apache-2.0) is vendored unmodified under unsloth/_vendor/clef with its license and a sha256 manifest. unsloth/models/clef.py imports the record encoding from it unchanged and adds a faster head forward.
• The backbone loads through FastModel, so Qwen3.5 uses the flash-linear-attention gated delta kernels, causal-conv1d and Unsloth's compiled modules. On Clef-flash, against the plain torch path on one B200:

• 4-bit uses Unsloth dynamic quantization (the vision tower and the gated delta in_proj layers stay in 16-bit, as in unsloth/Qwen3.8-27B-unsloth-bnb-4bit). Clef's backbone has a fine-tune merged into it, so it is quantized on load rather than taken from the stock Qwen 4-bit repos.
• The joint head is batched over the whole batch with no host syncs, its fp32 norm and pooling run in 2048-token chunks so long rows never hold an fp32 copy of the hidden states, and it compiles with no graph breaks. Its forward and backward take half the time at 3.5K tokens per row, and its own peak memory drops from 2.41 GB to 0.83 GB at 30K tokens. The head is only 1 to 3% of a QLoRA step, so whole-step time is unchanged.
• No vocabulary-sized logits are ever computed: the head reads the backbone's last hidden states, so no chunked cross-entropy is needed.
• float16 and bfloat16 both work. On GPUs without bfloat16 (T4) Qwen3.5 takes Unsloth's float32 path, as in the Qwen3.5 notebooks.
• Calibration fits the temperature against the gold labels. Fitting it to the soft consensus labels left fine-tuned models underconfident.
• Optional DecisionTrainer terms from Cloudflare's write-up: label_smoothing, brier_weight, ordinal_weight, permute_fields and kl_weight. None beat seed noise on accuracy and calibration together in our runs, so all are off by default.
• Clef needs an NVIDIA or AMD GPU. On a machine without one, Clef models are greyed out in Settings and training and the Decision API refuse them with a clear message.

Results

Clef-flash on LocalLLaMA/typed-decisions, 60 steps, LoRA r16, held out on 1,500 test decisions, one B200:

A QLoRA fine-tune saved merged and reloaded scores 78.0%. Clef (27B) QLoRA after 20 steps goes from 65.3% to 68.7% on 300 decisions with 26.4 GB peak.

Laya Multilingual on the same dataset, scored on its test split (2,000 decisions) on one NVIDIA L4:

Through Studio's Decision API over HTTP, stock Clef-flash answers the Clef README request in 84 to 160 ms once warm on a B200.

Other fixes

• Downloading a decision checkpoint now asks the Hub for its file list first. Asking for both layouts at once made Unsloth's incomplete snapshot check reject every decision repo.
• Saving a fine-tune over an existing folder is crash safe: files are staged and moved in with the completion marker last.
• A local eval file that is also a training file now warns and holds out from training instead of reporting training rows as held out.
• A cancel that arrives during calibration no longer saves the model.
• Laya never loads through FastLanguageModel or FastModel, so DecisionTrainer now applies Unsloth's gradient accumulation fix itself. Without it, transformers 5.0 to 5.5 (Studio's default) divided accumulated gradients by gradient_accumulation_steps twice.

Tested on

• The decision test suites pass on a B200 (bfloat16, 4-bit), and on Colab T4 (float16), L4, A100 and RTX PRO 6000. On the T4 the bfloat16 case is skipped because that GPU has no bfloat16.
• AMD Strix Halo (gfx1151, ROCm, Linux): tiny Clef trains in bfloat16, float16 and 4-bit with bitsandbytes, and Laya LoRA trains from 33.7% to 48.1% held-out accuracy in 20 steps. The two test failures it found (the gradient accumulation scale above, and a kernel check that only understood newer transformers) are fixed here. Windows on AMD cannot import Unsloth yet because there is no Triton wheel; that predates this PR.
• The CPU decision suite passes on both transformers 5.5 and 5.17.

Follow-ups: #12772 turns a plain language model (for example Qwen3.5-2B) into a decision model, and #12778 compiles the Laya encoder during training (about 1.23x faster LoRA steps).

Training needs Python 3.10 or newer and uses one GPU. It is not offered on Apple Silicon. If the Decision API is set to GPU, Laya answers on CPU while a run trains and moves back to the GPU when the run ends; Clef answers 503 while a run holds the GPU. Chat models, their training and the stock Decision API models work as before.

Screenshots

Before, on main: Laya is treated as a chat model.

After: the Train page shows Laya as a decision model, with a checkpoint picker and LoRA settings.

A finished run reports held-out accuracy and offers Use in Decision API.

Settings > API lists the fine-tunes next to the stock models.

UI: Decision API model choices

Settings > API > Decision API, the Model select (BEFORE = main at the merge base, AFTER = this branch). The three Laya checkpoints are unchanged; Clef-flash (19 GB) and Clef (55 GB) are added, and /api/settings/systemone lists 5 models instead of 3.