Studio: add a Disable Vision toggle to a model's Advanced Settings - #9383

Resync Vision after same-model reloads

When another tab or API client reloads the same model with a different disable_vision value, hydratingExistingModel remains false and loadedVisionDisabledByUser is already non-null, so this block leaves the editable disableVision value unchanged. The following blocks nevertheless update loadedDisableVision and the live image-gating state, leaving Advanced Settings showing the opposite of the running projector state; a later Apply or unrelated reload can then send the stale value and undo the external change. Preserve a genuinely pending local edit, but otherwise reseed this control when the incoming value differs from the previous loaded baseline.

Useful? React with 👍 / 👎.

Correct, and the divergence is mine rather than inherited. tensorParallel right above sets the control and its baseline in the SAME guarded spread, so the two cannot come apart. I split them deliberately -- loadedDisableVision has to track the running server or a failed switch rolls back to whatever the last seed happened to see -- and did not carry the consequence through: the baseline and loadedVisionDisabledByUser follow every poll while the control was seeded once.

Fixed in b6133b9. The rule moves out of the spread into shouldSeedVisionSwitch, next to resolveChatTemplateSeed, which is the pattern this file already uses for exactly this kind of guard:

if (previous.loadedVisionDisabledByUser === null) return true;   // never seeded
if (hydratingExistingModel) return true;                          // model/variant changed
if (previous.loadedDisableVision === null) return false;          // no baseline to compare
if (previous.loadedDisableVision === incoming) return false;      // nothing moved
return previous.disableVision === previous.loadedDisableVision;   // no pending local edit

Your caveat is the last line and it is the load-bearing one. "No pending edit" is measured against the OLD baseline, not against the incoming value, so a control the user moved and has not applied survives an external reload; only a control still sitting on its baseline gets resynced.

Eight cases pinned in a new vision-switch-seed.test.ts, plus a source-level check that the applier still routes through the resolver, since re-inlining the old guard would restore the divergence without failing any of the case tests. The old vision-toggle-gating.test.ts assertion pinned that inlined regex and is rewritten rather than dropped.

Three mutants. Two died immediately: never resyncing, and resyncing over a pending edit. The third -- deleting the loadedDisableVision === incoming early return -- survived, and it deserves the note: it is behaviourally equivalent through this call site, since the extra write puts back the value already there. I killed it by asserting the resolvers answer for that case rather than a store effect, on the grounds that the contract is what a future caller branches on. 4125 frontend tests pass, both tsc runs exit 0.

Charge retained inherited audio projectors

When a local GGUF is loaded with disable_vision=true while LLAMA_ARG_MMPROJ names an audio-only projector, the loader deliberately preserves that environment projector, but this estimator clears the projector charge based only on config.gguf_mmproj_file. In Auto GPU mode the inherited projector remains GPU-offloaded, so the training coexistence guard can approve an under-budgeted chat load and OOM the running training job. Include the retained environment projector in the estimate, or scrub it so the estimator and launch agree.

Useful? React with 👍 / 👎.

Correct, and it is the direct consequence of the previous fix: I taught the loader to keep an inherited audio-only projector and left the estimator reading config.gguf_mmproj_file alone, so a GPU-resident projector this config never names was billed nowhere. Under-charging here costs someone else their training run, which is the worst direction this guard can be wrong in.

Fixed in e26d42b. The estimator now charges LLAMA_ARG_MMPROJ when the launch will really open it, and exactly two things stop it: Studios own --mmproj, which overrides the env (argv after set_env) so one file loads rather than two, and the vision switch, asked through the loaders own _mmproj_env_is_audio_only so the two cannot answer differently.

Writing the tests turned up a second thing worth flagging, because my first version was wrong. I had also gated the charge on extra_args_disable_mmproj, and that is not a stopper: --no-mmproj / --no-mmproj-auto set params.no_mmproj, which suppresses Studios own resolution and the HF auto-download, but server-context.cpp:958 gates the load on a non-empty mmproj.path and never reads the field. So an inherited path loads straight through the opt-out. Same finding as the --no-mmproj-auto thread above, applied consistently this time. That clause is gone and a test pins its absence.

One deliberate widening to call out: this also charges an inherited projector when Vision is ON, which nothing charged before. That case was already under-budgeted for the same reason, it just was not what you reported. It can refuse a chat load that previously squeezed through, which is the safe direction for a guard whose job is protecting a running training job.

The charge is added to the local branchs return rather than to total_bytes, since that value decides local-vs-remote and must stay the main weights verdict. An inherited local projector alongside a REMOTE repo is still charged nowhere; that is unchanged and pre-existing, and I have left it rather than widen this further.

Four mutants, all killed: never charging, charging alongside Studios own, ignoring the audio-only rule, and re-adding the extras opt-out. 725 tests across the placement, batch-sizing, training-coexistence, gpu-memory, mmproj-fallback and server-args suites pass. The full backend suite on the previous head was clean at 432 names against a 432-name main baseline.

Charge the inherited projector after suppressing the configured one

When disable_vision=true, the configured projector is image-capable, and LLAMA_ARG_MMPROJ names an audio-only projector, the loader suppresses the configured path but deliberately retains the inherited audio projector; this new gate nevertheless skips the environment charge merely because config.gguf_mmproj_file is populated. Auto GPU admission therefore still underestimates VRAM and can approve a chat load that OOMs a concurrent training job. Fresh evidence in this revision is the added and not getattr(config, "gguf_mmproj_file", None) condition, which fails to account for the configured projector no longer winning once the switch removes its argv entry.

Useful? React with 👍 / 👎.

Correct, and it is a second, wider version of the same mistake. I gated the environment charge on config.gguf_mmproj_file being empty, which quietly assumes Studio's --mmproj always reaches the command line. argv beats the environment only when there IS argv.

Two things empty it while leaving the inherited path in place:

• the switch dropping an image-capable configured projector, which is exactly the combination you describe, and
• extras naming --no-mmproj, where llama_cpp.py skips the resolve entirely.

The second one I had actually argued for in my previous reply and then failed to apply: I said the opt-out does not unload an inherited path, which is true, but I did not notice that it also means Studio emits nothing of its own, so the inherited path wins there too.

Fixed in 0d984ee. The gate now asks what Studio emits rather than what the config names, and it reads the same answer the configured projector's own charge is built from, so the two cannot disagree about which single projector loads:

_studio_mmproj_on_argv = bool(
    getattr(config, "gguf_mmproj_file", None) and _dv_opens_projector
)

_dv_opens_projector is now bound before the switch rather than inside it, and the extras opt-out sets it False and clears _sized_attrs in the same place. That second part is a behaviour change beyond your report and worth naming: with --no-mmproj in the extras, the configured projector was being charged even though it never loads. It is now dropped. That is a reduction in the estimate, which is the risky direction for this guard, and I made it anyway because the file provably does not load -- charging it was the estimator disagreeing with the launch, which is the whole defect here.

Four mutants, all killed: restoring the old config.gguf_mmproj_file gate, assuming argv is always present, making the extras opt-out stop emptying the argv, and letting it empty the argv while keeping the charge. Two new tests, one per route to the empty command line, both measured against a baseline taken with the environment variable unset -- my first attempt compared against a baseline that itself charged the inherited projector and read a difference of zero.

813 tests across the placement, batch-sizing, training-coexistence, gpu-memory, mmproj-fallback, server-args and Metal suites pass.

Separately: this branch is now rebased onto main after #9063 was squash-merged, so the diff here is the Vision switch alone. I verified the rebase left no placement code behind, _shared_pool_mmproj, _mm_budgeted_gpus, _MMPROJ_FIT_FLOOR_CTX and _apple_model_size_fit all appear zero times in this PR's diff and are present in main.

Re-evaluate the inherited projector after paravirtual scrubbing

On a virtualized-Metal load with Vision disabled, an audio-only LLAMA_ARG_MMPROJ, and a remembered --mmproj-auto, _dv_env_mmproj_kept is true here, so the disabling override is omitted; however, the later _paravirtual_cpu_forced block unconditionally removes that environment path. The surviving --mmproj-auto can then rediscover an adjacent image projector, even though the user requested Vision off, while the capability probe still describes the removed audio-only file. Base this decision on the environment after the paravirtual scrub, or append the override after that scrub.

Useful? React with 👍 / 👎.

Correct on all three counts, and the root cause is one I introduced: _dv_env_mmproj_kept is computed from os.environ before the paravirtual block, so it describes an environment the launch does not end up with.

if _paravirtual_cpu_forced: pops both projector vars unconditionally and runs after the switch's block, so the audio-only file the switch kept is gone by launch. Two consequences, both the ones you name: the capability probe still reported an audio encoder the child does not have, and the --no-mmproj-auto override was skipped, leaving a remembered --mmproj-auto free to rediscover an adjacent image projector on a load the user asked to be text-only.

Fixed in afe47cf by folding the later scrub into the predicate rather than re-deriving it downstream:

_dv_env_mmproj_kept = (
    disable_vision
    and not _paravirtual_cpu_forced
    and _mmproj_env_is_audio_only(os.environ.get("LLAMA_ARG_MMPROJ"))
)

_paravirtual_cpu_forced is settled well before the probe, so one predicate now answers for the probe, the argv override and the scrub together, which is the property that was missing rather than any one of the three symptoms.

New test drives a virtualised device with --mmproj-auto in the extras and an audio-only inherited projector, and asserts all three: the var is gone, --no-mmproj-auto is emitted, and _mmproj_has_audio is False. Mutant killed by dropping the not _paravirtual_cpu_forced clause.

Record Vision state during Vulkan CPU recovery

When an automatic Vulkan launch crashes but the CPU replay succeeds, _apply_cpu_fallback_state returns success before reaching this assignment, so _disable_vision and _vision_disabled_by_user retain their previous/default values even though the recovered child was launched with Vision disabled. The load/status response then reports the wrong setting, causing the frontend to flip the control back on and making an unchanged disable_vision=true request fail runtime matching and reload the model; copy both Vision fields into the CPU-fallback state before its early return.

Useful? React with 👍 / 👎.

Correct. _apply_cpu_fallback_state is the last thing the recovery runs and its caller return Trues immediately after, before the load's own assignment of these two, so a recovered child reported the previous load's Vision state. That is worse now than it was a week ago: the status reseed resolver added earlier in this PR treats a moved baseline as an external reload, so a wrong disable_vision in the response is exactly the shape that flips the control back on.

Fixed in afe47cf. The helper takes both fields and records them next to _is_vision and _mmproj_has_audio, and both call sites pass them, using the same expression the load's own assignment uses since the replay carries the same switch and the same dropped projector.

Two tests. One drives the helper directly and asserts the two fields land while the rest of the recovery state still does. The other pins that both call sites pass them: only one is on the common path, so a keyword added to one and not the other is a silent half-fix, and the second needs a crash inside a replay this harness cannot stage. That one had to be tightened after it first matched the load's own self._vision_disabled_by_user = bool( and counted three uses where there are two.

Mark inherited projector suppressions as user-disabled

When a vision model has no resolved Studio projector but a readable image-capable LLAMA_ARG_MMPROJ supplies one, this branch removes that projector because the user disabled Vision, yet _dv_dropped_image_projector remains false because it is only set in the configured-projector branch. Consequently vision_disabled_by_user is reported false and the composer gives the generic missing/invalid-mmproj guidance even though turning the Vision switch back on restores the inherited projector; record this image-capable environment suppression as user-requested too.

Useful? React with 👍 / 👎.

Correct. _dv_dropped_image_projector was only ever set in the configured-projector branch, so an inherited image-capable projector that the switch scrubbed reported vision_disabled_by_user = false and the composer fell through to the generic missing/invalid-mmproj guidance -- pointing the user away from the one control that would restore it.

Fixed in afe47cf: the env scrub records the same signal the resolved path does, at the point the decision is made.

One deliberate narrowing. It is recorded only when the inherited path names a file that exists:

if _dv_env_mmproj and Path(_dv_env_mmproj).is_file():
    _dv_dropped_image_projector = True

A stale path drops nothing, so turning Vision back on changes nothing, and blaming the switch there sends the user to a control that cannot help -- the same reason the resolved path does not blame the switch for a missing file or a family mismatch. Unknown-but-present still reads image-capable, matching the rule the scrub itself just used.

Two tests, one per side, and both mutants died: removing the assignment fails the positive case, and widening it to if True: fails three of the existing "do not blame the switch" tests, which is what stops this becoming a blanket claim. 979 tests across the placement, paravirtual, mmproj-fallback, gpu-memory, Metal, batch-sizing, training-coexistence, server-args and TP suites pass.

Account for retained environment projectors before skipping the arbiter

When Manual GPU Memory uses zero layers with speculation off, Vision is disabled, the configured projector is image-capable, and LLAMA_ARG_MMPROJ names an audio-only projector, the loader suppresses the configured file but deliberately retains the inherited projector. This predicate nevertheless reports no projector because _load_keeps_a_projector inspects only the config, and zero_vram_chat_load does not detect LLAMA_ARG_MMPROJ, so the route skips acquire_for(CHAT) and can launch a GPU-offloaded audio projector alongside the current Images/Video owner. Include the retained environment projector in the zero-VRAM decision.

Useful? React with 👍 / 👎.

Block images when retained audio keeps multimodality true

When Vision suppresses a configured image projector but an inherited audio-only projector is retained, the response legitimately has both has_audio_input=true and vision_disabled_by_user=true. That makes loadedIsMultimodal true, and for the catalog's vision model the earlier branch returns success before reaching this new check, so the attachment picker accepts images even though the running server has only an audio encoder. Distinguish runtime image capability from generic multimodality, or evaluate the truthful Vision-disabled state before allowing the attachment.

Useful? React with 👍 / 👎.