Skip to content

Adding unique_string support#129

Merged
bethac07 merged 11 commits intocp_plus_stardistfrom
cp_output_fix
Nov 19, 2025
Merged

Adding unique_string support#129
bethac07 merged 11 commits intocp_plus_stardistfrom
cp_output_fix

Conversation

@rajavishah
Copy link
Copy Markdown
Collaborator

No description provided.

@rajavishah rajavishah marked this pull request as draft September 23, 2025 18:06
@bethac07
Copy link
Copy Markdown
Contributor

Hmm, worth thinking about if should be in inputs too. I'm not mad about having it there, necessarily?

@bethac07
Copy link
Copy Markdown
Contributor

Also, can we have this target main and then rebase the cp_plus_stardist branch after? We want to get back into the habit of small PRs to main, rather than mega-PRs, if we can. TY!

@rajavishah
Copy link
Copy Markdown
Collaborator Author

Agree on aiming for smaller PRs to main going forward!
For this one though, my branch depends on files that only exist in cp_plus_stardist, so switching it to main would bring in that whole branch’s history. I’d suggest merging this into cp_plus_stardist first, then we can rebase and start doing smaller PRs to main from there.

@rajavishah
Copy link
Copy Markdown
Collaborator Author

Hmm, worth thinking about if should be in inputs too. I'm not mad about having it there, necessarily?

Might be! I can may-be think of a case where an algorithm could take both “raw images” and “mask images” as inputs, both .tif. Schema says type: image, format: tif for both, but without unique_string we can’t automatically tell them apart if they’re in the same folder.

@bethac07
Copy link
Copy Markdown
Contributor

bethac07 commented Oct 1, 2025

Other than also implementing it in inputs, anything else left needed to convert from draft to ready?

@rajavishah
Copy link
Copy Markdown
Collaborator Author

I need to test this passing a folder of images (for all algorithms) & inputs that's all

@rajavishah rajavishah marked this pull request as ready for review October 17, 2025 19:16
@rajavishah
Copy link
Copy Markdown
Collaborator Author

It's not ready for review yet - I am trying to resolve rebase conflcits

@rajavishah rajavishah marked this pull request as draft October 17, 2025 19:19
…ration

Now loads a single image as ndarray and multiple outputs as list of ndarrays
@rajavishah
Copy link
Copy Markdown
Collaborator Author

Great, Single and multiple output handling works correctly for algorithms whose Docker images contain an importable Python module, but fails for algorithms like classical_segmentation, where the entry point is a standalone script

@bethac07
Copy link
Copy Markdown
Contributor

Is it failing for a particular interface(s)? Feel free to add more details if you want help chasing it down.

- Replaced per-type output blocks with single dynamic loader
- Added dynamic workspace + display_data registration for all outputs
- Improved measurement saving and logging consistency
[fix] wildcard * support for outputs
@rajavishah
Copy link
Copy Markdown
Collaborator Author

rajavishah commented Oct 23, 2025

Nope, Now everything works fine across all interfaces, but the output mount path must be /data/output - otherwise, it fails.

Why?
In original cp-plugin (take RunCellpose), all inputs and outputs live under a single shared mount - /data. The module reads from /data/img so there’s no conflict because everything exists inside the same top-level directory.

In Bilayers, we take a slightly different approach - each output can go to a user-defined path (e.g. /bilayers/output, /bilayers/models, etc.). That’s fine in general, and it means multiple directories might be mounted.

However, if the user sets both input and output to the same container path (for example, --save_dir /bilayers/input_images), Docker ends up getting two mounts for the same location:

docker run \
  -v /host/input_images:/bilayers/input_images \
  -v /host/output_images:/bilayers/input_images \
  ...

When this happens, the second mount overrides the first one, so the input files become invisible inside the container.

Now, if we mount only unique folders (to avoid duplicate mounts), the container only sees what’s explicitly mounted - for eg.

docker run \
  -v /host/input_images:/bilayers/input_images \
   ....

In this case, only /bilayers/input_images exists inside the container.
So, if the algorithm tries to write outputs to a different path (say /host/input_images), that path isn’t mounted to the host at all - meaning any files written there stay inside the container and are lost when it exits
If we only mount one unique input folder, the plugin can’t “see” or persist outputs because the output path simply doesn’t exist on the host

P.S.: I've the draft-code that fails at this point for mounting (haven't pushed)

@rajavishah
Copy link
Copy Markdown
Collaborator Author

But otherwise, help me break the current plugin just at save_dir in any algo define path as /data/output

@bethac07
Copy link
Copy Markdown
Contributor

Sorry, I don't follow one small point here- why isn't the conclusion "save_dir can't be the input directory", rather than "save_dir must be one and only one thing"?

@rajavishah
Copy link
Copy Markdown
Collaborator Author

rajavishah commented Oct 23, 2025

Well, algorithms may allow to write in the same directory as input (eg. cellpose does) - to advocate that case

separated input/output mounts, added fallbacks, deduplication, and safe image saving
@rajavishah
Copy link
Copy Markdown
Collaborator Author

Okay, works fine now - when we give unique input & output directories!

Failed for Instanseg this time, (before commit e88849d), it was generating some outputs for the same set of images. But the failure is mostly related to the algorithm not plugin

Error Message : 
Attempted file (/bilayers/input_images/DNA.tif) load with reader: ... failed with error: No module named 'xmlschema'
NotImplementedError: Got 3D input, but bilinear mode needs 4D input

@rajavishah rajavishah marked this pull request as ready for review October 23, 2025 20:20
@rajavishah
Copy link
Copy Markdown
Collaborator Author

Outputs aren’t written to the default mounted folder because, like standard cp plugins, this module only produces data in memory - saving them to disk is handled separately via the SaveImages module if the user chooses.

@rajavishah rajavishah mentioned this pull request Oct 23, 2025
Copy link
Copy Markdown
Contributor

@bethac07 bethac07 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@bethac07 bethac07 merged commit 235ff84 into cp_plus_stardist Nov 19, 2025
2 checks passed
@bethac07 bethac07 deleted the cp_output_fix branch November 19, 2025 16:14
bethac07 added a commit that referenced this pull request Nov 19, 2025
* Moving existing src folder in src/bilayers

* Changed type hints like str | None to Optional[str] to support py=3.9

* [gradio] Improve mandatory parameter/input validation and error handling

- Validate both None and empty string inputs for mandatory parameters

- Update error messaging to clearly flag missing required values

- Refactor logic to skip optional parameters when no input is provided

* [jupyter] optional bug resolution

* unify top-level text creation

* linkml schema rule respective changes for modifying citations

* ruff fixes for the modified code

* removed script,module fields from spec file

* [docs] updation

* [mod] test config file

* [mod] docker-build workflow on self-hosted runners

* [mod] workflow on push branch changed to main

* [del] extra -f flag in build_interface nox session

* resolved most of warnings

* Fixed Documentation for now by removing LinkML docs

* CLI Usage according to updated folder structure

* Added tests

* [add] readme

* [add] stardist stuff

* basic skeleton, need to change

* [mod] category, module_name in cp_plugin

* [mod] cp plugin

* [updated] changes

* Measurer changed to Measurement

* [resolved] podman and docker in one logic and some small tweaks

* [update] temp folder preserve setting

* [mod] cp changes on new reviews

* [try] docs fix

* [basic] workflow changes

* [fix] imports

* Don't use label images as input for segmentation

* typos

* fix module name format

* build stardist_inference too

* Non "none with float/int" modules run

* Split numeric parameters that use "None" as a "donotset"

* [fix] handle booleans better

* [fix] use tmp_path helper for consistent temp file

* Adding unique_string support (#129)

* [add] unique_string at parsing level

* [mod] use glob-based matching for outputs

* [fix] '

* [fix] normalize bool to avoid string/True confusion

* [readd] somehow deleted stardist_inference.py

* chore: trigger GitHub recheck after merge

* [fix] correctly handle single vs multiple output files in plugin generation

Now loads a single image as ndarray and multiple outputs as list of ndarrays

* [fix] unify output handling

- Replaced per-type output blocks with single dynamic loader
- Added dynamic workspace + display_data registration for all outputs
- Improved measurement saving and logging consistency

* [add] unique_string support for inputs

[fix] wildcard * support for outputs

* [add] dynamic mount handling

separated input/output mounts, added fallbacks, deduplication, and safe image saving

---------

Co-authored-by: Nodar Gogoberidze <[email protected]>
Co-authored-by: bethac07 <[email protected]>
Co-authored-by: Beth Cimini <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants