Testing Strategy #38

gnodar01 · 2024-09-10T18:00:19Z

gnodar01
Sep 10, 2024
Maintainer

We currently haven't built out our testing suite, but we plan to do so in the future (after schema validation and documentation).

The current workflow is this:

decide on a change (e.g. add flag that lets config.yml author to specify arg=val instead of existing arg val syntax)
implement it in code (parser-generator)
modify an existing config.yml or create a new one, and run bilayers to build
launch an interface (e.g. Gradio), and modify the various widgets (i.e. provide values, select options, etc) in such a way that it will end up producing the corresponding CLI command under test (e.g. algo_name arg=val [other stuff ...])
do a bunch of print statements and eyeball to make sure it looks right
hope you didn't miss anything
go back to 2. if you spot an error

Creating tests in Bilayers that automates the above process is a bit tricky. First, we have the change we made to the config.yml syntax. That can be tested via schema validation, i.e. modify the yaml schema to allow for arg=val syntax. Second, we have the changes we made to the parser-generator, i.e. has the parser been properly modified to account for the new feature, and do the generator + gradio/jupyter templates generate the correct logic (such that the logic correctly transforms the widget values to a valid CLI command). The second is harder to test, because it requires manual interaction at the interface level (step 4.)

A possible solution here is to not rely on interaction with Gradio/Jupyter specifically, but instead to create a third test-specific virtual interface. This virtual interface won't have a GUI. Instead it will be a collection of Python classes that match up with all of our interface types (e.g. Checkbox, Integer, Files, Textbox, Radio, Float, etc.) They'll also have internal methods that simulate user behavior. For instance, the Checkbox class will have a selectRandom which will randomly "check" some amount of the options, and selectNth to select the nth checkbox.

The template for the virtual interface will have very similar logic to generate_cli_command/construct_cli_command in the Gradio and Jupyter templates (in fact we should probably consolidate the internal logic of those two in a single template that the Gradio and Jupyter and virtual interface templates just import, so that it really will be identical).

The CLI command will not be for any real algorithm (e.g. cellpose). Instead it will be a pseudo algorithm (or perhaps a collection of them). The job of the pseudo algorithm is "simply" to take in the CLI args, parse them (via argparse, or better yet docopt, or possibly a third thing if docopt is insufficient in some way), and succeed if they are correctly specified. Note that this is different the the validation that happens at the schema validation step. It will be python code, so it can do arbitrarily complex things like interaction between args. The pseudo algorithm is essentially the test suite (or alt. each pseudo algorithm in the collection is a unit test). For instance the pseudo algorithm will have an argument parser that looks for pseudo_alg [opts] arg1=val1. It expects the required argument of arg1 to have a value val1, and for the CLI command to use the arg=val syntax.

TLDR: we generate a cli command for a pseudo algorithm based off of a simulated user interaction with a virtual interface.

gnodar01 · 2024-09-10T18:00:24Z

gnodar01
Sep 10, 2024
Maintainer Author

A note on docopt: It is great. It is so much easier to use than argparse. The short version is you write the CLI help text, and it generates the appropriate parser for it. The "negative" is that the help text has to be POSIX-compliant CLI grammar (i.e. ieee std 1003.1). For the cases we want to test that somehow can't be specified via POSIX-compliant grammar, we can fallback to the builtin argparse (although as far as I understand if you use argparse it always generates a default help-text that is compatible with docopt, so unless we're testing something really funky we should not need to fall back to it).

For example cellpose -h produces a decently complex help text. It's complex in the sense that the number of options is large. Some minor corrections are needed (there are 3 unmatched opening ( that need to be closed). After those corrections, the help text is valid (correctly, although pretty badly specified). The following produces a parser for cellpose:

import sys
import docopt

cellpose_help_text = '''
usage: cellpose [-h] [--version] [--verbose] [--use_gpu] [--gpu_device GPU_DEVICE] [--check_mkl] [--dir DIR]
                [--image_path IMAGE_PATH] [--look_one_level_down] [--img_filter IMG_FILTER]
                [--channel_axis CHANNEL_AXIS] [--z_axis Z_AXIS] [--chan CHAN] [--chan2 CHAN2] [--invert]
                [--all_channels] [--pretrained_model PRETRAINED_MODEL] [--add_model ADD_MODEL] [--unet]
                [--nclasses NCLASSES] [--no_resample] [--net_avg] [--no_interp] [--no_norm] [--do_3D]
                [--diameter DIAMETER] [--stitch_threshold STITCH_THRESHOLD] [--min_size MIN_SIZE] [--fast_mode]
                [--flow_threshold FLOW_THRESHOLD] [--cellprob_threshold CELLPROB_THRESHOLD] [--anisotropy ANISOTROPY]
                [--exclude_on_edges] [--save_png] [--save_tif] [--no_npy] [--savedir SAVEDIR] [--dir_above]
                [--in_folders] [--save_flows] [--save_outlines] [--save_ncolor] [--save_txt] [--train] [--train_size]
                [--test_dir TEST_DIR] [--mask_filter MASK_FILTER] [--diam_mean DIAM_MEAN]
                [--learning_rate LEARNING_RATE] [--weight_decay WEIGHT_DECAY] [--n_epochs N_EPOCHS]
                [--batch_size BATCH_SIZE] [--min_train_masks MIN_TRAIN_MASKS] [--residual_on RESIDUAL_ON]
                [--style_on STYLE_ON] [--concatenation CONCATENATION] [--save_every SAVE_EVERY] [--save_each]
cellpose parameters
optional arguments:
  -h, --help            show this help message and exit
  --version             show cellpose version info
  --verbose             show information about running and settings and save to log
hardware arguments:
  --use_gpu             use gpu if torch with cuda installed
  --gpu_device GPU_DEVICE
                        which gpu device to use, use an integer for torch, or mps for M1
  --check_mkl           check if mkl working
input image arguments:
  --dir DIR             folder containing data to run or train on.
  --image_path IMAGE_PATH
                        if given and --dir not given, run on single image instead of folder (cannot train with this
                        option)
  --look_one_level_down
                        run processing on all subdirectories of current folder
  --img_filter IMG_FILTER
                        end string for images to run on
  --channel_axis CHANNEL_AXIS
                        axis of image which corresponds to image channels
  --z_axis Z_AXIS       axis of image which corresponds to Z dimension
  --chan CHAN           channel to segment; 0: GRAY, 1: RED, 2: GREEN, 3: BLUE. Default: 0
  --chan2 CHAN2         nuclear channel (if cyto, optional); 0: NONE, 1: RED, 2: GREEN, 3: BLUE. Default: 0
  --invert              invert grayscale channel
  --all_channels        use all channels in image if using own model and images with special channels
model arguments:
  --pretrained_model PRETRAINED_MODEL
                        model to use for running or starting training
  --add_model ADD_MODEL
                        model path to copy model to hidden .cellpose folder for using in GUI/CLI
  --unet                run standard unet instead of cellpose flow output
  --nclasses NCLASSES   if running unet, choose 2 or 3; cellpose always uses 3
algorithm arguments:
  --no_resample         disable dynamics on full image (makes algorithm faster for images with large diameters)
  --net_avg             run 4 networks instead of 1 and average results
  --no_interp           do not interpolate when running dynamics (was default)
  --no_norm             do not normalize images (normalize=False)
  --do_3D               process images as 3D stacks of images (nplanes x nchan x Ly x Lx)
  --diameter DIAMETER   cell diameter, if 0 will use the diameter of the training labels used in the model, or with
                        built-in model will estimate diameter for each image
  --stitch_threshold STITCH_THRESHOLD
                        compute masks in 2D then stitch together masks with IoU>0.9 across planes
  --min_size MIN_SIZE   minimum number of pixels per mask, can turn off with -1
  --fast_mode           now equivalent to --no_resample; make code run faster by turning off resampling
  --flow_threshold FLOW_THRESHOLD
                        flow error threshold, 0 turns off this optional QC step. Default: 0.4
  --cellprob_threshold CELLPROB_THRESHOLD
                        cellprob threshold, default is 0, decrease to find more and larger masks
  --anisotropy ANISOTROPY
                        anisotropy of volume in 3D
  --exclude_on_edges    discard masks which touch edges of image
output arguments:
  --save_png            save masks as png and outlines as text file for ImageJ
  --save_tif            save masks as tif and outlines as text file for ImageJ
  --no_npy              suppress saving of npy
  --savedir SAVEDIR     folder to which segmentation results will be saved (defaults to input image directory)
  --dir_above           save output folders adjacent to image folder instead of inside it (off by default)
  --in_folders          flag to save output in folders (off by default)
  --save_flows          whether or not to save RGB images of flows when masks are saved (disabled by default)
  --save_outlines       whether or not to save RGB outline images when masks are saved (disabled by default)
  --save_ncolor         whether or not to save minimal "n-color" masks (disabled by default)
  --save_txt            flag to enable txt outlines for ImageJ (disabled by default)
training arguments:
  --train               train network using images in dir
  --train_size          train size network at end of training
  --test_dir TEST_DIR   folder containing test data (optional)
  --mask_filter MASK_FILTER
                        end string for masks to run on. use "_seg.npy" for manual annotations from the GUI. Default:
                        _masks
  --diam_mean DIAM_MEAN
                        mean diameter to resize cells to during training -- if starting from pretrained models it cannot
                        be changed from 30.0
  --learning_rate LEARNING_RATE
                        learning rate. Default: 0.2
  --weight_decay WEIGHT_DECAY
                        weight decay. Default: 1e-05
  --n_epochs N_EPOCHS   number of epochs. Default: 500
  --batch_size BATCH_SIZE
                        batch size. Default: 8
  --min_train_masks MIN_TRAIN_MASKS
                        minimum number of masks a training image must have to be used. Default: 5
  --residual_on RESIDUAL_ON
                        use residual connections
  --style_on STYLE_ON   use style vector
  --concatenation CONCATENATION
                        concatenate downsampled layers with upsampled layers (off by default which means they are added)
  --save_every SAVE_EVERY
                        number of epochs to skip between saves. Default: 100
  --save_each           save the model under a different filename per --save_every epoch for later comparsion
  '''

def main():
    args = docopt.docopt(cellpose_help_text, version='0.0.1')

if __name__ == '__main__':
    main()

See the homepage for more. The official repo is not well maintained, but this fork is and is called docopt-ng.

0 replies

rajavishah · 2024-09-12T16:11:13Z

rajavishah
Sep 12, 2024
Maintainer

Thanks! Whole Testing-Strategy gist is well-put. At the moment, a lot of time is spent on manually testing everything repeatedly, so this approach would help reduce that.

One point you raised about -

The template for the virtual interface will have very similar logic to generate_cli_command/construct_cli_command in the Gradio and Jupyter templates (in fact we should probably consolidate the internal logic of those two in a single template that the Gradio and Jupyter and virtual interface templates just import, so that it really will be identical).

I feel, some portion of the logic has different behavior in handling types -

When type==Files

Gradio has a behavior where it needs to make a folder and copy files to the newly created folder. And also, when it finds a path_of_folder in String it needs to create folder at that path (i.e. output_dir_set behavior, I'm pointing to)
Whereas, in Jupyter we are only supporting Volume-Mount so we don't explicitly need to create folders

return in both functions

Gradio needs to return folder_name, and output_folder_name in addition to the cli_command

In Gradio, checkboxes are checked as booleans (isinstance(value, bool)), while in Jupyter, they are checked as widgets.Checkbox. Also, similar for each type, they both are validated differently

0 replies

rajavishah · 2024-11-26T18:51:21Z

rajavishah
Nov 26, 2024
Maintainer

Currently testing is implemented in the below way -

Validating the incoming config file
Validating the formation of generate_cli_command is as expected

For the Part 1, we have used LinkML.
Since, we have written our config_file schema in LinkML, it's easy to lint it, and validate any incoming config files against schema.
For that, we have updated certain rules and added all the newly added flags in schema. Moreover, we have 3 test_config_files which tests most of the permutation n combinations for Validations. Out of which one file tests/test_algorithm/correct_validation_config.yaml is correct at the validation level. But it has some test cases, those should fail whilst individually testing interfaces. But all in all those 3 files are aggregation of possible combinations for each type.
All 3 files are run with a pytest function. The script for that is under in tests/test_config (i.e. test_validate_configs.py)

For Part 2, we discussed let's keep that portion until we implement a new interface. Analyze what manual work we face again and again, and implement testing strategy for Part 2. The simple choice in mind is as of now is implementing functional tests for each algorithm under their specific folder using pytest, with tests/test_algorithm/correct_validation_config.yaml as the dummy_spec_file.

The current testing folder structure is -

tests
├── test_algorithm
           ├── __init__.py
           ├── correct_validation_config.yaml
           ├── empty_config.yaml
           ├── incorrect_validation_config.yaml
├── test_config
           ├──  __init__.py
           ├──  test_validate_configs.py
           ├──  validate_schema.yaml
├── test_interface_name
           ├──  test_functional_name.py
.
.
.
.
└── __init__.py

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bilayer-containers

Testing Strategy #38

Uh oh!

{{title}}

Uh oh!

Replies: 3 comments

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

bilayer-containers

Testing Strategy #38

Uh oh!

gnodar01 Sep 10, 2024 Maintainer

Replies: 3 comments

Uh oh!

Uh oh!

gnodar01 Sep 10, 2024 Maintainer Author

Uh oh!

rajavishah Sep 12, 2024 Maintainer

Uh oh!

rajavishah Nov 26, 2024 Maintainer

gnodar01
Sep 10, 2024
Maintainer

gnodar01
Sep 10, 2024
Maintainer Author

rajavishah
Sep 12, 2024
Maintainer

rajavishah
Nov 26, 2024
Maintainer