Skip to content

Conversation

@nfrasser
Copy link
Contributor

Fixes #47

Use fromfile_prefix_chars option in ArgumentParser to allow specifying a long list of arguments in a file.

Example invocation before (still supported):

topaz denoise -o denoised_micrographs micrographs/mic1.mrc micrographs/mic2.mrc micrographs/mic3.mrc

Example invocation after:

topaz denoise -o denoised_micrographs @mics.txt

Where mics.txt has the following contents:

micrographs/mic1.mrc
micrographs/mic2.mrc
micrographs/mic3.mrc

This would be really beneficial to have when processing thousands of micrographs on systems that have a limit on the maximum number of allowed command-line arguments. It would also be good for debugging purposes e.g., show full command used by CryoSPARC Topaz wrapper instead of a truncated command that excludes the paths for legibility.

@tbepler please let me know if you have any feedback!

@tbepler
Copy link
Owner

tbepler commented Mar 13, 2024

I like this. Was not aware argparse had this option. We need to make sure this doesn't cause any unexpected errors and is compatible with some pending unreleased updates though.

@Guillawme
Copy link
Contributor

This will be a welcome improvement! On several systems, I have run into the "too many command-line arguments" limitation, and there is no other solution than splitting the dataset in several batches to pick separately (pretty annoying to do, and it makes the job graph look more complicated than it really is).

@olibclarke
Copy link

This would be great to have! We frequently run into this with large datasets, necessitating splitting into smaller batches. This isn't much of an issue for experienced users, but it trips up a lot of first time Topaz users in my experience.

Copy link
Collaborator

@DarnellGranberry DarnellGranberry left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for finding this!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Input micrographs as file

5 participants