Closes #936 Check directories for reruns/crashed imports#937
Closes #936 Check directories for reruns/crashed imports#937MichaelStritt merged 7 commits intodevelopfrom
Conversation
|
@jan-petr & @HenkMutsaerts: I don't really know how this is supposed to work in some cases. For NII2BIDS it is somewhat easy, because we can just check if the rawdata ASL or T1 folder already exists and delete it for the reruns or when the pipeline crashed. For BIDS2Legacy this is problematic, because we do the import on subject level and as I said before, as long as we don't have subject directories in legacy, this is kind of impossible. For DCM2NII I know there was a previous option with bOverwrite and so on. Maybe we can use that. Other than that I don't know how I should check for existing directories if one directory can be the top of the other one (T1 on top of ASL e.g.) or if we don't exactly know all of the output scans of DCM2NII beforehand. Nevertheless, I think the implemented code is an improvement for NII2BIDS and there are some simplifications for BIDS2Legacy. |
HenkMutsaerts
left a comment
There was a problem hiding this comment.
For the rerun, I would:
- base rerunning and overwriting or skipping per subject only on the existence of the lock files, identical to how this is managed in the processing modules. So if a lock/status file doesn't exist, this means that the current subject wasn't correctly run, and the previous rawdata should be deleted
- ensure that the lock/status files are only created after successful import of a single subject
- always overwrite existing rawdata or derivatives for the current subject (since overwriting/skipping is managed by the lock/status files)
So instead of checking folders, files, what if they exist partly, etc, etc, you only rely on the lock files (unless you create a bCompletelyRerunImport parameter of course, in that case you delete all lock files). Makes the code/program also much more transparent.
And that doesn't work on your side? If one of the import steps crashes for me, then the lock file is not created and on a rerun it reruns the sub-module. I think we should make it cleaner and remove some files/folders for the import reruns, because I'm not sure if every xASL_Move command or the used tools work with overwrite, but other than that this part should be fine. Please actually describe the defect/failure behavior. |
3e79423 to
edc2b08
Compare
|
Still think we should implement a sub/ses/run structure in the xASL derivatives and optimize the temp structure, so that reruns can be done in an easier way by removing pre-existing data from pipeline crashes etc. |
This comment has been minimized.
This comment has been minimized.
Yep good point for version 2.0 |
HenkMutsaerts
left a comment
There was a problem hiding this comment.
Some suggestions, hope this helps you also understanding this from a user perspective, which can be very useful in software development!
10deab0 to
54d1583
Compare
Linked issue
Closes #936