@MichaelStritt @jan-petr Why do we want the temp folder in the derivatives? Temp should in fact be rawdata. derivatives is the output of a processing pipeline, not of a dcm2bids conversion. Let's keep temp as it is. If we need to change anything to move the import module to a full module, we should remove the temp step and integrate it within rawdata. Let's discuss next week.

@MichaelStritt @jan-petr Why do we want the temp folder in the derivatives? Temp should in fact be rawdata. derivatives is the output of a processing pipeline, not of a dcm2bids conversion. Let's keep temp as it is. If we need to change anything to move the import module to a full module, we should remove the temp step and integrate it within rawdata. Let's discuss next week.

We've actually discussed this already on Monday. rawdata is BIDS and thus shouldn't contain any temp or lock files. So the only logical place to put those two is in derivatives because we don't want them hanging around elsewhere. We can of course move temp to rawdata as this is created only for a brief moment... So we can have this discussion on Monday, but it would be great to agree on keeping lock in derivatives so that Michael can proceed with the lock-files.

@jan-petr: The funny thing right now is that it crashes a few times and you get a nice amount of warnings, but the results are still fine. Let me explain. Right now numOf = x.modules.import.numOf; in xASL_imp_DCM2NII_Subject is not updated correctly. I saw warnings pop up if we have two visits for the first but only a single visit for the second subject. This issue is somewhat related to the way that xASL_Iteration runs the pipeline, too. Right now it tries to execute xASL_imp_NII2BIDS once for every subject, because we run xASL_module_Import for each subject, but xASL_imp_NII2BIDS itself iterates over the subjects, which is why the workflow crashes. After the first run of the Import module we already have the correct results though, so the crash happens but the correct results are there. We probably need to move the iteration over the subjects away from the import functions and do that using xASL_Iteration, but then we also need to make sure that things like deleting temp and so on only happen after everything is done, so that has to go to ExploreASL_ImportMaster. A more or less problematic thing remains. In xASL_imp_NII2BIDS we iterate over SubjectSessions, so the combined thing. We have to iterate over sessions only if we run the import module for each subject individually. For DCM2NII it's working without any crashes, but still not correct. There the workflow creates all the temp directories and files and after that it tries to do it again but sees that they are there already and skips it. Again, this happens because we run the import module for each subject. This is the way xASL_Iteration was written though and probably what you guys need to run larger studies. Could be possible that I need some help with that stuff, because it seems that with this request we opened pandoras box...

Testing

@jan-petr: I just retested. Seems to work just fine on windows, but maybe I need to change filesep to backslash in the last commit for linux support. The differences below are there because I removed GeneratedBy from dataPar, because we wanted that anyway, right?

>> load('...\ExploreASL\Testing\results.mat')
>> flavors.comparisonTable

ans =

  3×4 table

                 flavor                  dataset              name                                   message                      
    _________________________________    _______    ________________________    __________________________________________________

    'GE_PCASL_3Dspiral_14.0LX_1'         'Both'     'Different file content'    'Different file content: \ExploreASL\dataPar.json'
    'Philips_PCASL_3DGRASE_5.4.0.2_1'    'Both'     'Different file content'    'Different file content: \ExploreASL\dataPar.json'
    'Siemens_PCASL_3DGRASE_VE11C_2'      'Both'     'Different file content'    'Different file content: \ExploreASL\dataPar.json'

>> flavors.loggingTable

ans =

  0×3 empty table

Don't use filesep, but fullfile - that's OS non-specific...

also - Philips_PCASL_2DEPI_PARREC_1 did create temp-files, but never rawdata.

Philips_PCASL_2DEPI_PARREC_1 doesn't work - rawdata not created and subsequently, the comparison script crashes.

@jan-petr: the problem was that I seemingly added a superfluous underscore because of the _1 in the tokens. It should work now. I also fixed the escaping, which should basically only be necessary for windows anyway. af1e66b 👍

It still seems to complain about Lock-files in the comparison. And it displays a difference in Generated by, while for this purpose, it should ignore it - that's important for import and processing to note version differences, but not for Flavor reference:

Yeah, wasn't my intention to leave that in there. There's probably just a small mistake somewhere where I do the regular expressions. I'll give it a quick try on the AMC server.

Edit: Oh, okay, so this goes wrong for Koens Hadamard dataset? Is the import already working for that one?

Edit 2: @jan-petr better after this d37c17c & e118d58 ? I just did a stupid mistake with the escaping it seems...

Known issues:

1. 7T dataset: BIDS->Legacy is almost good, but it copies only ASL and not M0 - error is in xASL_bids_BIDS2Legacy_ParseModality, where in the second iteration of the first loop starts looking for m0scan, but doesn't find them as the Reference varaible still contains ASL and wasn't updated - probably things went wrong when reorganizing the code...
2. GE_PCASL_3Dspiral_25.0LX_ADNI3_027_S_5079_1 - issue with visits 👍 6d44934
3. Philips_PASL_2DEPI_5.4.1.0_ADNI3_006_S_6681_1 - issue with visits 👍 6d44934
4. Philips_PCASL_2DEPI_PARREC_1
   1. I think something is wrong here related to xASL_imp_AddVisitNames within xASL_imp_DetermineStructureFromSourcedata, because the visit is added as _1 instead of ASL_1
   2. visit is added as "1" and not as "_1" - creating "sub-011" instead of "sub-01_1",
   3. also - issues with copying everything 'perf' stuff to BIDS.
   4. One minor bug fixed 7f40cd4. 👍
5. Legacy comparison should skip the following (shortened):
   Dataset: derivativesReference
Missing: /ExploreASL/lock
Missing: /ExploreASL/lock/xASL_module_Import/Sub103/xASL_module_Import/999_ready.status
Missing: /ExploreASL/log/import_summary_Sub103.csv
Missing: /ExploreASL/log/xASL_module_Import_Sub103.log
Missing: /ExploreASL/sub-Sub103_1/bids_report_sub-Sub103.json
File: /ExploreASL/dataPar.json
Extra field: GeneratedBy
Different value: x (array)