Check continuity of fast-track, esp. multi-session datasets and w/cohort normalization

[oesteban]

#219 and #223 will likely break some parts of the anatomical fast track. Let's make sure it works before cutting 0.7.

[nuKs]

Hello,

Fast-track seems to break with my dataset on smriprep 0.7 / fmriprep 20.2.0, notably due to bids' build_path not considering potential session/runs within the name.

I developed a dirty-hack which rely on BIDSLayout instead and works for me.

commit: nuKs@bd72f83
branch: https://github.com/nuKs/smriprep/commits/nuKs/fix-fast-track

I did not PR as I am not sure it's the best way to fix this, and if this works on other use cases.
Notably, I didn't change anything related to Freesurfer files as I didn't have preprocessed sample files.
It's tested on a single subject/session dataset, I still need to try it out on multi session dataset as well.

The included logging of missing files is convenient in any case.

[nuKs]

The patch also fixes anats within two sessions. The use case appears to be different as for single session anat, no template has to be created, while 2 sessions are averaged. In both case, "run-" is kept.

Here are the outputs for anat derivatives (smriprep) of subsets of the day2day dataset,

with single session:

└── smriprep
    ├── dataset_description.json
    ├── logs
    │   ├── CITATION.bib
    │   ├── CITATION.html
    │   ├── CITATION.md
    │   └── CITATION.tex
    ├── sub-01
    │   ├── figures
    │   │   ├── sub-01_desc-about_T1w.html
    │   │   ├── sub-01_desc-summary_T1w.html
    │   │   ├── sub-01_ses-01_run-1_desc-conform_T1w.html
    │   │   ├── sub-01_ses-01_run-1_dseg.svg
    │   │   ├── sub-01_ses-01_run-1_space-MNI152NLin2009cAsym_T1w.svg
    │   │   ├── sub-01_ses-01_run-1_space-MNI152NLin6Asym_T1w.svg
    │   │   └── sub-01_ses-01_run-1_space-OASIS30ANTs_T1w.svg
    │   └── ses-01
    │       └── anat
    │           ├── sub-01_ses-01_run-1_desc-brain_mask.json
    │           ├── sub-01_ses-01_run-1_desc-brain_mask.nii.gz
    │           ├── sub-01_ses-01_run-1_desc-preproc_T1w.json
    │           ├── sub-01_ses-01_run-1_desc-preproc_T1w.nii.gz
    │           ├── sub-01_ses-01_run-1_dseg.nii.gz
    │           ├── sub-01_ses-01_run-1_from-MNI152NLin2009cAsym_to-T1w_mode-image_xfm.h5
    │           ├── sub-01_ses-01_run-1_from-MNI152NLin6Asym_to-T1w_mode-image_xfm.h5
    │           ├── sub-01_ses-01_run-1_from-OASIS30ANTs_to-T1w_mode-image_xfm.h5
    │           ├── sub-01_ses-01_run-1_from-T1w_to-MNI152NLin2009cAsym_mode-image_xfm.h5
    │           ├── sub-01_ses-01_run-1_from-T1w_to-MNI152NLin6Asym_mode-image_xfm.h5
    │           ├── sub-01_ses-01_run-1_from-T1w_to-OASIS30ANTs_mode-image_xfm.h5
    │           ├── sub-01_ses-01_run-1_label-CSF_probseg.nii.gz
    │           ├── sub-01_ses-01_run-1_label-GM_probseg.nii.gz
    │           ├── sub-01_ses-01_run-1_label-WM_probseg.nii.gz
    │           ├── sub-01_ses-01_run-1_space-MNI152NLin2009cAsym_desc-brain_mask.json
    │           ├── sub-01_ses-01_run-1_space-MNI152NLin2009cAsym_desc-brain_mask.nii.gz
    │           ├── sub-01_ses-01_run-1_space-MNI152NLin2009cAsym_desc-preproc_T1w.json
    │           ├── sub-01_ses-01_run-1_space-MNI152NLin2009cAsym_desc-preproc_T1w.nii.gz
    │           ├── sub-01_ses-01_run-1_space-MNI152NLin2009cAsym_dseg.nii.gz
    │           ├── sub-01_ses-01_run-1_space-MNI152NLin2009cAsym_label-CSF_probseg.nii.gz
    │           ├── sub-01_ses-01_run-1_space-MNI152NLin2009cAsym_label-GM_probseg.nii.gz
    │           ├── sub-01_ses-01_run-1_space-MNI152NLin2009cAsym_label-WM_probseg.nii.gz
    │           ├── sub-01_ses-01_run-1_space-MNI152NLin6Asym_desc-brain_mask.json
    │           ├── sub-01_ses-01_run-1_space-MNI152NLin6Asym_desc-brain_mask.nii.gz
    │           ├── sub-01_ses-01_run-1_space-MNI152NLin6Asym_desc-preproc_T1w.json
    │           ├── sub-01_ses-01_run-1_space-MNI152NLin6Asym_desc-preproc_T1w.nii.gz
    │           ├── sub-01_ses-01_run-1_space-MNI152NLin6Asym_dseg.nii.gz
    │           ├── sub-01_ses-01_run-1_space-MNI152NLin6Asym_label-CSF_probseg.nii.gz
    │           ├── sub-01_ses-01_run-1_space-MNI152NLin6Asym_label-GM_probseg.nii.gz
    │           ├── sub-01_ses-01_run-1_space-MNI152NLin6Asym_label-WM_probseg.nii.gz
    │           ├── sub-01_ses-01_run-1_space-OASIS30ANTs_desc-brain_mask.json
    │           ├── sub-01_ses-01_run-1_space-OASIS30ANTs_desc-brain_mask.nii.gz
    │           ├── sub-01_ses-01_run-1_space-OASIS30ANTs_desc-preproc_T1w.json
    │           ├── sub-01_ses-01_run-1_space-OASIS30ANTs_desc-preproc_T1w.nii.gz
    │           ├── sub-01_ses-01_run-1_space-OASIS30ANTs_dseg.nii.gz
    │           ├── sub-01_ses-01_run-1_space-OASIS30ANTs_label-CSF_probseg.nii.gz
    │           ├── sub-01_ses-01_run-1_space-OASIS30ANTs_label-GM_probseg.nii.gz
    │           └── sub-01_ses-01_run-1_space-OASIS30ANTs_label-WM_probseg.nii.gz
    └── sub-01.html

double session:

└── smriprep
    ├── dataset_description.json
    ├── logs
    │   ├── CITATION.bib
    │   ├── CITATION.html
    │   ├── CITATION.md
    │   └── CITATION.tex
    ├── sub-01
    │   ├── anat
    │   │   ├── sub-01_run-1_desc-brain_mask.json
    │   │   ├── sub-01_run-1_desc-brain_mask.nii.gz
    │   │   ├── sub-01_run-1_desc-preproc_T1w.json
    │   │   ├── sub-01_run-1_desc-preproc_T1w.nii.gz
    │   │   ├── sub-01_run-1_dseg.nii.gz
    │   │   ├── sub-01_run-1_from-MNI152NLin2009cAsym_to-T1w_mode-image_xfm.h5
    │   │   ├── sub-01_run-1_from-MNI152NLin6Asym_to-T1w_mode-image_xfm.h5
    │   │   ├── sub-01_run-1_from-OASIS30ANTs_to-T1w_mode-image_xfm.h5
    │   │   ├── sub-01_run-1_from-T1w_to-MNI152NLin2009cAsym_mode-image_xfm.h5
    │   │   ├── sub-01_run-1_from-T1w_to-MNI152NLin6Asym_mode-image_xfm.h5
    │   │   ├── sub-01_run-1_from-T1w_to-OASIS30ANTs_mode-image_xfm.h5
    │   │   ├── sub-01_run-1_label-CSF_probseg.nii.gz
    │   │   ├── sub-01_run-1_label-GM_probseg.nii.gz
    │   │   ├── sub-01_run-1_label-WM_probseg.nii.gz
    │   │   ├── sub-01_run-1_space-MNI152NLin2009cAsym_desc-brain_mask.json
    │   │   ├── sub-01_run-1_space-MNI152NLin2009cAsym_desc-brain_mask.nii.gz
    │   │   ├── sub-01_run-1_space-MNI152NLin2009cAsym_desc-preproc_T1w.json
    │   │   ├── sub-01_run-1_space-MNI152NLin2009cAsym_desc-preproc_T1w.nii.gz
    │   │   ├── sub-01_run-1_space-MNI152NLin2009cAsym_dseg.nii.gz
    │   │   ├── sub-01_run-1_space-MNI152NLin2009cAsym_label-CSF_probseg.nii.gz
    │   │   ├── sub-01_run-1_space-MNI152NLin2009cAsym_label-GM_probseg.nii.gz
    │   │   ├── sub-01_run-1_space-MNI152NLin2009cAsym_label-WM_probseg.nii.gz
    │   │   ├── sub-01_run-1_space-MNI152NLin6Asym_desc-brain_mask.json
    │   │   ├── sub-01_run-1_space-MNI152NLin6Asym_desc-brain_mask.nii.gz
    │   │   ├── sub-01_run-1_space-MNI152NLin6Asym_desc-preproc_T1w.json
    │   │   ├── sub-01_run-1_space-MNI152NLin6Asym_desc-preproc_T1w.nii.gz
    │   │   ├── sub-01_run-1_space-MNI152NLin6Asym_dseg.nii.gz
    │   │   ├── sub-01_run-1_space-MNI152NLin6Asym_label-CSF_probseg.nii.gz
    │   │   ├── sub-01_run-1_space-MNI152NLin6Asym_label-GM_probseg.nii.gz
    │   │   ├── sub-01_run-1_space-MNI152NLin6Asym_label-WM_probseg.nii.gz
    │   │   ├── sub-01_run-1_space-OASIS30ANTs_desc-brain_mask.json
    │   │   ├── sub-01_run-1_space-OASIS30ANTs_desc-brain_mask.nii.gz
    │   │   ├── sub-01_run-1_space-OASIS30ANTs_desc-preproc_T1w.json
    │   │   ├── sub-01_run-1_space-OASIS30ANTs_desc-preproc_T1w.nii.gz
    │   │   ├── sub-01_run-1_space-OASIS30ANTs_dseg.nii.gz
    │   │   ├── sub-01_run-1_space-OASIS30ANTs_label-CSF_probseg.nii.gz
    │   │   ├── sub-01_run-1_space-OASIS30ANTs_label-GM_probseg.nii.gz
    │   │   └── sub-01_run-1_space-OASIS30ANTs_label-WM_probseg.nii.gz
    │   ├── figures
    │   │   ├── sub-01_desc-about_T1w.html
    │   │   ├── sub-01_desc-summary_T1w.html
    │   │   ├── sub-01_run-1_desc-conform_T1w.html
    │   │   ├── sub-01_run-1_dseg.svg
    │   │   ├── sub-01_run-1_space-MNI152NLin2009cAsym_T1w.svg
    │   │   ├── sub-01_run-1_space-MNI152NLin6Asym_T1w.svg
    │   │   └── sub-01_run-1_space-OASIS30ANTs_T1w.svg
    │   ├── ses-01
    │   │   └── anat
    │   │       └── sub-01_ses-01_run-1_from-orig_to-T1w_mode-image_xfm.txt
    │   └── ses-02
    │       └── anat
    │           └── sub-01_ses-02_run-1_from-orig_to-T1w_mode-image_xfm.txt
    └── sub-01.html

[nuKs]

@oesteban what are your thoughts on this? Do you have had any feedback about fast-track since the update? I could be missing something as well.

edit: I have the same issue with the IBC dataset with smriprep 0.7.1 and fmriprep 20.2.1, some subjects have 2 anats so it is averaged and thus are working fine (these folders exist: /sub-xx/anat + 2x /sub-xx/ses-yy/anat) while some subjects have only 1 anat and fail (only this folder exists: /sub-xx/ses-yy/anat, /sub-xx/anat is missing).

[nuKs]

My patch seems to work, though not up to date with the fmriprep trunk (no config python file anymore etc).

I did not compare my patch output with traditional fmriprep non-fast track output so i am not 100% sure everything is fine (might pick the wrong template etc).

I can integrate it with the latest version if you think it's OK.

edit: actually seems not to work on subject with single anat session. (nevermind, was actually related to how I extract the session)

[oesteban]

Hi @nuKs - thanks for your patience, we are seriously limited in time lately - What would you need from us to push this forward? Just check out your branch and test on our settings?

Do you have examples that make the fast-track fail?

[nuKs]

Hi, no worries, thanks for sharing your work!

What my patch does is to rely on pybids layout instead of bids build_path in order to retrieve the fast_track derivatives files.
Pybids layout does adapt to any bids-valid naming scheme, while build_path does not, but rather directly assumes the file name.

If you have worked on these bits, a simple quick look at my patch code, or even just the way I've described it, may allow you to see if there is any red flag regarding what I did. I made the assumption that fmriprep must have some additional step that picks up the right derivative within the list, and I didn't check this step's code so I am not 100% sure I patched the right way considering the derivative list returned by the patched function is now less restrictive than it used to be. I am perhaps being overly cautious, If you don't see any reason why the patch would cause issues, then everything should be fine,.

So far, examples that make fast-track fails include:

• subjects with a single T1 (as the path is different since no template is made)
• subjects with T1 containing run-x segments within the name.

In my case, IBC has been failing (without fix) for some subjects, modified/reduced day2day datasets too.
IBC openneuro: https://openneuro.org/datasets/ds002685/versions/1.3.1
day2day upon request: https://bmcneurosci.biomedcentral.com/articles/10.1186/s12868-017-0383-y

I'll cleanup/update everything to the latest smriprep version so you can test and merge if you wish to.

[bpinsard]

Jumping in that old convo: I noticed that the fast-track would fail if you process a single anat in a multi-session dataset (and results is output in a session folder) and then try to use that output as anat fast-track for fMRIPrep. Because collect_derivatives does not look into sessions (even if the pattern would allow it), it will not find the preprocessed data and fMRIPrep. I think going with pybids would indeed provide more flexible search options.

[nuKs]

Had made that PR if you want to check in, a bit old though.
#272