Skip to content

Instantly share code, notes, and snippets.

@yibeichan

yibeichan/GSoC2022_summary.md

Last active March 31, 2023 11:33
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save yibeichan/2c9879906dcc8836b78dce7d7acdaf4b to your computer and use it in GitHub Desktop.
Save yibeichan/2c9879906dcc8836b78dce7d7acdaf4b to your computer and use it in GitHub Desktop.
Download ZIP
GSoC 2022 Final Evaluation
Raw
GSoC2022_summary.md

Converting existing scientific workflows to the new dataflow engine Pydra (Python)

Contributor

Name: Yibei Chen (@yibeichan)

Email: yibeichan@gmail.com

Mentors

Satra Ghosh (@satra)

Dorota Jarecka (@djarecka)

Hao-Ting Wang (@htwangtw)

Chris Markiewicz (@effigies)

Project Info

The goal of this project was to convert existing neuroimaging analytical workflows that are written in other languages to the new dataflow engine Pydra. The main deliverables of the project are comprehensive Pydra workflows that use interfaces from neuroimaging packages such as FSL, SPM, ANTs, AFNI, FreeSurfer.

Outcome Details

During the past ten weeks, I have contributed in four GitHub Repos: pydra-tutorial, pydra-fsl, pydra-fitlins, and pydra

  1. converting pydra tutorial into jupyterbook - done (link)
  2. converting first-level GLM (Nilearn) to Pydra - done (link)
    • see this PR
  3. converting two-level GLM (Nilearn) to Pydra - hasn't been merged yet (PR)
    • this notebook hasn't passed the tests in GHA but works fine locally for @yibeichan (Python 3.7 & 3.8) and @effigies (Python 3.10), though @djarecka (Python 3.8) still has problems with the notebook locally.
    • it is also related to Pydra that error messages are not very clear enough to identify where (which node) the error is (e.g., this PR made by @effigies)
  4. converting fitlins to Pydra - hasn't been merged yet (PR)
    • we encountered some typing issues that are related to python version and needed a decision
    • currently the converting is conducted in a half-dynamic and half-static way. This repo is related to GLM in Nilearn and so we decided to make it fully dynamic after converting the two-level GLM (Nilearn) to Pydra.
  5. converting FSL to Pydra - hasn't been merged yet (PR)
    • in this task, I have improved the converter.py, converted all interfaces that are needed for three-level GLM, and documented the wiki
    • we encountered some Large File Storage (LFS) issues that needed changes in GHA (see this PR)
    • we also encountered some issues that are related to the Pydra pacakge, see this PR by @yibeichan and PRs from @djarecka (e.g., here and here)

Summary

My goal/task was to convert as many workflows as possible; however, unexpected errors/bugs have occurred from time to time and have slowed down the progress. Meanwhile, the journey itself has been very rewarding and I deeply appreciate every of my mentors for their patience and help. Thanks for bringing me into an open, equal, and meaningful community.

I will keep working on this project after GSoC.

@djarecka
Copy link

@yibeichan - Thank you for all your PRs! Below some comments and updates:

  1. I will work on my issue with my local environment, but I'm happy to merge the PR if the git-annex issue is solved. I think you should merge the master branch (left the comment in the PR)
  2. I merged the PR and will update GA actions to remove py3.7, I guess it's ok to drop it
  3. I've merged the main PR (forgot about the issues with files), so you should merge master to the LFS branch, so the issue with bigger files should be fixed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment