Replies: 16 comments 12 replies
-
|
A possible project Project 4: benchmarking and improving the performance of the library.Idea: Benchmark and improve (if possible) some core functionalities of the library. As the library becomes more and more complex with new solvers/functionalities, it would be great if the core of the library has the best performance (in term of compute time), e.g. physics classes, some adjoint operators, some autodiff functions, ... Help: We can start by creating a utility function that measures the compute time/peak memory usage/... based on torch (eg torch. Timer). Technical challenges: function-by-function profiling, time consuming and diffucult task to improve the performance. Impact: This makes the library faster! Steps:
Closes: #601 |
Beta Was this translation helpful? Give feedback.
-
Project 1: Multiview physicsIdea: Implement a new physics operator that combines measurements from different 'views' of the same object. Define a linear multiview operator which measures where Help: Use torch's grid_sample function to implement the transformations Technical challenges: If time allows, it would be nice to write some basic code that estimates the transformations Impact: This operator would be useful for multiview imaging problems such as multiframe super-resolution, where we have multiple measurements of the same object from different viewpoints. Steps:
|
Beta Was this translation helpful? Give feedback.
-
Project 2: Custom autodiff for linear solversIdea: Implement a custom Seeing the linear solver as a function We would like to compute the closed-form gradients with respect to inputs Impact: This option would be useful for unrolled reconstruction methods where we require Technical challenges: Matrix Steps:
|
Beta Was this translation helpful? Give feedback.
-
Project 3: Latent space modelsIdea: It is becoming increasingly popular to solve inverse problems in where Technical challenges: Find a modular and backward-compatible way of integrating this new functionality into Impact: This would enable state-of-the-art reconstruction methods based on latent space models. Steps
|
Beta Was this translation helpful? Give feedback.
-
Project 5: Spatial UnwrappingIdea: Include a new physics model for spatial unwrapping. Spatial unwrapping is a non-linear inverse problem of great interest in different communities (phase unwrapping from optical interferometry, MRI, fringe pattern profilometry, InSAR, and recently, modulo imaging). This is modeled with the form: where For this case, it is interesting to analyze that the forward model and the fidelity terms have different operators! Technical Challenge: Propose and implement Impact: This would enable different spatial unwrapping problems to have access to state-of-the-art restoration algorithms in image processing. Steps:
|
Beta Was this translation helpful? Give feedback.
-
Project 6: Connect SPyRiT to deepinvIdea: Enable the use of the SPyRiT package together with deepinv. SPyRiT is an open-source PyTorch-based package for single-pixel imaging. This integration would allow users to leverage the capabilities of SPyRiT (e.g., forward models corresponding to actual imaging configurations, access to experimental datasets) while utilizing the optimization algorithms available in deepinv. Technical Challenge: Propose and implement Physics subclasses ? Impact: This would allow people in the optics community to access a wider variety of state-of-the-art reconstruction algorithms. Conversely, it would allow people in the inverse problems community to easily access more realistic forward models and the corresponding experimental datasets. Steps:
|
Beta Was this translation helpful? Give feedback.
-
Project 7: LintingIdea: DeepInverse is written in high quality Python, and avoids common pitfalls and bugs (such as #628). Steps:
Further notes: ruff-pylint correspondance |
Beta Was this translation helpful? Give feedback.
-
Project 8: External integrationsIdea: This project has two streams. Stream 1: interfaces for external libs:
Stream 2: DeepInverse plugins for existing industry software: e.g. ImageJ (microscopy) Impact: Stream 1: we open up DeepInverse to practitioners/researchers in more industries by making it more seamless to load and inference industry-specific images Stream 2: we spread awareness of DeepInverse by providing easy-to-use deep learning functionality for users of existing imaging software in various industries. |
Beta Was this translation helpful? Give feedback.
-
Project 9: Refactor TrainerIdea: The Trainer code is expanding and the current implementation might not be the most optimal in terms of:
Context for existing discussions and ideas (@jscanvic):
Technical Challenge: Tackling this problem will require some high-level Python design thinking, not just simple scientific coding. Potential solutions:
|
Beta Was this translation helpful? Give feedback.
-
Project 10: PnP priors with general restoration models.Idea:Building on the Plug-and-Play module, the idea is to use other restoration operators in order to solve inverse problems. Reference papers:
Plan of Work
|
Beta Was this translation helpful? Give feedback.
-
Project 11: Stochastic PnP methods.Idea: implement PnP methods that add noise or apply more general random transformations before denoising / restoration.Reference papers:
The goal of this project is to implement a method that add stochasticity inside PnP algorithms. The team can for example choose between SNORE (add nose before denoising) or FIRE (apply more general transformations) and will work in collaboration with the team of Project 10. The team can choose to either integrate general stochastic updates inside the DeepInv Optim module or to directly build an example file for the method of their choice. |
Beta Was this translation helpful? Give feedback.
-
Project 12 - SotA superresolutionIdea:SR methods in the library are suboptimal today. I think doing some re-evaluations / benchmarks of the library for SR would be of interest, as well as adding latest SOTA methods. Below are some SOTA models that are currently used in the literature, by order of perf/age: Plan of work:
TechnicalityIf the implementation of deepinv is matching perfectly that of the diffusers library, things should be rather straightforward. However, things are likely going to be bumpy 🙃 |
Beta Was this translation helpful? Give feedback.
-
Project 13 - PnP Latent Diffusion Inverse SolversIdea: Modern PnP SOTA methods for image restoration exploit Latent Diffusion Models (LDMs) and distilled versions of them (e.g., Consistency Models) to inject prior information during the restoration process. The idea is to implement these methods (LDPS/PSLD, P2L, TReg, LATINO-PRO) under the same framework, mimicking what has already been implemented with the DPS algorithm. The diffusers library can provide the implementation of the VAEs and score-networks for the priors (Stable Diffusion v1.5, Stable Diffusion XL, and DMD2). All these methods are already implemented in this repo; the key point would be to integrate them in deepinv. Plan of work
Impact: Allow researchers/practitioners to easily run SOTA restoration methods all in the same environment, enabling customizations to specific uses. Some of these methods have never been implemented with high-resolution priors (1024x1024 res.) on publicly available repositories before, and I believe that the community would appreciate this effort. |
Beta Was this translation helpful? Give feedback.
-
|
Project 14: Connect Pytomography to deepinv Enable the use of the Pytomography package together with deepinv. Pytomography is an open-source PyTorch-based package for mainly SPECT/CT reconstruction. This integration would allow users to leverage the capabilities of Pytomography (e.g. exact ECT forward models corresponding to actual imaging configurations) while utilizing the optimization algorithms available in deepinv. Technical Challenge: Propose and implement Physics subclasses ? Impact: This would allow people in the SPECT community to access a wider variety of state-of-the-art reconstruction algorithms. Conversely, it would allow people in the inverse problems community to easily access more realistic forward models for 3D problems. |
Beta Was this translation helpful? Give feedback.
-
|
Project 15:[Application] MRI reconstruction for accelerated non-Cartesian imaging
|
Beta Was this translation helpful? Give feedback.
-
|
Project 16: Add PET data and physics model Idea: Plan of work:
|
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Hackathon projects
Some projects for the hackathon that will take place on September 8-10 in Marseille, France.
Beta Was this translation helpful? Give feedback.
All reactions