A possible project

Project 4: benchmarking and improving the performance of the library.

Idea: Benchmark and improve (if possible) some core functionalities of the library.

As the library becomes more and more complex with new solvers/functionalities, it would be great if the core of the library has the best performance (in term of compute time), e.g. physics classes, some adjoint operators, some autodiff functions, ...

Help: We can start by creating a utility function that measures the compute time/peak memory usage/... based on torch (eg torch. Timer).
Then if possible, use pytorch profiler to profile, eg an unfolder solver.

Technical challenges: function-by-function profiling, time consuming and diffucult task to improve the performance.

Impact: This makes the library faster!

Steps:

1. Write utility functions (eg in deepinv.utils.perf) for time/memory quick report
2. Profile some core functions (e.g. Trainer)
3. Detect the botteneck and improve.

Closes: #601

Project 1: Multiview physics

Idea: Implement a new physics operator that combines measurements from different 'views' of the same object.

Define a linear multiview operator which measures $J$ measurement vectors $y_1,\dots,y_J$ of a single image $x$, following the model

$y_j = AT_jx \text{ for } j=1,\dots,J$

where $A$ is an arbitrary forward operator (eg downsampling), and $T_j$ are affine transformations defined by $2\times 3$ matrices, which include translations, rotations and shearing.

Help: Use torch's grid_sample function to implement the transformations $T_j$.
You can also have a look at a preliminary implementation as part of a tutorial here.

Technical challenges: If time allows, it would be nice to write some basic code that estimates the transformations $T_j$ from the measurements $y_j$.

Impact: This operator would be useful for multiview imaging problems such as multiframe super-resolution, where we have multiple measurements of the same object from different viewpoints.

Steps:

1. Write the physics operator, and make sure it works for a simple problem
2. Write some pytests checking with different base operators $A$.
3. Write a sphinx gallery demo showing how to use the operator for multi-frame super-resolution.

Project 3: Latent space models

Idea: It is becoming increasingly popular to solve inverse problems in
the latent space of some VQ-GAN autoencoder $g$, that is solving the non-linear inverse problem

$y = A g(z)$

where $A$ is the (generally) linear physics and $z$ is the latent code, such that
$\hat{x}=g(z)$ is the reconstructed image in pixel space.
This problem can be solved via any of the methods in the library, including diffusion, PnP, unrolling, etc.

Technical challenges: Find a modular and backward-compatible way of integrating this new functionality into
the library, without having to rewrite most reconstruction methods.

Impact: This would enable state-of-the-art reconstruction methods based on latent space models.

Steps

1. Find a SOTA implementation of a VQ-GAN autoencoder, and adapt a minimal version of it to work with deepinv.
2. Explore different ways of integrating this into the library, either by creating new non-linear composed physics $A \circ g$, or by modifying existing methods to work with latent space models.
3. Write a sphinx gallery demo showing how to use the new functionality.
4. Write tests to check that everything works as expected.

Project 5: Spatial Unwrapping

Idea: Include a new physics model for spatial unwrapping. Spatial unwrapping is a non-linear inverse problem of great interest in different communities (phase unwrapping from optical interferometry, MRI, fringe pattern profilometry, InSAR, and recently, modulo imaging). This is modeled with the form:

$$\mathbf{y} = M(x,k) = \mathbf{x} - k \cdot q(\mathbf{x}/k)$$

where $k$ is the folding factor and $q(\cdot)$ is the quotient function, which could be the round or floor operators, among others. In practice, since the modulo is a non-linear operator, this problem is solved with linear optimization algorithms under the assumption of the Itoh condition by minimizing the following measurement consistency term through application of the finite discrete difference operator $\Delta$:

$$\min_{\mathbf{x}} \Vert M( \Delta \mathbf{y},k) - \Delta \mathbf{x} \Vert_2^2$$

For this case, it is interesting to analyze that the forward model and the fidelity terms have different operators!

Technical Challenge: Propose and implement DataFidelity and Physics subclasses for the specific requirements of spatial unwrapping that are compatible with current restoration algorithms in the library.

Impact: This would enable different spatial unwrapping problems to have access to state-of-the-art restoration algorithms in image processing.

Steps:

1. Implement the Physics for spatial unwrapping with variable quotient function (round or floor)
2. Propose and discuss the best way to include Itoh Fidelity for spatial unwrapping (A custom DataFidelity?)
3. Validate compatibility with restoration algorithms such as PGD, FISTA, PnP, ADMM, etc.
4. Write demos for specific inverse problems such as phase unwrapping and modulo imaging
5. Write tests to check overall integrity

Project 10: PnP priors with general restoration models.

Idea:

Building on the Plug-and-Play module, the idea is to use other restoration operators in order to solve inverse problems.

Reference papers:

• A Restoration Network as an Implicit Prior
• FiRe: Fixed-points of Restoration Priors for Solving Inverse Problems

Plan of Work

1. Model Selection
   Start with an open-weight restoration network trained for a corruption other than denoising. A possible candidate is the RAM general restoration model, already available in the library.

2. PnP Integration
   Implement an example where this restoration model is used within the existing PnP prior module and in a standard optimization algorithm, already implemented in the library.

3. Parameter Tuning
   Optimize hyperparameters to achieve good performance on a selected inverse problem of your choice.

4. Algorithmic Exploration
   Extend the study by testing the approach within alternative optimization algorithms, possibly in collaboration with the team working on stochastic Plug-and-Play methods.

Project 11: Stochastic PnP methods.

Idea: implement PnP methods that add noise or apply more general random transformations before denoising / restoration.

Reference papers:

• Plug-and-Play image restoration with Stochastic deNOising REgularization
• FiRe: Fixed-points of Restoration Priors for Solving Inverse Problems

The goal of this project is to implement a method that add stochasticity inside PnP algorithms. The team can for example choose between SNORE (add nose before denoising) or FIRE (apply more general transformations) and will work in collaboration with the team of Project 10.

The team can choose to either integrate general stochastic updates inside the DeepInv Optim module or to directly build an example file for the method of their choice.

Project 12 - SotA superresolution

Idea:

SR methods in the library are suboptimal today. I think doing some re-evaluations / benchmarks of the library for SR would be of interest, as well as adding latest SOTA methods. Below are some SOTA models that are currently used in the literature, by order of perf/age:

Plan of work:

1. Implement HYPIR
2. (Optional) Implement DiffBIR
3. (Optional) Implement StableSR
4. (Optional) Implement ESRGAN/SRGAN - also showcases popular GANs. We already have the GAN framework in deepinverse.

Technicality

If the implementation of deepinv is matching perfectly that of the diffusers library, things should be rather straightforward. However, things are likely going to be bumpy 🙃

Project 13 - PnP Latent Diffusion Inverse Solvers

Idea:

Modern PnP SOTA methods for image restoration exploit Latent Diffusion Models (LDMs) and distilled versions of them (e.g., Consistency Models) to inject prior information during the restoration process. The idea is to implement these methods (LDPS/PSLD, P2L, TReg, LATINO-PRO) under the same framework, mimicking what has already been implemented with the DPS algorithm. The diffusers library can provide the implementation of the VAEs and score-networks for the priors (Stable Diffusion v1.5, Stable Diffusion XL, and DMD2).

All these methods are already implemented in this repo; the key point would be to integrate them in deepinv.

Plan of work

1. Think about the structure of classes we will need to implement the methods
2. Check what's already available in deepinv (e.g. the Reconstructor class) and what we can evolve from this, identify new classes to implement
3. Specify the tests we want to run (Deblur, SR, Inpainting...) and test all the models
4. (If time allows) Create examples/tutorials (e.g. a notebook that runs multiple methods and compares the results)

Impact:

Allow researchers/practitioners to easily run SOTA restoration methods all in the same environment, enabling customizations to specific uses. Some of these methods have never been implemented with high-resolution priors (1024x1024 res.) on publicly available repositories before, and I believe that the community would appreciate this effort.

Project 14: Connect Pytomography to deepinv
Idea:

Enable the use of the Pytomography package together with deepinv.

Pytomography is an open-source PyTorch-based package for mainly SPECT/CT reconstruction. This integration would allow users to leverage the capabilities of Pytomography (e.g. exact ECT forward models corresponding to actual imaging configurations) while utilizing the optimization algorithms available in deepinv.

Technical Challenge:

Propose and implement Physics subclasses ?
3D problem => high VRAM usage

Impact:

This would allow people in the SPECT community to access a wider variety of state-of-the-art reconstruction algorithms. Conversely, it would allow people in the inverse problems community to easily access more realistic forward models for 3D problems.

Project 15:[Application] MRI reconstruction for accelerated non-Cartesian imaging

• MRI-NUFFT bindings
• PnP with HQS model w/ and wo/ pre-conditioners (https://arxiv.org/abs/2411.01955), example
• Wavelet / TV based CS reconstructions (FISTA)
• EBM: Try to bring some EBM models up .. towards a comprehensive benchmark on prospective data.