This repository hosts my master's thesis "3D Instance Segmentation of an Unlabeled Modality via Cyclic Segmentation GANs" (grade 1.0). The thesis is based on a project targeting the segmentation of neuronal nuclei in Expansion Microscopy (ExM). It is the result of a collaboration between Prof. Hanspeter's Viusal Computing Group (VCG) at Harvard, where I was a research fellow from October 2021 to July 2022, and Prof. Edward Boyden's Synthetic Neurobiology Group (SNG) at MIT. My corresponding first-authored paper for this project is currently under review. An early version of this paper can be found here. We published this version so our collaborators from MIT's SNG can already use our method.
In this work and its companion paper, we propose a novel Cyclic Segmentation Generative Adversarial Network (CySGAN}) for 3D domain adaptive instance segmentation. We developed CySGAN for nuclei segmentation in connectomics. Instance segmentation of 3D data is critical for connectomics as the field aims to identify and track the entire neural structures of the human brain. While manual segmentation is not feasible, automatic annotation usually requires training data with expert annotations that are expensive and time-consuming to obtain. In addition, imaging methods with sufficient resolution for connectomics and available annotated data generally have weak points like high costs and low throughput while suffering from stitching and alignment artifacts. At the same time, new imaging techniques that have the potential to alleviate some of these weak points are slow to be adopted precisely because of the lack of expert annotations. As a result, 3D instance segmentation of unlabeled image domains is desirable. However, existing methods are predominantly 2D and designed for semantic segmentation. Moreover, almost all existing methods segment a new modality either by using pre-trained models optimized on diverse training data or by performing image translation and segmentation sequentially with two relatively independent networks. CySGAN performs image translation and instance segmentation simultaneously using a unified weight-sharing network. The weight-sharing between the segmentation and the translation component has multiple benefits. First, both components are aware of the final objective at all times. Second, the segmentation introduces structural constraints on the image transformation. Third, the computational complexity is significantly lower compared to sequential models. At the time of inference, we can even remove the image transformation layer, reducing the computational cost of CySGAN to that of a standard segmentation model. We build up on CycleGAN's adversarial and cyclic consistency losses for the image translation. We facilitate the image translation through implicit structural constraints induced by the segmentation components. The segmentation builds on supervised losses that leverage the segmentation from the labeled domain. We complement the supervised losses with self-supervised structural consistency and structure-based adversarial objectives, leveraging unlabeled images of the target domain. We evaluate our method on a newly collected densely annotated Expansion Microscopy (ExM) nuclei dataset that matches a publicly available EM nuclei dataset. The unpaired Electron Microscopy (EM) and the ExM datasets both image parts of the zebrafish brain. The ExM domain is our target domain and the EM is our source domain.
Our CySGAN outperforms both pre-trained and sequential models, as well as a modified 2D semantic segmentation model that we manually adapted to 3D instance segmentation due to the lack of available frameworks. The implementation of CySGAN and the newly collected ExM dataset named NucExM are publicly available at here.
There are two versions of the code. The first is my research code developed with Slurm for Harvard's computer clusters and based on the original CycleGAN implementation. The second is the community code in the Pytorch Connectomics project. This is a comprehensive and easy-to-use version of CySGAN that my supervisor Zudi Lin added to his public repository based on the knowledge, experiments, and code I created.