NFCMTL: Auto NailFold Capillaroscopy through a Multi-Task Learning Model

This is the official codebase for our paper:

NFCMTL: Auto NailFold Capillaroscopy through a Multi-Task Learning Model

---

🚀 Highlights

• Unified framework for NFC capillary segmentation, classification, and keypoint detection
• Uses Multiscale Vision Transformer (MViT) + FPN + Mask-RCNN
• Applies uncertainty-weighted multi-task loss to NFC tasks
• Outperforms prior SOTA in all major tasks

---

🏗️ Overview

We address key challenges in nailfold capillaroscopy (NFC) by designing a joint learning model. Compared to traditional and fragmented pipelines, NFCMTL unifies them into a streamlined inference graph as shwon below.

---

📦 Model Architecture

• Backbone: Multiscale image visual encoding using MViTv2-T pretrained on MViTv2-T.
  • Input: 3-channel RGB image
  • Output: Multiscale feature maps at 1/4, 1/8, and 1/16 resolutions
• Neck: Feature Pyramid Network (FPN)
• Heads: Mask-RCNN-based architecture with 3 output heads:
  • Classification: Type of each capillary loop
  • Segmentation: Pixel-wise instance masks
  • Keypoints: 8 landmark predictions for vessel apex and limbs

---

🧮 Multi-Task Loss with Uncertainty

We adopt Kendall et al.'s uncertainty-based formulation to dynamically balance task contributions:

$$\mathcal{L}_{\text{total}} = \sum_{i} \frac{1}{2\sigma_i^2} \mathcal{L}_i + \log \sigma_i$$

Where:

• $\mathcal{L}_i$ is the task-specific loss
• $\sigma_i$ is the learned task uncertainty

This adaptive scheme improves training stability and robustness, especially under annotation noise or scale imbalance.

---

🧬 Dataset

We use the public ANFC dataset ([Zhao et al., ISBI 2024]) with:

• 321 NFC images, high-res RGB
• Capillary instance masks, class labels, and 9 anatomical keypoints

Keypoints include apex (U, D), arterial limb (LL, LR), venous limb (RL, RR), bases (LB, RB), and optional conjunction point (X).

---

🔍 Results

Task	Metric	NFCMTL (Ours)	SOTA (Zhao et al.)
Segmentation	Sensitivity ↑	0.821	0.653
Classification	Accuracy ↑	0.930	0.800
Venous Diameter	MAE ↓	0.978	0.989
Arterial Diameter	MAE ↓	0.850	0.849
Apical Diameter	MAE ↓	1.459	1.674

---

🔧 Getting Started

Installation

git clone <https this repo>
cd NFCMTL
conda create -n nfcmtl python=3.11
conda activate nfcmtl
pip install -r requirements.txt

Install Detectron2 based on your CUDA version.

---

Data Preparation

1. Request dataset according to ANFC dataset.
2. Put COCO-formatted annotations in the desired directory as follows:

├── train
│   ├── *.jpg
│   ├── anfc_coco_train.json
├── val
│   ├── *.jpg
│   ├── anfc_coco_val.json

Training

python main.py --data-path /path/to/data/ANFC-coco

---

Evaluation

To evaluate specific tasks, uncomment and run one of the following inside main.py:

nfcmtl_test(cfg, mode="segmentation")
nfcmtl_test(cfg, mode="classification")
nfcmtl_test(cfg, mode="keypoints")

Or visualize predictions:

visualize(cfg, data_path, output_dir)

---

🧾 Citation

If you find this work helpful, please cite:

bibtex coming soon.

Questions and issues are welcome via GitHub!