Lightweight Multi-Scale Framework for Human Pose and Action Classification

Description

This repository contains the implementation of our lightweight modular deep learning architecture for human pose and action classification. The code is built on a Swin Transformer backbone for hierarchical feature extraction, combined with proposed attention modules, including Enhanced Spatial Attention (SPA), Context-Aware Channel Attention (CCAM), and Dual Weighted Cross Attention (DWCA). These modules enable effective fusion of spatial and channel-level features, improving classification accuracy while maintaining a very low parameter count.

Dataset Information

Both Yoga-82 and Stanford 40 Actions are publicly available datasets provided for academic research. Please refer to their respective sources for licensing and terms of use.

• Dataset Name: Yoga-82

• Description: The Yoga-82 dataset consists of 16800 images in 82 classes.

• Source: (https://www.kaggle.com/datasets/akashrayhan/yoga-82)

• Dataset Name: Stanford 40 actions

• Description: The Stanford 40 actions dataset consists of 9532 images in 40 classes.

• Source: (http://vision.stanford.edu/Datasets/40actions.html)

Prepare Dataset

Verify the dataset structure Make sure your dataset is organized as follows:

../                                   
├── data/               
|   ├── dataloader.py
|   ├── Stanford 40 actions/
│   |   ├── train/
│   |   │   ├── applauding
│   |   │   .  
│   |   │   .  
│   |   │   .  
│   |   |   └── writing_on_a_book
│   |   └── test/
│   |       ├── applauding
│   |       .  
│   |       .  
│   |       .  
│   |       └── writing_on_a_book
|   └── Yoga-82/
│       ├── train/
│       │   ├── Akarna_Dhanurasana
│       │   .  
│       │   .  
│       │   .  
│       |   └── Yogic_sleep_pose
│       ├── valid/
│       │   ├── Akarna_Dhanurasana
│       │   .  
│       │   .  
│       │   .  
│       |   └── Yogic_sleep_pose
│       └── test/
│           ├── Akarna_Dhanurasana
│           .  
│           .  
│           .  
│           └── Yogic_sleep_pose
├── model/
|   ├── CCAM.py
|   ├── DWCA.py
|   ├── MSR.py
|   ├── SPA.py
|   └── backbone_utils.py
├── README.md           
├── test_stanford.py
├── test_yoga82.py
├── train_stanford.py
└── train_yoga82.py             

Code Information

backbone_utils.py: Contains utilities for extracting hierarchical features from the frozen Swin Transformer backbone.

SPA.py: Implementation of the Enhanced Spatial Attention module.

CCAM.py: Implementation of the Context-Aware Channel Attention module.

DWCA.py: Implementation of the Dual Weighted Cross Attention module.

MSR.py: Main model file where all of the modules utilized on the backbone feature maps for robust classification.

data_loader.py: Implementation of data normalization, data augmentation and spliting data into tarin, test and validation.

train.py: Script for training the model on Yoga-82 or Stanford 40 Actions dataset.

test.py: Script for evaluating the trained model on the chosen dataset.

Requirements

-Python 3.11.13

-PyTorch version: 2.6.0

-CUDA version: 12.4

-CUDA device: Tesla P100

-TensorBoard version: 2.18.0

-NumPy version: 1.26.4

-scikit-learn version: 1.2.2

-timm version: 1.0.15

Usage Instruction

to train the model use the following command:

Stanford 40 actions

Training

To train the model, use the following command:

python train_stanford.py --bsz 16
                        --lr 1e-3
                        --niter 60
                        --data_dir data/Stanford 40 actions
                        --logpath log/"your_experiment_name"

Testing

To test the model, use the following command:

python test_stanford.py  --bsz 16
                        --data_dir data/Stanford 40 actions
                        --ckpt log/"your_experiment_name"

Yoga-82

Training

To train the model, use the following command:

python train_yoga82.py --bsz 16
                        --lr 1e-3
                        --niter 60
                        --data_dir data/Stanford 40 actions
                        --logpath log/"your_experiment_name"

Testing

To test the model, use the following command:

python test_yoga82.py  --bsz 16
                        --data_dir data/Stanford 40 actions
                        --ckpt log/"your_experiment_name"

Arguments

• --bsz: Batch size for training.

• --lr: Learning rate for the optimizer.

• --niter: Number of training iterations (epochs).

• --data_dir: Directory where the dataset is located.

• --logpath: Directory to save the best model checkpoint.

  Citation

If you use this repository in your work, please cite this repository or our paper.