Low Rank Adaptations for Effective Machine Unlearning

Anonymous submission

Abstract datasets. These limitations entail the need for efficient un-
learning methods given access only to the pre-trained model
Growing privacy regulations have made machine unlearn-
ing an essential process for removing the influence of spe- and the data points requested for deletion.
cific data points from trained models. While retraining on Several methods have been proposed to address this chal-
the remaining dataset is a straightforward solution, it incurs lenge. (Graves, Nagisetty, and Ganesh 2020) introduced a
high computational costs and requires access to the retained technique that stores the index of training examples and
dataset, which may not always be practical. Existing unlearn- per-batch parameter updates, and unlearns by applying the
ing methods, such as gradient ascent, often suffer from unsta- stored gradients in the opposite direction, effectively revers-
ble optimization and catastrophic forgetting. Recent studies ing the influence of the data point on the model parameters.
have demonstrated that by training fewer parameters, Low- However, this requires extensive storage and may lead to a
Rank Adaptation (LoRA) constrains updates to prevent sig-
nificant divergence from the base model, effectively miti-
model that poorly approximates the one that would have ex-
gating catastrophic forgetting. Building on this insight, we isted if those parameter updates had not been applied.
propose a novel framework, NegLoRA that leverages LoRA To reduce storage needs and computational overhead,
to enhance the efficiency and effectiveness of machine un- (Golatkar, Achille, and Soatto 2020) introduce NegGrad,
learning. Experimental results across various metrics indicate which applies negative gradient updates for the samples to
that NegLoRA outperforms baseline methods in unlearning be forgotten, equivalently moving in the direction of increas-
accuracy, generalization, and robustness to inference attacks ing loss for those samples. This approach aims to hinder the
while being computationally efficient. Our code is available model’s ability to classify the ”forget” samples correctly.
at [Link] However, it often leads to unstable training objectives and
can lead to catastrophic forgetting, impairing the model’s
Introduction generalizability on the primary task by inadvertently eras-
Deep Neural Networks have demonstrated remarkable ac- ing useful features learned from other data. Moreover, re-
curacies across numerous tasks by harnessing vast datasets. cent studies (Kurmanji et al. 2023) have shown that diverg-
However, this success brings the challenge of data privacy, ing algorithms such as gradien ascent, can cause uncharac-
as these models may inadvertently memorize training ex- teristially high errors on deleted examples making the model
amples, making them vulnerable to inference attacks that susceptible to Membership Inference Attacks.
could reveal sensitive information potentially harming user Recent studies such as (Hu et al. 2021) show that model
privacy. Privacy regulations like the General Data Protection adaptation intrinsically involves low-rank rank changes in its
Regulation (GDPR) and the California Consumer Privacy weights. Building on this conclusion, Low Rank Adaptation
Act (CCPA) grant individuals the ”Right to be forgotten” al- (LoRA) freezes the pretrained model’s weights and injects
lowing them to request the removal of their data from mod- trainable rank decomposition matrices to reduce the num-
els and services. To address these concerns, Machine Un- ber of parameters updated during fine-tuning, which signifi-
learning methods systematically remove data points from a cantly decreases computational costs. Furthermore, by train-
model, ensuring that once the data has been erased the model ing fewer parameters LoRA constraints the finetuned model
behaves as though it had never encountered that information. from diverging significantly from the base model (Biderman
The naive but optimal method method involves training et al. 2024), effectively acting as a regularizer that mitigates
the model from scratch using the remaining training set, af- catastrophic forgetting more than other regularization tech-
ter removing the data points to be forgotten. This procedure, niques, while achieving similar or even better performance
termed ”exact unlearning”, guarantees that the weights of than full fine-tuning in many cases.
the resulting model aren’t influenced by the instances to for- Building on LoRA’s parameter efficiency and regulariza-
get, but proves to be impractical for large-scale models and tion benefits, we propose NegLoRA, an unlearning method
datasets due to substantial computational costs. Moreover, which leverages Low-Rank Adaptation to apply targeted
loading the original dataset may be unfeasible due to data negative gradient updates. By efficiently guiding the model’s
retention policies or limitations in storage capacity for large parameters away from the influence of specific data points,
NegLoRA ensures effective forgetting while preserving high : Trainable
accuracy on both retained and test datasets. This approach 11 X
: Frozen
also significantly reduces computational and memory de-
mands through parameter-efficient low-rank updates, mak- : Sparse

MLP A B MLP
ing it a scalable and effective solution for unlearning. MLP Head

To further enhance efficiency, (Jia et al. 2024) leverages Norm Norm A B

the Lottery Ticket Hypothesis (Frankle and Carbin 2019) to

introduce sparsity by pruning specific weights or neurons Multi-Head
A B
Multi-Head
Class
Macaw
Attention Attention
in the trained model before unlearning, reducing overfitting Forget Data Snake
Pigeon
...
and computational costs. However, applying gradient ascent Norm Norm

to sparse models leads to a steep decline in the model’s ac-

curacy as sparsity increases. To address this, we propose Embedded Patches

Sparse NegLoRA using LoRA to unlearn in sprase models

which mitigates the adverse effects of gradient ascent, effec-
Figure 1: Sparse NegLoRA on ViT
tively balancing unlearning efficiency, computational cost,
and maintaining overall model performance.
Methodology
Background and Notation We focus on the problem of class unlearning in multi-class
classification tasks, where the aim is to remove information
LoRA about a specific class, called the forget class, from a trained
Low-Rank Adaptation (LoRA) (Hu et al. 2021) is a tech- model.
nique for efficiently fine-tuning large models by introducing
low-rank parameter updates, reducing computational and NegGrad
memory costs. Instead of updating the entire weight matrix GA: (Graves, Nagisetty, and Ganesh 2020) (Thudi et al.
W ∈ Rd×k , LoRA factorizes the update as W ′ = W +∆W , 2022) (Golatkar, Achille, and Soatto 2020) We define a loss
where ∆W is approximated by the product of two low- function L(ϕ(x; w), y) on Df that quantifies the model’s
rank matrices B ∈ Rd×r and A ∈ Rr×k with rank r ≪ performance on the data to be forgotten. Our objective is
min(d, k). This can be expressed as: to maximize this loss to reduce the model’s reliance on Df .
Here, the loss function L for unlearning can be defined as:
W ′ = W + BA,
1 X
where only the matrices A and B are trained, while W L(ϕ(x; w), y) = − log P (y|x; w),
|Df |
remains fixed. During fine-tuning, LoRA only modifies A (x,y)∈Df
and B, allowing the model to adapt to new data with mini-
mal parameter adjustments, which avoids overfitting and re- where P (y|x; w) is the model’s predicted probability for
duces catastrophic forgetting of previously learned informa- the correct label y given input x. We perform iterative up-
tion. By introducing low-rank updates, LoRA achieves an dates to wD using gradient ascent:
efficient, scalable adaptation, making it particularly suitable
for applications requiring continual learning or unlearning, w ← w + η∇w L(ϕ(x; w), y),
as it focuses parameter updates on specific subspaces of the where η is the learning rate. ϕ(x; S(wD )), approximates
model. ϕ(x; wDr ), aligning with privacy requirements for data re-
moval.
Unlearning
NegLoRA
Let ϕ(·) ∈ RK be a model, with parameters w (weights) While the vanilla NegGrad approach is successful in un-
trained using dataset D. The k-th component of the vector learning Df , it frequently leads to reduced performance
ϕ(x; w) in response to an image x approximates the log- on Dr , decreasing the model’s overall utility. (Zhang et al.
posterior, i.e., ϕ(x; w)k ≈ log P (y = k|x),, up to a normal- 2024) determine that gradient ascent approaches can lead to
izing constant. catastrophic collapse of the model due to the divergent na-
Given a model ϕ(x; w), where x is an input and w rep- ture of the algorithm as well as the unbounded nature of its
resents the model parameters, we aim to selectively unlearn loss function. To address this, we propose NegLoRA, which
a subset of data Df ⊂ D while retaining knowledge from exploits the fundamental regularization properties of Low
Dr = D \ Df . After training on dataset D, the model pa- Rank Adaptations by following a first adapt, then ascend
rameters are denoted as wD , resulting in a model ϕ(x; wD ). paradigm. By training fewer parameters LoRA constrains
Our goal is to design a scrubbing function S that modi- the fine-tuned model from diverging significantly from the
fies wD to produce updated parameters S(wD ), such that base model. By prioritizing ”forgetting” in our loss func-
ϕ(x; S(wD )) ≈ ϕ(x; wDr ), effectively removing the influ- tion and using low-rank updates, NegLoRA achieves higher
ence of Df and mimicking a model trained only on Dr . unlearning accuracy while maintaining model performance
 Unlearning Accuracy (UA) MIA-Efficacy Remaining Accuracy (RA) Testing Accuracy (TA) RTE
Model # Params
1 Epoch 2 Epochs 1 Epoch 2 Epochs 1 Epoch 2 Epochs 1 Epoch 2 Epochs (/epoch)
ImageNet100
Retrain 21.7M 100.00 100.00 100.00 100.00 62.07 62.07 55.48 55.48 1.0000
GA(full) 21.7M 97.34 0.00 84.45 85.23 54.28 39.07 47.99 34.75 0.0081
GA (headatt) 1.86M 97.19 100.00 84.06 85.08 54.50 39.56 48.42 35.22 0.0081
GA (LoRA) 20K 100.00 100.00 96.80 97.27 61.98 60.62 55.37 54.38 0.0056
CIFAR100
Retrain 21.7M 100.00 100.00 100.00 100.00 97.98 97.98 60.37 60.37 1.0000
GA (full) 21.7M 100.00 100.00 100.00 100.00 83.13 58.20 44.78 33.31 0.025
GA (headatt) 1.86M 100.00 100.00 100.00 100.00 95.94 94.98 55.77 49.15 0.025
GA LoRA 20K 98.86 100.00 100.00 100.00 96.87 95.80 57.76 50.43 0.017
CIFAR10
Retrain 21.7M 100.00 100.00 100.00 100.00 96.90 96.90 85.92 85.92 1.0000
GA (full) 21.7M 97.19 100.00 100.00 100.00 54.50 39.56 48.42 35.22 0.025
GA (headatt) 1.86M 97.50 99.88 100.00 100.00 96.22 92.80 84.80 79.10 0.025
GA LoRA 20K 93.57 100 100.00 100.00 96.53 96.50 85.01 85.38 0.017

Figure 2: Results of ViT When Tested Using Various Unlearning Approaches (in percent accuracy)

and making the model more robust to inference attacks. Re- Models
cent works (Geva et al. 2021; Dai et al. 2022) reveal that We trained a Vision Transformer (ViT) (Dosovitskiy et al.
FFN layers in Transformer blocks serve as key-value mem- 2021) and a ResNet50 (He et al. 2015) on the aforemen-
ories, storing factual knowledge. Effective unlearning re- tioned datasets. Check Appendix for more details
quires modifying these layers, but direct updates are compu-
tationally inefficient due to their large parameter size. To ad- Baseline: We evaluate two baselines to compare against
dress this, we integrate LoRA modules into the FFN layers, our proposed LoRA-based unlearning method:
enabling low-rank, parameter-efficient updates. By making • Retrain: Retrain the model on the retained dataset Dr
only the LoRA components trainable, our approach achieves
• NegGrad: Perform Gradient Ascent on the full set of
targeted unlearning while mitigating catastrophic forgetting
model weights.
and maintaining model utility and efficiency
• Partial NegGrad: Gradient Ascent on a subset of the
Sparse NegLoRA Recent works (Jia et al. 2024; Shah model’s weights.
et al. 2024; Mehta et al. 2022) show that model sparsity can
improve unlearning by focusing updates on a subset of pa- For Partial NegGrad in the ViT, Gradient Ascent is ap-
rameters, reducing overfitting and computational cost. How- plied to the classifier head and the feed-forward network
ever, (Jia et al. 2024) highlights that naive gradient ascent (FFN) and attention output projection layers in the last trans-
degrades model performance as sparsity increases. To ad- former block. For Resnet, it is applied to the convulutional
dress this, we propose Sparse NegLoRA, which follows a and linear layers.
”prune first, adapt, then ascend” approach. First, we prune a Sparse NegLoRA To evaluate the effectiveness of Sparse
subset of model weights, then fine-tune the pruned model to NegLoRA, we apply our method to fine-tuned sparse mod-
restore performance. Next, Low-Rank Adaptation (LoRA) is els at varying levels of sparsity. Specifically, we experiment
applied for efficient updates, followed by gradient ascent to with sparsity levels of 50%, 75%, 90%, and 95%, using L2
unlearn Df . Sparse NegLoRA mitigates catastrophic forget- pruning. Sparse NegLoRA is then compared against the es-
ting while maintaining efficiency and model performance. tablished baselines on both Vision Transformer (ViT) and
ResNet architectures.

Experimental Setup Evaluation Metrics

Our experiments aim to evaluate to evaluate the effective- Our goal is to ensure that the model, after unlearning, not
ness of low-rank updates through LoRA in selectively for- only removes specific data points but also retains its perfor-
getting specific data points while preserving the accuracy of mance on the remaining data and remains generalizable to
the model on the remaining dataset. Furthermore, we ana- unseen samples. Additionally, we aim to achieve this in a
lyze the ability of applying LoRA within sparse models, to parameter-efficient manner to reduce computational costs as
mitigate catastrophic forgetting in gradient ascent, while en- well as memory requirements. Our metrics include:
suring effective unlearning. 1. Unlearning accuracy: UA = 1-Acc(Df ) is the accuracy of
the unlearned model on the forget dataset. A higher UA
Datasets indicates more effective unlearning of the forget dataset.
For evaluation, we used the CIFAR-10 and CIFAR-100 2. Retention Accuracy (RA): This is the accuracy of the un-
datasets (Krizhevsky 2009) along with a subset of the Ima- learned model on the retain dataset, indicating how well
geNet dataset by randomly sampling 100 classes, which we the model maintains performance on data that should not
refer to as ImageNet100. Check Appendix for more details. be forgotten
 Figure 3: Sparse NegLoRA on ViT

3. Test Accuracy (TA): The accuracy of the unlearned the original training data is available at the time of unlearn-
model on a separate testing dataset containing the re- ing. For our proposed method, we require access to no more
maining classes, used to assess the model’s generalizabil- than the unlearning data Df , allowing seamless application
ity after unlearning. for practitioners under real-world scenarios.
4. Membership Inference Attack Efficacy (MIA-Efficacy):
Measures the vulnerability of the unlearned model to Results
membership inference attacks on the forget dataset Df
using a confidence-based MIA predictor. A higher MIA- Table 2 presents the accuracy metrics for ViT, compar-
Efficacy indicates less information about Df in the ing NegLoRA with the baselines. NegLoRA consistently
model. achieves superior Remaining Accuracy (RA) and Testing
Accuracy (TA) while maintaining near-perfect Unlearning
Related Work Accuracy and MIA-Efficacy. Its performance remains robust
egardless of the size of the dataset or the number of classes.
Machine Unlearning: Similarly, Appendix Table 4 demonstrates that NegLoRA
The problem of machine unlearning (MU) was formally in- outperforms baselines even on non-transformer architec-
troduced by (Cao and Yang 2015) in response to stricter pri- tures (Resnet) achieving higher RA and TA, while maintain-
vacy regulations, a decade of research has since followed. ing comparable MIA-Efficacy Furthermore, as shown in 3,
MU techniques can be classified into two broad categories: applying NegLoRA to sparse models effectively mitigates
exact and approximate unlearning. Exact unlearning typi- catastrophic collapse. It preserves unlearning accuracy at op-
cally involves retraining the model from scratch after the timal levels while delivering significantly higher Remaining
removal of specific data points. (Bourtoule et al. 2020) intro- Accuracy (RA) and Testing Accuracy (TA) compared to the
duced the SISA framework, which partitions data into shards baselines. Additionally, NegLoRA achieves MIA-Efficacy
and slices, with each shard serving as a weak learner that comparable to other methods, demonstrating its robustness
can be quickly retrained upon an unlearning request. How- even under high sparsity conditions.
ever, the computational intensity of these methods empha-
size the need for more efficient methods. (Neel, Roth, and Conclusion
Sharifi-Malvajerdi 2020) adopt similar DP-inspired defini-
tions of unlearning based on the goal of indistinguishabil- This study addresses the challenge of efficient machine un-
ity from the retrain-from-scratch model. (Cha et al. 2024) learning in light of growing privacy regulations and the
uses utilizing adversarial examples to overcome loss of need for adaptable AI systems. We introduced NegLoRA, a
utility at the representation-level. (Tarun et al. 2024) and novel approach for efficient and effective machine unlearn-
(Chundawat et al. 2023) learn error minimization and error ing using Low-Rank Adaptation (LoRA) combined with
maximization-based noise matrices which are used to fine- gradient ascent. Our extensive experiments across ViT and
tune the trained model in order to do unlearning. (Thudi et al. ResNet architectures demonstrate that NegLoRA consis-
2022) propose a regularizer to reduce the ‘verification er- tently achieves near-perfect unlearning performace, while
ror’, which is an approximation to the distance between the maintaining accuracy on the remaining classes even under
unlearned model and a retrained-from-scratch model. (Shah varying levels of sparsity. Additionally, NegLoRA demon-
et al. 2024) uses a Discrete Key-Value Bottleneck to perform strates significantly higher robustness towards inference at-
unlearning in a model with inherent sparse representations. tacks paving the road for more privacy-preserving and scal-
However, existing methods assume that at least a subset of able AI systems without sacrificing utility.
 References Shah, V.; Träuble, F.; Malik, A.; Larochelle, H.; Mozer, M.;
Biderman, D.; Portes, J.; Ortiz, J. J. G.; Paul, M.; Greengard, Arora, S.; Bengio, Y.; and Goyal, A. 2024. Unlearning via
P.; Jennings, C.; King, D.; Havens, S.; Chiley, V.; Frankle, Sparse Representations. arXiv:2311.15268.
J.; Blakeney, C.; and Cunningham, J. P. 2024. LoRA Learns Tarun, A. K.; Chundawat, V. S.; Mandal, M.; and Kankan-
Less and Forgets Less. arXiv:2405.09673. halli, M. 2024. Fast Yet Effective Machine Unlearning.
Bourtoule, L.; Chandrasekaran, V.; Choquette-Choo, C. A.; IEEE Transactions on Neural Networks and Learning Sys-
Jia, H.; Travers, A.; Zhang, B.; Lie, D.; and Papernot, N. tems, 35(9): 13046–13055.
2020. Machine Unlearning. arXiv:1912.03817. Thudi, A.; Deza, G.; Chandrasekaran, V.; and Papernot, N.
Cao, Y.; and Yang, J. 2015. Towards Making Systems For- 2022. Unrolling SGD: Understanding Factors Influencing
get with Machine Unlearning. 2015 IEEE Symposium on Machine Unlearning. arXiv:2109.13398.
Security and Privacy, 463–480. Wu, K.; Zhang, J.; Peng, H.; Liu, M.; Xiao, B.; Fu, J.; and
Cha, S.; Cho, S.; Hwang, D.; Lee, H.; Moon, T.; and Lee, Yuan, L. 2022. TinyViT: Fast Pretraining Distillation for
M. 2024. Learning to Unlearn: Instance-wise Unlearning Small Vision Transformers. arXiv:2207.10666.
for Pre-trained Classifiers. arXiv:2301.11578. Zhang, R.; Lin, L.; Bai, Y.; and Mei, S. 2024. Negative Pref-
Chundawat, V. S.; Tarun, A. K.; Mandal, M.; and Kankan- erence Optimization: From Catastrophic Collapse to Effec-
halli, M. 2023. Zero-Shot Machine Unlearning. IEEE tive Unlearning. arXiv:2404.05868.
Transactions on Information Forensics and Security, 18:
2345–2354.
Appendix
Dai, D.; Dong, L.; Hao, Y.; Sui, Z.; Chang, B.; and Wei,
F. 2022. Knowledge Neurons in Pretrained Transformers. Limitations and Future Scope
arXiv:2104.08696. A promising avenue for future research is the exploration
Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, of various membership inference attack methods to fur-
D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; ther validate our hypothesis. Another potential application
Heigold, G.; Gelly, S.; Uszkoreit, J.; and Houlsby, N. 2021. of this study is the extension of NegLoRA to more com-
An Image is Worth 16x16 Words: Transformers for Image plex models such as Multimodal Large Language Models
Recognition at Scale. arXiv:2010.11929. (MLLMs) where parameter efficiency is of the utmost im-
Frankle, J.; and Carbin, M. 2019. The Lottery Ticket portance for unlearning. While computational constraints
Hypothesis: Finding Sparse, Trainable Neural Networks. limited our ability to explore this direction, scaling our ap-
arXiv:1803.03635. proach to multimodal models could advance the develop-
Geva, M.; Schuster, R.; Berant, J.; and Levy, O. 2021. ment of adaptable, privacy-preserving AI systems capable
Transformer Feed-Forward Layers Are Key-Value Memo- of handling diverse and complex datasets.
ries. arXiv:2012.14913.
Training Details and Hyperparameters
Golatkar, A.; Achille, A.; and Soatto, S. 2020. Eternal Sun-
shine of the Spotless Net: Selective Forgetting in Deep Net- We perform all our experiments on an NVIDIA T4(x2)
works. arXiv:1911.04933. GPU.
Graves, L.; Nagisetty, V.; and Ganesh, V. 2020. Amnesiac Datasets
Machine Learning. arXiv:2010.10981.
CIFAR 10 CIFAR-10 is a widely used dataset in computer
He, K.; Zhang, X.; Ren, S.; and Sun, J. 2015. Deep Residual
vision and machine learning. It comprises 60,000 32x32
Learning for Image Recognition. arXiv:1512.03385.
color images in 10 different classes, with 6,000 images per
Hu, E. J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, class. The dataset is divided into 50,000 training images and
S.; Wang, L.; and Chen, W. 2021. LoRA: Low-Rank Adap- 10,000 testing images. CIFAR-10 represents a diverse range
tation of Large Language Models. arXiv:2106.09685. of everyday objects, such as airplanes, automobiles, birds,
Jia, J.; Liu, J.; Ram, P.; Yao, Y.; Liu, G.; Liu, Y.; Sharma, and cats, making it a challenging task for image classifica-
P.; and Liu, S. 2024. Model Sparsity Can Simplify Machine tion.
Unlearning. arXiv:2304.04934. Data Augmentations: random cropping to 32x32 with 4-
Krizhevsky, A. 2009. Learning multiple layers of features pixel padding, random horizontal flipping, and per-channel
from tiny images. Technical report. normalization with a mean of [0.4919, 0.4822, 0.4465] and
Kurmanji, M.; Triantafillou, P.; Hayes, J.; and Triantafil- standard deviation of [0.2023, 0.1994, 0.2010]. At test time,
lou, E. 2023. Towards Unbounded Machine Unlearning. we resize to 32x32 and normalize.
arXiv:2302.09880. CIFAR-100 CIFAR-100 is a complex extension of
Mehta, R.; Pal, S.; Singh, V.; and Ravi, S. N. 2022. Deep Un- CIFAR-10, containing 100 classes with 600 images per
learning via Randomized Conditionally Independent Hes- class, split into 500 training images and 100 testing images
sians. arXiv:2204.07655. per class. Each class is labeled with a ”fine” label. The in-
Neel, S.; Roth, A.; and Sharifi-Malvajerdi, S. 2020. Descent- creased number of classes make CIFAR-100 an intriguing
to-Delete: Gradient-Based Methods for Machine Unlearn- dataset and poses a more significant challenge for models to
ing. arXiv:2007.02923. forget specific classes while retaining knowledge of others.
Data Augmentations: random cropping to 32x32 with 4- Config Value
pixel padding, 50% random horizontal flipping, and per-
channel normalization with a mean of [0.5071, 0.4865, patch size 16
0.4409] and standard deviation of [0.2673, 0.2564, 0.2762]. optimizer AdamW
At test time, we resize to 32x32 and normalize. base learning rate 10−3
learning rate schedule warmup + cosine decay
ImageNet-100 ImageNet-100 is a subset of the ImageNet weight decay 0.05
dataset containing 100 diverse classes, with 1300 training momentum β1 , β2 = 0.9, 0.999
images and 50 validation images per class. The dataset pro- batch size 256
vides a balanced mix of complexity and scale, allowing for warm-up epochs 20
rigorous testing of a model’s ability to forget specific data warm-up learning rate 1 × 10−6
points while retaining generalization across the remaining training epochs 90
classes Data Augmentations: For training, we use random
resized cropping (224x224) with a scale range of (0.05, 1.0), Table 2: ViT/16 Training Configuration Table
random horizontal flipping, RandAugment (n=9, m=0.5),
per-channel normalization with mean [0.485, 0.456, 0.406]
and standard deviation [0.229, 0.224, 0.225], and random Config Value
erasing with a 25% probability. At test time, we resize to optimizer AdamW
224x224 and apply the same normalization. base learning rate 10−1
Data Augmentations learning rate schedule cosine decay
weight decay 0.05
n01968897 n01770081 n01818515 n02011460 n01496331 momentum β1 , β2 = 0.9, 0.999
n01847000 n01687978 n01740131 n01537544 n01491361
batch size 128
n02007558 n01735189 n01630670 n01440764 n01819313
n02002556 n01667778 n01755581 n01924916 n01751748 training epochs 200
n01984695 n01729977 n01614925 n01608432 n01443537 dropout rate 0.1
n01770393 n01855672 n01560419 n01592084 n01914609
n01582220 n01667114 n01985128 n01820546 n01773797 Table 3: Resnet50 Training Configuration Table
n02006656 n01986214 n01484850 n01749939 n01828970
n02018795 n01695060 n01729322 n01677366 n01734418
n01843383 n01806143 n01773549 n01775062 n01728572 Config Value
n01601694 n01978287 n01930112 n01739381 n01883070
n01774384 n02037110 n01795545 n02027492 n01531178 optimizer Adam
n01944390 n01494475 n01632458 n01698640 n01675722 batch size 256
n01877812 n01622779 n01910747 n01860187 n01796340 LoRA rank 16
n01833805 n01685808 n01756291 n01514859 n01753488 LoRA alpha 32
n02058221 n01632777 n01644900 n02018207 n01664065 LoRA dropout 0.1
n02028035 n02012849 n01776313 n02077923 n01774750 epochs 1/2
n01742172 n01943899 n01798484 n02051845 n01824575
n02013706 n01955084 n01773157 n01665541 n01498041
n01978455 n01693334 n01950731 n01829413 n01514668 Table 4: LoRA Finetuning Configuration Table

Table 1: List of ImageNet-100 classes

• Training Phase: An MIA predictor is trained on a bal-
anced dataset sampled from the model’s remaining train-
Models ing set (Dr ) and an external test set (Dtest ). This predic-
ViT Vision Transformer (ViT), introduced by (Dosovit- tor learns to classify samples based on features such as
skiy et al. 2021), adapts the transformer architecture to prediction confidence.
image classification by treating images as sequences of • Testing Phase: The trained MIA predictor evaluates the
patches. We consider TinyViT from (Wu et al. 2022) with forgetting dataset (Df ), aiming to classify samples as ei-
approximately 21M parameters, as it is a compact version ther “training” or “non-training.”
of ViT designed to be parameter-efficient while maintaining The MIA-Efficacy measures the success of unlearning by
high performance. quantifying how well the predictor identifies the forgetting
Resnet ResNet50 Introduced by (He et al. 2015), it facili- samples as ”non-training”. It is formally defined as:
tates the training of deep networks through residual connec-
TN
tions, which mitigates the problem of vanishing gradients. MIA-Efficacy =
|Df |
Detailed Evaluation Metrics where TN is the number of forgetting samples correctly
Membership Inference Attacks Membership Inference classified as non-training, and |Df | is the total number of
Attacks attempt to determine whether a specific data point samples in Df . A high MIA-Efficacy indicates effective un-
was part of the model train data. It operates in two phases: learning.
 Unlearning Accuracy (UA) MIA-Efficacy Remaining Accuracy (RA) Testing Accuracy (TA) RTE
Model # Params
1 Epoch 2 Epochs 1 Epoch 2 Epochs 1 Epoch 2 Epochs 1 Epoch 2 Epochs (secs/epoch))
CIFAR100
Retrain 23.5M 100.00 100.00 100.00 00 00 00 00 00 1.000
NegGrad 23.5M 100.00 100.00 28.60 45.40 58.05 52.91 45.55 41.47 0.025
Partial NegGrad 1.86M 100.00 100.00 58.00 48.97 60.94 45.96 47.23 37.06 0.025
NegLoRA 608K 100.00 100.00 100.00 100.00 98.97 98.66 74.19 72.21 0.017
CIFAR10
Retrain 23.5M 100.00 0.00 100.00 100.00 38.79 39.07 38.10 34.75 1.000
NegGrad 23.5M 94.16 98.08 0.00 0.10 73.64 49.40 70.62 48.82 0.025
Partial NegGrad 20.5M 89.36 95.66 0.00 0.00 80.27 68.34 76.06 66.17 0.025
NegLoRA 20.5K 5.58 1.56 96.44 87.84 98.17 95.69 92.92 90.20 0.017

Figure 4: Results of Resnet50 When Tested Using Various Unlearning Approaches (in percent accuracy)

Run Time Efficiency We evaluate the efficiency by com-

paring how much faster an unlearning method is relative to
retraining, i.e the run-time efficiency (RTE) of an unlearning
method U is defined as:

RT(UR )
RTE(U ) = , where RTE(U ) ∈ [0, +∞),
RT(U )
• RT(U ) represents the time (in seconds) required by
method U to complete unlearning.
• RT(UR ) represents the time (in seconds) required for re-
training the model from scratch (denoted as UR).
All methods are evaluated using the same hardware and
resources for consistency. The RTE of retraining from
scratch is always 1. An MU method with RTE(U ) > 1 is
faster than retraining, while a method with RTE(U ) < 1
is slower than retraining. Ideally, the run-time efficiency
(RTE) of a machine unlearning method should be lower than
the naive approach of retraining the model from scratch.