Summary

This PR fixes an incorrect weight initialization method in the LinearLoRA class. The xavier initialization was incorrectly using torch.nn.init.uniform_() instead of the proper xavier_normal_() function.

Problem

In the LinearLoRA class located in nemo_automodel/components/_peft/lora.py, when init_method="xavier" is specified, the code was using a generic uniform distribution initialization instead of the Xavier initialization method. This semantic mismatch could lead to suboptimal initialization and potentially affect model convergence and fine-tuning performance.

Solution

File: nemo_automodel/components/_peft/lora.py

Changed the initialization of lora_A weights from:

torch.nn.init.uniform_(self.lora_A.weight.data)

To the correct Xavier initialization:

nn.init.xavier_normal_(self.lora_A.weight.data)

Impact

• Affects: LoRA weight initialization when using xavier method
• Scope: Models using LoRA fine-tuning with xavier initialization
• Behavior Change: The initialization distribution will now follow the Xavier/Glorot normal distribution instead of a simple uniform distribution, which is the expected behavior for xavier initialization