Skip to content

About

No description, website, or topics provided.

Resources

Stars

5 stars

Watchers

3 watching

Forks

Repository files navigation

DMaS-LLaMa-Lite: 1.7B-Parameter LLaMa Model Training Code

This repository contains the training scripts and configuration files for DMaS-LLaMa-Lite, a 1.7B-parameter model inspired by the LLaMa architecture and trained from scratch on approximately 20 billion tokens of curated high-quality data.

The overall training code is modified from BuildNanoGPT, while the LLaMa implementation is adapted from hengjiUSTC's learn-llm.


Model Overview

DMaS-LLaMa-Lite is a 1.7B-parameter LLaMa-based language model pretrained with the following key highlights:

  • High-Quality Training Data: Curated from the FineWeb-Edu dataset, emphasizing educational and coherent content.
  • Training Stability: Insights include the importance of optimizer state restoration and managing hardware transitions.
  • Efficient Performance: Competitive downstream task results achieved with significantly fewer tokens than comparable models.

For more details, refer to our paper:

Experience of Training a 1.7B-Parameter LLaMa Model From Scratch
Miles Q Li, Benjamin Fung, Shih-Chia Huang (2024)
arXiv preprint link


Citation

If you use this repository or the pre-trained model, please cite our work:

@INPROCEEDINGS{li2025training,
  author={Li, Miles Q. and Fung, Benjamin C. M. and Huang, Shih-Chia},
  booktitle={2025 International Joint Conference on Neural Networks (IJCNN)},
  title={Training Dynamics of a 1.7B LLaMa Model: A Data-Efficient Approach},
  year={2025},
  volume={},
  number={},
  pages={1-10},
  keywords={Training;Analytical models;Refining;Benchmark testing;Throughput;Data models;Hardware;Stability analysis;Trajectory;Tuning},
  doi={10.1109/IJCNN64981.2025.11228044}}

Resources


Acknowledgments

This work builds on open-source projects, including:

We thank the community for their valuable contributions and tools that make this research possible.

About

No description, website, or topics provided.

Resources

Stars

5 stars

Watchers

3 watching

Forks

Releases

Packages

Used by

Contributors

Languages