Skip to content

arch: refactor LLM_TENSOR_NAMES #18051

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
ngxson merged 7 commits into from
Dec 16, 2025
Merged

arch: refactor LLM_TENSOR_NAMES #18051

ngxson merged 7 commits into from
Dec 16, 2025
+1,897 −2,284

Conversation

@ngxson
Copy link
Collaborator

@ngxson ngxson commented Dec 15, 2025 •
edited
Loading

Motivation

while working on #18042 , I want to add a new arch GLM4V which is mostly the same as GLM4, but use M-RoPE instead of normal RoPE.

However, I feel like sometimes it can be quite too redundant to duplicate the list of tensors and cgraph (for cgraph, we can reuse the same file, but switch rope type using template)

This PR propose a new way to organize model tensors naming, bring it more aligned with the convert_hf_to_gguf.py script, while also allow multiple models to reuse the same mapping.

TODO in a follow-up PR: also update the mapping in convert_hf_to_gguf.py to allow models to reuse the same mappings

This PR was mostly generated using this script: https://gist.github.com/ngxson/a20411de6bc66a84a4d354b254bfe4da

Before

static const std::map<llm_arch, std::map<llm_tensor, const char *>> LLM_TENSOR_NAMES = {
    {
        LLM_ARCH_LLAMA,
        {
            { LLM_TENSOR_TOKEN_EMBD,      "token_embd" },
            { LLM_TENSOR_OUTPUT_NORM,     "output_norm" },
            { LLM_TENSOR_OUTPUT,          "output" },
            { LLM_TENSOR_ROPE_FREQS,      "rope_freqs" },
            { LLM_TENSOR_ATTN_NORM,       "blk.%d.attn_norm" },
            { LLM_TENSOR_ATTN_Q,          "blk.%d.attn_q" },
            ...

After

static const std::map<llm_tensor, const char *> LLM_TENSOR_NAMES = {
    { LLM_TENSOR_TOKEN_EMBD,                             "token_embd" },
    { LLM_TENSOR_OUTPUT_NORM,                            "output_norm" },
    { LLM_TENSOR_OUTPUT_NORM_LFM2,                       "token_embd_norm" }, // fix for wrong tensor name
    { LLM_TENSOR_OUTPUT,                                 "output" },
...

static std::set<llm_tensor> llm_get_tensor_names(llm_arch arch) {
    switch (arch) {
        case LLM_ARCH_LLAMA:
        case LLM_ARCH_DECI:
        case LLM_ARCH_MISTRAL3:
            return {
                LLM_TENSOR_TOKEN_EMBD,
                LLM_TENSOR_OUTPUT_NORM,
                LLM_TENSOR_OUTPUT,
                LLM_TENSOR_ROPE_FREQS,
                LLM_TENSOR_ATTN_NORM,
                ...

@ngxson ngxson requested a review from ggerganov December 15, 2025 09:09
@ngxson ngxson requested a review from CISC as a code owner December 15, 2025 09:09
@github-actions github-actions bot added the documentation Improvements or additions to documentation label Dec 15, 2025
@ngxson ngxson added the refactoring Refactoring label Dec 15, 2025
@ngxson ngxson mentioned this pull request Dec 15, 2025
Merged
@ngxson ngxson merged commit 7f2b2f3 into ggml-org:master Dec 16, 2025
67 of 68 checks passed
@ngxson ngxson mentioned this pull request Dec 16, 2025
Merged
@loci-dev loci-dev mentioned this pull request Dec 16, 2025
Open
@tdakhran tdakhran mentioned this pull request Dec 17, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ggerganov ggerganov ggerganov approved these changes

@CISC CISC Awaiting requested review from CISC CISC is a code owner

Assignees

No one assigned

Labels

documentation Improvements or additions to documentation refactoring Refactoring

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ngxson @ggerganov