Skip to content

Unable to create a context #283

@vlamanna

Description

@vlamanna

Issue description

Create a context fail on Linux / AMD64 with CUDA

Expected Behavior

The following code returns a context:

const context = await model.createContext();

Actual Behavior

The following works fine on my Mac M1 environment but fails every time on linux/amd64 with CUDA. It returns the following error: Context is disposed.

I've tried adding:
console.log(this._ctx.getContextSize()
right after this line:
https://github.com/withcatai/node-llama-cpp/blob/beta/src/evaluator/LlamaContext/LlamaContext.ts#L82

And even there, I get the same error.

It works fine on 3.0.0-beta.32.

Steps to reproduce

On linux/amd64 with CUDA, run the following:
const context = await model.createContext();

My Environment

Dependency Version
Operating System Linux
CPU Intel Xeon
Node.js version 20.11.0
Typescript version 5.5.3
node-llama-cpp version 3.0.0-beta.40

Additional Context

No response

Relevant Features Used

  • Metal support
  • CUDA support
  • Grammar

Are you willing to resolve this issue by submitting a Pull Request?

Yes, I have the time, but I don't know how to start. I would need guidance.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions