Skip to main content

Llama CPP

Compatibility

Only available on Node.js.

This module is based on the node-llama-cpp Node.js bindings for llama.cpp, allowing you to work with a locally running LLM. This allows you to work with a much smaller quantized model capable of running on a laptop environment, ideal for testing and scratch padding ideas without running up a bill!

Setup​

You'll need to install major version 3 of the node-llama-cpp module to communicate with your local model.

npm install -S node-llama-cpp@3
tip

See this section for general instructions on installing integration packages.

npm install @langchain/community @langchain/core

You will also need a local Llama 3 model (or a model supported by node-llama-cpp). You will need to pass the path to this model to the LlamaCpp module as a part of the parameters (see example).

Out-of-the-box node-llama-cpp is tuned for running on a MacOS platform with support for the Metal GPU of Apple M-series of processors. If you need to turn this off or need support for the CUDA architecture then refer to the documentation at node-llama-cpp.

A note to LangChain.js contributors: if you want to run the tests associated with this module you will need to put the path to your local model in the environment variable LLAMA_PATH.

Guide to installing Llama3​

Getting a local Llama3 model running on your machine is a pre-req so this is a quick guide to getting and building Llama 3.1-8B (the smallest) and then quantizing it so that it will run comfortably on a laptop. To do this you will need python3 on your machine (3.11 is recommended), also gcc and make so that llama.cpp can be built.

Getting the Llama3 models​

To get a copy of Llama3 you need to visit Meta AI and request access to their models. Once Meta AI grant you access, you will receive an email containing a unique URL to access the files, this will be needed in the next steps. Now create a directory to work in, for example:

mkdir llama3
cd llama3

Now we need to go to the Meta AI llama-models repo, which can be found here. In the repo, there are instructions to download the model of your choice, and you should use the unique URL that was received in your email. The rest of the tutorial assumes that you have downloaded Llama3.1-8B, but any model from here on out should work. Upon downloading the model, make sure to save the model download path, this will be used for later.

Converting and quantizing the model​

In this step we need to use llama.cpp so we need to download that repo.

cd ..
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp

Now we need to build the llama.cpp tools and set up our python environment. In these steps it's assumed that your install of python can be run using python3 and that the virtual environment can be called llama3, adjust accordingly for your own situation.

cmake -B build
cmake --build build --config Release
python3 -m venv llama3
source llama3/bin/activate

After activating your llama3 environment you should see (llama3) prefixing your command prompt to let you know this is the active environment. Note: if you need to come back to build another model or re-quantize the model don't forget to activate the environment again also if you update llama.cpp you will need to rebuild the tools and possibly install new or updated dependencies! Now that we have an active python environment, we need to install the python dependencies.

python3 -m pip install -r requirements.txt

Having done this, we can start converting and quantizing the Llama3 model ready for use locally via llama.cpp. A conversion to a Hugging Face model is needed, followed by a conversion to a GGUF model. First, we need to locate the path with the following script convert_llama_weights_to_hf.py. Copy and paste this script into your current working directory. Note that using the script may need you to pip install extra dependencies, do so as needed. Then, we need to convert the model, prior to the conversion let's create directories to store our Hugging Face conversion and our final model.

mkdir models/8B
mkdir models/8B-GGUF
python3 convert_llama_weights_to_hf.py --model_size 8B --input_dir <dir-to-your-model> --output_dir models/8B --llama_version 3
python3 convert_hf_to_gguf.py --outtype f16 --outfile models/8B-GGUF/gguf-llama3-f16.bin models/8B

This should create a converted Hugging Face model and the final GGUF model in the directories we have created. Note that this is just a converted model so it is also around 16Gb in size, in the next step we will quantize it down to around 4Gb.

./build/bin/llama-quantize ./models/8B-GGUF/gguf-llama3-f16.bin ./models/8B-GGUF/gguf-llama3-Q4_0.bin Q4_0

Running this should result in a new model being created in the models\8B-GGUF directory, this one called gguf-llama3-Q4_0.bin, this is the model we can use with langchain. You can validate this model is working by testing it using the llama.cpp tools.

./build/bin/llama-cli -m ./models/8B-GGUF/gguf-llama3-Q4_0.bin -cnv -p "You are a helpful assistant"

Running this command fires up the model for a chat session. BTW if you are running out of disk space this small model is the only one we need, so you can backup and/or delete the original and converted 13.5Gb models.

Usage​

import { LlamaCpp } from "@langchain/community/llms/llama_cpp";

const llamaPath = "/Replace/with/path/to/your/model/gguf-llama3-Q4_0.bin";
const question = "Where do Llamas come from?";

const model = await LlamaCpp.initialize({ modelPath: llamaPath });

console.log(`You: ${question}`);
const response = await model.invoke(question);
console.log(`AI : ${response}`);

API Reference:

  • LlamaCpp from @langchain/community/llms/llama_cpp

Streaming​

import { LlamaCpp } from "@langchain/community/llms/llama_cpp";

const llamaPath = "/Replace/with/path/to/your/model/gguf-llama3-Q4_0.bin";

const model = await LlamaCpp.initialize({
  modelPath: llamaPath,
  temperature: 0.7,
});

const prompt = "Tell me a short story about a happy Llama.";

const stream = await model.stream(prompt);

for await (const chunk of stream) {
  console.log(chunk);
}

/*


 Once
  upon
  a
  time
 ,
  in
  the
  rolling
  hills
  of
  Peru
 ...
 */

API Reference:

  • LlamaCpp from @langchain/community/llms/llama_cpp
;

Was this page helpful?


You can also leave detailed feedback on GitHub.