Skip to main content

ChatOllama

Ollama allows you to run open-source large language models, such as Llama 2, locally.

Ollama bundles model weights, configuration, and data into a single package, defined by a Modelfile. It optimizes setup and configuration details, including GPU usage.

This example goes over how to use LangChain to interact with an Ollama-run Llama 2 7b instance as a chat model. For a complete list of supported models and model variants, see the Ollama model library.

Setup​

Follow these instructions to set up and run a local Ollama instance. Then, download the @langchain/ollama package.

tip

See this section for general instructions on installing integration packages.

npm install @langchain/ollama

Usage​

import { ChatOllama } from "@langchain/ollama";
import { StringOutputParser } from "@langchain/core/output_parsers";

const model = new ChatOllama({
  baseUrl: "http://localhost:11434", // Default value
  model: "llama3",
});

const stream = await model
  .pipe(new StringOutputParser())
  .stream(`Translate "I love programming" into German.`);

const chunks = [];
for await (const chunk of stream) {
  chunks.push(chunk);
}

console.log(chunks.join(""));

/*
The translation of "I love programming" into German is:

Ich liebe Programmieren.

Here's a breakdown of the translation:

* Ich = I
* liebe = love
* Programmieren = programming (note: this word is the infinitive form, which is often used in informal contexts or when speaking about one's profession or hobby)

If you want to make it sound more formal or use the correct verb conjugation for "I", you can say:

Ich bin ein großer Fan von Programmieren.

This translates to:

I am a big fan of programming.

In this sentence, "bin" is the first person singular present tense of the verb "sein", which means "to be". The phrase "ein großer Fan" means "a big fan".
*/

API Reference:

tip

See a LangSmith trace of the above example here

Tools​

Ollama now offers support for native tool calling. The example below demonstrates how you can invoke a tool from an Ollama model.

import { tool } from "@langchain/core/tools";
import { ChatOllama } from "@langchain/ollama";
import { z } from "zod";

const weatherTool = tool((_) => "Da weather is weatherin", {
  name: "get_current_weather",
  description: "Get the current weather in a given location",
  schema: z.object({
    location: z.string().describe("The city and state, e.g. San Francisco, CA"),
  }),
});

// Define the model
const model = new ChatOllama({
  model: "llama3-groq-tool-use",
});

// Bind the tool to the model
const modelWithTools = model.bindTools([weatherTool]);

const result = await modelWithTools.invoke(
  "What's the weather like today in San Francisco? Ensure you use the 'get_current_weather' tool."
);

console.log(result);
/*
AIMessage {
  "content": "",
  "tool_calls": [
    {
      "name": "get_current_weather",
      "args": {
        "location": "San Francisco, CA"
      },
      "type": "tool_call"
    }
  ],
  "usage_metadata": {
    "input_tokens": 177,
    "output_tokens": 30,
    "total_tokens": 207
  }
}
*/

API Reference:

tip

You can see the LangSmith trace of the above example here

Since ChatOllama supports the .bindTools() method, you can also call .withStructuredOutput() to get a structured output from the tool.

import { ChatOllama } from "@langchain/ollama";
import { z } from "zod";

// Define the model
const model = new ChatOllama({
  model: "llama3-groq-tool-use",
});

// Define the tool schema you'd like the model to use.
const schema = z.object({
  location: z.string().describe("The city and state, e.g. San Francisco, CA"),
});

// Pass the schema to the withStructuredOutput method to bind it to the model.
const modelWithTools = model.withStructuredOutput(schema, {
  name: "get_current_weather",
});

const result = await modelWithTools.invoke(
  "What's the weather like today in San Francisco? Ensure you use the 'get_current_weather' tool."
);
console.log(result);
/*
{ location: 'San Francisco, CA' }
*/

API Reference:

tip

You can see the LangSmith trace of the above example here

JSON mode​

Ollama also supports a JSON mode that coerces model outputs to only return JSON. Here's an example of how this can be useful for extraction:

import { ChatOllama } from "@langchain/ollama";
import { ChatPromptTemplate } from "@langchain/core/prompts";

const prompt = ChatPromptTemplate.fromMessages([
  [
    "system",
    `You are an expert translator. Format all responses as JSON objects with two keys: "original" and "translated".`,
  ],
  ["human", `Translate "{input}" into {language}.`],
]);

const model = new ChatOllama({
  baseUrl: "http://localhost:11434", // Default value
  model: "llama3",
  format: "json",
});

const chain = prompt.pipe(model);

const result = await chain.invoke({
  input: "I love programming",
  language: "German",
});

console.log(result);

/*
AIMessage {
  "content": "{\n\"original\": \"I love programming\",\n\"translated\": \"Ich liebe Programmieren\"\n}",
  "response_metadata": { ... },
  "usage_metadata": {
    "input_tokens": 47,
    "output_tokens": 20,
    "total_tokens": 67
  }
}
*/

API Reference:

tip

You can see a simple LangSmith trace of this here

Multimodal models​

Ollama supports open source multimodal models like LLaVA in versions 0.1.15 and up. You can pass images as part of a message's content field to multimodal-capable models like this:

import { ChatOllama } from "@langchain/ollama";
import { HumanMessage } from "@langchain/core/messages";
import * as fs from "node:fs/promises";

const imageData = await fs.readFile("./hotdog.jpg");
const chat = new ChatOllama({
  model: "llava",
  baseUrl: "http://127.0.0.1:11434",
});
const res = await chat.invoke([
  new HumanMessage({
    content: [
      {
        type: "text",
        text: "What is in this image?",
      },
      {
        type: "image_url",
        image_url: `data:image/jpeg;base64,${imageData.toString("base64")}`,
      },
    ],
  }),
]);
console.log(res);

/*
  AIMessage {
    content: ' The image shows a hot dog with ketchup on it, placed on top of a bun. It appears to be a close-up view, possibly taken in a kitchen setting or at an outdoor event.',
    name: undefined,
    additional_kwargs: {}
  }
*/

API Reference:

This will currently not use the image's position within the prompt message as additional information, and will just pass the image along as context with the rest of the prompt messages.

Was this page helpful?


You can also leave detailed feedback on GitHub.