Skip to main content

ChatGroq

Groq is a company that offers fast AI inference, powered by LPU™ AI inference technology which delivers fast, affordable, and energy efficient AI.

This will help you getting started with ChatGroq chat models. For detailed documentation of all ChatGroq features and configurations head to the API reference.

Overview​

Integration details​

ClassPackageLocalSerializablePY supportPackage downloadsPackage latest
ChatGroq@langchain/groq❌❌✅NPM - DownloadsNPM - Version

Model features​

See the links in the table headers below for guides on how to use specific features.

Tool callingStructured outputJSON modeImage inputAudio inputVideo inputToken-level streamingToken usageLogprobs
✅✅✅❌❌❌✅✅✅

Setup​

To access ChatGroq models you’ll need to create a Groq account, get an API key, and install the @langchain/groq integration package.

Credentials​

In order to use the Groq API you’ll need an API key. You can sign up for a Groq account and create an API key here. Then, you can set the API key as an environment variable in your terminal:

export GROQ_API_KEY="your-api-key"

If you want to get automated tracing of your model calls you can also set your LangSmith API key by uncommenting below:

# export LANGCHAIN_TRACING_V2="true"
# export LANGCHAIN_API_KEY="your-api-key"

Installation​

The LangChain ChatGroq integration lives in the @langchain/groq package:

tip

See this section for general instructions on installing integration packages.

yarn add @langchain/groq @langchain/core

Instantiation​

Now we can instantiate our model object and generate chat completions:

import { ChatGroq } from "@langchain/groq";

const llm = new ChatGroq({
  model: "mixtral-8x7b-32768",
  temperature: 0,
  maxTokens: undefined,
  maxRetries: 2,
  // other params...
});

Invocation​

const aiMsg = await llm.invoke([
  {
    role: "system",
    content:
      "You are a helpful assistant that translates English to French. Translate the user sentence.",
  },
  { role: "user", content: "I love programming." },
]);
aiMsg;
AIMessage {
  "content": "I enjoy programming. (The French translation is: \"J'aime programmer.\")\n\nNote: I chose to translate \"I love programming\" as \"J'aime programmer\" instead of \"Je suis amoureux de programmer\" because the latter has a romantic connotation that is not present in the original English sentence.",
  "additional_kwargs": {},
  "response_metadata": {
    "tokenUsage": {
      "completionTokens": 73,
      "promptTokens": 31,
      "totalTokens": 104
    },
    "finish_reason": "stop"
  },
  "tool_calls": [],
  "invalid_tool_calls": []
}
console.log(aiMsg.content);
I enjoy programming. (The French translation is: "J'aime programmer.")

Note: I chose to translate "I love programming" as "J'aime programmer" instead of "Je suis amoureux de programmer" because the latter has a romantic connotation that is not present in the original English sentence.

Json invocation​

const messages = [
  {
    role: "system",
    content:
      "You are a math tutor that handles math exercises and makes output in json in format { result: number }.",
  },
  { role: "user", content: "2 + 2 * 2" },
];

const aiInvokeMsg = await llm.invoke(messages, {
  response_format: { type: "json_object" },
});

// if you want not to pass response_format in every invoke, you can bind it to the instance
const llmWithResponseFormat = llm.bind({
  response_format: { type: "json_object" },
});
const aiBindMsg = await llmWithResponseFormat.invoke(messages);

// they are the same
console.log({
  aiInvokeMsgContent: aiInvokeMsg.content,
  aiBindMsg: aiBindMsg.content,
});
{
  aiInvokeMsgContent: '{\n"result": 6\n}',
  aiBindMsg: '{\n"result": 6\n}'
}

Chaining​

We can chain our model with a prompt template like so:

import { ChatPromptTemplate } from "@langchain/core/prompts";

const prompt = ChatPromptTemplate.fromMessages([
  [
    "system",
    "You are a helpful assistant that translates {input_language} to {output_language}.",
  ],
  ["human", "{input}"],
]);

const chain = prompt.pipe(llm);
await chain.invoke({
  input_language: "English",
  output_language: "German",
  input: "I love programming.",
});
AIMessage {
  "content": "That's great! I can help you translate English phrases related to programming into German.\n\n\"I love programming\" can be translated to German as \"Ich liebe Programmieren\".\n\nHere are some more programming-related phrases translated into German:\n\n* \"Programming language\" = \"Programmiersprache\"\n* \"Code\" = \"Code\"\n* \"Variable\" = \"Variable\"\n* \"Function\" = \"Funktion\"\n* \"Array\" = \"Array\"\n* \"Object-oriented programming\" = \"Objektorientierte Programmierung\"\n* \"Algorithm\" = \"Algorithmus\"\n* \"Data structure\" = \"Datenstruktur\"\n* \"Debugging\" = \"Debuggen\"\n* \"Compile\" = \"Kompilieren\"\n* \"Link\" = \"Verknüpfen\"\n* \"Run\" = \"Ausführen\"\n* \"Test\" = \"Testen\"\n* \"Deploy\" = \"Bereitstellen\"\n* \"Version control\" = \"Versionskontrolle\"\n* \"Open source\" = \"Open Source\"\n* \"Software development\" = \"Softwareentwicklung\"\n* \"Agile methodology\" = \"Agile Methodik\"\n* \"DevOps\" = \"DevOps\"\n* \"Cloud computing\" = \"Cloud Computing\"\n\nI hope this helps! Let me know if you have any other questions or if you need further translations.",
  "additional_kwargs": {},
  "response_metadata": {
    "tokenUsage": {
      "completionTokens": 327,
      "promptTokens": 25,
      "totalTokens": 352
    },
    "finish_reason": "stop"
  },
  "tool_calls": [],
  "invalid_tool_calls": []
}

API reference​

For detailed documentation of all ChatGroq features and configurations head to the API reference: https://api.js.langchain.com/classes/langchain_groq.ChatGroq.html

Was this page helpful?


You can also leave detailed feedback on GitHub.