Skip to main content

ChatOpenAI

OpenAI is an artificial intelligence (AI) research laboratory.

This guide will help you getting started with ChatOpenAI chat models. For detailed documentation of all ChatOpenAI features and configurations head to the API reference.

Overview​

Integration details​

ClassPackageLocalSerializablePY supportPackage downloadsPackage latest
ChatOpenAI@langchain/openaiβŒβœ…βœ…NPM - DownloadsNPM - Version

Model features​

See the links in the table headers below for guides on how to use specific features.

Tool callingStructured outputJSON modeImage inputAudio inputVideo inputToken-level streamingToken usageLogprobs
βœ…βœ…βœ…βœ…βŒβŒβœ…βœ…βœ…

Setup​

To access OpenAI chat models you’ll need to create an OpenAI account, get an API key, and install the @langchain/openai integration package.

Credentials​

Head to OpenAI’s website to sign up for OpenAI and generate an API key. Once you’ve done this set the OPENAI_API_KEY environment variable:

export OPENAI_API_KEY="your-api-key"

If you want to get automated tracing of your model calls you can also set your LangSmith API key by uncommenting below:

# export LANGCHAIN_TRACING_V2="true"
# export LANGCHAIN_API_KEY="your-api-key"

Installation​

The LangChain ChatOpenAI integration lives in the @langchain/openai package:

yarn add @langchain/openai @langchain/core

Instantiation​

Now we can instantiate our model object and generate chat completions:

import { ChatOpenAI } from "@langchain/openai";

const llm = new ChatOpenAI({
model: "gpt-4o",
temperature: 0,
// other params...
});

Invocation​

const aiMsg = await llm.invoke([
{
role: "system",
content:
"You are a helpful assistant that translates English to French. Translate the user sentence.",
},
{
role: "user",
content: "I love programming.",
},
]);
aiMsg;
AIMessage {
"id": "chatcmpl-ADItECqSPuuEuBHHPjeCkh9wIO1H5",
"content": "J'adore la programmation.",
"additional_kwargs": {},
"response_metadata": {
"tokenUsage": {
"completionTokens": 5,
"promptTokens": 31,
"totalTokens": 36
},
"finish_reason": "stop",
"system_fingerprint": "fp_5796ac6771"
},
"tool_calls": [],
"invalid_tool_calls": [],
"usage_metadata": {
"input_tokens": 31,
"output_tokens": 5,
"total_tokens": 36
}
}
console.log(aiMsg.content);
J'adore la programmation.

Chaining​

We can chain our model with a prompt template like so:

import { ChatPromptTemplate } from "@langchain/core/prompts";

const prompt = ChatPromptTemplate.fromMessages([
[
"system",
"You are a helpful assistant that translates {input_language} to {output_language}.",
],
["human", "{input}"],
]);

const chain = prompt.pipe(llm);
await chain.invoke({
input_language: "English",
output_language: "German",
input: "I love programming.",
});
AIMessage {
"id": "chatcmpl-ADItFaWFNqkSjSmlxeGk6HxcBHzVN",
"content": "Ich liebe Programmieren.",
"additional_kwargs": {},
"response_metadata": {
"tokenUsage": {
"completionTokens": 5,
"promptTokens": 26,
"totalTokens": 31
},
"finish_reason": "stop",
"system_fingerprint": "fp_5796ac6771"
},
"tool_calls": [],
"invalid_tool_calls": [],
"usage_metadata": {
"input_tokens": 26,
"output_tokens": 5,
"total_tokens": 31
}
}

Custom URLs​

You can customize the base URL the SDK sends requests to by passing a configuration parameter like this:

import { ChatOpenAI } from "@langchain/openai";

const llmWithCustomURL = new ChatOpenAI({
temperature: 0.9,
configuration: {
baseURL: "https://your_custom_url.com",
},
});

await llmWithCustomURL.invoke("Hi there!");

The configuration field also accepts other ClientOptions parameters accepted by the official SDK.

If you are hosting on Azure OpenAI, see the dedicated page instead.

Custom headers​

You can specify custom headers in the same configuration field:

import { ChatOpenAI } from "@langchain/openai";

const llmWithCustomHeaders = new ChatOpenAI({
temperature: 0.9,
configuration: {
defaultHeaders: {
Authorization: `Bearer SOME_CUSTOM_VALUE`,
},
},
});

await llmWithCustomHeaders.invoke("Hi there!");

Disabling streaming usage metadata​

Some proxies or third-party providers present largely the same API interface as OpenAI, but don’t support the more recently added stream_options parameter to return streaming usage. You can use ChatOpenAI to access these providers by disabling streaming usage like this:

import { ChatOpenAI } from "@langchain/openai";

const llmWithoutStreamUsage = new ChatOpenAI({
temperature: 0.9,
streamUsage: false,
configuration: {
baseURL: "https://proxy.com",
},
});

await llmWithoutStreamUsage.invoke("Hi there!");

Calling fine-tuned models​

You can call fine-tuned OpenAI models by passing in your corresponding modelName parameter.

This generally takes the form of ft:{OPENAI_MODEL_NAME}:{ORG_NAME}::{MODEL_ID}. For example:

import { ChatOpenAI } from "@langchain/openai";

const fineTunedLlm = new ChatOpenAI({
temperature: 0.9,
model: "ft:gpt-3.5-turbo-0613:{ORG_NAME}::{MODEL_ID}",
});

await fineTunedLlm.invoke("Hi there!");

Generation metadata​

If you need additional information like logprobs or token usage, these will be returned directly in the .invoke response within the response_metadata field on the message.

tip

Requires @langchain/core version >=0.1.48.

import { ChatOpenAI } from "@langchain/openai";

// See https://cookbook.openai.com/examples/using_logprobs for details
const llmWithLogprobs = new ChatOpenAI({
logprobs: true,
// topLogprobs: 5,
});

const responseMessageWithLogprobs = await llmWithLogprobs.invoke("Hi there!");
console.dir(responseMessageWithLogprobs.response_metadata.logprobs, {
depth: null,
});
{
content: [
{
token: 'Hello',
logprob: -0.0004740447,
bytes: [ 72, 101, 108, 108, 111 ],
top_logprobs: []
},
{
token: '!',
logprob: -0.00004334534,
bytes: [ 33 ],
top_logprobs: []
},
{
token: ' How',
logprob: -0.000030113732,
bytes: [ 32, 72, 111, 119 ],
top_logprobs: []
},
{
token: ' can',
logprob: -0.0004797665,
bytes: [ 32, 99, 97, 110 ],
top_logprobs: []
},
{
token: ' I',
logprob: -7.89631e-7,
bytes: [ 32, 73 ],
top_logprobs: []
},
{
token: ' assist',
logprob: -0.114006,
bytes: [
32, 97, 115,
115, 105, 115,
116
],
top_logprobs: []
},
{
token: ' you',
logprob: -4.3202e-7,
bytes: [ 32, 121, 111, 117 ],
top_logprobs: []
},
{
token: ' today',
logprob: -0.00004501419,
bytes: [ 32, 116, 111, 100, 97, 121 ],
top_logprobs: []
},
{
token: '?',
logprob: -0.000010206721,
bytes: [ 63 ],
top_logprobs: []
}
],
refusal: null
}

Tool calling​

Tool calling with OpenAI models works in a similar to other models. Additionally, the following guides have some information especially relevant to OpenAI:

strict: true​

As of Aug 6, 2024, OpenAI supports a strict argument when calling tools that will enforce that the tool argument schema is respected by the model. See more here: https://platform.openai.com/docs/guides/function-calling.

Requires @langchain/openai >= 0.2.6

Note: If strict: true the tool definition will also be validated, and a subset of JSON schema are accepted. Crucially, schema cannot have optional args (those with default values). Read the full docs on what types of schema are supported here: https://platform.openai.com/docs/guides/structured-outputs/supported-schemas.

Here’s an example with tool calling. Passing an extra strict: true argument to .bindTools will pass the param through to all tool definitions:

import { ChatOpenAI } from "@langchain/openai";
import { tool } from "@langchain/core/tools";
import { z } from "zod";

const weatherTool = tool((_) => "no-op", {
name: "get_current_weather",
description: "Get the current weather",
schema: z.object({
location: z.string(),
}),
});

const llmWithStrictTrue = new ChatOpenAI({
model: "gpt-4o",
}).bindTools([weatherTool], {
strict: true,
tool_choice: weatherTool.name,
});

// Although the question is not about the weather, it will call the tool with the correct arguments
// because we passed `tool_choice` and `strict: true`.
const strictTrueResult = await llmWithStrictTrue.invoke(
"What is 127862 times 12898 divided by 2?"
);

console.dir(strictTrueResult.tool_calls, { depth: null });
[
{
name: 'get_current_weather',
args: { location: 'current' },
type: 'tool_call',
id: 'call_hVFyYNRwc6CoTgr9AQFQVjm9'
}
]

If you only want to apply this parameter to a select number of tools, you can also pass OpenAI formatted tool schemas directly:

import { zodToJsonSchema } from "zod-to-json-schema";

const toolSchema = {
type: "function",
function: {
name: "get_current_weather",
description: "Get the current weather",
strict: true,
parameters: zodToJsonSchema(
z.object({
location: z.string(),
})
),
},
};

const llmWithStrictTrueTools = new ChatOpenAI({
model: "gpt-4o",
}).bindTools([toolSchema], {
strict: true,
});

const weatherToolResult = await llmWithStrictTrueTools.invoke([
{
role: "user",
content: "What is the current weather in London?",
},
]);

weatherToolResult.tool_calls;
[
{
name: 'get_current_weather',
args: { location: 'London' },
type: 'tool_call',
id: 'call_EOSejtax8aYtqpchY8n8O82l'
}
]

Structured output​

We can also pass strict: true to the .withStructuredOutput(). Here’s an example:

import { ChatOpenAI } from "@langchain/openai";

const traitSchema = z.object({
traits: z
.array(z.string())
.describe("A list of traits contained in the input"),
});

const structuredLlm = new ChatOpenAI({
model: "gpt-4o-mini",
}).withStructuredOutput(traitSchema, {
name: "extract_traits",
strict: true,
});

await structuredLlm.invoke([
{
role: "user",
content: `I am 6'5" tall and love fruit.`,
},
]);
{ traits: [ `6'5" tall`, 'love fruit' ] }

Prompt caching​

Newer OpenAI models will automatically cache parts of your prompt if your inputs are above a certain size (1024 tokens at the time of writing) in order to reduce costs for use-cases that require long context.

Note: The number of tokens cached for a given query is not yet standardized in AIMessage.usage_metadata, and is instead contained in the AIMessage.response_metadata field.

Here’s an example

import { ChatOpenAI } from "@langchain/openai";

const modelWithCaching = new ChatOpenAI({
model: "gpt-4o-mini-2024-07-18",
});

// CACHED_TEXT is some string longer than 1024 tokens
const LONG_TEXT = `You are a pirate. Always respond in pirate dialect.

Use the following as context when answering questions:

${CACHED_TEXT}`;

const longMessages = [
{
role: "system",
content: LONG_TEXT,
},
{
role: "user",
content: "What types of messages are supported in LangChain?",
},
];

const originalRes = await modelWithCaching.invoke(longMessages);

console.log("USAGE:", originalRes.response_metadata.usage);
USAGE: {
prompt_tokens: 2624,
completion_tokens: 263,
total_tokens: 2887,
prompt_tokens_details: { cached_tokens: 0 },
completion_tokens_details: { reasoning_tokens: 0 }
}
const resWitCaching = await modelWithCaching.invoke(longMessages);

console.log("USAGE:", resWitCaching.response_metadata.usage);
USAGE: {
prompt_tokens: 2624,
completion_tokens: 272,
total_tokens: 2896,
prompt_tokens_details: { cached_tokens: 2432 },
completion_tokens_details: { reasoning_tokens: 0 }
}

API reference​

For detailed documentation of all ChatOpenAI features and configurations head to the API reference: https://api.js.langchain.com/classes/langchain_openai.ChatOpenAI.html


Was this page helpful?


You can also leave detailed feedback on GitHub.