Mid-Conversation Changes

Put a stable tool list at the start of each request so the provider can keep its prompt cache. Add a tool that appears mid-chat after that prefix.

On some GPT and Claude models, TanStack AI sends the added tool inside the conversation. The start stays the same, and the cache keeps working. You change no code.

Quick setup#

This setup takes about 5 minutes.

1. Pick a model with a channel#

The channel exists on these models:

Every other model and adapter sends the full lists on each call.

2. Add a tool during the run#

This route starts with one tool. A middleware adds a second tool after the first model call:

import {
  chat,
  chatParamsFromRequest,
  toServerSentEventsResponse,
  toolDefinition,
} from "@tanstack/ai";
import type { ChatMiddleware } from "@tanstack/ai";
import { openaiText } from "@tanstack/ai-openai";
import { z } from "zod";

const getWeather = toolDefinition({
  name: "get_weather",
  description: "Get the current weather for a city",
  inputSchema: z.object({ city: z.string() }),
}).server(async ({ city }) => ({ city, temperature: 21 }));

const getForecast = toolDefinition({
  name: "get_forecast",
  description: "Get the forecast for a city",
  inputSchema: z.object({ city: z.string() }),
}).server(async ({ city }) => ({ city, tomorrow: "sunny" }));

const addForecast: ChatMiddleware = {
  name: "add-forecast",
  onConfig: (ctx, config) => {
    if (ctx.phase !== "beforeModel" || ctx.iteration === 0) return;
    if (config.tools.some((tool) => tool.name === "get_forecast")) return;
    return { tools: [...config.tools, getForecast] };
  },
};

export async function POST(request: Request) {
  const { messages, threadId } = await chatParamsFromRequest(request);
  const stream = chat({
    adapter: openaiText("gpt-6-astra"),
    messages,
    threadId,
    tools: [getWeather],
    middleware: [addForecast],
  });
  return toServerSentEventsResponse(stream);
}

The client does not change. Ask about the weather, and the second model call reads the start of the request from the cache.

How it works#

The second model call of the example sends this:

Some more rules:

• The channel works only when the first call already had a tool.
• A system prompt that you add at the end of the list works the same way. It goes into the conversation as a developer message (GPT) or a system message (Claude).

Advanced#

Check your model#

To check in code, read midConversationChannels on the adapter:

import { openaiText } from "@tanstack/ai-openai";

const adapter = openaiText("gpt-6-astra");
console.log(adapter.midConversationChannels); // { tools: true, systemPrompts: true }

It is undefined when the model has no channel, or when the channels are off.

Turn it on or off#

The channels are on by default only when the adapter talks to the API of the provider directly. A gateway or a proxy can drop the extra fields. So the channels are off with:

• a custom baseURL or a custom fetch.
• a proxy URL in OPENAI_BASE_URL or ANTHROPIC_BASE_URL.
• an Anthropic adapter on your own client (createAnthropicChatWithClient, anthropicVertexText).

Set midConversationChannels in the adapter config to choose:

import { anthropicText } from "@tanstack/ai-anthropic";
import { openaiText } from "@tanstack/ai-openai";

// Send the full lists on every call.
export const fullLists = openaiText("gpt-6-astra", {
  midConversationChannels: false,
});

// Use the channels behind a proxy that sends the request on with no change.
export const throughProxy = anthropicText("claude-opus-5-5", {
  baseURL: "https://llm-proxy.example.com",
  midConversationChannels: true,
});

On a model with no channel, true changes nothing.

Keep the record when you store messages#

Before each model call, chat() compares the tools and the system prompts with the earlier calls. It saves a small record on the first assistant message of the call, in the midConversationChange field. The record holds tool names and short prompt hashes, not tool definitions.

The next turn reads these records:

• If you save messages yourself, keep the midConversationChange field.
• With withPersistence, you do nothing. The store keeps the record, also when useChat sends the same message back without it.
• With plain useChat and no store, the records last for one chat() call. Each new turn starts with no record, and the cache breaks one time for each turn.

Limits#

These changes still break the cache one time:

• A tool that you remove, or a tool that keeps its name but gets a new definition.
• A system prompt that you edit, move, or add anywhere but the end.
• A provider tool, such as webSearchTool(), in the request. That request sends the full tools list. Prompt changes still go into the conversation.
• The first tool, when the recorded start had no tools. That request sends the full tool list.