Ollama

The Ollama adapter provides access to local models running via Ollama, allowing you to run AI models on your own infrastructure with full privacy and no API costs.

Installation

shell

npm i @tanstack/ai-ollama

shell

pnpm add @tanstack/ai-ollama

shell

yarn add @tanstack/ai-ollama

shell

bun add @tanstack/ai-ollama

Basic Usage

import { chat } from "@tanstack/ai";
import { ollamaText } from "@tanstack/ai-ollama";

const stream = chat({
  adapter: ollamaText("llama3"),
  messages: [{ role: "user", content: "Hello!" }],
});

Basic Usage - Custom Host

import { chat } from "@tanstack/ai";
import { createOllamaChat } from "@tanstack/ai-ollama";

const adapter = createOllamaChat("llama3", "http://your-server:11434");

const stream = chat({
  adapter,
  messages: [{ role: "user", content: "Hello!" }],
});

Configuration

import { createOllamaChat } from "@tanstack/ai-ollama";

// Custom host (URL string)
const adapter = createOllamaChat("llama3", "http://your-server:11434");

// Custom client config (e.g., custom headers, fetch)
const adapter2 = createOllamaChat("llama3", {
  host: "http://your-server:11434",
  headers: { Authorization: "Bearer ..." },
});

Behind a proxy

Route every request through a gateway, such as Cloudflare AI Gateway or a corporate proxy, with baseURL and defaultHeaders. These two option names are the same on every TanStack AI adapter, so one gateway config works for all of them.

import { createOllamaChat } from "@tanstack/ai-ollama";

const adapter = createOllamaChat("llama3", {
  baseURL: "https://gateway.example.com/ollama",
  defaultHeaders: { Authorization: `Bearer ${process.env.GATEWAY_TOKEN}` },
});

host and headers are aliases of the same two options. If you set both forms, baseURL and defaultHeaders win.

Need more than a fixed header, such as a log of each model call? This adapter supports wrapFetch. An adapter with your own Ollama client ignores it and the abort signal.

Available Models

To see available models on your Ollama instance:

shell

ollama list

Popular Models

llama3 / llama3.1 / llama3.2 - Meta's Llama models
mistral / mistral:7b - Mistral AI models
mixtral - Mixtral MoE model
codellama - Code-focused Llama
phi3 - Microsoft's Phi models
gemma / gemma2 - Google's Gemma models
qwen2 / qwen2.5 - Alibaba's Qwen models
deepseek-coder - DeepSeek coding model

Example: Chat Completion

import { chat, toServerSentEventsResponse } from "@tanstack/ai";
import { ollamaText } from "@tanstack/ai-ollama";

export async function POST(request: Request) {
  const { messages } = await request.json();

  const stream = chat({
    adapter: ollamaText("llama3"),
    messages,
  });

  return toServerSentEventsResponse(stream);
}

Example: With Tools

import { chat, toServerSentEventsResponse, toolDefinition } from "@tanstack/ai";
import { ollamaText } from "@tanstack/ai-ollama";
import { z } from "zod";

const getLocalDataDef = toolDefinition({
  name: "get_local_data",
  description: "Get data from local storage",
  inputSchema: z.object({
    key: z.string(),
  }),
});

const getLocalData = getLocalDataDef.server(async ({ key }) => {
  // Access local data
  return { data: "..." };
});

export async function POST(request: Request) {
  const { messages } = await request.json();

  const stream = chat({
    adapter: ollamaText("llama3"),
    messages,
    tools: [getLocalData],
  });

  return toServerSentEventsResponse(stream);
}

Note: Tool support varies by model. Models like llama3, mistral, and qwen2 generally have good tool calling support.

Tool choice

Ollama ignores toolChoice. Ollama has no tool choice, so the model decides to call a tool or not, also with 'none' or 'required'.

Want no tool calls for a request? Do not pass tools. For the other providers, see Choose when the model calls a tool.

Model Options

Ollama supports various provider-specific options. Unlike the other providers, Ollama nests its sampling and runner parameters inside an options object within modelOptions — temperature, top_p, and num_predict (the token-limit key) all live under modelOptions.options:

import { chat } from "@tanstack/ai";
import { ollamaText } from "@tanstack/ai-ollama";

const stream = chat({
  adapter: ollamaText("llama3:latest"),
  messages: [{ role: "user", content: "Hello!" }],
  modelOptions: {
    options: {
      temperature: 0.7,
      top_p: 0.9,
      top_k: 40,
      num_predict: 1000, // Max tokens to generate
      repeat_penalty: 1.1,
      num_ctx: 4096, // Context window size
      num_gpu: -1, // GPU layers (-1 = auto)
    },
  },
});

If you previously passed temperature / topP / maxTokens at the root of chat(), note that for Ollama they map to modelOptions.options.temperature, modelOptions.options.top_p, and modelOptions.options.num_predict. See Moving Sampling Options into modelOptions.

Advanced Options

All sampling and runner parameters are nested under modelOptions.options:

modelOptions: {
  options: {
    // Sampling
    temperature: 0.7,
    top_p: 0.9,
    top_k: 40,
    min_p: 0.05,
    typical_p: 1.0,

    // Generation
    num_predict: 1000,
    repeat_penalty: 1.1,
    repeat_last_n: 64,
    penalize_newline: false,

    // Performance
    num_ctx: 4096,
    num_batch: 512,
    num_gpu: -1,
    num_thread: 0, // 0 = auto

    // Memory
    use_mmap: true,
    use_mlock: false,

    // Mirostat sampling
    mirostat: 0, // 0 = disabled, 1 = Mirostat, 2 = Mirostat 2.0
    mirostat_tau: 5.0,
    mirostat_eta: 0.1,
  },
}

Reasoning

Running a thinking model like Qwen 3 or gpt-oss? Set reasoning on chat():

import { chat } from "@tanstack/ai";
import { ollamaText } from "@tanstack/ai-ollama";

const stream = chat({
  adapter: ollamaText("qwen3:8b"),
  messages: [{ role: "user", content: "Plan a database migration." }],
  reasoning: "off",
});

The adapter sends it as think:

Model	Levels	Sent as
qwen3, deepseek-r1, deepseek-v3.1	off, high	think: false or true
gpt-oss	low, medium, high	think: "low" and so on
a model name this package does not list	off, high	think: false or true

The other listed models take no reasoning.

Summarization

Summarize long text content locally:

import { summarize } from "@tanstack/ai";
import { ollamaSummarize } from "@tanstack/ai-ollama";

const result = await summarize({
  adapter: ollamaSummarize("llama3"),
  text: "Your long text to summarize...",
  maxLength: 100,
  style: "concise", // "concise" | "bullet-points" | "paragraph"
});

console.log(result.summary);

Embeddings

Generate embedding vectors locally with any Ollama embedding model:

import { embed } from "@tanstack/ai";
import { ollamaEmbedding } from "@tanstack/ai-ollama";

const result = await embed({
  adapter: ollamaEmbedding("nomic-embed-text"),
  input: ["a red guitar", "a blue drum kit"],
});

console.log(result.embeddings[0]?.vector);

Known models (nomic-embed-text, mxbai-embed-large, all-minilm, snowflake-arctic-embed, bge-m3, embeddinggemma) get autocomplete, and any other model name is accepted. Pull the model first with ollama pull nomic-embed-text. Output dimensions are fixed per model — the top-level dimensions option is not supported.

See the Embeddings guide for the full API.

Setting Up Ollama

1. Install Ollama

shell

# macOS
brew install ollama

# Linux
curl -fsSL https://ollama.com/install.sh | sh

# Windows
# Download from https://ollama.com

2. Pull a Model

shell

ollama pull llama3

3. Start Ollama Server

shell

ollama serve

The server runs on http://localhost:11434 by default.

Running on a Remote Server

import { createOllamaChat } from "@tanstack/ai-ollama";

const adapter = createOllamaChat("llama3", "http://your-server:11434");

To expose Ollama on a network interface:

shell

OLLAMA_HOST=0.0.0.0:11434 ollama serve

Environment Variables

Optionally set the host in environment variables:

shell

OLLAMA_HOST=http://localhost:11434

API Reference

ollamaText(model)

Creates an Ollama text/chat adapter using OLLAMA_HOST from the environment (defaults to http://localhost:11434).

Parameters:

model - Model name (e.g. "llama3", "mistral:7b")

createOllamaChat(model, hostOrConfig?)

Creates an Ollama text/chat adapter with an explicit host or client config.

Parameters:

model - Model name
hostOrConfig? - Either an OLLAMA_HOST-style URL string, or an OllamaClientConfig object (e.g. { baseURL, defaultHeaders }). host and headers are aliases.

ollamaSummarize(model) / createOllamaSummarize(model, hostOrConfig?)

Creates an Ollama summarization adapter — same signature shape as the chat adapter.

Benefits of Ollama

✅ Privacy - Data stays on your infrastructure
✅ Cost - No API costs after hardware
✅ Customization - Use any compatible model
✅ Offline - Works without internet
✅ Speed - No network latency for local deployment

Limitations

Image Generation: Ollama does not support image generation. Use OpenAI or Gemini for image generation.
Performance: Depends on your hardware (GPU recommended for larger models)