OpenAI compatibility

Turn existing OpenAI code into a RunInfra call by changing the base URL, the key, and the model id. Examples for LangChain, LiteLLM, LlamaIndex, the AI SDK, and plain fetch.

If your code already speaks the OpenAI API, three values change and nothing else does.

Python

import os

from openai import OpenAI

client = OpenAI(
    base_url="https://api.runinfra.ai/v1",        # <- change
    api_key=os.environ["RUNINFRA_GATEWAY_KEY"],   # <- change
)

response = client.chat.completions.create(
    model="nemotron-3-5-lightning-30b",               # <- change
    messages=[{"role": "user", "content": "Hello"}],
    max_tokens=16384,
)

OpenAI model names do not alias to RunInfra models. Pass a model id from GET /v1/models. Streaming, tools, and structured output use the same request and response shapes where the selected model’s page lists that capability. Streams terminate with data: [DONE], tool calls appear on the assistant message, and usage appears on the response.

Set an output budget

This is the one setting that decides whether a reasoning model has room to answer. Reasoning tokens are billed output that count toward max_tokens.

For nemotron-3-5-lightning-30b, start with at least 4,096 output tokens.

Where a recommended minimum is published, the model’s page in the Model Library carries it. You pay for tokens generated, not for the ceiling you set, so a generous budget costs nothing extra. If the answer comes back empty, check finish_reason: "length" means raise the budget, not that the model produced nothing.

Where the model’s thinking arrives

RunInfra emits the model’s thinking as reasoning on the message and on each stream delta. Some OpenAI-compatible clients and providers use the name reasoning_content instead, so a client that reads only that name sees the answer but not the thinking. Measured per client:

Client	Visible answer	Model’s thinking
AI SDK (@ai-sdk/openai-compatible)	text / text-delta	reasoningText, and reasoning-delta parts
OpenAI SDK, Node and Python	message.content / delta.content	message.reasoning / delta.reasoning
LiteLLM	message.content	message.reasoning_content
LangChain (langchain-openai)	content	not exposed, dropped by the integration
Plain fetch	delta.content	delta.reasoning

The answer is correct in every client above. Only access to the thinking differs, so pick accordingly if you want to render it.

Prompt caching and session affinity

Prefix caching is automatic server behavior on models with a published cached-input price. You do not select cache placement, retention, or reuse per request: prompt_cache_options and prompt_cache_retention are rejected with 400 hosted_parameter_not_supported.

To maximize hit rate for multi-turn conversations and agent sessions, give repeat requests a consistent session hint so the session’s cached prefix is reused:

• Header: x-session-id: <your-conversation-or-task-id> (x-parent-session-id and the legacy x-session-affinity are also accepted)
• Body field: prompt_cache_key (the OpenAI SDK spelling), or a stable user

All of these work on /v1/chat/completions and /v1/responses; the hint is used to keep your session’s requests and cached prefix together, and it never reaches the model. Without one, repeat requests from the same workspace are still kept together, so steady sessions already get a warm cache; an explicit hint makes that grouping exact for your own conversation or task ids. Cached tokens appear in usage.prompt_tokens_details.cached_tokens on /v1/chat/completions and in usage.input_tokens_details.cached_tokens on /v1/responses, and bill at the cached-input rate published on that model’s page. The model page and public pricing page show any measured cache hit-rate figures that are currently published.

Framework configuration

These framework integrations returned complete answers from https://api.runinfra.ai/v1 in checks on 2026-08-14. The examples now select models listed as available on September 21, 2026. Versions used: ai 7.0.65, @ai-sdk/openai-compatible 3.0.30, openai 7.4.0 (Node) and 2.54.0 (Python), langchain-openai, litellm.

LangChain

import os
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    model="nemotron-3-5-lightning-30b",
    base_url="https://api.runinfra.ai/v1",
    api_key=os.environ["RUNINFRA_GATEWAY_KEY"],
    max_tokens=4096,
)

print(llm.invoke("Write a TypeScript retry helper.").content)

for chunk in llm.stream("Write a TypeScript retry helper."):
    print(chunk.content, end="", flush=True)

LiteLLM

LiteLLM reaches RunInfra through its openai/ provider prefix, and exposes the thinking as reasoning_content.

import os
import litellm

response = litellm.completion(
    model="openai/nemotron-3-5-lightning-30b",
    api_base="https://api.runinfra.ai/v1",
    api_key=os.environ["RUNINFRA_GATEWAY_KEY"],
    messages=[{"role": "user", "content": "Write a TypeScript retry helper."}],
    max_tokens=16384,
)
print(response.choices[0].message.content)

Add stream=True for deltas, reading chunk.choices[0].delta.content and skipping the usage frame, which has no choices.

LlamaIndex

Use OpenAILike from llama-index-llms-openai-like, with is_chat_model=True. The OpenAI class raises Unknown model for any id it does not list.

import os
from llama_index.llms.openai_like import OpenAILike

llm = OpenAILike(
    model="qwen3-8-27b",
    api_base="https://api.runinfra.ai/v1",
    api_key=os.environ["RUNINFRA_GATEWAY_KEY"],
    is_chat_model=True,
)

print(llm.complete("What is RunInfra?").text)

AI SDK

@ai-sdk/openai-compatible reads reasoning_content and falls back to reasoning, which is the name we emit, so both the answer and the thinking come through.

// Server-side only: never expose RUNINFRA_GATEWAY_KEY in browser code.
import { createOpenAICompatible } from "@ai-sdk/openai-compatible";
import { streamText } from "ai";

const apiKey = process.env.RUNINFRA_GATEWAY_KEY;
if (!apiKey) throw new Error("Set RUNINFRA_GATEWAY_KEY first.");

const runinfra = createOpenAICompatible({
  name: "runinfra",
  baseURL: "https://api.runinfra.ai/v1",
  apiKey,
});

const result = streamText({
  model: runinfra.chatModel("nemotron-3-5-lightning-30b"),
  maxOutputTokens: 4096,
  prompt: "Write a TypeScript retry helper.",
});

for await (const part of result.fullStream) {
  if (part.type === "reasoning-delta") process.stdout.write(part.text);
  if (part.type === "text-delta") process.stdout.write(part.text);
}

generateText works the same way, with the thinking on reasoningText.

maxOutputTokens is the AI SDK’s name for the output budget above.

usage.outputTokenDetails.reasoningTokens reports 0 even when the model reasoned, because this endpoint does not yet break reasoning out of completion_tokens. Total outputTokens is correct and is what you are billed for.

Plain fetch, no SDK

const response = await fetch("https://api.runinfra.ai/v1/chat/completions", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.RUNINFRA_GATEWAY_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    model: "nemotron-3-5-lightning-30b",
    messages: [{ role: "user", content: "Write a TypeScript retry helper." }],
    max_tokens: 4096,
    stream: true,
    stream_options: { include_usage: true },
  }),
});

if (!response.ok) {
  const { error } = await response.json();
  throw new Error(`${response.status} ${error.code}: ${error.message}`);
}

const reader = response.body.getReader();
const decoder = new TextDecoder();
let buffer = "";

while (true) {
  const { done, value } = await reader.read();
  if (done) break;

  buffer += decoder.decode(value, { stream: true });
  const lines = buffer.split("\n");
  buffer = lines.pop() ?? "";

  for (const line of lines) {
    if (!line.startsWith("data: ")) continue;
    const payload = line.slice(6).trim();
    if (payload === "[DONE]") continue;

    const chunk = JSON.parse(payload);
    const delta = chunk.choices[0]?.delta;
    // The model's thinking arrives as `reasoning`, the answer as `content`.
    if (delta?.reasoning) process.stdout.write(delta.reasoning);
    if (delta?.content) process.stdout.write(delta.content);
  }
}

A data: line can be split across two network reads, so the trailing partial line stays in buffer rather than being parsed. Dropping that carry is the usual cause of intermittent JSON.parse failures on a stream that works in testing.

Where RunInfra differs from OpenAI

Which endpoints exist

Model APIs serve POST /v1/chat/completions, POST /v1/responses, POST /v1/messages, POST /v1/messages/count_tokens, GET /v1/models, GET /v1/models/{model}, and GET /v1/credits. Chat completions and Responses are OpenAI-compatible. Messages and token counting use the Anthropic-compatible contract.

Not supported anywhere: /v1/completions (legacy, non-chat), /v1/files, /v1/assistants, /v1/threads, and /v1/batches. /v1/responses is a chat-completions compatibility adapter and does not implement state, hosted tools (web search, file search), conversation storage, or background jobs.

On a model whose page lists tool calling, function tools work end to end on /v1/responses, in both spellings: declare tools flat ({"type": "function", "name", "parameters"}, the Responses shape) or nested under function (the chat shape). The model’s calls come back as function_call output items with call_id, name, and arguments; send results back as function_call_output input items with the matching call_id (string or content-part array), and resend prior output items in input since the endpoint is stateless. Streams carry the full item lifecycle (response.output_item.added, response.content_part.added, text and function_call_arguments deltas, the matching done events, and a spec-complete response.completed, which ends every stream), so Codex-style clients that build items from output_item.done work unchanged. Harness fields: store is accepted and always treated as false, nothing is retrievable later; metadata, include (reasoning.encrypted_content, message.output_text.logprobs, or empty), truncation: "disabled", parallel_tool_calls, service_tier, and client_metadata are accepted; truncation: "auto" and previous_response_id are refused with the reason.

On /v1/responses, output_text carries the final answer only. The model’s thinking arrives as its own reasoning output item ahead of the message (raw text under content as reasoning_text parts, summary empty), and on streams as response.reasoning_text.delta and response.reasoning_text.done events. A completion that spends its whole token budget thinking returns an empty output_text with the thinking preserved in the reasoning item, so raise max_output_tokens if you see that.

A reply that max_output_tokens or a content filter cut short is not whole, and the response says so: status is "incomplete" and incomplete_details.reason is max_output_tokens or content_filter. A non-streaming body carries it directly. A stream still ends on response.completed, never on response.incomplete, and the response inside that event carries the same status and reason. Check status before you treat an answer as complete. Item status is advisory: only the item being written when the reply stopped is incomplete, so parse a function call’s arguments before you run it.

What happens to a field you send

Fields on the forwarding allowlist reach the model unchanged. Fields that are not on it are dropped before the request leaves the gateway, silently rather than as an error, so a request shape that works today keeps working.

Chat completions lists the allowlist field by field, and names the few that carry gateway behavior. It is the one authoritative list, so this page does not restate it.

Errors and retries

These OpenAI-compatible routes use the OpenAI envelope, { error: { message, type, code } }, with six error.type values: invalid_request_error, authentication_error, permission_error, not_found_error, rate_limit_error, and api_error. There is no server_error type; 5xx responses carry api_error. The Messages routes use the Anthropic envelope. Every status and code is in Errors, and the triage is in Troubleshooting.

Retry behavior belongs to your client configuration. Do not blindly retry a charge-bearing request after a partial stream may have reached your app; use an idempotency key instead. Quote the X-Request-Id response header on any support ticket.

Hugging Face clients

huggingface_hub and @huggingface/inference, verified.

Tool calling

A complete multi-turn loop.

Limits

Context, output, concurrency, tokens per minute.