Skip to content
All articles
AI Engineering

Open-Source LLM vs OpenAI API: How to Choose Your Model Strategy

A constraints-first framework for the open-source vs hosted API decision, the multi-provider middle path most products actually need, and a production support bot that swaps between Gemini, OpenAI, and Anthropic behind one function.

Rushit Jivani5 min read

The short version

  • Choose from constraints, not ideology: data residency and massive steady volume argue for self-hosting; small teams, spiky traffic, and frontier quality argue for hosted APIs.
  • The strategy most products actually need is neither camp: hosted models behind one adapter function, two or three providers, and your embedding model pinned so the vector index never needs a rebuild.
  • The case study: a support bot that defaults to Gemini and can answer through OpenAI or Anthropic per request — same message list, same retrieval — while embeddings stay on one pinned model and high-stakes paths keep authored fallbacks.

"Should we use an open-source LLM or the OpenAI API?" is usually asked as a philosophy question — ownership and independence versus convenience and lock-in. Answered that way, it produces strategy documents and no shipping software. It is better asked as an engineering question: which constraints do we actually have, and what does each path cost to satisfy them? That version has concrete answers, and for most products it also has a third option the binary framing hides.

This guide works through the real trade-offs, the decision framework, and then the strategy I shipped on a production support bot — which is neither camp's answer, and has survived provider outages precisely because of that.

What each path actually involves

Hosted APIs (OpenAI, Anthropic, Google) sell you tokens through an endpoint. You get frontier-quality models, zero infrastructure, instant upgrades — and you accept per-token pricing, provider outages as your outages, data leaving your network under the provider's terms, and models that get deprecated on someone else's schedule.

Self-hosted open-weight models (Llama, Mistral, Qwen and descendants) give you weights to run on your own GPUs. You get data that never leaves, no per-token meter, total version control — and you accept the real bill: GPU provisioning and utilization management, inference-server operations, an on-call rotation that now includes "the model server," quality that trails the frontier for most tasks, and upgrades that are your migration projects. "Free weights" have a payroll line.

The hidden third axis is operational maturity: a hosted API failure is a status page and a retry; a self-hosted failure at 2 a.m. is your pager. Teams consistently underprice this.

How you should choose

Start from constraints, not preferences:

Open-source vs hosted: decide from constraints

    Data cannot leave your infrastructureSelf-hosted open-weight model — accept the ops burden
    Spiky, modest traffic; small teamHosted APIs — you rent the GPUs and the on-call rota
    Afraid of vendor lock-in or outagesHosted, but behind one adapter function with 2–3 providers
    “Owning weights” with no requirement behind itNot a reason. Name the constraint first
The question is never which model is best in a vacuum. It is which constraint you actually have — residency, ops capacity, lock-in — and what it costs to satisfy.

Choose self-hosting when a constraint forces it: data that legally or contractually cannot leave your infrastructure; enormous steady volume where utilized GPUs genuinely beat per-token pricing (do that math honestly — idle GPUs destroy the comparison); or an edge/offline deployment target. These are real cases, and if you have one, you know.

Choose hosted when none does — which is most products: small team, spiky traffic, quality-sensitive outputs, speed of iteration mattering more than marginal token cost.

And then de-risk hosted properly, because the legitimate fears behind the open-source argument — lock-in, outages, deprecations — have a cheaper mitigation than running your own GPUs:

The multi-provider adapter: the strategy most products need

Put every model call behind one function you own: it takes the message list and an optional provider name, constructs the right client, normalizes the response to plain text, and alerts with the provider's name on failure. The rules that make it work:

  1. Normalize at the boundary. Callers get a string back, whichever provider ran. Provider quirks (content-block lists, stop reasons) die inside the adapter.
  2. Pin your embedding model separately — and treat it as permanent. Vectors from different models are not comparable; swapping the embedder means re-embedding the entire index. Chat models should be swappable per request; the embedding model changes only as a deliberate migration. This asymmetry is the most practically useful fact in the whole debate.
  3. Keep specialized calls (image, transcription) on their own named models rather than coupling them to the chat default — they have different alternatives and different failure modes.
  4. Let failure degrade by design. The adapter alerts and throws; callers decide what failure means on their path — a fallback provider, or better, authored fallback text on high-stakes paths (why that beats any provider strategy).

A note on tooling honesty: using LangChain or similar SDKs — which are open-source libraries — does not make yours "an open-source LLM stack." The library constructs clients; the models are still hosted. Say which models you call. (And a multi-provider adapter is not LangGraph either; routing logic is a separate decision.)

How I implemented it on a real project

The support bot on the events platform I work on (the full build) answers through three hosted providers and self-hosts nothing. That was not a bake-off verdict against open-source models — we never ran one. No constraint forced self-hosting: no residency requirement, no steady-volume GPU math, a small team. What we did have was the lock-in and outage fear, and the adapter answered it for an afternoon of work instead of an ops rotation.

One function, three clients

The chat function takes the question, recent history, page context, retrieved FAQ text, and an optional provider. Gemini is the default (a fast hosted model); OpenAI and Anthropic are the alternatives; anything unrecognized is treated as OpenAI. The message list is identical for all three — system rules, then up to the last 16 turns, then the new question. Clients are constructed lazily per provider via LangChain adapters, keys come from Secret Manager (never the browser), and content-block responses are joined to plain text so callers see one shape. On failure: an alert with the provider name, then the caller's own fallback logic. Swap the provider live:

Swap the provider, keep everything else — try it

always the same

  • 1. system message: rules + page context + FAQ hits
  • 2. the last 16 turns of history
  • 3. the new question

chosen per request

Gemini

a fast hosted Gemini model. Key loaded from the secret manager; a failure alerts and the caller's fallback takes over.

Pinned regardless: one OpenAI embedding model feeds the FAQ index, and image / transcript calls stay on their own models. Switching chat never re-embeds a thing.

The message list and the retrieval pipeline never change. Only the client constructed at the end does. Embeddings stay pinned so the index never needs a rebuild.

What is pinned, what floats

The division of labor is deliberate:

JobModelSwappable?
Support answersGemini / OpenAI / AnthropicPer request
FAQ vectorsOne pinned OpenAI embedding modelMigration only
Flyer images, transcriptsNamed OpenAI modelsIndependent of chat
Dispute classifierGemini, OpenAI backup, JSON-onlyFixed pair, fail-closed
Routing refund vs generalRegular expressionsNot a model

Pointing chat at Anthropic re-embeds nothing, because the FAQ index lives in the pinned embedding space. The dispute classifier runs its own two-provider chain with temperature 0 and a confidence floor (that pipeline) — a provider pair chosen for the task, not inherited from the chat default.

The property that actually mattered

During a provider outage, the bot's riskiest path — refund verdicts — kept answering correctly, because its states are decided in code and its fallback paragraphs are authored text. The multi-provider switch made the general path resilient; the architecture made the critical path resilient. That ordering is the real lesson: provider flexibility is the second line of defense. Owned truth — your knowledge store, your record reads, your written fallbacks — is the first, and it is portable across every strategy in this guide, including a future self-hosted one if a constraint ever arrives.

Pitfalls to avoid

  • Self-hosting to "own the stack" with no forcing constraint. You buy an ops burden and trail frontier quality, to mitigate risks an adapter mitigates for less.
  • Scattering openai.chat(...) calls through the codebase. Lock-in is not the vendor; it is call sites. One adapter function is the exit door.
  • Letting the embedding model drift with the chat choice. Mixing embedding spaces silently breaks retrieval scores. Pin it, document it, migrate it deliberately.
  • Automatic silent retry across all providers on every path. High-stakes paths should fail into authored text, not into a different model's guess. Make failure behavior per-path.
  • Claiming open-source because your SDK is. Credibility costs more than the sentence saved.

FAQ

High, steady, predictable volume — enough to keep purchased or reserved GPUs well-utilized around the clock. Spiky or modest traffic almost always prices better per token on hosted APIs once you count idle hardware and the people running it.

  • #OpenAI
  • #OpenSourceLLM
  • #LangChain
  • #Guide
  • #CaseStudy
Rushit Jivani headshot

Written by

Rushit Jivani

Senior Software Architect & AI Application Engineer. Architecture that shows up in crash rate, performance and how fast the team ships.