The short version
- Choose from constraints, not ideology: data residency and massive steady volume argue for self-hosting; small teams, spiky traffic, and frontier quality argue for hosted APIs.
- The strategy most products actually need is neither camp: hosted models behind one adapter function, two or three providers, and your embedding model pinned so the vector index never needs a rebuild.
- The case study: a support bot that defaults to Gemini and can answer through OpenAI or Anthropic per request — same message list, same retrieval — while embeddings stay on one pinned model and high-stakes paths keep authored fallbacks.
"Should we use an open-source LLM or the OpenAI API?" is usually asked as a philosophy question — ownership and independence versus convenience and lock-in. Answered that way, it produces strategy documents and no shipping software. It is better asked as an engineering question: which constraints do we actually have, and what does each path cost to satisfy them? That version has concrete answers, and for most products it also has a third option the binary framing hides.
This guide works through the real trade-offs, the decision framework, and then the strategy I shipped on a production support bot — which is neither camp's answer, and has survived provider outages precisely because of that.
What each path actually involves
Hosted APIs (OpenAI, Anthropic, Google) sell you tokens through an endpoint. You get frontier-quality models, zero infrastructure, instant upgrades — and you accept per-token pricing, provider outages as your outages, data leaving your network under the provider's terms, and models that get deprecated on someone else's schedule.
Self-hosted open-weight models (Llama, Mistral, Qwen and descendants) give you weights to run on your own GPUs. You get data that never leaves, no per-token meter, total version control — and you accept the real bill: GPU provisioning and utilization management, inference-server operations, an on-call rotation that now includes "the model server," quality that trails the frontier for most tasks, and upgrades that are your migration projects. "Free weights" have a payroll line.
The hidden third axis is operational maturity: a hosted API failure is a status page and a retry; a self-hosted failure at 2 a.m. is your pager. Teams consistently underprice this.
How you should choose
Start from constraints, not preferences:
Open-source vs hosted: decide from constraints
Choose self-hosting when a constraint forces it: data that legally or contractually cannot leave your infrastructure; enormous steady volume where utilized GPUs genuinely beat per-token pricing (do that math honestly — idle GPUs destroy the comparison); or an edge/offline deployment target. These are real cases, and if you have one, you know.
Choose hosted when none does — which is most products: small team, spiky traffic, quality-sensitive outputs, speed of iteration mattering more than marginal token cost.
And then de-risk hosted properly, because the legitimate fears behind the open-source argument — lock-in, outages, deprecations — have a cheaper mitigation than running your own GPUs:
The multi-provider adapter: the strategy most products need
Put every model call behind one function you own: it takes the message list and an optional provider name, constructs the right client, normalizes the response to plain text, and alerts with the provider's name on failure. The rules that make it work:
- Normalize at the boundary. Callers get a string back, whichever provider ran. Provider quirks (content-block lists, stop reasons) die inside the adapter.
- Pin your embedding model separately — and treat it as permanent. Vectors from different models are not comparable; swapping the embedder means re-embedding the entire index. Chat models should be swappable per request; the embedding model changes only as a deliberate migration. This asymmetry is the most practically useful fact in the whole debate.
- Keep specialized calls (image, transcription) on their own named models rather than coupling them to the chat default — they have different alternatives and different failure modes.
- Let failure degrade by design. The adapter alerts and throws; callers decide what failure means on their path — a fallback provider, or better, authored fallback text on high-stakes paths (why that beats any provider strategy).
A note on tooling honesty: using LangChain or similar SDKs — which are open-source libraries — does not make yours "an open-source LLM stack." The library constructs clients; the models are still hosted. Say which models you call. (And a multi-provider adapter is not LangGraph either; routing logic is a separate decision.)
How I implemented it on a real project
The support bot on the events platform I work on (the full build) answers through three hosted providers and self-hosts nothing. That was not a bake-off verdict against open-source models — we never ran one. No constraint forced self-hosting: no residency requirement, no steady-volume GPU math, a small team. What we did have was the lock-in and outage fear, and the adapter answered it for an afternoon of work instead of an ops rotation.
One function, three clients
The chat function takes the question, recent history, page context, retrieved FAQ text, and an optional provider. Gemini is the default (a fast hosted model); OpenAI and Anthropic are the alternatives; anything unrecognized is treated as OpenAI. The message list is identical for all three — system rules, then up to the last 16 turns, then the new question. Clients are constructed lazily per provider via LangChain adapters, keys come from Secret Manager (never the browser), and content-block responses are joined to plain text so callers see one shape. On failure: an alert with the provider name, then the caller's own fallback logic. Swap the provider live:
Swap the provider, keep everything else — try it
always the same
- 1. system message: rules + page context + FAQ hits
- 2. the last 16 turns of history
- 3. the new question
chosen per request
Gemini
a fast hosted Gemini model. Key loaded from the secret manager; a failure alerts and the caller's fallback takes over.
Pinned regardless: one OpenAI embedding model feeds the FAQ index, and image / transcript calls stay on their own models. Switching chat never re-embeds a thing.
What is pinned, what floats
The division of labor is deliberate:
| Job | Model | Swappable? |
|---|---|---|
| Support answers | Gemini / OpenAI / Anthropic | Per request |
| FAQ vectors | One pinned OpenAI embedding model | Migration only |
| Flyer images, transcripts | Named OpenAI models | Independent of chat |
| Dispute classifier | Gemini, OpenAI backup, JSON-only | Fixed pair, fail-closed |
| Routing refund vs general | Regular expressions | Not a model |
Pointing chat at Anthropic re-embeds nothing, because the FAQ index lives in the pinned embedding space. The dispute classifier runs its own two-provider chain with temperature 0 and a confidence floor (that pipeline) — a provider pair chosen for the task, not inherited from the chat default.
The property that actually mattered
During a provider outage, the bot's riskiest path — refund verdicts — kept answering correctly, because its states are decided in code and its fallback paragraphs are authored text. The multi-provider switch made the general path resilient; the architecture made the critical path resilient. That ordering is the real lesson: provider flexibility is the second line of defense. Owned truth — your knowledge store, your record reads, your written fallbacks — is the first, and it is portable across every strategy in this guide, including a future self-hosted one if a constraint ever arrives.
Pitfalls to avoid
- Self-hosting to "own the stack" with no forcing constraint. You buy an ops burden and trail frontier quality, to mitigate risks an adapter mitigates for less.
- Scattering
openai.chat(...)calls through the codebase. Lock-in is not the vendor; it is call sites. One adapter function is the exit door. - Letting the embedding model drift with the chat choice. Mixing embedding spaces silently breaks retrieval scores. Pin it, document it, migrate it deliberately.
- Automatic silent retry across all providers on every path. High-stakes paths should fail into authored text, not into a different model's guess. Make failure behavior per-path.
- Claiming open-source because your SDK is. Credibility costs more than the sentence saved.
FAQ
High, steady, predictable volume — enough to keep purchased or reserved GPUs well-utilized around the clock. Spiky or modest traffic almost always prices better per token on hosted APIs once you count idle hardware and the people running it.

Written by
Rushit Jivani
Senior Software Architect & AI Application Engineer. Architecture that shows up in crash rate, performance and how fast the team ships.