Where Should the LLM Live?
Cloud behind a BFF, edge inference, or on-device WebLLM — the architecture choices that decide latency, privacy, cost, and whether you leak an API key.
Before you argue about streaming parsers and tool-call cards, answer a blunter question: where does the model run? That one decision locks your latency budget, your privacy story, your bill, and whether a curious user can yank your OpenAI key out of DevTools.
This guide is the map. Direct browser → vendor API is almost never the answer. Most products put a BFF or AI gateway in the middle. Edge and on-device are real options for specific slices of the product — not replacements for the default path.
Architecture wireframe
browser → model
Cloud
Direct → LLM API
Anthropic, OpenAI — don't
BFF / AI gateway
key stays server-side
LLM API
Anthropic, OpenAI
Self-hosted
vLLM, TGI
Edge
edge inference
Cloudflare, Vercel
Closer to users; great for small models and filters
On-device
WebLLM / ONNX
runs in the browser
Privacy wins; model download + device limits apply
| Approach | Latency | Data privacy | Cost | Good for |
|---|---|---|---|---|
| Cloud API, direct | High | Data leaves the client | Per token | Nothing — it exposes your API key |
| Cloud API behind a BFF | High, plus one hop | Your server controls the flow | Per token + server | Most products |
| Self-hosted model | Medium | Full control | Infrastructure | Regulated / sensitive data |
| Edge inference | Medium | Depends on provider | Per request / plan | Content moderation, light models |
| On-device (WebLLM) | Low after load | Stays on device | User CPU/GPU | Offline, privacy-first demos |
Hover or click a row / column to highlight the matching approach.
01The forbidden path: browser → vendor API
Shipping an Anthropic or OpenAI key in client JavaScript is a security incident waiting for a GitHub gist. Anyone can open Network/DevTools, copy the key, and burn your budget. There is no clever obfuscation that fixes this.
Referer and origin checks are not authentication. They slow down the dumbest scrapers. Treat any key that touches the browser as public.
Interview line: "I never call the vendor from the browser. Keys stay server-side behind a BFF or gateway."
02Default: Cloud API behind a BFF / AI gateway
The client talks to your backend. Your backend holds the vendor key, enforces auth, rate limits, prompt assembly, logging, and cost controls, then streams tokens back to the browser (usually SSE).
- Latency: high (model) plus one hop (your server). Still fine for chat.
- Privacy: you control what leaves the client and what you log.
- Cost: per-token vendor bill plus whatever runs your gateway.
- Good for: almost every product you will actually ship.
From the BFF you can route to a hosted LLM API or a self-hosted stack (vLLM, TGI) without changing the frontend contract. That flexibility is why seniors start here.
03Self-hosted when regulation (or cost at scale) demands it
Same BFF shape — different inference backend. You run the weights on your infra. Full data control, infrastructure cost instead of pure per-token, and operational burden (GPUs, autoscaling, model upgrades).
Frontend implication: almost none. You still consume a streaming HTTP API. The interview signal is knowing when to recommend it (regulated data, predictable high volume) versus when it's premature optimization.
04Edge inference for the thin slice
Cloudflare Workers AI, Vercel edge functions with small models, etc. Lower hop distance, good for classification, moderation, rewrite-as-you-type, and other short prompts. Privacy and cost depend on the provider. Rarely your primary chat brain — often a pre/post filter in front of the big model.
05On-device: WebLLM / ONNX in the browser
The model downloads and runs on the user's GPU/CPU. Best privacy story and offline potential. Worst cold start (model download), weakest quality for a given size, and painful battery/thermal realities on mobile.
Use it for demos, privacy-first notebooks, or features that must work offline. Don't promise ChatGPT-class quality from a 1–3B on-device model without setting expectations in the UI.
Draw the three boxes — Cloud (BFF), Edge, On-device — then say which one is the product default and which two are accelerators. Interviewers at AI companies care that you can choose, not that you memorize every vendor name.
Key Takeaways
- 01Never put vendor API keys in the browser. Direct client → LLM API is a non-starter.
- 02Default architecture: browser → BFF/AI gateway → hosted or self-hosted model, with streaming back to the client.
- 03Self-host when privacy/regulation or scale economics demand it — the frontend contract can stay identical.
- 04Edge and on-device are specialized lanes (filters, offline, privacy demos), not the default chat brain.
- 05Be ready to compare latency, privacy, cost, and "good for" in one whiteboard table.