LLM Integration Guide for Business Apps

LLM integration is the work of connecting a large language model — GPT, Claude, Gemini, Llama, or an open-weights alternative — to your business application so it can summarize, classify, draft, extract, or answer questions using your data. The API call itself is trivial. The engineering that makes it accurate, affordable, and safe is not. Most production integrations we see land between $10,000 and $80,000 to build, and the projects that fail almost always fail on architecture choice, data handling, or cost control — not on the model.

This guide walks through the three integration architectures, how to pick a model, how to do the cost-per-query math before you commit, and the mistakes that turn a two-month project into a nine-month one.

The Three LLM Integration Architectures (and When Each Fits)

Every LLM integration is one of three patterns — or a combination of them. Choosing the wrong one is the single most expensive mistake in this space.

1. Direct API integration (prompting only). Your application sends a prompt to a hosted model and uses the response. All the "knowledge" comes from the model's training plus whatever context you stuff into the prompt. This is the right pattern for summarization, drafting, classification, extraction, rewriting, and translation — tasks where the input contains everything the model needs. Build cost: $5,000–$20,000. Timeline: 2–6 weeks.

2. Retrieval-Augmented Generation (RAG). Before calling the model, your system searches your own documents — policies, product docs, tickets, contracts — and injects the relevant passages into the prompt. This is the pattern for "answer questions about our stuff": internal knowledge assistants, support bots, document Q&A. Build cost: $20,000–$80,000 depending on data volume and quality. We cover this pattern in depth in our plain-English RAG explainer.

3. Fine-tuning. You train a model variant on hundreds or thousands of your own examples so it internalizes a style, format, or narrow skill. Fine-tuning does not reliably teach a model new facts — that is RAG's job. It teaches behavior: your brand voice, your JSON output schema, your classification taxonomy. Cost: $15,000–$60,000+ including data preparation, which is usually the largest line item.

Pattern	Best for	Typical build cost	Knowledge freshness
Direct API	Summarize, draft, classify, extract	$5K–$20K	Whatever is in the prompt
RAG	Q&A over your documents	$20K–$80K	As fresh as your index
Fine-tuning	Style, format, narrow skills	$15K–$60K+	Frozen at training time

Roughly 60–70% of the business use cases we scope need only direct API integration with good prompt engineering. If a vendor jumps straight to fine-tuning, ask why. The full decision logic deserves its own article — see Fine-Tuning vs RAG: Which Does Your AI Project Need?

How to Choose a Model Without Getting Lost in Benchmarks

Leaderboard scores are a weak predictor of how a model performs on your task. A practical selection process looks like this:

Start with a frontier model (the current top tier from OpenAI, Anthropic, or Google) to establish a quality ceiling. If the best available model can't do the task acceptably, no amount of engineering will save the project.
Build a small evaluation set — 50–200 real examples from your business with known-good answers. This costs a few days of subject-matter-expert time and is the highest-ROI investment in the entire project.
Step down until quality breaks. Run the same eval on smaller, cheaper models. Many classification and summarization tasks lose almost nothing moving from a frontier model to a mid-tier one at 5–20x lower cost.
Decide hosted vs open-weights. Hosted APIs win on speed-to-market and model quality. Open-weights models (Llama, Mistral, Qwen) running on your own infrastructure win when data cannot leave your environment or when volume is high enough that GPU hosting beats per-token pricing — typically only at millions of requests per month.

Route different tasks to different models. A common production setup uses a cheap model for triage and classification and a frontier model only for the 10–20% of requests that need it. This alone often cuts inference spend by more than half.

Do the Cost-per-Query Math Before You Build

LLM pricing is per token (roughly 0.75 words per token), billed separately for input and output. The math is simple, and skipping it is how teams end up with a surprise five-figure monthly bill.

Worked example for a support-assistant query:

System prompt + retrieved context: ~3,000 input tokens
User question: ~100 input tokens
Model answer: ~400 output tokens

At typical mid-tier hosted-model rates (order of $0.25–$3 per million input tokens and $1–$15 per million output tokens depending on tier), that query costs somewhere between a twentieth of a cent and two cents. At 50,000 queries per month, your inference bill lands roughly between $50 and $1,000/month — the spread is driven almost entirely by which model tier you chose and how much context you inject.

Three levers control this number:

Context discipline. Every retrieved chunk you inject is billed on every call. Tight retrieval (3–5 relevant chunks) instead of "dump everything" can cut input tokens by 70%.
Prompt caching. Most providers discount repeated prompt prefixes (your system prompt, static instructions) by 50–90%. Structure prompts so the static part comes first.
Model routing. As above — cheap model by default, expensive model on demand.

Budget inference at 2–3x your initial estimate for year one. Usage grows faster than teams expect once a feature actually works.

Security and PII: The Part Legal Will Ask About

Sending customer data to a third-party model API is a data-processing relationship, and it needs the same diligence as any other vendor. What we implement on nearly every business integration:

Zero-retention API agreements. Major providers offer enterprise terms where prompts are not stored or used for training. Get this in writing before any production data flows.
PII redaction at the boundary. Detect and mask names, emails, account numbers, and health identifiers before the prompt leaves your infrastructure, then re-insert them in the response. This is a solved engineering problem and typically adds $5,000–$15,000 to a build.
Prompt-injection defenses. If your LLM reads untrusted content — emails, web pages, uploaded documents — that content can contain instructions that hijack the model. Treat all retrieved text as data, constrain the model's tools and permissions, and never let an LLM execute privileged actions without a deterministic policy check.
Audit logging. Log every prompt and response (with PII masked) so you can investigate bad outputs and demonstrate compliance.
Regional and residency controls. EU customer data may need EU inference endpoints; regulated industries may need VPC-isolated or self-hosted deployment.

None of this is exotic, but retrofitting it after launch costs 3–5x more than designing it in.

The Five Most Common LLM Integration Mistakes

No evaluation set. Teams ship on vibes, then discover accuracy problems from angry users instead of from a test suite. Build the eval before you build the feature.
Fine-tuning to fix a knowledge problem. Fine-tuning does not teach facts reliably. If the model needs to know your return policy, retrieve the policy — don't train on it.
Ignoring data quality. RAG over messy, outdated, duplicated documents produces confident answers from the wrong source. Data quality decides AI success more than model choice does.
No fallback design. Models time out, rate-limit, and occasionally return garbage. Production integrations need retries, a fallback model, output validation, and a graceful "I can't answer that" path.
Treating it as a demo, not a product. The demo is 20% of the work. Monitoring, cost controls, guardrails, versioned prompts, and regression testing are the other 80%.

When NOT to Integrate an LLM

Honesty check — an LLM is the wrong tool when:

The task is deterministic. Tax calculation, inventory math, rule-based routing: regular code is cheaper, faster, and never hallucinates.
You need guaranteed correctness. LLMs are probabilistic. If a wrong answer is unacceptable and unreviewable, keep a human or a deterministic system in the loop.
Volume is tiny. If a task happens ten times a week, a checklist or a template beats a five-figure integration.
Your data isn't ready. If the documents an assistant would rely on are wrong or stale, fix the data first — the model will only amplify the mess.

A two-week AI proof of concept against a real evaluation set is the cheapest way to find out which side of this line your use case falls on.

Frequently Asked Questions

How much does LLM integration cost?

A direct API integration (summarization, drafting, classification) typically costs $5,000–$20,000 to build. A RAG-based assistant over your own documents runs $20,000–$80,000. Fine-tuning projects start around $15,000, with data preparation usually the biggest cost. Ongoing inference costs range from under $100 to a few thousand dollars per month for most mid-market workloads, driven by model tier, query volume, and context size.

Do I need to fine-tune a model to use my company's data?

Usually not. If you want the model to answer questions from your documents, retrieval (RAG) is the right mechanism — it keeps knowledge fresh and shows sources. Fine-tuning is for teaching behavior: tone, output format, or a narrow classification skill. Many production systems use neither and rely on well-engineered prompts alone.

Which LLM is best for business applications?

There is no single best model. Start with a frontier model to establish your quality ceiling, test cheaper models against a 50–200 example evaluation set from your own data, and use the smallest model that passes. Most businesses end up routing between two or three models by task.

Is it safe to send customer data to an LLM API?

It can be, with the right controls: an enterprise agreement with zero data retention, PII redaction before prompts leave your infrastructure, audit logging, and regional endpoints where residency rules apply. For the strictest environments, open-weights models can run entirely inside your own cloud.

How long does an LLM integration project take?

A prompt-based feature ships in 2–6 weeks. A production RAG assistant typically takes 6–12 weeks including data cleanup and evaluation. Fine-tuning adds 2–4 weeks, mostly for dataset preparation. The long pole is almost always data readiness and evaluation, not the integration code.

LLM Integration: A Practical Guide for Business Applications