Why LLM Budgets Regularly Spiral Out of Control
A manufacturing company in Bavaria launches a corporate LLM project in March: an internal assistant system for technical customer support, built on a leading cloud provider. The initial cost estimate: €800 per month in API fees. Result after eight months: monthly costs exceeding €6,400, plus one-time development costs of €47,000 that never appeared in the original business case.
This scenario is no outlier. In practice, companies systematically underestimate the total cost of internal language models — because they scope and time their calculations too narrowly. API prices per token are transparent and easy to understand. The actual cost drivers are not: poorly optimised prompts that consume ten times more tokens than necessary; missing caching infrastructure; underestimated integration effort; and an operating model that — after the first rollout — binds more MLOps capacity than planned.
The result: many LLM projects are a technical success but an economic problem. LLM cost calculation for enterprises therefore does not mean multiplying a token price by a usage estimate. It means capturing all four cost categories — before the first budget is approved.
LLM Cost Calculation for Enterprises: The TCO Framework
The following framework structures the total cost of an internal LLM system into four categories. Each category has its own calculation figures, optimisation levers, and typical share of total budget — based on project experience from DACH mid-market companies.
Cost Driver 1: Direct Inference Costs
This is the only cost item that most budget plans capture. Inference costs are incurred with every LLM request and are typically billed per token (input + output). With leading cloud providers, prices for powerful models currently range from $2 to $15 per million input tokens and $8 to $60 per million output tokens — depending on model class and provider.
Practical calculation benchmark: a mid-market company with 150 active users making 10–20 LLM requests daily generates approximately 30–60 million tokens per month. With a premium model ($12/1M input tokens, $48/1M output tokens) and an average input/output ratio of 3:1, this results in a monthly inference cost of $900–$1,800 — without any optimisation.
Important note: inference costs represent only 20–40% of the actual TCO in practice. The remaining 60–80% is hidden across the three following categories.
Cost Driver 2: Infrastructure and Hosting
Organisations integrating an LLM into their own infrastructure incur additional hosting costs — even if the model itself is called via an external API. These include: the vector store for RAG systems (e.g. Qdrant, Weaviate, Pinecone), an orchestration layer (e.g. LangChain, LlamaIndex, custom middleware), compute resources for pre-processing and embedding calculations, and storage for context, session history, and logs.
Typical monthly hosting costs for a mid-market corporate LLM system: €300 to €1,200 on cloud hosting (AWS, Azure, GCP), depending on user volume and architectural complexity. Self-hosted open-weight models (e.g. Llama 3, Mistral) require GPU instances; an A100 instance on AWS costs approximately $3.50 per GPU hour — at 720 hours/month and two GPUs, this yields a base cost of approximately $5,000 per month, regardless of actual usage volume.
For organisations considering self-hosting: the break-even versus cloud APIs is experienced to lie at approximately 5–10 million tokens per day — a volume most mid-market companies do not reach in their first year of operation.
Cost Driver 3: Integration and Development
This is the most frequently underestimated cost item. Integrating an LLM into existing enterprise systems (ERP, CRM, DMS, intranet) is not a plug-and-play operation. It encompasses: API connectors to source systems, data preprocessing and chunking for the vector store, access rights management (who may query which documents?), user interface (chat window, Slack bot, Teams integration), and the complete evaluation and testing infrastructure.
Realistic development costs for a mid-market first system: €25,000 to €80,000 one-time, depending on system complexity and internal development capacity. When external service providers are engaged, costs typically fall in the upper range. Amortised over 24 months, this translates to €1,050–€3,300 per month — a cost item that is entirely absent from pure API price analyses.
Important for the calculation: integration costs are not linear. A second LLM project on the same infrastructure typically costs 40–60% less than the first, because infrastructure, governance frameworks, and development know-how are already in place.
Cost Driver 4: Ongoing Operations and Hidden Costs
LLM systems are not "set and forget" applications. After rollout, ongoing operational costs emerge that appear in no initial budget: monitoring and alerting (response quality, latency, cost anomalies), regular evaluation cycles (hallucination rate, user satisfaction), index updates when the knowledge base changes, prompt optimisation when new model versions become available, and support and internal training as user groups grow.
Practical experience: the ongoing operation of a well-configured corporate LLM system consumes 0.5 to 1.5 person-days per month in IT — for simple systems that were already shipped with solid monitoring and evaluation workflows. Without this foundation, the effort doubles quickly.
Hidden costs are cost effects that only become visible after 6–12 months: model upgrades (when a provider introduces a new model and deprecates the old one), rising user acceptance (leading to unexpected token growth), and compliance efforts arising from the EU AI Act, which introduces documentation and audit obligations for LLM systems.
Cost Optimisation: The Four Most Important Levers
Once an LLM system is in operation, four adjustment levers significantly influence effective costs — without degrading the user experience.
Lever 1: Prompt Optimisation and Context Trimming
The system prompt and the supplied context are the largest token cost drivers in RAG systems. Uncontrolled implementations deliver the entire retrieved document context to the model — even when 80% of it is irrelevant. Precise chunking, reranking of retrieved passages, and compressed system prompts can reduce input token consumption by 30–50% without measurably degrading response quality.
Lever 2: Semantic Caching
In enterprise environments, many requests are semantically similar or identical: "What is our holiday policy?" is asked by 50 different users in 50 different ways. Semantic caching stores LLM responses to common requests and delivers cached results without invoking the model. In practice, this eliminates 20–40% of monthly API calls — while simultaneously delivering lower latency for end users.
Lever 3: Model Tiering
Not every request requires the most powerful model. A multi-tier system routes simple requests (fact lookups, short summaries, yes/no decisions) to a smaller, cheaper model and reserves the large model for complex reasoning tasks. The price differential between a mini model and a full-size model can be a factor of 10–20 — with appropriate routing, 40–70% of inference costs can be saved.
Lever 4: Usage Monitoring and Budget Alerting
The most effective cost control instrument is not a technical optimisation tool, but visibility. Organisations that know which use cases, departments, and user groups consume how many tokens can intervene proactively — before uncontrolled growth leads to budget overruns. Recommendation: usage dashboard from day one, monthly cost review with the responsible business owner, automated alerting when 80% of the monthly budget is reached.