04 - Operate & Measure

Operate, Measure & Maintain

Observability, telemetry, cost optimization and ongoing maintenance for AI systems already in production - so they keep working as your data and usage change.

Talk to an engineer See the practices

In short

23Minds operates AI in production for teams that need their systems to keep working as data, usage and providers shift. The work covers LLM observability (tracing, eval pipelines, drift detection); telemetry and analytics for AI products (what to measure, what to alert on); LLM cost and model optimization audits that compare model prices with actual usage and the answer quality you need; and ongoing application maintenance for AI-touched systems. The cost work starts from provider price lists, checked on 25 September 2026. At Anthropic a prompt cache read costs 0.1x the base input token price and a five-minute cache write 1.25x. OpenAI bills cached input on GPT-5.6 and newer models at 0.1x as well, and Google's Gemini API lists cached tokens at 10% of the input price for current models. Model retirements set the maintenance calendar: Anthropic gives at least 60 days' notice for publicly released models, OpenAI at least six months for generally available ones.

Who it's for

Built for teams in these situations.

Engineering teams running LLM features in production with growing cost surprises
Product leaders who cannot tell whether their AI feature is actually working for users
CFOs and finance partners asking why the LLM bill keeps growing
DevOps and SRE teams adding LLM-backed services to their on-call rotation

Problems we solve

The situations that bring teams to this area.

LLM spend is growing faster than traffic and no one can tell why
There is no signal whether prompt changes made the product better or worse
An LLM provider deprecated a model and the team has no migration plan
Customer complaints about AI quality cannot be tied back to specific behavior in production

Practices in this area

Where Operate & Measure meets your roadmap.

01LLM ObservabilityTracing, eval pipelines, drift detection - see what your agentic stack is actually doing.Entry packagefrom EUR 5,200LLM Observability Starter · up to 4 weeks from scoping 02Product Analytics for AI FeaturesWhat to measure, what to alert on, what to put on the dashboard for AI products.Entry packagefrom EUR 4,000AI Event Taxonomy & Dashboards · up to 3 weeks from scoping 03LLM Cost OptimizationAn audit of LLM spend and the changes that can lower it: prompt caching, model routing, prompt compression. The report compares cost and quality before and after.Entry packageEUR 4,200LLM Cost Audit · up to 3 weeks from scoping 04AI Application MaintenanceOngoing maintenance for AI-touched systems: provider migrations, prompt tuning, dependency upgrades.Entry packagefrom EUR 2,600Maintenance Handover Pack · up to 2 weeks from scoping

FAQ

Questions procurement teams ask.

Whatever the measurement on your system shows; we do not promise a percentage up front. The savings come from four mechanisms: a higher prompt cache hit rate, routing tasks to smaller models where tests confirm the required quality, prompt compression and removing redundant retries. The audit package ends with a written report on the measured effect of the three changes it implements. At Anthropic and on OpenAI's newest models a cache read costs a tenth of the input price.

OpenTelemetry-based where possible. We integrate with Datadog, Honeycomb, Grafana, Posthog and dedicated LLM observability tools (Langfuse, Arize, Helicone) depending on what is already in place. The OpenTelemetry semantic conventions for generative AI moved to their own repository in semconv 1.42.0 (12 June 2026) and still have Development status, so attribute names can change between releases.

Eval pipelines: golden test sets, side-by-side comparisons against a baseline, regression alerts on metrics that matter. We design the eval rubric for your product specifically. Answer quality can change even when you do not change the application: Chen, Zaharia and Zou measured GPT-4's accuracy at identifying prime numbers falling from 84% in March 2023 to 51% in June 2023.

Yes. Provider migrations (e.g., Claude 4.6 to 4.7, GPT-4 to GPT-4o) are a standard maintenance engagement: we re-baseline evals, run regression tests and migrate prompts with measured before-after on quality and latency. Provider rules: Anthropic announces the retirement of a publicly released model at least 60 days ahead, OpenAI gives at least six months for generally available models and at least three for specialised variants, and Google lists shutdown dates as the earliest possible. In practice, Anthropic retired nine model versions on six dates between October 2025 and August 2026, with 60 to 189 days' notice.

Both work. Retainer fits teams with continuous AI features in production. Fixed engagements (provider migration, cost audit, eval setup) fit teams with a specific need.

Sources

Where the dates and numbers come from.

Talk to an engineer.

Describe where you are with operate & measure. A reply within one business day.

Talk to an engineer

Szczecin - ul. Wawrzyniaka 6WWarszawaZielona GóraKraków