02 - Evaluate & Secure / LLM Evaluation

LLM Evaluation

Independent evaluation of the AI already running in your product or your back office - answer quality, retrieval grounding, agent task success and whether the AI step genuinely shortens the process - scored against a rubric that stays with you.

Talk to an engineer See the package

Fixed-price entry packageLLM Evaluation: one solutionEUR 7,0001 AI solution (LLM feature or agent) · up to 4 weeks from scopingWhat is included

In short

A number for how well it works

Every case from your real traffic scored against a rubric the business owner agreed to.

Failure patterns, not anecdotes

The report groups what goes wrong, with reproductions and a fix per pattern.

The eval set stays yours

Handed over wired as a CI gate, so every prompt or model change is scored the same way.

23Minds evaluates the quality of AI solutions that are already live: LLM features in a product, retrieval-based (RAG) answers, and agentic workflows that run without a person watching.

Show the full summary

The engagement builds an evaluation set from your real cases, agrees a scoring rubric with the people who own the business outcome, scores every case, and delivers a written report with a measured baseline, the failure patterns behind the score and a recommended fix per pattern. The unit of work is one evaluation - one test case scored against the rubric - so the scope and the invoice describe the same thing. The evaluation set and the rubric stay with you and get wired into CI as a quality gate that catches regression the next time a model, prompt or vendor changes. The service is tool-neutral: it runs on the evaluation stack you already have or helps you choose one, and at high volume it shows where self-hosting costs less than SaaS.

Why 23Minds

What makes this service defensible.

Rubric agreed with the business owner
Correctness, grounding, task success, format, cost and latency, weighted by the person who owns the outcome.
Roughly 150 real cases in the package
Built from your traffic and edge cases, in your tooling, owned by you afterwards.
Judge model from a different family
The scorer is not the model being scored, so the score does not agree with itself.
Fixed price per feature or agent
One LLM feature or one agent workflow, four weeks.

How we work

The process, phase by phase.

1
1. Scope + rubric
Agree what 'good' means for this solution with the people who own the outcome. Define the quality dimensions, their weights and the pass threshold. Fix the number of evaluations in scope.
Week 1
2
2. Evaluation set build
Assemble test cases from real traffic, known failures and the edge cases nobody tests. Add expected answers or grading criteria per case. Wire the set into your evaluation tooling or into one we help you pick.
Week 1-2
3
3. Scoring + analysis
Score every case against the rubric using automated grading, model grading and human review where the rubric needs judgment. Group failures into patterns and trace each pattern to its cause: retrieval, prompt, model choice, tool wiring or process design.
Week 2-3
4
4. Report + handover
Deliver the baseline scorecard and failure-pattern report, walk the product and engineering owners through it, and hand over the evaluation set and rubric wired into CI as a quality gate your team can run without us.
Week 3-4
5
5. Continuous evaluation (optional)
Retainer: we maintain and extend the evaluation set, re-run it after every model, prompt or vendor change, and report the regression before your users find it.
Ongoing

How to start

Pick the way in that fits your budget and timing.

Every practice has a free first step, an entry package with a written deliverable and its price listed here, and a full project or retainer quoted after a first call. Once the scope is agreed, the price is fixed, with no hidden costs.

1
Step 1 · Free
Intro call or self-assessment
60 minutes · 1 engineer · no obligation
Free
Show details
- Intro call with an engineer, or the online self-assessment
- A written next step within one business day
- No obligation and no sales deck
Talk to an engineer
Recommended
2
Step 2 · Fixed price
LLM Evaluation: one solution
1 AI solution (LLM feature or agent) · up to 4 weeks from scoping
EUR 7,000 net
Fixed price, no hidden costs; extras only from the Not included list.
Show details
- Scoring rubric agreed with you for one LLM feature or agent
- Evaluation set of roughly 150 test cases
- Baseline run against the current system
- The set wired as a gate in your CI
- Not included: Fixing the system, prompt and retrieval rework, building the AI feature, large-scale human labelling, the continuous evaluation retainer, tool licences.
Ask for this package
3
Step 3 · Project or retainer
Full scope, quoted after a first call
Full scope · phased delivery · quoted per phase
from EUR 14,000 net, quoted after a first call
Show details
- Evaluation across several features or agents
- Continuous evaluation retainer
- Regression check after every model or prompt change
Start the conversation

Sounds familiar?

The situations that bring teams to this service.

The LLM feature is live and the only quality signal is a complaint or a support ticket
RAG answers read fluently, but nobody has verified they are grounded in the right source document
Agentic workflows report success while a human quietly finishes the job, and nobody counts how often

Show the other problems (2)

A prompt or model change ships and users discover the regression before the team does
The team adopted a free evaluation tool and stalled on two things: building the test set and agreeing what 'good' means

Who it's for

Product teams that shipped an LLM feature and have no way to tell whether the answers are actually good
Companies running an internal AI assistant with flat adoption and no measure of whether it helps anyone
Teams about to change model, prompt framework or vendor and afraid of a silent regression
Operations and shared-service leads who put an AI step into a process and need to prove it shortened the process instead of moving the work
Heads of engineering, data and support who started with a free evaluation tool, got traces flowing and stalled on the test set

Failure modes

Where this goes wrong, and what we do about it.

1. The judge agrees with itselfWhat stays with you: agreement rate between judge and human raters, the labelled sample, and the rubric that produced it
What happens
A model from the same family scores the answers that another model produced, so shared blind spots cancel out and the score comes back reassuring. Practitioners call this a second failure mode rather than a verification layer, and it is the default setup in most evaluation stacks because it is the cheapest one to build.
What we do about it
We calibrate the judge against human labels on a sample before trusting a single automated score, and we report the agreement rate. Where agreement is poor we fix the rubric rather than the model, because a rubric people cannot apply consistently is not a measurement.
2. Everyone eyeballs the output and calls it evaluatedWhat stays with you: frozen evaluation set drawn from your traffic, with expected outcomes and the baseline scores
What happens
A change ships because it looked better in a handful of prompts someone typed by hand. There is no held-out set, so the next change is judged against a different informal sample and nobody can say whether quality moved.
What we do about it
We build a case set from your real traffic, including the failures, and freeze it so successive versions are comparable. Every case carries the expected outcome and who decided it.
3. The workflow reports success while a person quietly finishes the jobWhat stays with you: end-to-end completion rate with interventions counted, broken down per step and per user journey
What happens
An agentic step marks itself complete, and a human silently corrects the result downstream. The dashboard shows a high completion rate because nobody counts the correction. The measured improvement is real to the tool and invisible to the business.
What we do about it
We measure at the point where the work is actually finished, not where the agent stops, and we count human interventions as failures of the step that needed them.

What you get

Deliverables, not deliverable-shaped slides.

Evaluation set built from your real traffic and edge cases, in your tooling, owned by you after the engagement
Scoring rubric with weighted dimensions (correctness, grounding, task success, format and tone, cost and latency) agreed with the business owner, not just with engineering
Baseline scorecard: every case scored, results broken down per dimension, per user journey and per model or prompt version
Failure-pattern report with reproductions, root cause and a recommended fix for each pattern
CI quality gate: the evaluation set running as a blocking check on prompt, model, retrieval and vendor changes
Process effectiveness read-out showing where the AI step shortens the process and where it moves work to a human, plus a tooling recommendation with a self-hosting versus SaaS cost comparison at your volume

FAQ

Questions procurement teams ask.

The unit of work is one evaluation: one test case scored against the agreed rubric. We fix the number of evaluations, the rubric and the reporting in the scope, so you can see what you are paying for. Fixed fee per engagement, scoped to the number of evaluations and the number of AI surfaces in scope. We share a tight range on the intro call once scope is clear.

Three to four weeks from kick-off to handover. From you we need one person who owns the business outcome to agree the rubric, one engineer for access and tooling, a sample of real traffic or logs, and any known failure cases you have already collected. The rubric workshop and the handover session take about half a day each; the rest runs on our side.

The Independent AI Code Audit covers the code: correctness, security, maintainability and test fitness of what your team and your AI tools wrote. LLM Evaluation covers the behaviour of the deployed solution: whether the answers are right and grounded, whether the agent finishes the task, whether the process actually got shorter. Perfectly clean code can still produce a system that answers badly, and the reverse is also true. Teams frequently buy both, usually in that order.

Not necessarily. We can work on anonymised or synthetic cases derived from your real ones, and many engagements run that way for regulated clients. Where production access is granted, we work under NDA, on your infrastructure, with the data staying in your environment. Standard NDA, and we can operate on isolated infrastructure if the engagement requires it.

We are tool-neutral and work on what you already run: Braintrust, Langfuse, LangSmith, Confident AI, Promptfoo, Arize Phoenix, W&B Weave, or a plain test harness in your own repository. If nothing is in place we help you choose, and at high evaluation volume we show you where self-hosting an open-source platform costs less than SaaS. The evaluation set is written so it can move between tools, because tying your quality baseline to a single vendor is a risk in a market that keeps consolidating.

Every evaluation platform has a free tier, so most clients arrive having tried one. The tool is rarely where teams get stuck. They get stuck on the two things the tool does not give you: a test set that represents what users actually ask, and a written definition of 'good' that the business owner and the engineers both sign. We start exactly there, and the tool you already picked keeps running underneath.

Yes, and that is usually where the interesting numbers are. We score task success rate (did the agent complete the job end to end), human takeover rate (how often a person silently finishes it), step-level failures across sessions, turns, tool calls and subagents, and process effect: how much time the AI step removes from the process versus how much it moves to someone else.

The evaluation set, the rubric and the CI gate are yours, in your repository and your tooling, and your team can run and extend them without us. If you would rather not own the maintenance, we run it as a continuous retainer: we keep the set current, re-run it after each model, prompt or vendor change, and report regression against your baseline. That retainer sits alongside LLM Observability, which watches the same quality signals in live traffic.

Guides

Guides that lead to this service.

Talk to an engineer.

Describe where you are with LLM evaluation. A reply within one business day.

Talk to an engineer

Szczecin - ul. Wawrzyniaka 6WWarszawaZielona GóraKraków

LLM Evaluation

A number for how well it works

Failure patterns, not anecdotes

The eval set stays yours

What makes this service defensible.

The process, phase by phase.

1. Scope + rubric

2. Evaluation set build

3. Scoring + analysis

4. Report + handover

5. Continuous evaluation (optional)

Pick the way in that fits your budget and timing.

Intro call or self-assessment

LLM Evaluation: one solution

Full scope, quoted after a first call

The situations that bring teams to this service.

Where this goes wrong, and what we do about it.

Deliverables, not deliverable-shaped slides.

Questions procurement teams ask.

Often shipped together with LLM Evaluation.

Independent AI Code Audit

LLM Observability

Production Readiness

Guides that lead to this service.

Talk to an engineer.