Skip to content
A chess game in progress under a single lamp, the pieces casting long shadows across the board.

Service 10 · AI Cognitive Testing

See how it reasons before your users do

A benchmark score says a model passed someone else’s test. You need to know how it reasons on yours.

Book a readiness reviewHow we work
  • Language, perception and adaptive reasoning, each scored on its own
  • Logic, composition and theory of mind, probed directly
  • Coverage mapped to a cognitive taxonomy
  • Where it breaks, mapped before your users find it

Example record

Your model, ability by ability

So you know which ability failed, and on which case.

A drop on reworded cases is the usual sign of recall rather than reasoning.

cognitive battery · claims assistantSample log

  • Applied logic over policy clauses

    multi-step deductions, each step checked

    Holds

  • Compositional generalisation

    known rules in unseen combinations

    Partial

  • Theory of mind

    what the customer believes against what is true

    Fails

  • Memory across a long case file

    facts from early in the context

    Holds

  • Reworded variants of passed cases

    same case, different wording

    Drops
Wooden blocks of different geometric shapes laid out on a dark table, lit from one side.

Beyond conversation

A conversation can pass. Reasoning has to be tested

Cognitive Turing tests look past fluent chat. You see language, perception and adaptive reasoning scored one by one.

AI Test Automation

Why a benchmark score misleads

Public benchmarks are useful, and easy to overfit. One score says little about your domain.

  • Benchmark items leak into training data

  • Recognition tasks reward pattern matching over reasoning

  • One number hides which ability failed

  • Your cases are rarely in a public test set

Capabilities

What your battery covers

So each result points to one ability you can fix or accept.

01Applied logicMulti-step deduction over your rules, with each step checked as well as the final answer.
02Compositional generalisationKnown rules combined in ways the model has not seen, to separate reasoning from recall.
03Theory of mindHow well the model tracks what your user believes, wants or has misunderstood.
04Memory and contextRecall and use of facts from early in a long case, with distractors in between.
05Perception across modalitiesDocuments, images and tables read together, where the task needs more than text.
06Adaptive reasoningHow the model updates when new evidence contradicts its first answer.

How your battery is built

Taxonomies such as CAIE (Cognitive and Artificial Intelligence Evaluation) group 90+ cognitive features across memory, learning and decision-making. We check your battery against one, so it tests the abilities your use case depends on.

  • Cases drawn from your domain experts and production traffic

  • Each case varied: reworded, reordered, with distractors added

  • Rubric scoring, with a named reviewer on judgement calls

  • Results grouped by capability, each with its cases

  • Rerun on each model or prompt change, as a release gate

Model

What we fix, and what you decide

What we fix

  1. F1A battery built before tuning starts
  2. F2Reworded variants for each case, to separate reasoning from recall
  3. F3Failures reported by capability, with the cases behind them

What you decide

  1. C1Which models and providers are compared
  2. C2Which abilities matter for your use case
  3. C3The threshold that blocks a release

Related

Where to go next

Next step

Send us the model and your hardest cases.

A capability map comes back, showing where it reasons, recalls and breaks. What you get is a release threshold you can set on evidence and defend.

Book a readiness review →