A practical guide for technical buyers
Before hiring an AI SME, engineer or architect, ask them to show how they turn a requirement into a result that can be tested.
By Phil Steadman · AI PhD researcher, technical scrum master and IT engineer · 17 September 2026
Give candidates the same non-sensitive scenario, the same constraints and a clear time limit. Agree any paid assessment in advance. A worked example should reveal their reasoning and practical contribution.
For example, ask how they would help staff find answers in an approved set of internal documents. Use synthetic documents for the assessment. Describe who will use the system, which sources are permitted, what must remain private and what happens when the answer is missing.
Ask what they need to know before selecting a tool. Look for questions about users, the current workflow, the cost of errors, available data and the intended outcome.
Evidence to request: a short problem statement, assumptions and reasons an AI solution might or might not fit.
Request a walkthrough of a relevant example. Ask which parts they designed, implemented or tested, what failed and what changed as a result.
Evidence to request: a permitted demonstration, a sanitised design or an explanation of test results. Respect client confidentiality.
Ask how they would evaluate correct answers, unsupported answers, missing information and attempts to retrieve information a user cannot access.
Evidence to request: test cases, expected behaviour and a plan to record failures. A fluent answer alone does not establish correctness.
My approach is to make the task explicit: role, context, approved inputs, constraints, expected output, acceptance tests and a definition of done. This makes a proposed AI workflow easier to challenge and repeat.
In the document example, a recipe should specify that answers are grounded in the approved sources, access permissions apply to retrieval, and the system must acknowledge missing evidence. The implementation still needs testing; writing the instruction is only one part of the work.
A useful technical discussion examines the failure, considers possible causes, changes one part of the approach and reruns relevant tests. Ask how they prevent the fix from breaking previously correct behaviour and how they record unresolved limitations.
An AI SME may advise on feasibility and evaluation. An engineer needs to implement and integrate. An architect needs to explain decisions across systems and operating constraints. Assess the person's ability against the responsibilities of the proposed role.
For a delivery team, also identify who owns testing, deployment, access approval and ongoing support. Make these responsibilities explicit in the brief.
Compare AI specialist and team engagement options →These questions form a starting framework for assessment; the checks must be adapted to the actual risk and intended use.
Share a non-sensitive outline of your requirement. We can discuss a suitable role brief, assessment approach or delivery team.
Discuss your AI requirement →