Checklist
Twelve questions to ask before you hire an AI development company
Twelve questions a team that has shipped and operated AI can answer with a number, a mechanism or a name, and the weak answers to listen for.
Published Updated 8 min read
What these twelve questions are for
The questions worth asking an AI development company are the ones that cannot be answered with a slide. How do they know a change made the system better and not quietly worse. What does it do when it does not know. Who owns the code. Who is paged at 2am. Have they ever told a client not to build the thing.
A team that has shipped and operated AI answers in specifics: a metric, a mechanism, a person. A reseller answers in adjectives.
We sell the thing these questions interrogate, so read them with us in mind too. Several are uncomfortable for any vendor, this one included, and the text says so. Write down each answer as it was given, not as you would have phrased it. The gap is the finding.
1. How will you know if a change made this better and not worse?
Really asking: whether output quality is measured at all, or whether they ship on a demo that worked once.
Good answer: "We keep a graded set of real questions with known-good answers. Every change runs against it before merge, and a drop below threshold blocks the merge." Ask how many cases, who wrote them, and whether you can add one.
Weak answer: "We test thoroughly before release." Or "the models keep getting better, so quality improves over time." Neither is a measurement. The honest caveat: a test set written by the builders flatters the system, so the right to add cases matters more than the size of the suite. This is the eval and reliability layer most teams skip.
2. What does it do when it does not know?
Really asking: whether there is a path that returns nothing, and whether anyone measured how often taking it was correct.
Good answer: "It abstains. When retrieval turns up nothing that supports an answer, it says so rather than composing one, and we measure abstain precision: of the times it held back, how often was that right." One production system we run reports 1.000 on its own test set. The number means little without the set, so ask for both.
Weak answer: "We use a very strong model, so that is rare." Rare is not a design. Any model answers questions it cannot support unless something stops it, which is the subject of how AI systems are kept from making things up.
3. When it answers, can I see where the answer came from?
Really asking: whether a citation points at something real, or whether the model wrote a plausible-looking footnote.
Good answer: "Every claim carries the document, page or row it came from, and a check runs before serving to confirm the claim appears there." Mechanisms have names. On one system we run, a seven-stage verifier compares each claim against its retrieved source and fails closed to an abstain. Ask to click a citation in a live system and watch the source open.
Weak answer: "It provides sources." Ask whether the source is retrieved or generated. If the model writes the citation, it can be wrong in exactly the way the answer can.
4. Who owns the code, and can my team run it without you?
Really asking: whether you are buying an asset or renting access to a system only they can operate.
Good answer: "It is built in your repositories, under your license, from the first commit. Here are the deployment docs, and your engineers can stand the whole thing up on your own accounts today." That is our standard, and worth checking with anyone: ask which repository the code sits in now, not at the end.
Weak answer: "You own your data and your configuration." That sentence hides a lot. Data plus configuration, minus the code, is a hostage arrangement, and it is the real cost inside the build versus buy decision for AI agents.
5. Where does our data go, and how do you keep it apart from other clients' data?
Really asking: two things. Which third parties see your data, and whether one tenant's documents can surface in another's answers.
Good answer: named services and named regions. "Embeddings go to this provider, inference runs in your cloud account under your keys, nothing is retained for training." On separation, a measurement: one system we run holds zero cross-tenant leaks across both of its embedding spaces, retested on every change. The caveat is real. Zero leaks means zero on the cases tested, and a leak test only covers the pairs someone thought to write.
Weak answer: "It is all enterprise grade and fully encrypted." Encryption at rest answers a different question.
6. Who is actually going to write this code?
Really asking: whether the people on the call are the people on the keyboard, or whether the work goes to a team in another company you will never meet.
Good answer: names and a headcount. "Two engineers, both employees, one of them is on this call, nothing is subcontracted." Ask for those names in the contract.
Weak answer: "We have a global delivery network of 200 plus engineers." That is an org chart, not an answer. This one is awkward for a small firm, and we are one. A small team cannot subcontract you into a surprise, but it carries a bus factor a large firm does not. Ask what happens if their lead is out for a month.
7. It breaks at 2am. What happens, and who is paged?
Really asking: whether anyone owns the system after launch, and what support means in hours and dollars.
Good answer: a named person, a response window, and monitoring that fires before your users notice. "Alerts go to this rota, we respond inside four hours, here is the runbook your team also has." Ask what pages, not only who.
Weak answer: "We offer 24/7 support" from a shop with no rota behind it. This is a hard question for small firms, ours included. A firm without a night rota should say so and give you a real response window instead of implying a shift that does not exist. An honest best-effort window survives the first incident; a promise does not.
8. Is this fixed price or hourly, and what makes the number change?
Really asking: where the cost risk sits, and what triggers a change order.
Good answer: a number, a written scope, and a stated list of what counts as new scope. Ours is a fixed-scope build with the price agreed before work starts, then a flat monthly rate to operate it, no open-ended retainer. Whatever you are quoted, ask for one change order they raised on a past project and what it cost.
Weak answer: "We work in sprints, so it depends on scope." A rate card is not a price. Fixed price has its own failure mode: a vendor who underbids makes it back on change orders, so read the scope document harder than the number.
9. Tell me about a project that did not work.
Really asking: whether they will tell you an inconvenient truth before you sign, which is the only signal about whether they will tell you one after.
Good answer: a specific story with a diagnosis and a cost. "The knowledge we needed to ground the system had never been written down. We found that in week two, stopped, and refunded the remainder." Expect vagueness about client identity and precision about mechanism. That is the right shape.
Weak answer: "Every project has its challenges, but we always find a way." Or a failure story in which the client is the villain. Every vendor has this rehearsed, ours included, so listen for whether the failure is actually theirs.
10. What happens when the model you build on is deprecated or repriced?
Really asking: how much of the system is welded to one vendor's API and one vendor's price list.
Good answer: "The model sits behind an interface we can swap, and here is the eval suite we would run against the replacement to see what regressed." Ask whether they have actually done a swap and what broke. Prompts tuned hard against one model often degrade on another, and a team that has migrated will name the parts that hurt.
Weak answer: "We are model agnostic." Nobody is. Ask how many engineering hours a swap would take, who pays for them, and whether that estimate has ever been tested against reality.
11. What does the last week of the engagement look like?
Really asking: whether there is a handover with a defined end, or a soft slide into a permanent invoice.
Good answer: a list of artefacts and a test. "You get the repositories, deployment docs, the eval suite and the runbook, plus a session where your engineer ships a change while we watch." Ours is written that way: a handover that leaves your team able to run the system. Ask what happens if you never renew.
Weak answer: "We will always be here for you." Warm, and expensive. With no defined exit, the engagement is a retainer with extra steps. Ask them to describe the handover as though you were about to fire them.
12. Have you ever told a client not to build the thing they asked for?
Really asking: whether they hold a bar a project can fail, or whether every problem turns out to be an AI problem.
Good answer: a concrete case and a reason. "Their approval workflow had to be right every time, and grounding is not a guarantee, so we told them to write rules and keep the model out of it." We state it as a term: where the data to ground a system does not exist, or the workflow needs a guarantee AI cannot give, we say so before you spend a dollar. Ask for their version.
Weak answer: "Every business can benefit from AI somewhere." That is a sales position, not an assessment.
What to do with the answers
Score as you go. A specific answer names a mechanism, a metric or a person. A weak answer names a category. Four or more categories and you are talking to someone who will subcontract the parts that decide whether this works.
Two follow-ups beat any single question above. Ask to see an eval run that failed, and ask to click one citation through to the source behind it. Both take five minutes for a vendor who built the thing, and cannot be done at all by one who did not.
If you want to run this list on us, tell us the workflow that hurts and ask all twelve.
Questions people ask
How do I choose an AI development company?
Choose on evidence you can check, not on a portfolio. Ask how the vendor measures whether a change improved the system, what the system does when it does not know, who owns the code, and who is paged when production breaks. A team that has operated AI in production answers with a metric, a mechanism or a name; a reseller answers with adjectives. The two fastest checks are asking to see a failing eval run and asking to click one citation through to the document it came from.
What should I ask an AI vendor about security?
Ask where the data goes, which third-party services see it, whether anything is retained or used for training, and how one client's documents are kept out of another client's answers. Named services, a named region, a retention period and a separation test are answers. 'Enterprise grade and encrypted' is not, because encryption at rest says nothing about who can query the index. If the system is multi-tenant, ask for the leak test itself, since a leak measurement only covers the pairs someone thought to write.
How do I tell a real AI team from a reseller?
A real engineering team can show the parts nobody puts in a demo: the eval suite, a run that failed, the abstain path, and the repository the code sits in today. A reseller talks about model partnerships and moves on to case studies. The quickest test is to ask who will write the code and get names back, then ask what happens when the underlying model is deprecated. Vagueness on either usually means the work is going to a subcontracted team.
Keep reading
- 4 August 2026How to stop AI hallucinationsFour mechanisms that cut invented answers, the number to demand for each, and the eight questions that expose a vendor who has never measured one.
- 4 August 2026Build vs buy AI agents, and the fourth option nobody sells youFour routes to an AI agent, what each one costs you in speed, control and lock-in, and the five-step rule that picks between them.