Red flags when hiring an AI development agency
What to watch for in a pitch — from demo theatre to vague teams — written by an agency that would rather lose badly-fitting work.
We lose pitches, and the ones we are glad to lose are the ones where the client was buying something we do not think works. Here is what we would look for if we were on the other side of the table.
The demo is too smooth
An AI demo that never stumbles has been rehearsed on inputs chosen to succeed. Ask them to run it live on your data, unedited, including the awkward cases. The reaction to that request is itself informative.
Make the demo prove something
- 1
Send your own data ahead
Unedited, including the messy records you are embarrassed by.
- 2
Ask them to run it live
Not a recording. The reaction to the request is itself informative.
- 3
Ask what it got wrong
And what they would change to fix it.
- 4
Ask what they'd not build
A vendor who talks you out of scope is showing you their judgement.
Nobody mentions evaluation
If a vendor cannot tell you how they will measure whether the system works — specifically, numerically, on your data — they intend to ship on impressions. That is the single most reliable predictor of a project that disappoints.
The team is described in the abstract
'Our team of senior engineers' is not an answer. Ask who by name, what they have built, and how much of their week this gets. Pitch teams that do not overlap with delivery teams are common and worth knowing about in advance.
Confidence about your data without seeing it
Anyone who commits to accuracy figures before examining your data is guessing. The honest answer at pitch stage is a range and a set of assumptions, plus a proposal to narrow it during discovery.
Signals worth reading
Walk away
- A demo that never stumbles, on data you didn't provide
- No answer to how success will be measured
- "Our team of senior engineers" — no names, no availability
- Accuracy committed to before anyone saw your data
Good signs
- Offers a paid discovery before quoting the build
- Shows you an evaluation report from real work
- Tells you which parts you shouldn't build
- Can describe a project that went badly and what changed
Questions worth asking
The ones that most reliably separate teams who have shipped from teams who have not:
- What happens to the code and the models if we part ways?
- Show me an evaluation report from a real project
- What will this cost to run per month at our volume?
- Which parts of this would you advise us not to build?
- Tell me about a project that went badly and what you changed afterwards
And a flag in the other direction
A vendor who tells you your idea is not worth building, or that a simpler non-AI solution would do, is demonstrating something a portfolio cannot. That advice is usually worth more than the proposal.