How to evaluate a technology vendor when you are not technical

The demo is a performance, and you are evaluating the performance, not the product. The data on screen was prepared. The path being clicked is the one that works. The questions the presenter invites are the ones with good answers. None of this requires the vendor to be dishonest. It only requires them to do what everyone does when selling, which is to show the strongest version of the thing. The problem is that you will not live with the strongest version. You will live with the version that has your messy data, your edge cases, and your worst week.

This is part four of a six-part series on technology modernization for executives. Part three covered choosing to buy. This part covers how to evaluate who you buy from when you cannot read the code yourself, which describes most of the people who actually sign the contract.

Why the demo cannot be trusted, and what to do instead

A demo answers the question "can this software do this thing under ideal conditions." That is not the question you need answered. You need to know what happens under your conditions, when something goes wrong, and three years from now when you are locked in. None of those are demonstrable in a scripted session, so stop trying to evaluate them there. Use the demo only to disqualify, never to select. If it fails on the happy path, it is out. If it passes, you have learned almost nothing, and the real evaluation has not started.

The questions that actually separate vendors

These are not technical questions. Anyone in the room can ask them, and the quality of the answer, not the polish of it, is the signal.

1. Show me what happens when this fails. Ask to see an error, a failed integration, a recovery. A vendor confident in the product will show you the failure handling because it is good. A vendor who deflects to "that rarely happens" is telling you the failure handling is the weak part, which is exactly the part you will meet.

2. Who actually does the integration and the data migration, you or us? The answer determines who owns the largest, riskiest cost from part two. If the answer is "a partner," ask which one, then evaluate that partner as hard as the vendor, because that is who will actually be in your environment.

3. What happens at renewal? Ask directly what the price does in year two and year five, and what it costs to leave. A vendor who is vague about exit is pricing your captivity into the relationship, and you want that priced now, while you still have leverage, not later, when you have none.

4. Show me a customer who left, and tell me why. Every vendor has churned customers. One who claims otherwise is not credible. One who can speak honestly about why a customer left is showing you self-awareness and giving you the real edges of the product.

The reference check that yields the truth

The references the vendor provides are selected to say good things, so the reference call as normally run is theatre. Run it differently. Do call the provided references, but do not ask whether they are happy. Ask what they would do differently, what surprised them after go-live, and what the bad quarter looked like. Every implementation has a bad quarter, and a reference who cannot describe theirs is either not being candid or did not go live in any meaningful way.

Then do the thing that actually works: ask each reference who else they know running the same platform, and call those people instead. The vendor did not select them, so they have no reason to perform. That second ring of references is where the truth about a vendor lives.

The answers that should worry you

You do not need technical judgment to recognize these. Worry when a vendor will not put exclusions in writing. Worry when every weakness is reframed as a strength without acknowledgment. Worry when the answer to a hard question is the name of a feature rather than a description of a behavior. Worry when the people in the sales room will not be the people in your implementation, and the implementation people are unavailable until after signature. And worry most when the vendor is more fluent about their roadmap than their current product, because a roadmap is a story and the current product is the thing you are buying.

The discipline

Vendor selection failures are rarely a failure to understand the technology. They are a failure to ask non-technical questions that have uncomfortable answers, and to keep asking after the demo made everyone feel good. The executive who cannot read the code still has the most important capability in the room, which is the willingness to ask "show me what happens when this breaks" and to notice whether the answer is a behavior or a deflection.

If you are in a vendor process and want a second read on what the demos are not showing you, that is a conversation I have with selection committees before the contract, not after.

This is part four of a six-part series. Part five covers what happens after you sign: why modernization programs fail, and the warning signs visible from the boardroom.