A health system signs a contract with an AI integration vendor after a polished sales process: an impressive demo, a compelling case study from a similarly sized organization, aggressive but plausible-sounding pricing, and a project timeline that matched what the CIO had promised the board. Fourteen months later, the relationship has become the primary source of frustration for the entire IT department. The vendor’s demo environment turned out to be substantially more capable than what they could actually build against this organization’s specific EHR version. Support tickets take four days to get a first response, not the four hours promised in the sales deck. When the organization asks what happens to their custom model configurations if they ever want to leave, the answer is vague in a way that suggests the configurations don’t actually belong to them at all.
None of this was visible during the evaluation process, because the evaluation process focused on the questions that are easy to ask and easy for a vendor to answer well in a sales conversation: what features does the product have, what does it cost, what does the demo look like. The questions that actually predict whether a three-year technology partnership will succeed or become a slow-motion liability are different questions, and they are uncomfortable enough that most procurement processes never quite get around to asking them with the specificity required.
Evaluating an AI integration and system modernization partner requires assessment across five distinct categories that go well beyond the feature-and-price comparison most RFPs are built around. This article walks through each category, the specific questions to ask, and the red flags that should give any healthcare organization pause before signing a multi-year contract that will be extraordinarily difficult to unwind once your operational workflows depend on it.
Every AI vendor can show an impressive demo against a clean, generic dataset in a controlled environment. What matters is whether they can show a live, working integration against your specific EHR version, with your specific configuration quirks, at something close to your operational scale. This is the single question that most reliably separates vendors who have genuinely built healthcare integrations from vendors who have built an impressive AI capability and are hoping the integration work will be manageable once you’re under contract.
Ask specifically: have you deployed against our exact EHR version, not just the same vendor’s platform in general, and can we speak with that customer directly rather than reading a case study summary. Ask what percentage of their implementations required custom engineering work beyond their standard integration toolkit, and how long that custom work typically took versus the original estimate. A vendor who can only point to a generic demo environment, or who describes every past integration as smooth and on-schedule with no exceptions, is either unusually lucky or not being fully candid about what integration work with real healthcare systems actually involves. Real integration work has friction; a vendor who denies any friction ever occurring has either not done much of it or is not telling you the whole story.
Every vendor will produce a HIPAA compliance attestation and a business associate agreement. These are necessary and completely insufficient on their own, because they establish the legal framework without answering the practical questions that determine your actual risk exposure. Where does our protected health information physically reside — which data centers, which regions, and does that location satisfy our specific regulatory and contractual obligations? Who at the vendor organization can access our data, under what circumstances, and is that access logged and auditable in a way we can independently verify rather than simply trusting their word for it?
A specific and important question in the current environment: does any part of their pipeline send our data to a third-party AI model provider, and if so, under what data-sharing terms, and is that disclosed clearly or buried in a subcontractor list nobody reads carefully. Vague answers on data residency, evasiveness about subprocessor relationships, or an unwillingness to let your security team conduct a real technical review rather than reading a marketing-oriented compliance summary are all significant red flags. This is an area where the cost of due diligence upfront is trivial compared to the cost of a data governance failure discovered after the fact.
For each use case, the ethics review board examines four specific questions. What populations could this AI disadvantage if its training data underrepresents them? What level of explainability does this use case actually require, and does the proposed model meet that bar? What is the patient impact if the AI is wrong, and is that impact reversible? Who in clinical operations has veto authority if this AI starts behaving in ways that concern them? The output of this review isn’t a binary approve-or-reject decision. It’s a specific set of conditions that have to be met before and during deployment — particular monitoring, particular escalation paths, particular guardrails.
A vendor’s aggregate accuracy statistics, generated against their own validation dataset, tell you relatively little about how the model will perform against your specific patient population, payer mix, and clinical specialty distribution. The question to ask directly: will you share validation data broken out by demographic and clinical subgroup, not just an aggregate number, and will you support us running our own validation against a sample of our historical data before we commit to production deployment?
A vendor who responds with ‘our methodology is proprietary, but trust the number’ is asking you to take a significant operational and clinical risk on faith. Reasonable vendors protect their actual model architecture and training methodology as proprietary IP while still being willing to share subgroup-level validation performance and to support an independent validation exercise on your own data before go-live. The distinction matters: protecting how the model was built is normal business practice; refusing to demonstrate how well it actually performs on populations like yours is a transparency problem that should concern you regardless of how impressive the aggregate marketing numbers look.
Sales conversations describe support in aspirational terms: 24/7 coverage, rapid response, a dedicated success manager. What matters is the specific, contractual answer to a concrete scenario: the AI system fails or produces clearly wrong output at 2am on a Saturday during a period that affects patient care or claims processing. What is the actual contractual response time, who specifically gets paged, and what escalation path exists if the first-line support team cannot resolve it?
Ask for the name and role of your actual escalation contact — not a generic support ticket queue, but a specific person or specific small team who owns your account and can be reached directly for a genuine emergency. Ask what the SLA penalty structure looks like if response times are missed repeatedly, and whether that penalty has ever actually been invoked with a reference customer. A vendor with no named escalation contact, or one who deflects specific SLA questions with generic reassurances about their commitment to customer success, is telling you what your actual support experience will look like once you’re a customer rather than a prospect they’re still trying to close.
This is the category evaluation teams most consistently skip, because nobody wants to plan for the relationship failing while they’re still in the excitement of signing it. It is also the category that determines your actual negotiating leverage for the entire life of the contract, because a vendor who knows you cannot realistically leave has very little incentive to compete for your continued business on price, service quality, or innovation. If we leave in two years, do we retain ownership of our trained model configurations, our custom data mappings, and our historical performance data, or does everything we’ve built together stay locked inside their platform?
Ask for the specific data export format and process, and ask how long a full data export actually takes in practice, not in theory. Ask what happens to your integration work — do you have to rebuild the entire connection to your EHR from scratch with a new vendor, or does a meaningful part of that integration work transfer? A vendor whose answer amounts to ‘everything is proprietary to our platform’ is telling you that switching costs will be prohibitively high regardless of how the relationship performs, which should factor directly into how hard you negotiate on every other term, because you are effectively signing up for a much longer commitment than the stated contract length.
These five categories work best when they are converted into a structured scorecard used consistently across every vendor under consideration, rather than treated as a loose set of conversation topics that get raised inconsistently depending on which member of the evaluation team happens to think of them in a given meeting. A simple approach: for each category, assign a score of one to five based on the specificity and credibility of the vendor’s answers, with a five reserved for vendors who provide concrete evidence — a live reference call, an actual technical document, a specific named contact — rather than a confident verbal assurance. Weight the five categories according to what matters most for your specific situation; an organization with a complex, heavily customized EHR environment should weight integration depth more heavily, while an organization in a tightly regulated state should weight data governance more heavily.
This scorecard approach does two things that an unstructured conversation cannot. First, it forces the evaluation team to actually ask the uncomfortable questions in every vendor conversation, rather than letting a strong sales relationship with one vendor cause the team to skip questions they would have asked a less charismatic competitor. Second, it creates a documented record that the organization can point back to later — both to justify the selection decision to the board, and, if the relationship does go poorly, to understand at what point warning signs were visible and whether they were adequately weighted in the original decision.
The evaluation process itself should take real time — typically four to eight weeks for a meaningful AI integration decision, not the two-week sprint that internal pressure to move quickly often produces. Reference calls need to be scheduled and conducted without the vendor present, technical staff need real access to test integration claims rather than relying on vendor-narrated demos, and legal and security review of the data governance answers needs enough time to actually be substantive rather than a rubber stamp on a template BAA. Organizations that compress this timeline under deadline pressure are the ones most likely to discover, a year into the relationship, that the questions they should have asked during evaluation are now the questions defining a painful and expensive vendor transition.
Across all five categories, one single question does more filtering work than any other: can I talk to a reference customer running this in production for twelve or more months, at a scale similar to ours, on our specific EHR platform. This question is uncomfortable for vendors to answer honestly when the true answer is no, and the responses tend to sort into three clear buckets. Vendors who immediately produce a specific reference customer and facilitate a direct, unsupervised conversation are demonstrating real confidence in their track record. Vendors who offer only curated case studies, written testimonials, or references who turn out to be in an early pilot phase rather than mature production are signaling that your organization may effectively be an early-stage customer regardless of how the sales materials frame the maturity of their product. Vendors who become evasive or change the subject when this specific question is asked directly are the clearest and most reliable red flag in the entire evaluation process. Being an early reference customer for a genuinely promising technology is not automatically a mistake — sometimes the innovation and attention that comes with being a priority early account is worth the additional risk. But that should be a decision your organization makes with full information, with pricing and support commitments that explicitly account for the elevated risk, not a reality you discover fourteen months into a contract you signed believing you were buying mature, proven infrastructure.
BTC builds scalable AI governance operating models for healthcare organizations. From policy frameworks and ethics review processes to monitoring infrastructure and incident response protocols, we help leaders turn governance from a compliance exercise into infrastructure that lets AI scale safely and quickly.
Comments