How to Verify an AI Vendor's ROI Claims Before You Sign

A vendor's deck says conversions jumped last quarter. You are about to sign a year-long contract on the strength of that one slide. Before you do, you need to know three things the deck will not tell you on its own: a jump measured against what, against what baseline, and verified how. This is the checklist we tell prospects to run against us and against every other vendor on their shortlist, because a number nobody can reproduce is not evidence, it is marketing copy with a percent sign on it.

Why do AI sales vendor numbers need independent verification in the first place?

AI sales claims need verification because a dashboard can combine activity, incomplete trials and outcomes without showing which records support the total. Aktify’s buyer checklist raises this same question and recommends checking current customer references and individual lead records. That is evidence of a public buyer concern, not a survey of the market or proof that a particular vendor will fail.

The regulatory record gives buyers another reason to ask for evidence. In August 2025, the FTC finalized an order against Workado, requiring support for advertised accuracy and efficacy claims about its AI content detector. That case concerns a different product category, so it does not establish how an AI sales system performs. It does show why a claim should come with evidence relevant to its intended use.

We run our own managed AI sales work under a conversions-up-or-you-do-not-pay guarantee, so we get asked to defend our own numbers constantly, and we think that is the right amount of scrutiny for anyone selling an AI sales system, including us. The rest of this article is the checklist we would want a buyer to run against any vendor before signing, written from the buyer's seat, not the seller's.

What is the difference between a vanity metric and an outcome metric?

A vanity metric moves in the vendor's favor almost no matter what the AI does. An outcome metric only moves when something a buyer actually cares about happens. Before you read a single number on a pitch deck, sort every metric the vendor shows you into one of these two buckets.

Treat a reported booking as the start of an evidence trail. Can you trace it to an eligible lead, a held meeting and the next agreed sales stage? Keep canceled meetings and duplicate leads visible. A larger activity total can be useful for operations, but it cannot substitute for the outcome you agreed to buy.

Ask the vendor, in writing, to show you both numbers side by side for every claim on the deck: the activity number and the outcome number it is supposed to predict. If they cannot produce the outcome number, or if it is visibly smaller than the activity number implied, that is your answer before you negotiate price.

Who should you actually talk to before you sign?

Not the two logos on the case study slide. Ask for reference calls with current, paying customers that match your business on deal size, sales cycle length, and ideally the same industry. A vendor that only offers written testimonials or a logo wall, and resists a live call, is telling you something by what it is avoiding.

On the reference call, ask the customer these four questions directly, not the vendor:

1. "When did you actually go live, and how long did it take to see a conversion number you trusted?" Compare the answer against what the vendor's own sales rep told you to expect.
2. "Can you see individual lead-level results, or only the aggregate dashboard the vendor gives you?" If the customer only sees a rolled-up number, they cannot verify the vendor's claim any more than you can from a pitch deck.
3. "What is your actual retention or renewal status with this vendor, past the first three months?" Ask what caused the customer to renew or leave; renewal alone does not prove conversion lift.
4. "If you had to switch vendors tomorrow, would you get your data back in a usable format?" You want to hear whether this was ever tested, not just promised.

Consider a hypothetical professional education business whose membership keeps falling while several teams handle marketing and AI follow-up. If nobody can connect an individual workflow to a result, the owner cannot tell which work to keep. A reference call should test for that accountability gap. Use our AI agency accountability audit to organize the records you would request.

How do you set a baseline before the AI ever touches a lead?

You cannot prove lift without knowing what "before" looked like, and most businesses do not have this written down anywhere a vendor's number can be checked against it. Do this before you sign, not after the pilot starts, because a baseline measured mid-pilot is not a baseline, it is a guess with hindsight.

1. Pull your last 90 days of lead-to-close data from your own CRM, filtered to the same lead source and volume the pilot will touch; do not use a company-wide average if the pilot only runs on inbound web leads.
2. Calculate your current conversion rate at each stage the vendor claims to improve (contact, qualified, booked-meeting, close). Write down the actual percentage and denominator, not a rounded estimate.
3. Document your current response time, lead creation to first contact, since speed to lead is a common lever vendors claim credit for.
4. Freeze this baseline in writing, dated, attached to the contract before any AI activity touches those leads.
5. Agree on the outcome metric and measurement window in writing, long enough for your sales cycle and enough eligible leads to support a useful comparison. Treat 30, 60 or 90 days as planning options, not proof of sufficient data.
6. Re-run the same calculation at the end of the window, same lead source and stage definitions, and compare the two numbers yourself before accepting the vendor's dashboard figure.

This is the same discipline we expect a buyer to hold us to, and it is the backbone of the structure we describe in how performance-based AI engagements are written: no written baseline means there is no honest way to measure the "up" in conversions-up-or-you-do-not-pay.

Is the vendor's lift number tested against a holdout group, or just a before/after comparison?

This is the question most buyers never ask, and it is the one that separates a real measurement from a story. A before/after comparison tells you conversions went up after the vendor started working your leads. It does not tell you they would not have gone up anyway, from seasonality or a sales process change that had nothing to do with the AI.

A randomized holdout assigns eligible leads to either the AI workflow or the existing process, then compares outcomes over the same window. Random assignment helps reduce selection bias. A matched comparison uses similar groups without random assignment and can still contain differences you did not measure. Neither design fixes missing outcomes or a sample too small to support the conclusion. The NIST AI Risk Management Framework provides broader guidance for evaluating AI risks; the pilot questions below are our practical application to a sales purchase. Ask the vendor to explain its design plainly:

• "Was this lift number tested against a holdout group or a matched comparison period, or is it a straight before/after?"
• "Where does the conversion signal actually come from, your own CRM activity, a self-reported dashboard, or a probabilistic model?"
• "What is the sample size behind this percentage, and what is the margin of error?"

If the vendor only has a before/after number, that is not disqualifying, but it means you should run your own holdout during the pilot: hold back a portion of comparable leads from the AI and let your existing process handle them, so you have your own control group.

Before accepting the report, select a completed opportunity and trace it backward: outcome, stage-change timestamp, meeting record, contact event, original lead source and group assignment. Repeat with a lost opportunity and a canceled booking. Missing records are a reason to repair the measurement process before treating the headline as reliable. Our guide to proving AI sales conversion lift explains the evidence trail after launch.

Also separate conversion lift from return on investment. More meetings do not necessarily create more profit. For your internal calculation, compare incremental contribution from completed sales with the full cost of implementation, ongoing service, staff review and rework. Record which inputs are measured and which remain assumptions. Do not count every sale touched by AI as incremental revenue. If the pilot has not reached the point where customers buy, report the earlier-stage result and leave revenue ROI unresolved.

What should you ask the vendor to put in writing before you sign?

This is the step most buyers skip, because it feels adversarial. Do it anyway. A verbal claim that never makes it into the contract is not a claim you can hold anyone to later.

Use this original worksheet as a starting point for the terms to request. Have the people responsible for procurement and the agreement review the final wording:

AI SALES VENDOR VERIFICATION TERMS (request before signing)

1. Metric definition: [exact outcome metric, e.g. "booked qualified
   meetings that reach opportunity stage," not "engagements" or "replies"]
2. Baseline: [your pre-pilot number, dated, attached as an exhibit]
3. Measurement window: [30/60/90 days, start and end date]
4. Methodology disclosure: [holdout group / before-after / other,
   stated in writing, with sample size]
5. Audit rights: [your right to pull lead-level data, not just the
   vendor's aggregate dashboard, during and after the pilot]
6. Remedy if the metric is missed: [refund, credit, exit clause,
   stated specifically, not "we'll work with you"]
7. Data ownership and export: [you keep call recordings, transcripts,
   reply history and lead scores; stated export format]
8. Disclosure compliance: [what AI-disclosure language the vendor
   uses on calls/chats by default, and whether it is configurable
   to your state's requirements]

A vendor may propose different terms for legitimate reasons, including customer privacy or limits on available data. Ask what alternative evidence it can provide. If the final agreement leaves the metric, audit access or remedy undefined, you still have an unresolved buying risk.

What does this look like end to end?

Sort the deck's metrics into vanity or outcome, get your reference calls, freeze your own baseline, pin down the vendor's methodology, send the written terms template, and at the end of the window recalculate the outcome metric yourself from your own CRM data instead of accepting the vendor's dashboard number. If a vendor cannot survive that sequence, you have saved yourself a year of contract and a much harder conversation later. If it can, you now have your own evidence, and that evidence is yours to keep whether you renew or switch. Our own managed AI sales work for service businesses with existing sales teams, including setting the baseline and the measurement window with you before anything goes live, runs under our managed services guarantee: conversions go up, or you do not pay.

Our sister brand RizzDial, an AI calling platform, has a deeper look at tracking the actual numbers once a system is live in measuring AI dialer ROI, which pairs well with the baseline and measurement window steps above if calling is the channel you are piloting.

What else do buyers ask before signing with an AI sales vendor?

What is a reasonable price range to expect from an AI sales vendor?

Pricing structures vary by vendor (license fee, per-seat, usage based, or performance based), and we are not going to quote competitors' numbers here. The more useful question is what you get for the price: whether the metric, baseline and remedy are in writing, not just the monthly rate. Ask every vendor on your shortlist for a written quote tied to the specific metric you agreed to measure.

How long should a vendor's pilot or trial period run before I trust the numbers?

Choose the pilot window around your sales cycle and eligible lead volume. Thirty, 60 or 90 days can be planning options, but none guarantees a reliable result. Set the window and the rule for an inconclusive result in writing before the pilot starts, so nobody moves the finish line after seeing the numbers.

Can an AI sales vendor plug into our existing CRM, or do we have to switch systems?

A vendor that cannot write its activity and outcomes directly into your CRM's existing fields and pipeline stages is a vendor you should be cautious about, not one you should adapt your systems for. Ask for a live demo of the integration against your actual CRM, not a slide describing one.

Does the vendor need to disclose AI involvement on sales calls or chats?

Ask which AI-disclosure language the vendor uses for each channel, when it appears and whether it can be configured. Have your compliance reviewer verify the requirements for the jurisdictions and audiences you serve before launch. A configurable script is a feature to test, not proof that the campaign meets every applicable rule.

What should switching vendors cost us if this one does not work out?

Get the answer in writing before you sign, not after you decide to leave. Confirm in the contract that you keep ownership of call recordings, transcripts, reply history and lead scores, and that they export in a format your next vendor or your CRM can actually import.

So, what is the one question that decides whether you sign?

Can the vendor's conversion claim survive being written down, with a metric, a baseline, a measurement window and a remedy, in the same document you would ask a performance-based partner to sign? If a vendor's number only works as a slide and falls apart the moment you ask for the methodology behind it, you already have your answer. If it survives that test, you are not just trusting a pitch anymore, you are holding a vendor to the same standard we hold ourselves to with every managed engagement we run.