What to measure without fake ROI

Small businesses get sold on numbers they never collected themselves. A pilot doesn’t need an impressive-sounding improvement figure to be worth keeping. It needs evidence that the people doing the work would recognize and agree with.

Time spent on the task

Time is a useful measure when you’re comparing the same task before and during the pilot, on real instances. For example, you might time “quoting a standard job” both ways. A general sense that the business feels faster doesn’t tell you much.

Record minutes, or a range, across a handful of instances. If the range is wide, note that. Small businesses usually do have wide ranges, and a single average hides the job that took all afternoon because the customer kept changing the scope.

Count the review time too. A draft that takes two minutes to generate and fifteen to fix isn’t saving anything until you’ve added both together. Plenty of tools only look fast because they’ve moved the work onto whoever does the checking.

Mistakes you can point to

A mistake here means something specific, like a wrong part, a promise your business can’t keep, a tone you’d never send under your name, or a field that doesn’t match your system of record.

Keep a short log with the date, what was wrong, and whether someone caught it. By the end of the pilot you’ll be able to say something like “the draft missed the access note on two of eight jobs, and the reviewer caught both.” People will believe that sentence far more readily than “accuracy improved.”

Be careful about turning the log into a rate you couldn’t defend to the person who does the job. With eight instances, you have eight examples and a rough sense of direction, which isn’t enough to support a statistic.

What your team thinks

Ask the people reviewing the drafts a simple question: would you want this draft next month or not? Then ask what they still have to fix every time.

Trust is worth measuring. If your most experienced person won’t send a draft without rewriting it from scratch, the tool hasn’t been adopted yet, however good the demo looked. Write their reasons into the decision note. Comments like “It sounds like a brochure,” “It keeps forgetting the gate code,” or “Checking it takes as long as writing it” tell you exactly what needs to change.

You could run a survey with a five-point scale, but three short conversations will usually tell you more. On a small team, you can simply ask.

What to leave out for now

Skip revenue attribution, “hours given back to the business,” and any figure that depends on imagining a future where everything’s fully rolled out. Numbers like those are how pilots get oversold, and then dropped when a normal week doesn’t match the slide.

Leave out activity counts too, like messages sent to the bot, seats assigned, or logins. Someone opening an app doesn’t mean the work is being done differently. The adoption strategy is concerned with whether a workflow you’ve kept holds up once the whole team is using it, and login counts can’t tell you that.

When bigger numbers make sense

Once a workflow is a keep and more than one person has used it for a month, you can look at the same three measures again: time on the task, mistakes caught, and whether people still want the draft. At that point you’re looking at a trend you can stand behind.

If the workflow now spans several tools, needs shared memory, or should appear on a screen the whole team uses, use the stack map to sort out what’s a tool, what’s an agent, and what stays with a person. Plans for that kind of setup are on arcasya.ai. If you’d like help reading your log, bring it to a Free AI evaluation. We’ll work from what you recorded.