Checking AI's CRM recommendations against three buyer briefs
I set out to count how many of the CRMs an AI assistant recommends would suit the person asking, going by the vendors' own pages. I stopped before the comparison. What it took to verify one product is what I'm left with.
The newer AI tools we've got at our disposal have genuinely intrigued me this year, DataForSEO's data especially (which, I should say, this project didn't use), and reading other people's write-ups of similar experiments made me want to run one of my own. The question I set about answering was this: when you ask an AI assistant to recommend a CRM, how many of the ones it names would suit the person asking, going by what the vendor publishes about plans, prices and integrations.
What I wanted was a count. What I got was a comparison I couldn't finish, because my process couldn't verify enough products to make it stand up, so I stopped it rather than loosen the rules until it passed. It left me with a fairly clear idea of what a recommendation has to show before you can count it, and a checklist for anyone whose product pages are being read by these assistants.
One of the three buyer briefs I worked from was five people, £250 a month before VAT, who need to raise invoices inside the CRM and push customers and invoices into QuickBooks Online, because that's where the bookkeeper lives. Ask an assistant and it'll name several CRMs that "integrate with QuickBooks", and to be fair that's true as far as it goes. It doesn't tell you which plan, which records, which direction, or what five seats cost once the connector and the minimum term are added up. It took me five pages and a currency conversion to settle that for one product, and on another the per-seat price turned out to be beside the point.
What I did
The plan was a three-level count on the same set of answers: which CRMs the assistants mentioned, which they recommended, and which of the recommended ones met a buyer brief by what the vendor publishes. The first two levels are what the visibility tools already sell you. The third was the one I cared about, because a recommendation that doesn't fit the person asking isn't worth much to them, and in my opinion it isn't worth much to the vendor either. The general point, that being visible isn't the same as fitting the buyer, has been made before, by MaxAEO and CiteWorks among others, so I'm not claiming it. What I hadn't seen was anyone show what verifying fit took, so that third level is where the work went.
The products. A frame of 57 CRMs, screened from public lists by a functional rule rather than by whether I'd heard of them.
The briefs. I wanted the requirements to come from buyers rather than from me, so I read 25 posts in r/CRM, r/smallbusiness and r/sales where small teams asked for a CRM and said what they needed. Nine of them raised accounting or invoicing and three stated a budget I could use. From those I wrote three small-team scenarios, each for five seats, in the UK, in GBP, billed monthly:
| Brief | What the team needs | Budget a month, before tax |
|---|---|---|
| S1 | Outlook email logging and calendar sync | £150 |
| S2 | WhatsApp sending and receiving against the contact record | £150 |
| S3 | Invoices raised in the CRM; customers and invoices pushed to QuickBooks Online, UK edition | £250 |
The seat count, the budgets and the UK setting were my design choices, so they're not estimates of what the market wants.
The evidence. Only what the vendor publishes: pricing pages, feature grids, help articles. For each of the 171 product-and-brief pairs, a pipeline fetched the vendor's pages and archived them with the date and a hash of the text, a page ranker and selector chose which archived pages a model would see, and the model pulled out the facts that bore on the brief, each with a verbatim quote and the page it came from. I read and decided the pairs that mattered myself. A pair counted as meets or fails only when every requirement in the brief was settled one way or the other; a pair with anything still open stayed pending, conflicting or unknown. None of this is about how the software behaves once you're in it, and an external reviewer audited the work through nine rounds as it went.
The tests. Before collecting any answers I wrote down two readiness tests and what would happen if they failed: at least 90% of products coded with no more than 40% of pairs unresolved for a brief, which on 57 products means around 35 resolved, and at least five products per brief whose own documentation showed they met it. If a brief missed either test, it left the comparison, and I wasn't allowed to narrow the frame to the well-documented vendors or move a budget to make it pass.
Method CRM, five pages
Method CRM against the QuickBooks brief is the case that passed, so it's the one I can show in full. The integration badge says QuickBooks. Getting from that to "meets the brief" took five pages, in this order:
| Page | What it settled |
|---|---|
| The pricing page | CRM Pro at $59 per user per month, billed monthly. The $45 beside it is a three-month promotion, so it doesn't count. |
| The full-feature grid | "View and create invoices" is ticked for Pro and Enterprise and not for the cheaper Quick Start plan. The ticks are images, so my capture had to read the icon file names rather than any text, which was a bit of a pain. |
| The supported-tables help article | Customer and Invoice are listed under "Method to QuickBooks, Push", the direction the bookkeeper needs. |
| The invoice help article, and the supported-editions article | Invoices sync to QuickBooks unless you tick "wait for approval". QuickBooks Online UK, yes. |
| The billing FAQ, once expanded | Method bills in USD for everyone outside Canada. Five seats come to $295 a month, about £223 at the ECB rate of 2 October, which I'd frozen for the whole project, so inside the £250. |
All of those were captured on 7 and 8 October, and I made the decision by reading them, with the pipeline doing the fetching. It establishes that Method's published material supported this one brief on those dates, and nothing about how well the sync works, what support is like, or whether the price is the same today.
Creatio's minimum purchase
Creatio's pricing page, in its GBP state on 4 and 6 October, showed its Growth plan at £32 per user per month, which for five people is £160 and comfortably inside a £250 budget. Further down, in the price calculator, it said: "The minimum purchase for new customers is £8,300 per year. The standard contract term is 3 years." (The page's USD state, captured on 2 and 8 October, puts the same floor at $10,000 a year.)
| A year | A month | |
|---|---|---|
| Five Growth seats at £32 | £1,920 | £160 |
| Minimum purchase for a new customer | £8,300 | £691.67 |
That's a floor, so it isn't added on top, and five Growth seats sit well under it, which makes the floor the price, on every billing basis a new customer can choose. That settled all three briefs for Creatio without my needing to look at its QuickBooks connector at all. Pricing yourself for bigger buyers than a five-seat team is a perfectly reasonable thing to do, and Creatio says so plainly on its own page.
Readings that went wrong
The problem that kept turning a yes into an unknown was interpretation. The extraction would pull a sentence that really was on the page, and the sentence wouldn't establish what it appeared to.
A site search for Salesforce Sales Cloud came back with a page about SAP Concur, and on another pass a Service Cloud help index was offered as evidence about Sales Cloud. MiraSync, a third-party app, described a QuickBooks sync in the opposite direction to the one the brief needed. Another company's help centre, hosted on a zendesk.com subdomain, was being read as Zendesk Sell's own documentation until I added a rule that platform-hosted subdomains don't count as the vendor. In every one of those the quote was accurate and the reading was wrong until a person looked at it.
Two defects of my own
The first full coding run over all 57 products came back with 96 to 100% of pairs unresolved. The page ranker, which decides which archived pages the model gets to see, was still scoring them against the requirement ids from a set of briefs I'd withdrawn, so pages mentioning WhatsApp or QuickBooks were sitting in the archive and never being sent. Pipedrive, for one, had six relevant pages archived and none sent until that was fixed on 7 October.
The reviewer found the other one by looking at what the model was really being given. The page selector was spending its character allowance on pricing snapshots and the first few thousand characters of each page, so ActiveCampaign on the WhatsApp brief was sent twelve pricing captures and none of its WhatsApp help pages, and Method lost the full-feature grid that decided its case, which was repaired before the final extraction. Both defects cost a cycle each, and fixing them didn't change the final counts.
Why I stopped
After the full pass I ran one more thing: a bounded re-ask of the unresolved items, where the model was shown the exact passages and asked the one question again, with "not stated" allowed as an answer. The worklist was frozen first, at 97 items across 80 pairs, after dropping 24 requirements with no qualifying passage and leaving out the 14 pairs I'd already decided by hand. It cost three cents and answered "not stated" 88 times and "yes" nine times, three of which survived checking the quote against the archived page and replaced a feature fact. No complete pair was resolved, because every pair it touched still had a cost, seat or budget requirement nobody had settled.
Those are the final counts: 4 of 57 pairs resolved for the Outlook brief, 2 of 57 for WhatsApp and 3 of 57 for QuickBooks, against a test that needed around 35. There were three ways to make that pass: widen the budgets, narrow the frame to vendors with good documentation, or treat "unknown" as "no", and any of them would have produced a comparison I'd made up.
The briefs left the comparison, as the rule said they would, and the project stands closed for that analysis. The human check of the machine labels was never completed either, so the labels stay development outputs and nobody should quote them as a benchmark.
The model calls for the whole thing came to under ten dollars: $2.39 for three coding runs, $4.46 for source discovery, $0.96 for the final extraction and $0.03 for the re-ask. The limit was how much the pages said and how often the reading was wrong.
A checklist for your own pages
This is what I'd ask a B2B marketing team to confirm their documentation says for one real buying requirement, drawn from where my pairs got stuck. It isn't a tested way to rank higher or convert better; it's the list of things whose absence left a product unresolved.
- Which product, and which edition of the other system. "QuickBooks" without "Online" and "UK" didn't settle Agile CRM's case.
- Which plan includes the capability, in a grid a reader can tie to a column. Method had it. 17hats showed one $60 plan on one page and three packages on another, with nothing linking them.
- The action the buyer performs, not the integration's name. "View and create invoices" answered the question; "QuickBooks integration" didn't.
- Which records move, and in which direction, per record.
- Any connector, add-on or third-party app involved, with who sells it and what it costs. ActiveCampaign's WhatsApp is a separately priced add-on.
- Per-seat price, seat minimums, billing basis, term, and any minimum purchase.
- The total for the stated configuration, with usage charges listed as excluded.
- The currency the buyer will be billed in, and the date the page was checked.
- What the page still doesn't say. 17hats implied that customers transfer and never stated it, and that pair stayed unknown.
Still open
This project can't tell you how often AI recommendations satisfy briefs like these, and I'd want to see receipts like the ones above from anyone who says theirs can. It also doesn't show that vendor pages generally can't establish fit, that the unresolved CRMs are unsuitable, that AI recommendations are inaccurate, or that anything on a website moves an AI ranking. I didn't isolate what share of my unresolved pairs was a capture limit, an unpublished fact or my own misreading, and I didn't test whether human review at scale, asking the vendors, or trials would resolve more. Vendor pages change, so every figure above is as of its capture date, and the two I'd check first on any given day are Method's pricing page and Creatio's price calculator.