For as popular and prolific as publicly available generative AI platforms have become in recent years, it’s important to remember the potential pitfalls that come with them for B2B IT purchasing decisions. Lionfish offers a solution to traditional analyst research and generic AI search.
IT leaders evaluating enterprise software and technology vendors are increasingly turning to general-purpose AI models for quick answers, sometimes in place of the analyst research and market reports firms like Gartner have traditionally provided. Lionfish argues that shortcut carries real risk in a category where the wrong recommendation could mean months of wasted implementation time and real budget on the line.
Today, many IT buyers have come to rely on general-purpose artificial intelligence (AI) models when researching vendors and evaluating purchasing decisions. Undoubtedly, this is partly because of the ease with which users can get near-immediate answers to questions or prompts. Still, the sheer confidence with which these systems operate also plays an integral role in their popularity. Ask such a model about a vendor’s capabilities or market position, and it will answer with immense speed and certainty. However, in high-stakes B2B IT buying decisions, is that confidence informed by the latest product and vendor information?
Rob Smith, a former Gartner analyst and founder of Lionfish Tech Advisors, believes it often is not, at least in the space where IT buyers may rely on generic AI models for their vendor selection. While these systems operate with confidence regardless of whether they’re right, Smith argues that confidence isn’t a reliable signal of accuracy when it comes to vendor-specific claims.
Fortunately, Lionfish built Periscope, a product that rests on a specific claim: for B2B IT buying decisions, public AI models and traditional analyst research each miss something the other has, and closing the gap between them, rather than picking one, is what makes a recommendation trustworthy.
That is a more precise claim than most AI-versus-analyst comparisons make. Here is what goes wrong when AI answers a vendor-selection question on its own, what is out of date with analyst reports when they are updated on an annual basis , and how human verification is meant to sit between the two for the specific job of choosing an IT vendor.
Where Public AI Models Fall Short
Smith’s critique is not that the models are poorly built. It is that they are trained heavily on public data, and public data about IT vendors is often wrong or out of date. Vendor sites update on marketing timelines, not accuracy timelines: product names change, features get rebranded, pricing pages lag reality. A model trained on that inherits its blind spots. Smith points to a subtler bias, too: models trained on vendor-published content can tilt toward vendors that publish the most, rather than the ones best suited to a given buyer.
Where the Analyst Reports Fall Short
With traditional analyst research, a market report takes hundreds of hours to build. It goes through heavy internal and legal review before it ships, both because the findings are rigorous and because they carry financial consequences, since a vendor’s placement can move its valuation. That same cycle is why reports are only periodically updated, often once a year. . Smith argues that the underlying data can be out of date within three to six months of publishing.
Reviews and ratings sites do not reliably fill the gap either. Smith notes that vendors often incentivize positive reviews with small gift cards, which pushes aggregate scores up regardless of product quality. His rule of thumb: the reviews worth trusting are usually the negative ones, because almost nobody writes an unprompted rave about a service they pay for, but plenty of people speak up when they feel let down.
The Role of Human Verification
It sits between the two failure modes rather than trying to outbuild either. The Aquarium, Lionfish’s proprietary competitive and market intelligence platform pulls in AI-sourced data as a general model would. Still, every data point is checked by one of the firm’s roughly 30 former Gartner analysts before it is usable, and the underlying data refreshes at least daily, more often in fast-moving categories like agentic AI. That cadence is the direct answer to the annual report’s lag: instead of a snapshot that is already stale on release day, the system reflects what is truly true this week for the vendor a buyer is evaluating.
There is already a concrete example of the verified layer working. Veracode used a survey-automation feature inside the Aquarium, built on the same human-checked data, to finish an entire Gartner Magic Quadrant vendor survey in a single day. This process usually takes two to three weeks. It is narrow, built for vendors, but it is a real, verifiable illustration of AI and human-checked data working together rather than one replacing the other.
The More Useful Question to Ask
The tagline Michael Disabato, a former Gartner Research Vice President now with Lionfish Tech Advisors, uses is four words: use AI, but verify. It is not a claim that Lionfish has better AI; it almost certainly does not, and Smith does not argue otherwise. The claim is narrower and more defensible: a recommendation is only as good as the data behind it, and data checked by an analyst who has spent years covering a market beats data scraped from whatever vendors happened to publish that quarter. For an IT leader deciding which vendor to trust with a real technology purchase, that beats asking which model is smartest.
Final Thoughts
Lionfish is not betting that its AI is smarter. It is betting that verified data beats abundant data when a real purchase is on the line. Public AI brings speed, the analyst brings judgment, and the recommendation is only trustworthy when both touch it before a buyer acts.
FAQ
Why is general-purpose AI unreliable for B2B IT vendor selection?
It is trained largely on public data, including outdated or rebranded vendor pages, so it can produce a confident answer built on stale facts, with no built-in way to flag which claims are current.
Why can an annual analyst report be out of date?
It takes hundreds of hours, plus internal and legal review, before publishing, and Smith argues the underlying data can be out of date within three to six months of that publication, with a yearly cadence unable to reflect an event that happens the week after it ships
What is the Aquarium?
The Aquarium is Lionfish’s competitive and market intelligence platform. It ingests AI-sourced data, has former Gartner analysts verify each point, and refreshes at least daily, more often in fast categories like agentic AI.
