← Writing

The Gap Between Demo and Defensible

AI-native product management: fluency or skepticism?

2026-05-15 · Seth Manwaring

The phrase “AI-native” has been doing a lot of work this year. It usually means one of three things: a product built around an LLM (chat interface, generative content, conversational search), an engineering team using AI assistants in their daily workflow, or a vague aspiration that “AI” is somewhere in the strategy. What it rarely means is that the product was calibrated from its inception — or through significant refactoring — by a product organization with LLM-shaped capabilities, and whose definition of “ready to ship” reflects what those capabilities can and cannot defensibly do.

Over the last few years, I landed somewhere between Early Adopter and Pragmatist regarding AI. After some dabbling and using ChatGPT and Perplexity for research and writing, I ran logic and bias tests in ChatGPT, Gemini, Claude, Perplexity, and Grok. I was surprised at the number of failed simple logic tests just two years ago; the tools that passed moved on to the bias tests. As a result of these tests, I chose Perplexity as my go-to for drafting process docs, product requirements, and competitive analysis.

My relationship with AI changed dramatically in 2026. Instead of prompting for information, I tasked Claude Code to create files and build applications. Instead of asking about cost and revenue potential, I prompted Claude to generate reports and save them for my review.

I know I wasn’t alone in this journey. Most companies are looking for AI talent and several companies are already downsizing, citing AI as a factor. Cloudflare cut roughly 20% of staff and internal AI usage reportedly went up 600% in three months. An enormous portion of the workforce is picking up the tools and skills to become more efficient.

Side note: AI isn’t replacing workers, per se. But if one person makes effective use of AI to do the job of six people, then companies will reflexively downsize and hope to preserve their current productivity at a lower cost, or they can keep the headcount the same and massively expand productivity. Workers who don’t use AI will be replaced by workers who do. We saw a similar shift during the rise of automated software testing. Manual-only testers felt the same squeeze: a lot of teams now want a hybrid QA profile — someone who can do manual exploratory testing, then automate the stable, repetitive checks.

An AI-native Product Manager (or Product Owner) runs AI tools all day alongside Jira, Aha!, or Asana, performing research and synthesis. Claude Code generates system diagrams, ROI models, and prototypes. Perplexity does the first pass on competitive landscape questions. Grok pulls real-time signal about what’s happening now on a topic. Whisper transcribes customer interviews. The combination compresses what used to be a week of work into hours, and what used to be a day of work into an hour.

That’s the part that’s easy to say. The part that’s harder, and where I think most of the genuinely interesting questions live, is what happens to senior product judgment when these tools become ambient.

Three things that have actually changed in my work

The first is the cost curve on synthesis. My job is to absorb a lot of input from several sources — customer interaction, sales conversations, support tickets, engineering tradeoffs, market signals, regulatory updates — bringing key decisions to the surface. The bottleneck used to be reading. Now the bottleneck is judgment. I can feed a quarter’s worth of customer interviews into Claude and ask for thematic clustering in minutes; the thinking about which themes matter is the same thinking it always was, just no longer gated by the time it takes to do the reading. My source material is now in sharper focus. Highly specific prompts into AI produce clean insights the same way that clear, detailed requirements into Development produce desired outcomes.

The second is the cost curve on writing. PRDs, decision memos, stakeholder updates, executive narratives, competitive analysis, cost and revenue calculations — the documents a Product team produces are mostly synthesis exercises with a thin layer of original argument. The synthesis is cheap now. The argument is still hard. The work I do on a PRD and other documentation in 2026 is more editing than drafting. Think about that. Accelerate drafts so you can spend more energy on edits and rewrites.

The third is the cost curve on stress-testing. Before shipping any non-trivial decision I now run it through more lenses with an LLM:

  • What’s the most aggressive critique an engineer would make of this approach?
  • What’s the customer-trust risk if this fails in production?
  • What scrutiny will a skeptical executive apply?
  • What would a regulator ask in an audit conversation?
  • What’s the case for doing nothing instead?

The model isn’t 100% on any of these. Not yet, but it’s as good as a team of thoughtful colleagues who’ll engage with your draft on a Friday afternoon (or middle of the night) when no actual colleague has time. The judgment I bring to evaluate the model’s responses is the same judgment I’d bring to evaluate any colleague’s responses — but I get the responses more often and faster.

Where AI-augmented PMs go wrong

The most common failure mode I see in PMs using AI heavily is outsourcing judgment to a tool that is structurally incapable of having it. An LLM can summarize, cluster, draft, restructure, and surface patterns. It cannot decide which patterns are the load-bearing ones for this customer in this market under thisbusiness pressure. It doesn’t have a relationship with your customer. It can’t empathize; it can only imitate human sentiment. It doesn’t feel business pressure. PMs who treat the model’s output as conclusion rather than input ship product that looks coherent and isn’t.

The second failure mode is building demos that don’t survive contact with the customer’s responsibility surface. Most consumer LLM features feel magical because the cost of being wrong is low — the user gets a worse haiku, a slightly off recipe, a hallucinated trivia answer, and shrugs. In the industries where I’ve built, the cost of being wrong includes regulatory exposure, financial harm to the customer, company liability, and audit trails that have to hold up months later. The gap between this model is impressive in offline evaluation and this feature is reliable enough that the customer can defend its decisions to a regulator, an auditor, or their own bottom line is enormous. Most product orgs underestimate it because their team’s intuitions were calibrated on consumer demos.

The third failure mode is treating AI tools as a productivity hack rather than a product judgment recalibration. The first means “I’ll write the PRD twice as fast.” The second means “the questions I should be asking about this product have shifted because the capability surface has shifted.” Today most PMs are still in the first mode. The ones that have moved into the second will prove the real value of AI in company strategy.

What I look for now

When I evaluate product organizations from the outside — as a candidate thinking about where to spend the next several years, or as a leader thinking about how to build and guide my team — the question I find myself asking is: does this team understand the gap between demo-impressive and ship-ready, and does their definition of “done” reflect that gap? I expect many are still finding their way. The teams that don’t are the ones whose AI features will quietly underperform over the next two years while the teams that do build durable trust with their customers. And from the ten-thousand foot view, this will differentiate companies who successfully implement AI in their products and services.

The PMs I want to work with are the ones who use AI fluently and skeptically at the same time. Fluently, because that’s the operating reality of the work in the fourth industrial revolution. Skeptically, because the cost of unwarranted trust in model output compounds in regulated categories the way technical debt compounds in mature codebases.

The combination of both is what AI-native product management actually means.