I design AI-first products at Hints, a tool where sales reps talk to a chatbot over WhatsApp or Telegram, and the AI updates their CRM for them, may that be HubSpot, Salesforce, Pipedrive or any other tool. We're also building an agentic AI product now, where the AI doesn't just respond to commands but takes initiative: following up on stale deals, flagging missing data, suggesting next steps before the rep asks. The model part works well in both cases. What took us much longer to get right was everything around it: what happens when the AI misunderstands a deal stage? What happens when it's about to update a contact that another rep is editing? What happens when an agent acts on a stale assumption and nobody catches it? These aren't model problems. They're product design problems. And they're the reason AI products quietly die after the first week of adoption. AI-first products run on a probabilistic engine. The output may be excellent, average, or wrong, and nobody can really predict which. The design job is not to package the AI prettily. It's to turn that uncertainty into a task a person can start, steer, check, and trust. I think of it as four staples, and if any one of them breaks, the product breaks. This article is the checklist I use. The pattern taxonomy draws from Shape of AI, which I keep open as a reference tool. When something feels wrong in a product, I scan it to locate which category of failure I might be looking at. Three Technical Foundations Give every request a task_id task_id One identifier per user request, threaded through everything: creation, context assembly, model call, rendering, user reaction, errors. At Hints, we learned this early. A user reported that the AI was "randomly updating wrong deals." We opened the logs. Thousands of entries. No way to connect them to each other. We couldn't reconstruct what the AI had seen, what it had done, or why. That week we made task_id a non-optional requirement for every new feature going forward. task_id Log failures, not just successes The dashboard looks great. Users churn anyway. If you're not logging rejections and failed tasks, you are looking at a survivorship bias chart and calling it a success metric. I've sat in review meetings where the dashboard said 94% success rate and the support queue told a completely different story. The gap was rejections, users clicking away without interacting, that nobody was tracking. Separate reads from writes Reading data (pulling a contact from CRM, fetching a document) can happen automatically. Writing data (sending an email, updating a deal, creating a task) requires explicit confirmation. We design AI that integrates with HubSpot, Pipedrive, Jira, Notion and other tools. One accidental CRM update sent to the wrong deal, the rep loses trust, tells the team, and adoption drops across the org in a day. We gate every  action. No exceptions. It may seem conservative until the first time it saves you from a churning customer. Six Pattern Groups for the Full User Journey I organize AI design into six groups based on what the user needs at each stage. They need to start (wayfinders), steer (inputs), ground the AI in context (tuners), stay safe (governors), verify the output (trust builders), and understand what and where the AI is (identifiers). start steer ground the AI in context stay safe verify the output understand what and where the AI is The full taxonomy lives at Shape of AI. Here I'll focus on where I see products fail most often. Wayfinders: The Blank Input Problem User opens the product. Sees a text field. Doesn't know what's possible. Leaves. This is the most common AI product failure and the easiest to fix. Three options here. 1. Contextual suggestions: 3 to 6 starters shown during onboarding, after new contexts, and after pauses. You want to A/B test them and swap by context. At Hints, we noticed that suggestions relevant to what the rep was actually doing (just finished a call, just opened a deal) converted significantly better than generic ones. The difference was large enough that we now consider suggestion relevance a core metric, not a nice-to-have. 2. Templates are the next step: structured fields with defaults instead of open-ended prompts. A sales rep doesn't want to prompt-engineer a CRM update. They want to tap the meeting they just had, confirm the deal stage, and go. Version your templates though. I spent two days tracking down a quality drop that turned out to be a prompt template someone had edited without telling anyone. We now version them in git like code. 3. Follow-up suggestions after the output (short, contextual, one-tap refinements) let people iterate without starting from zero. Keep them tied to the last step. Also, don't try to offer the entire capability menu, offer the three most likely next actions. Inputs: When "Almost Right" Costs More Than "Wrong" The AI often generates something around 80% correct. The only action available is "Regenerate." Now the user gambles. Maybe the next output is worse. They try again, spend 3x the inference cost, and end up with a worse result than attempt one. Inline actions solve this, when you let users select a specific fragment (sentence, paragraph, field) and apply a targeted operation: shorten, rewrite, change tone, translate. While doing so, the best is to show the result as a suggestion next to the original, not overwriting automatically. Then, technically, each edit is a diff, not a full replacement. It's also important to store output and input versions, since concurrent editing conflicts will happen. Tuners: Context, Data, Constraints This is where the product gets powerful and where the security risks live. Attachments (files, URLs, pasted text) anchor responses in specific sources. When the AI references an attachment, cite the exact passage for accuracy, not just the filename. Connectors are where I've seen the biggest problems. We integrate with Slack, Notion, HubSpot, Jira, Pipedrive. Every one of those is an attack surface for prompt injection, OWASP's #1 risk for LLM applications. I've seen someone put ignore all previous instructions in a Jira ticket description, and the AI complied. In production, we now mark all external content as <untrusted> in prompts, gate writes behind confirmation, and sanitize for injection patterns. If you're building connectors and you haven't read the OWASP Top 10 for LLMs, I'd recommend reading it. ignore all previous instructions <untrusted> Parameters (settings: e.g. strictness, cost tradeoffs) need sensible defaults and progressive disclosure for power users. Don't overwhelm a new user with fifteen sliders, make the product work well for them out of the box. If toggling an option doubles the inference cost, say so. Being transparent about cost increases trust. Users don't mind paying more when they understand the value and the why. Governors: The Strict Part When the AI is about to modify real data, the design needs to be strict. Before an expensive or irreversible action, the AI should show an action plan: what it intends to do so the user can scan it and edit it. If execution deviates from the plan, explain why. This sounds obvious but many products skip the "explain deviations" part. Verification is the most under-engineered pattern I encounter. Almost every team implements it as just yes/no buttons. It should be a state machine: pending → confirmed → executed pending → confirmed → executed Plus three additional states: expired, abandoned, rolled_back, three pieces of metadata: expired abandoned rolled_back An idempotency key prevents double execution when someone clicks "Confirm" twice on a slow connection. An expiration timestamp invalidates stale confirmations: so if a user walks away, comes back, they don't confirm something outdated. A data version check then detects when the underlying record changed between proposal and confirmation. The stakes get even higher with agentic AI, which is what we're building into Hints now. When the AI isn't just responding to commands but proactively suggesting actions (follow up with this lead, update this deal stage, flag this stale opportunity), the verification layer has to be airtight. An agent that takes initiative without strict confirmation controls is an agent that eventually does something wrong on your behalf. Same as with attachments, citations need to be able to link to specific source passages, not document titles, in an ideal world, with a version hash and timestamp. Documents change. You need proof that the source matched at citation time. Memory only works if it's transparent. Show what's stored. Separate preferences ("always use bullet points") from facts ("our fiscal year starts in April"). Provide a "memory off" mode. Trust Builders and Identifiers Leave visible traces everywhere the AI touched something. Consistent across the product. Inconsistency in how AI presents itself erodes trust faster than a bad output. How the AI shows up across the product is becoming a real competitive differentiator. And again, being transparent about cost increases trust. Users truly don't mind paying more when they understand why. Four Metrics Four metrics that diagnose whether your AI product is working. Accept rate = is the AI useful at all. Edit rate = how close is the first attempt. Regeneration rate = are users steering or gambling, above ~25% means users are gambling instead of steering (that's a control problem, not a model problem). Confirmation rate = for agentic AI, is the AI proposing relevant actions. As we build the agentic side of Hints, this has become our most watched number. Read them as combinations to pinpoint the problem category. And track cost per task at p99, not the mean. The long tail shows problems weeks before the average moves. Where This Framework Doesn't Work This checklist works best for human-in-the-loop B2B products where the AI generates or transforms content and accuracy matters: CRM, legal, finance, and internal ops. It maps poorly to fully autonomous agents with no human in the loop. Full autonomy is a different design problem. Based on the Shape of AI pattern library and experience designing AI-first products at Hints. Based on the Shape of AI pattern library and experience designing AI-first products at Hints.

This story contains AI-generated text. The author has used AI either for research, to generate outlines, or write the text itself. 

Keep

The Design Checklist: Keeping AI Products Alive After Week One

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

The Noonification: How Often Do NFTs Pass The Howey Test? (1/13/2023)

Darwin's Hybrid Intelligence to Align AI & Human Goals for Startups & VCs

The Noonification: White Man (11/26/2022)

The Noonification: The Metaverse is a Sh*tshow (11/2/2022)

100 Days of AI Day 1: From Newsletter to Podcast, Leveraging AI for Audio Transformation

10 Ways AI Has Changed Our Lives

The Noonification: How Often Do NFTs Pass The Howey Test? (1/13/2023)

Darwin's Hybrid Intelligence to Align AI & Human Goals for Startups & VCs

The Noonification: White Man (11/26/2022)

The Noonification: The Metaverse is a Sh*tshow (11/2/2022)

100 Days of AI Day 1: From Newsletter to Podcast, Leveraging AI for Audio Transformation

10 Ways AI Has Changed Our Lives

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps