Designing an agentic support system

The problem

A regulated e-commerce pharmacy I consulted for sees a steady stream of support conversations, most of them variations of the same five questions — but the domain is sensitive: payments, prescriptions, medical context. A wrong confident answer is worse than no answer.

The team was drowning in status questions ("where is my order? did my payment arrive?") while genuinely hard cases were queued behind them.

Why not the vendor agent

The client was already running their support through HubSpot — happy with the product, less happy with the seat-based pricing. They tried HubSpot's built-in AI auto-resolution and found it too generic for this domain, and with per-resolution pricing (at the time $0.99, now $0.50) it didn't scale either.

So we decided to roll a custom solution: much cheaper per conversation (~$0.05) and under complete control — prompts, analytics, categories, auto-resolve flags per category, all of it.

The architecture

The whole thing lives in a mono-repository — frontend, backend, agent, and an ingestion worker as the four deployable services. The pipeline is roughly:

webhook → classify (category, sentiment, safe-to-auto-send)
  → resolve (extract order number → pull live order status)
  → draft (knowledge base context)
  → guardrails
  → human queue or auto-send

The load-bearing decision was this: order status is a service interface, not agent code. The client runs several different shop systems, and none of them speak the same language. Before we could touch any agent code, we built an ingestion layer that pulls every order from these shop systems into a single, normalized database — payment state, prescription state, shipping milestones, all queryable through one endpoint. The WISMO agent — where-is-my-order — is just its first consumer.

shop A ┐
shop B ┼─ ingestion ─→ normalized order DB ─→ WISMO endpoint ─┬─→ internal chat (team)
shop C ┘               payment · prescription                 ├─→ email agent (HubSpot)
                       · shipping milestones                  ├─→ voice agent (Fonio)
                                                              └─→ next: WhatsApp, chatbot

That was deliberate from the start, not a retrofit. We began with an internal chatbot so the support team could query order states directly, then exposed the WISMO endpoint for the HubSpot agent (authenticated) because we wanted to test HubSpot's built-in AI agent before committing to our own. The eventual consumers were already in mind: WhatsApp Business, a phone agent, a customer-facing chatbot. When the voice agent (Fonio) became channel #2, the go-live took days rather than months because the trustworthy endpoint already existed. Calls started flowing on day one, every transcript stored and analyzed.

The team got a second brain too: the same agent has an internal chat that ingests support SOPs (upload → retrieval → answer). The retrieval logic underneath will be the topic of the follow-up post.

Guardrails

We launched in draft-only mode: the agent wrote, a human reviewed and sent. Once we were happy with what we saw, we started flipping categories to live, one at a time.

The low point came a month or two into live operation. The auto-resolve rate had slipped to around 25% — the goal was 40–50% — and my spot checks kept turning up answers where the bot confidently claimed something that wasn't true. Sometimes I found them myself; sometimes someone from the support team pinged me about one. Every prompt tweak I shipped to fix one case could silently break another, and I had no way of knowing.

That was the moment I stopped treating quality as something to check by hand. Three decisions came out of it: giving feedback had to be effortless for the support team — one click, straight from the frontend they already work in; every prompt change had to be checked for regressions before shipping; and the auto-resolve rate had to move up, not down — measured, not felt.

Auto-send has two gates. A human flips categories live, one at a time. On top of that, the classifier judges every single mail: auto_send_safe can veto a mail even in a live category — mixed topics, legal threats, an upset customer. Held-back drafts show the reason. Wrongly auto-sending costs trust; wrongly holding back costs minutes.

Hallucination-sensitive rules are hard-coded, with both prompt instructions and data-side gates. Payment confirmations are only ever derived from bank-matched data, and a pending bank match below 100% confidence counts as no match — that gate is enforced in code, before the model ever sees the data.

Identity verification is deliberately separated from retrieval. Before any order data enters a response, the customer's identity is verified against a PII-hash match — email, date of birth, postal code. Customer data lives encrypted; lookups run over normalized hashes, not raw PII. All LLM and embedding calls route through EU endpoints — GDPR as an architecture decision, not an afterthought.

The feedback loop is a product feature. Every support agent can flag a bad draft in one click. Flags become frozen test cases. By that same afternoon the incident is red in the test suite, and it's green before the next deploy.

Quality is rated blind, by the support team, in the same UI they already work in. They grade the agent's drafts without seeing what the machine thought of itself, which matters because the grader is not immune to the anchoring bias a good wording induces.

None of this makes the system infallible — some answers still land wrong. The backstop is the customer's reaction: every incoming reply gets classified for sentiment, and if the model is not confident enough to auto-answer, it'll flag the conversation so it goes straight to a human. And where the agent can't verify something, it says so ("I couldn't check this right now") instead of inventing a confident reply.

The measurement discipline underneath all of this — golden sets, baselines, graders — is a story of its own; the follow-up post covers it.

Results, plainly

• 75% of WISMO tickets fully autonomous
• ~54% across the categories where auto-send is enabled (6 of 12 so far — autonomy is granted per category) — past the 40–50% goal, up from the 25% low point
• Per-conversation cost: $0.50 (theirs) vs $0.05 (ours)
• Voice channel: live and taking calls since day one; 61% still end in transfer-to-human — that's the current frontier
• Knowledge context cut by 83–97% through retrieval instead of stuffing, depending on the corpus

Advice — three takeaways

1. Build the data truth first, agents second. Channels multiply for free afterwards.
2. Grant autonomy per category, never globally — and remember: a right answer held back is better than a wrong answer going out to the customer.
3. Make it easy for people reading the answers to give feedback. Your test suite grows out of production, not out of your imagination.