Agent System Blueprint

The blueprint #

Eight layers. A message flows down the spine; knowledge flows in from the sides; nothing ships unverified. Click a layer to jump to its section.

pipeline & knowledge reasoning guardrails & humans

How a turn flows. A message enters through a channel adapter, its modality pipeline normalizes it (persisting media pending-first), and the turn assembler dedupes, filters self-echoes, and batches the burst into one turn under a conversation lock. The reasoning core routes the turn to a capability subset, assembles the prompt from the context engine's four memory classes, and runs a bounded generation loop against the capability system. The loop can only exit through the terminal tool, whose payload — reply blocks plus claims — must pass the verification gate: each claim is audited against the ledgers, failing once triggers a hinted retry, failing twice ships a safe fallback instead. The output system renders the blocks into each channel's native format and stamps a delivery receipt. Escalation transfers the thread to human integration; platform services underpin every layer.

Invariants #

The eight laws that make the blueprint hold together. Break one and a specific failure mode returns.

1. Stateless turn, durable context. No state lives in process memory; every turn reloads from stores. Any instance can serve any turn — crash-safe and horizontally scalable.
2. Single terminal exit. The only way to answer is the terminal tool with a typed contract. Prose outside it is not a reply; "forgot to call it" is a detectable, retryable error.
3. Every assertion needs proof. Claims are audited against ledgers before shipping. An agent that cannot prove "I saved your order" does not get to say it.
4. Pending-first media. Persist a placeholder row immediately, prepare asynchronously, gate the reply on readiness. Slow transcription never drops a message or races the reply.
5. One turn per conversation. A per-conversation lock, a quiet-window debounce, and a drain loop. Bursts become one coherent reply; concurrent webhooks never double-fire.
6. Channel-agnostic core. Adapters exist only at the edges — ingestion and rendering. The reasoning core cannot tell one messaging platform from another.
7. Authenticate, then bind. Credentials are selected by the authenticated secret, never by payload fields. A forged sender id cannot borrow another tenant's identity.
8. Receipts for every side effect. Deliveries, actions, and rejections all leave a queryable record. A send cut off mid-flight is visible, not silently lost.

1 Ingestion layer #

Channel adapters → modality pipelines → turn assembler. Everything after this layer is channel-blind.

Channel adapters

One adapter per messaging platform. Each does exactly three jobs: authenticate the caller (header secret, platform token echo, or per-tenant URL secret — verified constant-time), normalize the payload into the shared conversation model, and acknowledge fast (heavy work deferred past the response when the platform enforces an ack budget). Adapters also own platform quirks: at-least-once redelivery gets an idempotency key; platforms that echo your own outbound messages get a self-echo filter keyed on the authenticated sender identity.

Modality pipelines

Text — platforms often send both a plain and an HTML variant. Prefer the plain field only when non-empty, else strip tags and decode entities. Image — persist pending-first, re-host the file from the platform CDN into your own storage (platform URLs expire), link it to the message, and resolve a signed URL at reasoning time for the vision model. Audio — persist pending-first with a placeholder, then transcribe with a contextual prompt: the business name, a language hint (episodic memory wins over the identity default), domain keywords so jargon isn't misheard, and the last few conversation lines for reference resolution. Everything else — stickers, videos, files become labeled placeholders; a message with no text and no usable attachment is skipped entirely.

pending-first message row

{
  "role": "user",
  "content_type": "voice",
  "content": "(voice message — transcribing)",
  "metadata": {
    "source": "channel_webhook",
    "transcription_status": "pending",   // the reply gate keys on this
    "channel_account_id": "acct_4821…",  // tenant namespace for dedup
    "channel_message_id": "msg_88214…"   // idempotency key (unique index)
  }
}

transcription verdict — structured, not just text

{
  "text": "Quisiera saber el precio de 500 tarjetas de presentación",
  "quality": "ok",                        // "ok" | "uncertain" | "failed"
  "unclear_spans": [],                    // what the model couldn't make out
  "detected_language": "Spanish",         // mismatch → agent adapts + memory learns
  "note": null
}

The verdict drives behavior: ok reads as an ordinary message, uncertain nudges the agent to confirm the unclear spans instead of guessing, failed yields a placeholder and a polite "could you type that?". The audio itself travels as base64 input_audio (format derived from the MIME type) directly to a multimodal model — no separate speech service.

Turn assembler

Three mechanisms turn raw message events into exactly one agent run: a per-conversation lock (losers leave their row pending for the winner), a quiet-window collector (sleep, re-claim, repeat until a window passes silent — a burst becomes one composite turn), and a drain loop (after each reply, re-check for mid-turn arrivals and run another pass). A readiness gate holds the run until every pending media marker resolves.

2 Context engine #

Four memory classes, split by lifespan and mutability. The agent is stateless; its knowledge is layered.

Identity & policy

The distinctive move: behavioral policy is operator-editable plain language, not code. Trigger fields — "when to capture a lead", "when to escalate", "what's out of scope" — are written by the business owner and wired to tools by a static playbook section that also carries the safety scaffolding (never invent prices; fetch before quoting; always end with the terminal tool). Because this half of the prompt changes only on operator edits, it is emitted first and hash-tracked so provider prompt-caching stays effective.

Episodic memory

A cheaper utility model re-summarizes on a cadence — after every few assistant replies, or when the snapshot goes stale — never on the reply's critical path. Every field is hard-capped after parsing so a verbose model can't bloat future prompts. One anti-hallucination rule worth copying: the model records entity names, never ids — ids only ever come from retrieval tools.

episodic snapshot

{
  "summary_text": "Customer runs a boutique café; asked about branded packaging and loyalty cards…",  // ≤900 chars
  "preferred_language": "es",             // learned from voice verdicts / chat
  "last_intent": "place_order",           // ≤80
  "interested_service_names": ["Branded packaging", "Loyalty cards"],  // names, never ids — ≤12
  "contact_preference": "Chat, evenings",                    // ≤120
  "unresolved_question": "Bulk price for 500 units?",        // ≤240
  "next_action": "Send bulk quote after operator confirms",  // ≤180
  "turn_count": 14
}

Working state — the ledgers

Two append-only windows make the verification gate possible. The tool ledger keeps compact summaries of recent tool calls (a small sliding window; state-replacing tools overwrite their previous entry so the window always holds the latest facts, and stale entries fall out — forcing a re-fetch instead of a stale quote). The action ledger records transactional events in a separate, more durable window that survives pruning, so "I've saved your order" can still be proven several turns after the capture.

tool ledger entry + action ledger entry

// tool ledger — compact summary, replaced when the same tool runs again
{ "tool_name": "get_service_detail",
  "summary_json": { "service_id": "9b2f…", "name": "Branded packaging",
                    "price": "$1.80 / unit", "price_is_set": true,
                    "turnaround_days": 5, "min_order_quantity": 100 } }

// action ledger — durable proof of a side effect ("__action__" prefix survives pruning)
{ "tool_name": "__action__lead_captured",
  "summary_json": { "follow_up_id": "f3a1…", "intent": "place_order" } }

3 Reasoning core #

Route → assemble → generate. Bounded everywhere; the model can be wrong, but never unbounded.

Turn router

A fast, model-free classifier (regex over the latest message, in every supported language, plus conversation state) maps each turn to an intent class, and each class unlocks only the capabilities that turn needs. This is a cost and a safety measure — a greeting turn physically cannot capture a lead. Typical classes: greeting · browse · detail question · capture (contact info appeared with an item in context) · post-capture · explicit human request. Conversation state feeds routing too: an open task routes to post-capture; a recent detail fetch makes contact info mean "capture now".

Prompt assembler

Two segments, strictly ordered: the static prefix (identity & policy — cacheable, hash-tracked so cache drift is observable) and the dynamic tail (clock in the business timezone, entity profile, open tasks, episodic snapshot, tool ledger, route hint, tripwire signal, and — on retry — the precise rejection hint). Dynamic content never goes above static content; that would invalidate the provider's prompt cache every turn.

Generation loop

The model runs with the route's tool allowlist under three bounds: a step ceiling (≈8 tool steps), a stop condition (the moment the terminal tool fires), and a retry budget (one, with a targeted hint). After every step, the tool ledger is updated — that's how working state stays current mid-turn. Two recovery paths handle model sloppiness: a missing-terminal retry ("you must call it exactly once") and salvage parsing that recovers a reply the model wrote as prose or slightly-malformed JSON instead of a clean tool call.

4 Capability system #

Categorize tools by contract, not by domain. Domains change per project; contracts don't.

Three design rules that generalize:

• Operator-language fields. Every operator-facing string a tool writes (lead summaries, handoff reasons, notes) is written in the operator's configured language, regardless of the customer's — the console always reads natively.
• Evidence capture. Tools that surface citable content (photos, prices) record what they returned in per-turn state, so the verification gate can check the reply only cites what was actually fetched.
• Compound actions are one tool. "Capture a lead" resolves the entity, saves the contact, opens a task, and notifies — as a single atomic tool, not four calls the model must sequence correctly.

5 Verification gate #

The model files a typed report of what its reply asserts. The report is audited before anything ships.

The terminal tool's payload carries a claims array — one typed object per evidence-requiring assertion. Two claim families, two proof sources:

• Event claims ("I did X this turn") — proven by per-turn flags set when the action tool actually ran, or by an action-ledger entry from a recent turn.
• Reference claims ("I'm showing/quoting X") — proven by the tool ledger: the cited id must appear in what retrieval actually returned.

claim contract — the union

claims: [
  { "type": "lead_captured" },                              // event — capture_lead ran
  { "type": "handoff_triggered" },                          // event — refer_to_human ran
  { "type": "service_presented", "service_id": "9b2f…" },   // reference — id ∈ tool ledger
  { "type": "business_info_stated" },                       // reference — info block was loaded
  { "type": "follow_up_updated", "follow_up_id": "f3a1…" }, // event — updated this turn
  { "type": "customer_info_updated" },                      // event — ran this turn
  { "type": "images_shown", "photo_ids": ["ph1…", "ph2…"] } // reference — fetched this turn
]

The repair loop. An ungrounded claim rejects the whole reply. The agent gets one retry with a hint naming exactly which claim failed and why. A second failure ships a safe fallback — a polite hold message — instead of a fabrication, and the rejection is logged for the observability surface. The claims the model omits are policed too: prompt rules require declaring any evidence-bearing assertion, so an undeclared price quote is a prompt violation the validator's companion rules catch (price text must trace to a fetched row).

6 Output system #

One typed reply contract, rendered per channel, with a receipt for every send.

Reply contract

The reply is an ordered list of 1–8 typed blocks — a "message bubble" and an "image group" are the two block types. Text is length-capped per bubble; an image group bundles 1–10 URLs (which must trace to fetched photos) with an optional caption. Blocks are objects with an explicit discriminator, never bare strings — that single schema rule eliminates a whole class of malformed output.

reply contract — terminal tool payload

{
  "blocks": [
    { "type": "text",
      "text": "¡Hola! Aquí tienes algunas muestras de empaques personalizados." },  // ≤2000 chars
    { "type": "image_group",
      "caption": "Kraft boxes with gold-foil logo",                     // optional, ≤500
      "image_urls": ["https://cdn…/ph1.png", "https://cdn…/ph2.png"] }  // 1–10, claims-checked
  ],
  "claims": [
    { "type": "service_presented", "service_id": "9b2f…" },
    { "type": "images_shown", "photo_ids": ["ph1…", "ph2…"] }
  ]
}

Channel renderers

The same blocks compile to each platform's native shape. One renderer per channel, all lossless for the two block types:

same blocks → an album-capable bot API

// text block → sendMessage
{ "chat_id": 88214, "text": "¡Hola! Aquí tienes algunas muestras…" }

// image group → sendMediaGroup (native album; caption rides the first item)
{ "chat_id": 88214,
  "media": [
    { "type": "photo", "media": "https://cdn…/ph1.png",
      "caption": "Kraft boxes with gold-foil logo" },
    { "type": "photo", "media": "https://cdn…/ph2.png" }
  ] }

• Album-capable platforms — text maps to the message call; an image group maps to the native album call, so multi-image replies arrive as one bubble.
• Aggregator platforms — upload media in parallel, then attach with one combined call, so a multi-image reply is atomic: one media bubble, no half-sent groups.
• In-app webchat — blocks are inserted as message rows; the widget renders them live from a realtime subscription. Delivery is the database itself.

Delivery receipts

Out-of-band channels (where the reply is sent by API call, not returned in the webhook response) stamp a receipt on the assistant message around the send. A process killed mid-send leaves "sending" — visible in the console as "Not delivered / Partly delivered" instead of silently looking fine in the database. No blind auto-retry: without a platform-side idempotency key, a resend risks double-posting, so failures surface for a deliberate replay.

delivery receipt

"metadata": {
  "delivery": {
    "status": "partial",          // sending → sent | partial | failed
    "channel": "messenger",
    "account_id": "acct_4821…",
    "at": "2026-07-09T04:12:31Z",
    "sent_text": 1, "sent_image": 2,
    "errors": ["image(https://…/ph3.png): upload failed (…)"]
  }
}

7 Human integration #

The agent must know when to stop being the agent — and the operator must never be surprised.

• Handoff protocol. The escalation tool disables AI for that one thread (never globally), records the reason in the operator's language, opens a high-priority task, and pings the operator over a dedicated notification channel — deduped per run so nobody is double-paged. The agent still sends a final courtesy message so the customer isn't left hanging.
• Console. The operator configures the brain (identity & policy, domain catalog, media galleries, credentials) and monitors it (conversations with per-thread AI toggle, tasks, delivery logs, claim rejections, audit trail) in one place. Everything the agent writes for the operator arrives in the operator's language.
• Notification routing. Typed event kinds with a subscriber-resolution order: explicit recipients → kind-level subscriptions → entity-level assignments → all active subscribers. Delivery is fire-and-forget with a logged attempt trail.

8 Platform services #

The substrate every layer leans on.

• Model gateway. A single provider abstraction with named slots — chat (multimodal: text + vision + audio-in), utility (cheap background work: summaries, captions), embeddings (semantic search). Slots are operator-configurable; code never names a vendor model.
• Credential vault. Per-provider credentials encrypted at rest (AES-256-GCM envelopes), a short-TTL read cache busted on writes, environment fallback for local development, and one hard rule: secrets never enter any client-bound projection.
• Persistence. One relational store for conversations, messages, entities, tasks, memory, ledgers — plus object storage for re-hosted media and a vector index for semantic search. Unique indexes are the last line of idempotency (a duplicate insert must fail loudly, not merge silently).
• Observability. Per-turn traces (route, prompt sizes, cache-prefix hash, batch shape), claim-rejection logs, webhook delivery logs with sanitized headers, delivery receipts, and an operator audit trail. The rule of thumb: every "why did it do that?" must be answerable from stored data.

Implementation checklist #

Blueprint component → what you build. Eighteen pieces; each is small, and the contracts keep them independent.