13 KiB
Phase 3: Operator Experience Verification Report
Phase Goal: An operator can sign up, onboard their tenant through a web UI, connect their messaging channels, configure their AI employee, and manage their subscription — without touching config files or the command line Verified: 2026-03-24T00:00:00Z Status: human_needed Re-verification: Yes — after gap closure via plan 03-05
Summary of Gap Closure
All three automated-verifiable gaps from the initial verification have been closed:
Gap 1 (CLOSED) — Backend routers mounted. packages/gateway/gateway/main.py now imports portal_router, billing_router, channels_router, llm_keys_router, usage_router, and webhook_router from shared.api (lines 43-50) and registers all six with app.include_router() at lines 110-115. The module docstring was updated to document these routes. All Phase 3 backend APIs are now reachable at runtime.
Gap 2 (CLOSED) — Slack OAuth field names fixed.
packages/portal/app/api/slack/callback/route.tsline 44 now readsif (!data.ok)(wasif (!data.success)) — Slack OAuth callback now correctly identifies success.packages/portal/lib/api.tsSlackInstallResponseinterface (line 161-164) now declaresurl: stringinstead ofauthorize_url: string.packages/portal/app/(dashboard)/onboarding/steps/connect-channel.tsxreferencesslackInstallData?.urlthroughout (lines 79, 81, 135, 139) — the Add to Slack button is now enabled when the backend returns an install URL.
Gap 3 (CLOSED) — Budget alerts field name fixed.
packages/portal/lib/queries.tsBudgetAlertinterface (lines 224-230) now declarescurrent_usd: number.packages/portal/app/(dashboard)/usage/[tenantId]/page.tsxline 268 now referencesalert.current_usd.toFixed(2)— budget amounts render correctly.
Goal Achievement
Observable Truths
| # | Truth | Status | Evidence |
|---|---|---|---|
| 1 | Operator can connect Slack and WhatsApp via guided in-portal wizard | VERIFIED | Slack: connect-channel.tsx uses slackInstallData?.url (line 79/81/135), callback checks data.ok (line 44). WhatsApp: form submits to /api/portal/channels/whatsapp/connect. Both paths wired end-to-end. |
| 2 | New tenant completes onboarding (connect -> configure -> test) in under 15 minutes | UNCERTAIN | 3-step wizard exists and is fully wired. All blocker bugs removed. Time budget requires human verification. |
| 3 | Operator can subscribe, upgrade, cancel via Stripe — limits enforced automatically | VERIFIED | Billing page, SubscriptionCard, BillingStatus, mutation hooks all exist. All billing routers now mounted at /api/portal/billing/*. |
| 4 | Portal displays per-tenant agent cost and token usage without backend log access | VERIFIED | Usage dashboard with Recharts charts, all 4 query hooks wired. Budget alerts now use current_usd (matches backend). Usage router mounted at /api/portal/usage/*. |
Score: 18/18 individual must-have items verified (0 automated gaps remaining)
Required Artifacts
| Artifact | Status | Details |
|---|---|---|
packages/shared/shared/models/billing.py |
VERIFIED | TenantLlmKey (RLS, UNIQUE provider/tenant), StripeEvent models — substantive and complete |
packages/shared/shared/crypto.py |
VERIFIED | KeyEncryptionService with MultiFernet encrypt/decrypt/rotate — substantive |
packages/shared/shared/api/channels.py |
VERIFIED | Full Slack OAuth + WhatsApp connect + test endpoint — substantive |
packages/shared/shared/api/llm_keys.py |
VERIFIED | Full LLM key CRUD — substantive |
packages/shared/shared/api/billing.py |
VERIFIED | Stripe Checkout, Billing Portal, webhook handler — substantive |
packages/shared/shared/api/usage.py |
VERIFIED | All 4 usage endpoints with JSONB SQL aggregation — substantive |
migrations/versions/005_billing_and_usage.py |
VERIFIED | Adds all billing columns, tenant_llm_keys table, stripe_events table, composite index, RLS, grants |
packages/portal/app/api/slack/callback/route.ts |
VERIFIED | Checks data.ok (line 44) — OAuth success check now matches backend response |
packages/portal/app/(dashboard)/onboarding/page.tsx |
VERIFIED | 3-step stepper with URL searchParam-driven step state |
packages/portal/app/(dashboard)/onboarding/steps/connect-channel.tsx |
VERIFIED | Uses slackInstallData?.url throughout; WhatsApp form wired correctly |
packages/portal/app/(dashboard)/settings/api-keys/page.tsx |
VERIFIED | BYO key list, add dialog, delete with confirmation — wired to correct endpoints |
packages/portal/app/(dashboard)/billing/page.tsx |
VERIFIED | Subscription card, past-due banner, checkout success toast — substantive |
packages/portal/components/subscription-card.tsx |
VERIFIED | Status-driven action buttons, agent count adjuster, plan pricing |
packages/portal/components/billing-status.tsx |
VERIFIED | 6-state color-coded badge component |
packages/portal/app/(dashboard)/usage/[tenantId]/page.tsx |
VERIFIED | Budget alerts table uses alert.current_usd (line 268) — matches backend field |
packages/portal/components/usage-chart.tsx |
VERIFIED | Recharts BarChart with agent token breakdown |
packages/portal/components/provider-cost-chart.tsx |
VERIFIED | Horizontal BarChart by provider |
packages/portal/components/budget-alert-badge.tsx |
VERIFIED | ok/warning/exceeded badge |
packages/portal/components/message-volume-chart.tsx |
VERIFIED | BarChart by channel |
packages/gateway/gateway/main.py |
VERIFIED | Imports and mounts all 6 Phase 3 routers (lines 43-50, 110-115) |
Key Link Verification
| From | To | Via | Status | Details |
|---|---|---|---|---|
packages/orchestrator/orchestrator/agents/runner.py |
shared/models/audit.py |
log_llm_call metadata | WIRED | prompt_tokens, completion_tokens, cost_usd, provider all present in metadata dict |
packages/shared/shared/api/usage.py |
audit_events table | JSONB aggregate queries on metadata | WIRED | SQL queries on metadata->>'prompt_tokens', metadata->>'cost_usd' present |
packages/shared/shared/crypto.py |
PLATFORM_ENCRYPTION_KEY env var | MultiFernet key loaded at init | WIRED | MultiFernet present, reads settings.platform_encryption_key |
packages/shared/shared/api/llm_keys.py |
packages/shared/shared/crypto.py |
KeyEncryptionService.encrypt() on POST | WIRED | encrypt() called on api_key before storage; GET never decrypts |
packages/portal/app/(dashboard)/settings/api-keys/page.tsx |
/api/portal/tenants/{tenant_id}/llm-keys |
GET/POST/DELETE | WIRED | useLlmKeys, useAddLlmKey, useDeleteLlmKey hooks call correct endpoints |
packages/portal/app/(dashboard)/onboarding/steps/connect-channel.tsx |
/api/portal/channels/slack/install |
fetch to get OAuth URL, then window.location redirect | WIRED | useSlackInstallUrl fetches correct endpoint; response field is url; button uses slackInstallData?.url |
packages/portal/app/api/slack/callback/route.ts |
/api/portal/channels/slack/callback |
proxy the OAuth callback to FastAPI backend | WIRED | URL proxy correct; checks data.ok which matches backend {ok: true} response |
packages/portal/app/(dashboard)/onboarding/steps/test-message.tsx |
/api/portal/channels/{tenant_id}/test |
POST to send test message | WIRED | useSendTestMessage hook calls correct endpoint pattern |
packages/portal/app/(dashboard)/billing/page.tsx |
/api/portal/billing/checkout |
POST to create Checkout Session | WIRED | useCreateCheckoutSession mutation calls billing/checkout |
packages/portal/app/(dashboard)/billing/page.tsx |
/api/portal/billing/portal |
POST to create Billing Portal session | WIRED | useCreateBillingPortalSession mutation calls billing/portal |
packages/portal/app/(dashboard)/usage/[tenantId]/page.tsx |
/api/portal/usage/{tenantId}/summary |
TanStack Query hook | WIRED | useUsageSummary calls correct endpoint |
packages/portal/app/(dashboard)/usage/[tenantId]/page.tsx |
/api/portal/usage/{tenantId}/budget-alerts |
TanStack Query hook | WIRED | useBudgetAlerts calls correct endpoint; BudgetAlert.current_usd matches backend field |
| All portal API routes | gateway.main:app |
FastAPI include_router | WIRED | billing_router, channels_router, llm_keys_router, usage_router, portal_router, webhook_router all mounted (main.py lines 110-115) |
Requirements Coverage
| Requirement | Source Plan | Description | Status | Evidence |
|---|---|---|---|---|
| AGNT-07 | 03-01, 03-04 | Agent token usage tracked per-agent per-tenant with configurable budget limits | SATISFIED | budget_limit_usd field, compute_budget_status, usage aggregation endpoints all exist; usage_router mounted at lines 113; BudgetAlert.current_usd field name matches across stack |
| LLM-03 | 03-01, 03-02 | Tenant can provide BYO API keys (encrypted at rest) | SATISFIED | Encryption service, CRUD endpoints, and frontend UI all substantive and wired; llm_keys_router mounted at line 113 |
| PRTA-03 | 03-01, 03-02 | Operator can connect messaging channels via guided wizard | SATISFIED | WhatsApp path complete and wired; Slack path fully wired (field name bugs resolved, channel router mounted) |
| PRTA-04 | 03-02 | New tenants guided through structured onboarding | SATISFIED | 3-step wizard exists, is substantive, and all blocking wiring bugs resolved |
| PRTA-05 | 03-01, 03-03 | Operator can manage subscription and billing via Stripe | SATISFIED | Full UI and backend exists; billing_router and webhook_router mounted; Stripe integration complete |
| PRTA-06 | 03-01, 03-04 | Portal displays agent cost tracking and usage metrics | SATISFIED | Full dashboard exists; budget alerts display uses correct field name; usage_router mounted |
All 6 Phase 3 requirements are now SATISFIED. No orphaned requirements.
Anti-Patterns Found
No blocker anti-patterns remain. The three blocker patterns identified in the initial verification have all been resolved.
Human Verification Required
1. WhatsApp Test Message Delivery
Test: Connect a WhatsApp Business number, complete onboarding to Step 3, click "Send Test Message" for WhatsApp Expected: A real message appears in the configured WhatsApp Business account Why human: The test endpoint validates token config but does not send an actual WhatsApp message (channels.py line 471-476 explicitly skips the send). A human needs to determine if this is acceptable behavior or requires a real send.
2. Stripe Checkout End-to-End
Test: Using Stripe test mode keys, click Subscribe on the billing page, complete checkout with test card Expected: Portal redirects to Stripe checkout, payment processes, subscription activates, agent_quota updates, portal shows "active" status Why human: Requires live Stripe test keys, external browser flow, and webhook delivery — cannot verify from static analysis
3. Onboarding Time Budget
Test: Starting from a new tenant, complete the full onboarding sequence Expected: Connection, agent configuration, and test message complete in under 15 minutes Why human: Time-bounded user experience goal requires human execution
Gaps Summary
No automated gaps remain. All three previously-identified blockers are closed.
The only remaining items require human verification with live credentials and browser interaction (Stripe checkout flow, WhatsApp actual message delivery, onboarding time). These are inherently untestable from static code analysis and were flagged in the initial verification as well.
Verified: 2026-03-24T00:00:00Z Verifier: Claude (gsd-verifier)