adelorenzo/konstruct
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(03): research operator experience phase

Browse Source
This commit is contained in:
Adolfo Delorenzo
2026-03-23 20:54:13 -06:00
parent a8f48df305
commit c4ebcf0de4
1 changed files with 769 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

769
.planning/phases/03-operator-experience/03-RESEARCH.md Normal file
View File

@@ -0,0 +1,769 @@
# Phase 3: Operator Experience - Research
**Researched:** 2026-03-23
**Domain:** Slack OAuth V2, Stripe Subscriptions, BYO API Key Encryption, Cost Dashboard
**Confidence:** HIGH (core stack verified against official docs)
---
<user_constraints>
## User Constraints (from CONTEXT.md)
### Locked Decisions
- Slack connection via standard OAuth2 "Add to Slack" flow — operator clicks button, authorizes, tokens stored automatically
- WhatsApp connection: guided manual setup (Claude's discretion confirmed)
- After connecting a channel, the wizard MUST include a "send test message" step — required, not optional
- Test message verifies end-to-end connectivity before the agent goes live
- Onboarding sequence: Connect Channel → Configure Agent → Send Test Message
- Agent goes live automatically after the test message succeeds — no separate "Go Live" button
- Pricing model: per-agent monthly (e.g., $49/agent/month)
- 14-day free trial with full access, credit card required upfront
- Subscription management via Stripe: subscribe, upgrade (add agents), downgrade (remove agents), cancel
- LLM-03 resolved: BYO API keys IS in v1 scope (Phase 3)
- Cost metrics: token usage per agent, cost breakdown by LLM provider, message volume per agent/channel, budget alerts
- Budget alerts: visual indicator when approaching or exceeding per-agent budget limits (from AGNT-07)
### Claude's Discretion
- WhatsApp connection method (guided manual vs embedded signup)
- Stepper UI for onboarding (yes/no, visual style)
- Non-payment enforcement behavior
- BYO key scope (tenant-level settings page vs per-agent)
- Cost dashboard time range options
- Dashboard chart library (recharts, nivo, etc.)
- Stripe webhook event handling strategy (idempotency, retry)
### Deferred Ideas (OUT OF SCOPE)
None — discussion stayed within phase scope
</user_constraints>
---
<phase_requirements>
## Phase Requirements
| ID | Description | Research Support |
|----|-------------|-----------------|
| AGNT-07 | Agent token usage tracked per-agent per-tenant with configurable budget limits | Audit event JSONB metadata must store `prompt_tokens`, `completion_tokens`, `provider`; budget stored on Tenant model; alert threshold query pattern documented |
| LLM-03 | Tenant can provide their own API keys for supported LLM providers (BYO keys, encrypted at rest) | Fernet AES-128-CBC with HMAC-SHA256; envelope encryption pattern; new `tenant_llm_keys` table; LiteLLM routing integration |
| PRTA-03 | Operator can connect messaging channels (Slack, WhatsApp) via guided wizard | Slack OAuth V2 flow; required scopes; token storage in `channel_connections.config`; WhatsApp manual setup steps |
| PRTA-04 | New tenants are guided through structured onboarding (connect channel, configure agent, test message) | Stepper UI pattern; Next.js App Router multi-step page; test message endpoint |
| PRTA-05 | Operator can manage subscription plans and billing via Stripe integration | Stripe Checkout with per-seat quantity; Billing Portal for self-service; webhook event map; idempotency pattern |
| PRTA-06 | Portal displays agent cost tracking and usage metrics per tenant | SQL aggregate query on audit_events; JSONB path extraction; Recharts for visualization; time-range filtering |
</phase_requirements>
---
## Summary
Phase 3 adds the commercial and operational layer to the Konstruct portal: Slack OAuth, subscription billing, BYO key encryption, and a cost dashboard. All four areas are well-trodden territory with mature libraries — the risks are in integration details, not algorithmic complexity.
The largest architectural gap is in the audit trail: the existing `audit_events.metadata` JSONB field stores `model` and `iteration` but NOT `prompt_tokens`, `completion_tokens`, or `cost_usd`. These fields must be added to the audit logger before the cost dashboard can function. This is a prerequisite for PRTA-06 and AGNT-07 and needs to be Wave 0 work.
The second important finding is that WhatsApp Embedded Signup (Meta OAuth flow) is now the standard for BSP-level onboarding in 2026, but it requires a registered Facebook Business Verification and a BSP/Tech Provider program account. For v1 "guided manual setup" is the correct choice — it means operators manually create a WhatsApp Business App, get their phone number token, and paste credentials into the portal. This avoids the multi-week Meta verification process while shipping.
**Primary recommendation:** Build Slack OAuth → Stripe billing → BYO key encryption → cost dashboard in that order. Each is independently deployable. Start with the audit trail metadata migration as Wave 0.
---
## Standard Stack
### Core
| Library | Version | Purpose | Why Standard |
|---------|---------|---------|--------------|
| `stripe` (Python) | `>=12.0.0` | Stripe API, webhook verification, subscription management | Official Stripe Python SDK; `StripeClient` pattern is current API |
| `cryptography` (Python) | `>=47.0.0` | BYO key encryption via Fernet | pyca/cryptography is the Python standard; already used for bcrypt via `bcrypt` dep; Fernet is audited |
| `slack-bolt` (Python) | `>=1.22.0` | Slack OAuth installer, Events API | Already in CLAUDE.md tech stack; `OAuthFlow` handles token exchange |
| `stripe` (npm) | `>=17.0.0` | Stripe.js for frontend Checkout redirect | Official JS client |
| `recharts` | `>=2.15.0` | Cost dashboard charts | 17M weekly downloads vs Nivo's 2M; simpler JSX API; strong shadcn/ui alignment |
### Supporting
| Library | Version | Purpose | When to Use |
|---------|---------|---------|-------------|
| `@stripe/stripe-js` | `>=5.0.0` | Stripe Checkout redirect from browser | When creating Checkout Sessions from portal |
| `slack-sdk` (Python) | `>=3.35.0` | Lower-level Slack Web API calls (post test message) | For the "send test message" verification step |
### Alternatives Considered
| Instead of | Could Use | Tradeoff |
|------------|-----------|----------|
| Fernet (AES-128-CBC + HMAC) | AES-256-GCM via `cryptography.hazmat` | AES-256-GCM is stronger but requires manual MAC management; Fernet is audited, has MultiFernet key rotation, and AES-128-CBC + HMAC-SHA256 is sufficient for API key protection |
| Recharts | Nivo | Nivo has more chart types but 8x fewer downloads, worse documentation, and verbose API; Recharts is recommended for SaaS admin dashboards |
| Stripe Billing Portal (hosted) | Custom billing UI | Custom UI requires full payment method management; Billing Portal handles card updates, invoice history, cancellation in a Stripe-hosted page — use it |
**Installation:**
```bash
# Python (add to packages/shared/pyproject.toml)
uv add stripe cryptography
# Node (in packages/portal)
npm install recharts @stripe/stripe-js stripe
```
---
## Architecture Patterns
### Recommended Project Structure (new files only)
```
packages/
├── shared/
│ └── shared/
│ ├── models/
│ │ └── billing.py # TenantBilling, TenantLlmKey models
│ └── api/
│ ├── billing.py # Stripe webhooks + subscription endpoints
│ └── channels.py # Slack OAuth callback, channel connection
├── portal/
│ └── app/
│ ├── api/
│ │ └── slack/
│ │ └── callback/
│ │ └── route.ts # Slack OAuth redirect handler
│ └── (dashboard)/
│ ├── onboarding/
│ │ └── page.tsx # Connect Channel → Configure Agent → Test
│ ├── billing/
│ │ └── page.tsx # Subscription status + Billing Portal redirect
│ ├── usage/
│ │ └── [tenantId]/
│ │ └── page.tsx # Cost dashboard per tenant
│ └── settings/
│ └── api-keys/
│ └── page.tsx # BYO key management
migrations/
└── versions/
├── xxxx_add_billing_fields.py # stripe_customer_id, subscription_status, trial_ends_at on tenants
├── xxxx_add_tenant_llm_keys.py # tenant_llm_keys table
└── xxxx_add_token_fields.py # prompt_tokens, completion_tokens, cost_usd, provider on audit_events
```
### Pattern 1: Slack OAuth V2 Flow
**What:** Operator clicks "Add to Slack" → Slack authorization page → redirect back to portal callback → exchange code for bot token → store in `channel_connections`
**Scopes required (bot):**
- `app_mentions:read` — receive @mention events
- `channels:read` — list public channels
- `channels:history` — read channel message history
- `chat:write` — post messages (required for test message + agent replies)
- `groups:read` — private channels
- `im:read` / `im:write` / `im:history` — DM support
- `mpim:read` / `mpim:history` — multi-party DMs
**OAuth V2 flow:**
```
1. Operator visits /onboarding → clicks "Add to Slack"
2. Portal redirects to:
https://slack.com/oauth/v2/authorize
?client_id=<SLACK_CLIENT_ID>
&scope=app_mentions:read,channels:read,channels:history,chat:write,im:read,im:write,im:history
&redirect_uri=https://app.konstruct.ai/api/slack/callback
&state=<csrf_token:tenant_id>
3. User approves → Slack redirects to /api/slack/callback?code=xxx&state=yyy
4. FastAPI backend exchanges code:
POST https://slack.com/api/oauth.v2.access
client_id, client_secret, code, redirect_uri
5. Response contains:
{
"ok": true,
"access_token": "xoxb-...", ← bot token, store encrypted
"team": { "id": "T12345", "name": "Acme Corp" },
"bot_user_id": "U67890",
"scope": "app_mentions:read,..."
}
6. Store in channel_connections:
- channel_type: "slack"
- workspace_id: team.id
- config: { "bot_token": encrypt(access_token), "bot_user_id": ..., "team_name": ... }
```
**State parameter** must encode `tenant_id` + CSRF token (sign with HMAC-SHA256, verify on callback).
```python
# Source: https://docs.slack.dev/authentication/installing-with-oauth/
# Generate state
import hmac, hashlib, secrets, json, base64
def generate_oauth_state(tenant_id: str, secret: str) -> str:
nonce = secrets.token_urlsafe(16)
payload = json.dumps({"tenant_id": tenant_id, "nonce": nonce})
sig = hmac.new(secret.encode(), payload.encode(), hashlib.sha256).hexdigest()
return base64.urlsafe_b64encode(f"{payload}:{sig}".encode()).decode()
def verify_oauth_state(state: str, secret: str) -> str:
"""Returns tenant_id or raises ValueError."""
decoded = base64.urlsafe_b64decode(state.encode()).decode()
payload_str, sig = decoded.rsplit(":", 1)
expected = hmac.new(secret.encode(), payload_str.encode(), hashlib.sha256).hexdigest()
if not hmac.compare_digest(sig, expected):
raise ValueError("Invalid OAuth state")
return json.loads(payload_str)["tenant_id"]
```
### Pattern 2: Stripe Per-Agent Subscription
**What:** Operator subscribes → Checkout Session created with quantity=agent_count → redirected to Stripe → on success webhook, provision access.
**Key objects to persist on Tenant:**
- `stripe_customer_id` (String) — created once per tenant on first subscription
- `stripe_subscription_id` (String | None)
- `stripe_subscription_item_id` (String | None) — needed for quantity updates
- `subscription_status` (Enum: `trialing`, `active`, `past_due`, `canceled`, `unpaid`)
- `trial_ends_at` (DateTime | None)
- `agent_quota` (Integer) — number of paid seats
**Checkout Session creation (Python):**
```python
# Source: https://docs.stripe.com/payments/checkout/build-subscriptions
import stripe
client = stripe.StripeClient(api_key=settings.stripe_secret_key)
session = client.v1.checkout.sessions.create({
"mode": "subscription",
"customer": tenant.stripe_customer_id, # or create new
"line_items": [{
"price": settings.stripe_per_agent_price_id,
"quantity": agent_count, # number of agents being subscribed
}],
"subscription_data": {
"trial_period_days": 14,
},
"success_url": f"{settings.portal_url}/billing?session_id={{CHECKOUT_SESSION_ID}}",
"cancel_url": f"{settings.portal_url}/billing",
})
# Return session.url to frontend for redirect
```
**Quantity update when agents are added/removed:**
```python
# Source: https://docs.stripe.com/api/subscription_items/update?lang=python
client.v1.subscription_items.update(
tenant.stripe_subscription_item_id,
{"quantity": new_agent_count},
)
```
**Billing Portal session:**
```python
# Source: https://docs.stripe.com/customer-management/integrate-customer-portal
portal_session = client.v1.billing_portal.sessions.create({
"customer": tenant.stripe_customer_id,
"return_url": f"{settings.portal_url}/billing",
})
# Return portal_session.url to frontend
```
### Pattern 3: Stripe Webhook Handler
**Critical webhook events to handle:**
| Event | Action |
|-------|--------|
| `checkout.session.completed` | Store `subscription_id`, `subscription_item_id`, set status `trialing` or `active` |
| `customer.subscription.created` | Same as above if not using Checkout |
| `customer.subscription.updated` | Update `subscription_status`, `agent_quota`, `trial_ends_at` |
| `customer.subscription.deleted` | Set status `canceled`, deactivate all agents |
| `customer.subscription.trial_will_end` | Send alert email (3 days before trial ends) |
| `invoice.paid` | Set status `active`, re-enable agents if they were suspended |
| `invoice.payment_failed` | Set status `past_due`, send payment failure notification |
**FastAPI webhook endpoint:**
```python
# Source: https://docs.stripe.com/webhooks
from fastapi import APIRouter, Request, HTTPException
import stripe
webhook_router = APIRouter()
@webhook_router.post("/webhooks/stripe")
async def stripe_webhook(
request: Request,
session: AsyncSession = Depends(get_session),
) -> dict[str, str]:
payload = await request.body()
sig_header = request.headers.get("stripe-signature", "")
try:
event = stripe.WebhookEvent.construct_from(
stripe.Webhook.construct_event(
payload, sig_header, settings.stripe_webhook_secret
).to_dict(),
stripe.api_key,
)
except stripe.SignatureVerificationError:
raise HTTPException(status_code=400, detail="Invalid signature")
# Idempotency: check if event already processed
already_processed = await _check_event_processed(session, event["id"])
if already_processed:
return {"status": "already_processed"}
await _record_event_processed(session, event["id"])
await _dispatch_event(session, event)
return {"status": "ok"}
```
**Idempotency table:** Add a `stripe_events` table with `(event_id PRIMARY KEY, processed_at)` — INSERT with ON CONFLICT DO NOTHING; if 0 rows affected, skip processing.
**Non-payment enforcement:** When `subscription_status` becomes `past_due` after grace period (configurable, suggest 7 days), set `Agent.is_active = False` for all tenant agents. The gateway/orchestrator already gates on `is_active`, so no further changes needed.
### Pattern 4: BYO API Key Encryption (Envelope Encryption)
**What:** Tenant provides their OpenAI/Anthropic API key. We encrypt it before storing. The platform-level master encryption key is in environment variables (or secrets manager).
**Important:** Fernet uses AES-128-CBC + HMAC-SHA256, NOT AES-256. This is still cryptographically sound and the `cryptography` library is audited. CLAUDE.md specifies "AES-256" aspirationally — Fernet is the correct practical choice. Document this tradeoff in ADR-005.
**Schema — new table `tenant_llm_keys`:**
```sql
CREATE TABLE tenant_llm_keys (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
tenant_id UUID NOT NULL REFERENCES tenants(id) ON DELETE CASCADE,
provider TEXT NOT NULL, -- 'openai' | 'anthropic' | 'custom'
label TEXT NOT NULL, -- human-readable name
encrypted_key TEXT NOT NULL,
key_version INT NOT NULL DEFAULT 1, -- for rotation tracking
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
UNIQUE(tenant_id, provider) -- one key per provider per tenant
);
-- RLS enabled: same pattern as agents table
```
**Encryption service:**
```python
# Source: https://cryptography.io/en/latest/fernet/
from cryptography.fernet import Fernet, MultiFernet
import os
class KeyEncryptionService:
"""
Encrypts/decrypts tenant BYO API keys.
PLATFORM_ENCRYPTION_KEY env var must be a URL-safe base64 Fernet key.
For rotation: PLATFORM_ENCRYPTION_KEY_PREVIOUS holds the prior key.
"""
def __init__(self) -> None:
primary = Fernet(os.environ["PLATFORM_ENCRYPTION_KEY"])
keys = [primary]
if prev := os.environ.get("PLATFORM_ENCRYPTION_KEY_PREVIOUS"):
keys.append(Fernet(prev))
self._fernet = MultiFernet(keys)
def encrypt(self, plaintext: str) -> str:
return self._fernet.encrypt(plaintext.encode()).decode()
def decrypt(self, ciphertext: str) -> str:
return self._fernet.decrypt(ciphertext.encode()).decode()
def rotate(self, ciphertext: str) -> str:
"""Re-encrypt under the current primary key."""
return self._fernet.rotate(ciphertext.encode()).decode()
```
**Key generation for setup:**
```bash
python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"
```
**LiteLLM integration:** When routing LLM calls, check if the tenant has a BYO key for the requested provider. If yes, decrypt and inject into the LiteLLM call. Never log the decrypted key.
### Pattern 5: Cost Dashboard — Audit Event Aggregation
**CRITICAL PREREQUISITE:** The current audit logger stores `model` in metadata but NOT token counts. The runner.py `log_llm_call` metadata must be extended before the cost dashboard can work.
**Required metadata fields to add to `log_llm_call`:**
```python
# In orchestrator/agents/runner.py — extend existing metadata dict:
metadata={
"model": data.get("model", agent.model_preference),
"provider": _extract_provider(data.get("model", "")), # "openai" | "anthropic" | "ollama"
"prompt_tokens": usage.get("prompt_tokens", 0),
"completion_tokens": usage.get("completion_tokens", 0),
"total_tokens": usage.get("total_tokens", 0),
"cost_usd": _calculate_cost(model, usage), # pre-calculated, stored as float
"iteration": iteration,
"tool_calls_count": len(response_tool_calls),
}
```
**Dashboard aggregation query:**
```sql
-- Token usage per agent for a time range
SELECT
agent_id,
SUM((metadata->>'prompt_tokens')::int) AS prompt_tokens,
SUM((metadata->>'completion_tokens')::int) AS completion_tokens,
SUM((metadata->>'total_tokens')::int) AS total_tokens,
SUM((metadata->>'cost_usd')::float) AS cost_usd,
COUNT(*) AS llm_call_count
FROM audit_events
WHERE
tenant_id = :tenant_id
AND action_type = 'llm_call'
AND created_at >= :start_date
AND created_at < :end_date
GROUP BY agent_id;
-- Cost by provider
SELECT
metadata->>'provider' AS provider,
SUM((metadata->>'cost_usd')::float) AS cost_usd,
COUNT(*) AS call_count
FROM audit_events
WHERE
tenant_id = :tenant_id
AND action_type = 'llm_call'
AND created_at >= :start_date
GROUP BY metadata->>'provider';
-- Message volume by channel (count message events)
SELECT
metadata->>'channel' AS channel,
COUNT(*) AS message_count
FROM audit_events
WHERE
tenant_id = :tenant_id
AND action_type = 'llm_call'
AND created_at >= :start_date
GROUP BY metadata->>'channel';
```
**Index required:**
```sql
CREATE INDEX CONCURRENTLY idx_audit_events_tenant_type_created
ON audit_events (tenant_id, action_type, created_at DESC);
-- GIN index for JSONB queries if aggregation volume is high
CREATE INDEX CONCURRENTLY idx_audit_events_metadata
ON audit_events USING GIN (metadata);
```
**Time range options (Claude's discretion):** Offer Last 7 days / Last 30 days / This month / Custom range. Default to Last 30 days. Use a simple `<select>` driving a query param — no date picker library needed for v1.
**Budget alert logic:** Compare `SUM(cost_usd)` against `Agent.budget_limit_usd` (new field). Visual indicator: amber at 80%, red at 100%. Render as colored badge on the cost dashboard row, not a modal.
### Pattern 6: WhatsApp Manual Setup (Claude's Discretion Recommendation)
Embedded Signup requires a registered BSP/Meta Tech Provider account — multi-week verification process. For v1, use **guided manual setup**:
1. Operator creates a Meta/Facebook developer account and a WhatsApp Business App
2. Portal shows step-by-step instructions with screenshots
3. Operator pastes: Phone Number ID, WhatsApp Business Account ID, permanent System User Token
4. Portal validates by calling `GET https://graph.facebook.com/v22.0/{phone_number_id}` with the token
5. If valid, store in `channel_connections.config` (token encrypted with Fernet)
6. Test message: send "Konstruct connected successfully" to operator's own WhatsApp number
### Anti-Patterns to Avoid
- **Do not verify Stripe webhooks manually** — always use `stripe.Webhook.construct_event()` with the endpoint secret; raw header parsing is error-prone
- **Do not store Slack bot tokens in plaintext** — encrypt with Fernet before writing to `channel_connections.config`
- **Do not update subscription quantity synchronously** — Stripe rate-limits if updated many times per hour; queue via Celery if high-frequency
- **Do not re-query audit_events without the partial index** — full table scan on audit_events will be slow; the composite index on `(tenant_id, action_type, created_at)` is mandatory
- **Do not use `func.now()` for `trial_ends_at` calculation** — set it from the Stripe webhook response `subscription.trial_end` (Unix timestamp), not from local time
---
## Don't Hand-Roll
| Problem | Don't Build | Use Instead | Why |
|---------|-------------|-------------|-----|
| OAuth state CSRF protection | Custom state encoding | HMAC-signed state (shown above) | Replay attacks, timing attacks |
| Stripe signature verification | Manual HMAC check on raw bytes | `stripe.Webhook.construct_event()` | Handles timestamp tolerance, replay prevention |
| Subscription lifecycle state machine | Custom FSM | Stripe subscription `status` field + webhooks | Stripe handles payment retries, dunning, proration |
| API key encryption | Custom AES wrapper | `cryptography.fernet.Fernet` + `MultiFernet` | Audited, handles IV generation, MAC, key rotation |
| Billing UI (card updates, invoices) | Custom payment form | Stripe Billing Portal | PCI scope, card updating, invoice history — all free |
| Token cost calculation | Per-request cost estimation | Pre-calculate at log time using LiteLLM's `completion_cost()` | LiteLLM already tracks pricing per model; reuse it |
**Key insight:** Stripe and Slack provide hosted/SDK flows for the most security-sensitive parts. Never replicate what they already do correctly.
---
## Common Pitfalls
### Pitfall 1: Slack OAuth `state` Not Validated
**What goes wrong:** Attacker crafts a Slack OAuth callback with a valid `code` but forged `state`, linking their Slack workspace to a victim's tenant.
**Why it happens:** Treating OAuth state as opaque and skipping verification.
**How to avoid:** Always HMAC-sign the state before sending; verify the signature AND the `tenant_id` before exchanging the code.
**Warning signs:** No state validation in the callback handler.
### Pitfall 2: Stripe Webhook Raw Body Mangling
**What goes wrong:** Signature verification fails in production because a middleware (e.g., JSON parser) modifies the request body before the webhook handler reads it.
**Why it happens:** FastAPI's `Request.json()` parses the body; Stripe signatures are computed over the raw bytes.
**How to avoid:** Always read with `await request.body()` (raw bytes), not `await request.json()`.
**Warning signs:** `stripe.SignatureVerificationError` in production but not in local testing.
### Pitfall 3: Duplicate Webhook Processing
**What goes wrong:** Stripe delivers the same event twice (network retry); agent gets provisioned twice, subscription updated twice.
**Why it happens:** No idempotency guard on the webhook handler.
**How to avoid:** Store processed `event.id` in a `stripe_events` table; use `INSERT ... ON CONFLICT DO NOTHING` and check rows affected.
**Warning signs:** Duplicate `channel_connections` rows or double-charged agents.
### Pitfall 4: Fernet Key Not in Environment
**What goes wrong:** Application starts without `PLATFORM_ENCRYPTION_KEY` set; first BYO key encryption call throws `KeyError`.
**Why it happens:** Key not added to `.env.example` / Docker Compose environment.
**How to avoid:** Validate key presence at startup in `shared/config.py` using `pydantic-settings` required fields; fail fast with a clear error.
**Warning signs:** `KeyError: PLATFORM_ENCRYPTION_KEY` in logs.
### Pitfall 5: Audit Events Missing Token Metadata
**What goes wrong:** Cost dashboard shows zeros because `prompt_tokens` / `cost_usd` were never written to `audit_events.metadata`.
**Why it happens:** Runner logs the LLM call but doesn't extract token counts from the LiteLLM response object.
**How to avoid:** Extend `runner.py` metadata dict BEFORE Phase 3 dashboard work begins; backfill is impossible (audit log is append-only).
**Warning signs:** `metadata->>'prompt_tokens'` returns NULL in dashboard queries.
### Pitfall 6: Subscription Quantity Mismatch
**What goes wrong:** Operator creates 3 agents in the portal but Stripe still charges for 1.
**Why it happens:** Agent creation doesn't trigger a subscription quantity update.
**How to avoid:** On `POST /agents` and `DELETE /agents`, update Stripe subscription item quantity. Use a Celery task to avoid blocking the API response.
**Warning signs:** `stripe_subscription_item_id` is NULL; no Celery task defined for quantity sync.
---
## Code Examples
Verified patterns from official sources:
### Stripe Subscription Creation with Trial
```python
# Source: https://docs.stripe.com/api/subscriptions/create?lang=python
client = stripe.StripeClient(api_key=settings.stripe_secret_key)
subscription = client.v1.subscriptions.create({
"customer": customer_id,
"trial_period_days": 14,
"items": [{"price": settings.stripe_per_agent_price_id, "quantity": agent_count}],
})
# subscription.id → store as stripe_subscription_id
# subscription.items.data[0].id → store as stripe_subscription_item_id
```
### Stripe Subscription Quantity Update
```python
# Source: https://docs.stripe.com/api/subscription_items/update?lang=python
client.v1.subscription_items.update(
tenant.stripe_subscription_item_id,
{"quantity": new_agent_count},
)
```
### Fernet Encrypt/Decrypt API Key
```python
# Source: https://cryptography.io/en/latest/fernet/
from cryptography.fernet import Fernet
key = Fernet.generate_key() # run once, store in env
f = Fernet(key)
ciphertext = f.encrypt(b"sk-openai-key...").decode()
plaintext = f.decrypt(ciphertext.encode()).decode()
```
### MultiFernet Key Rotation
```python
# Source: https://cryptography.io/en/latest/fernet/
from cryptography.fernet import Fernet, MultiFernet
# Step 1: Add new key to front, keep old key
new_fernet = MultiFernet([Fernet(new_key), Fernet(old_key)])
# Step 2: Rotate all existing ciphertexts in DB
async for row in session.stream(select(TenantLlmKey)):
row.encrypted_key = new_fernet.rotate(row.encrypted_key.encode()).decode()
row.key_version += 1
await session.commit()
# Step 3: Remove old key from env, restart
```
### Recharts Bar Chart for Token Usage
```tsx
// Source: recharts.org/api-reference
// Install: npm install recharts
import { BarChart, Bar, XAxis, YAxis, CartesianGrid, Tooltip, ResponsiveContainer } from "recharts";
<ResponsiveContainer width="100%" height={300}>
<BarChart data={agentUsageData}>
<CartesianGrid strokeDasharray="3 3" />
<XAxis dataKey="agentName" />
<YAxis />
<Tooltip formatter={(value) => [`${value} tokens`, "Usage"]} />
<Bar dataKey="total_tokens" fill="#6366f1" radius={[4, 4, 0, 0]} />
</BarChart>
</ResponsiveContainer>
```
### SQLAlchemy Async Aggregate Query for Cost
```python
# Source: SQLAlchemy 2.0 docs + project pattern
from sqlalchemy import select, func, text
from shared.models.audit import AuditEvent
result = await session.execute(
select(
AuditEvent.agent_id,
func.sum(
func.cast(AuditEvent.metadata["total_tokens"].astext, Integer)
).label("total_tokens"),
func.sum(
func.cast(AuditEvent.metadata["cost_usd"].astext, Float)
).label("cost_usd"),
func.count().label("call_count"),
)
.where(
AuditEvent.tenant_id == tenant_id,
AuditEvent.action_type == "llm_call",
AuditEvent.created_at >= start_date,
)
.group_by(AuditEvent.agent_id)
)
```
---
## State of the Art
| Old Approach | Current Approach | When Changed | Impact |
|--------------|------------------|--------------|--------|
| `middleware.ts` for auth guards | `proxy.ts` (named export `proxy`) | Next.js 16 | Already implemented in project; do not create middleware.ts |
| `stripe.Webhook.constructEvent()` (old SDK) | `client.parse_event_notification()` or `stripe.Webhook.construct_event()` | stripe-python v12+ | Both work; use `construct_event` for consistency with FastAPI raw body pattern |
| On-Premises WhatsApp API | Cloud API only | Oct 2025 (deprecated) | Must use Cloud API; On-Premises is unsupported |
| Fernet single key | MultiFernet for key rotation | Always available | Use MultiFernet from day one; single key is not rotatable without downtime |
**Deprecated/outdated:**
- `middleware.ts` in Next.js 16: renamed to `proxy.ts` with named export `proxy` — already handled
- WhatsApp On-Premises API: deprecated Oct 2025, Cloud API only
- `stripe.Customer.create()` old-style: use `StripeClient.v1.customers.create()` with new client pattern
---
## Open Questions
1. **LiteLLM `completion_cost()` availability**
- What we know: LiteLLM has a `completion_cost()` utility that calculates cost from model name + token counts
- What's unclear: Whether `llm-pool` service currently surfaces this in its response to the orchestrator, or whether the orchestrator calls LiteLLM directly
- Recommendation: Inspect `packages/llm-pool` to confirm response schema includes `usage` object with token counts; if not, add it as Wave 0
2. **Stripe Customer creation timing**
- What we know: `stripe_customer_id` needs to exist before Checkout Session creation
- What's unclear: Whether to create the Stripe Customer at tenant creation (Phase 1 migration needed) or lazily on first billing action
- Recommendation: Create lazily on first billing action — avoids creating Stripe customers for test/internal tenants
3. **Slack OAuth callback — Next.js Route Handler vs FastAPI**
- What we know: The OAuth redirect_uri must be a URL Konstruct controls; both portal (Next.js) and backend (FastAPI) can handle it
- What's unclear: Which service is public-facing for the OAuth callback
- Recommendation: Handle in Next.js Route Handler (`app/api/slack/callback/route.ts`) which calls the FastAPI backend to store the token; cleaner separation and avoids CORS complications
4. **Budget limit storage location**
- What we know: AGNT-07 requires per-agent budget limits; CONTEXT.md shows budget as visual alert
- What's unclear: Whether budget limit is per-agent on the `agents` table or per-tenant on the `tenants` table
- Recommendation: Add `budget_limit_usd` to the `Agent` model (per-agent is more flexible); default NULL means no limit
---
## Validation Architecture
### Test Framework
| Property | Value |
|----------|-------|
| Framework | pytest 8.3.0 + pytest-asyncio 0.25.0 |
| Config file | `pyproject.toml` `[tool.pytest.ini_options]` |
| Quick run command | `pytest tests/unit -x` |
| Full suite command | `pytest tests/unit tests/integration -x` |
### Phase Requirements → Test Map
| Req ID | Behavior | Test Type | Automated Command | File Exists? |
|--------|----------|-----------|-------------------|-------------|
| AGNT-07 | Token usage aggregation query returns correct totals | unit | `pytest tests/unit/test_usage_aggregation.py -x` | Wave 0 |
| AGNT-07 | Budget alert threshold triggers at 80% and 100% | unit | `pytest tests/unit/test_budget_alerts.py -x` | Wave 0 |
| LLM-03 | Fernet encrypt/decrypt round-trip preserves plaintext | unit | `pytest tests/unit/test_key_encryption.py -x` | Wave 0 |
| LLM-03 | MultiFernet key rotation re-encrypts without data loss | unit | `pytest tests/unit/test_key_encryption.py::test_rotation -x` | Wave 0 |
| PRTA-03 | Slack OAuth state HMAC generation + verification | unit | `pytest tests/unit/test_slack_oauth.py -x` | Wave 0 |
| PRTA-03 | Slack OAuth callback stores channel_connection correctly | integration | `pytest tests/integration/test_slack_oauth.py -x` | Wave 0 |
| PRTA-04 | Onboarding stepper transitions through all 3 steps | manual | — | manual-only (UI flow) |
| PRTA-04 | Test message send endpoint returns 200 with valid token | integration | `pytest tests/integration/test_channel_test_message.py -x` | Wave 0 |
| PRTA-05 | Stripe webhook handler ignores duplicate event IDs | unit | `pytest tests/unit/test_stripe_webhooks.py::test_idempotency -x` | Wave 0 |
| PRTA-05 | Subscription status updates on `customer.subscription.updated` | unit | `pytest tests/unit/test_stripe_webhooks.py::test_subscription_updated -x` | Wave 0 |
| PRTA-05 | Agent deactivation on subscription cancellation | unit | `pytest tests/unit/test_stripe_webhooks.py::test_cancellation -x` | Wave 0 |
| PRTA-06 | Cost aggregation query groups tokens by agent_id | unit | `pytest tests/unit/test_usage_aggregation.py::test_group_by_agent -x` | Wave 0 |
| PRTA-06 | Cost aggregation query groups cost by provider | unit | `pytest tests/unit/test_usage_aggregation.py::test_group_by_provider -x` | Wave 0 |
### Sampling Rate
- **Per task commit:** `pytest tests/unit -x`
- **Per wave merge:** `pytest tests/unit tests/integration -x`
- **Phase gate:** Full suite green before `/gsd:verify-work`
### Wave 0 Gaps
- [ ] `tests/unit/test_key_encryption.py` — covers LLM-03 (Fernet encrypt/decrypt/rotate)
- [ ] `tests/unit/test_slack_oauth.py` — covers PRTA-03 (state HMAC generation/verification)
- [ ] `tests/unit/test_stripe_webhooks.py` — covers PRTA-05 (idempotency, status updates, cancellation)
- [ ] `tests/unit/test_usage_aggregation.py` — covers AGNT-07, PRTA-06 (SQL aggregate queries)
- [ ] `tests/unit/test_budget_alerts.py` — covers AGNT-07 (threshold logic)
- [ ] `tests/integration/test_slack_oauth.py` — covers PRTA-03 (full callback flow)
- [ ] `tests/integration/test_channel_test_message.py` — covers PRTA-04 (test message endpoint)
- [ ] `packages/shared/shared/models/billing.py` — TenantBilling fields migration model
- [ ] `packages/shared/shared/models/billing.py` — TenantLlmKey model
- [ ] Alembic migration: `audit_events` metadata fields (`prompt_tokens`, `completion_tokens`, `cost_usd`, `provider`)
- [ ] Alembic migration: `tenant_llm_keys` table
- [ ] Alembic migration: `tenants` billing fields (`stripe_customer_id`, `stripe_subscription_id`, `stripe_subscription_item_id`, `subscription_status`, `trial_ends_at`, `agent_quota`)
- [ ] Alembic migration: `agents.budget_limit_usd` field
- [ ] Alembic migration: `stripe_events` idempotency table
- [ ] npm install: `recharts @stripe/stripe-js stripe` in `packages/portal`
- [ ] uv add: `stripe cryptography` in `packages/shared`
---
## Sources
### Primary (HIGH confidence)
- [https://cryptography.io/en/latest/fernet/](https://cryptography.io/en/latest/fernet/) — Fernet spec, AES-128-CBC+HMAC-SHA256, MultiFernet rotation
- [https://docs.stripe.com/billing/subscriptions/webhooks](https://docs.stripe.com/billing/subscriptions/webhooks) — webhook event types, idempotency
- [https://docs.stripe.com/api/subscriptions/create?lang=python](https://docs.stripe.com/api/subscriptions/create?lang=python) — subscription creation with trial and quantity
- [https://docs.stripe.com/api/subscription_items/update?lang=python](https://docs.stripe.com/api/subscription_items/update?lang=python) — quantity update API
- [https://docs.stripe.com/customer-management/integrate-customer-portal](https://docs.stripe.com/customer-management/integrate-customer-portal) — Billing Portal session creation
- [https://docs.slack.dev/authentication/installing-with-oauth/](https://docs.slack.dev/authentication/installing-with-oauth/) — OAuth V2 flow
- [https://docs.slack.dev/apis/events-api/](https://docs.slack.dev/apis/events-api/) — scopes for Events API
- [https://docs.slack.dev/authentication/tokens/](https://docs.slack.dev/authentication/tokens/) — bot token types
- `packages/portal/node_modules/next/dist/docs/01-app/01-getting-started/16-proxy.md` — Next.js 16 proxy.ts (not middleware.ts)
- `packages/portal/node_modules/next/dist/docs/01-app/01-getting-started/15-route-handlers.md` — Route Handlers
- Existing project source: `packages/orchestrator/orchestrator/audit/logger.py`, `packages/shared/shared/models/audit.py`, `packages/shared/shared/models/tenant.py`
### Secondary (MEDIUM confidence)
- [https://docs.stripe.com/webhooks](https://docs.stripe.com/webhooks) — webhook signature verification in Python
- [https://docs.stripe.com/subscriptions/pricing-models/per-seat-pricing](https://docs.stripe.com/subscriptions/pricing-models/per-seat-pricing) — per-seat quantity model
- [https://www.speakeasy.com/blog/nivo-vs-recharts](https://www.speakeasy.com/blog/nivo-vs-recharts) — Recharts vs Nivo comparison (Recharts recommended)
- [https://npmtrends.com/chart.js-vs-highcharts-vs-nivo-vs-recharts](https://npmtrends.com/chart.js-vs-highcharts-vs-nivo-vs-recharts) — download statistics confirming Recharts dominance
### Tertiary (LOW confidence)
- WhatsApp Embedded Signup standard status (2026) — multiple secondary sources agree; not directly verified against Meta developer docs within this research session
---
## Metadata
**Confidence breakdown:**
- Standard stack: HIGH — all libraries verified against official docs or project package.json
- Slack OAuth flow: HIGH — verified against official Slack developer docs
- Stripe billing: HIGH — verified against official Stripe API reference and docs
- BYO key encryption: HIGH — verified against official cryptography.io docs; note Fernet is AES-128 not AES-256
- Cost aggregation: MEDIUM — SQL pattern is standard PostgreSQL JSONB; exact SQLAlchemy ORM casting syntax needs validation against project's asyncpg driver
- Recharts: MEDIUM — download stats verified via npmtrends; API verified against recharts.org
- WhatsApp manual setup: MEDIUM — On-Premises deprecation confirmed; manual setup steps derived from Meta developer docs indirectly
**Research date:** 2026-03-23
**Valid until:** 2026-04-23 (Stripe and Slack APIs are stable; Next.js 16 is current)