From cac01b7ff9ee7d73a219d793b579689f9ecf45a0 Mon Sep 17 00:00:00 2001 From: Adolfo Delorenzo Date: Thu, 26 Mar 2026 09:29:24 -0600 Subject: [PATCH] docs(phase-10): complete Agent Capabilities phase execution --- .planning/ROADMAP.md | 2 +- .planning/STATE.md | 2 +- .../10-agent-capabilities/10-VERIFICATION.md | 155 ++++++++++++++++++ 3 files changed, 157 insertions(+), 2 deletions(-) create mode 100644 .planning/phases/10-agent-capabilities/10-VERIFICATION.md diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index 71149e9..6706022 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -144,7 +144,7 @@ Phases execute in numeric order: 1 -> 2 -> 3 -> 4 -> 5 -> 6 -> 7 -> 8 -> 9 -> 10 | 7. Multilanguage | 4/4 | Complete | 2026-03-25 | | 8. Mobile + PWA | 4/4 | Complete | 2026-03-26 | | 9. Testing & QA | 3/3 | Complete | 2026-03-26 | -| 10. Agent Capabilities | 3/3 | Complete | 2026-03-26 | +| 10. Agent Capabilities | 3/3 | Complete | 2026-03-26 | --- diff --git a/.planning/STATE.md b/.planning/STATE.md index 0968bd8..4f17142 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -4,7 +4,7 @@ milestone: v1.0 milestone_name: milestone status: completed stopped_at: "Completed 10-03: Knowledge Base portal page, file upload, URL ingest, RBAC, i18n" -last_updated: "2026-03-26T15:24:12.697Z" +last_updated: "2026-03-26T15:29:17.215Z" last_activity: 2026-03-23 — Completed 03-02 onboarding wizard, Slack OAuth, BYO API keys progress: total_phases: 10 diff --git a/.planning/phases/10-agent-capabilities/10-VERIFICATION.md b/.planning/phases/10-agent-capabilities/10-VERIFICATION.md new file mode 100644 index 0000000..c997ef7 --- /dev/null +++ b/.planning/phases/10-agent-capabilities/10-VERIFICATION.md @@ -0,0 +1,155 @@ +--- +phase: 10-agent-capabilities +verified: 2026-03-25T22:00:00Z +status: passed +score: 15/15 must-haves verified +re_verification: false +--- + +# Phase 10: Agent Capabilities Verification Report + +**Phase Goal:** Connect the 4 built-in agent tools to real external services so AI Employees can actually search the web, query a knowledge base of uploaded documents, make HTTP API calls, and check calendar availability +**Verified:** 2026-03-25 +**Status:** PASSED +**Re-verification:** No — initial verification + +--- + +## Goal Achievement + +### Observable Truths + +All must-haves are drawn from plan frontmatter across plans 10-01, 10-02, and 10-03. + +#### Plan 10-01 Truths + +| # | Truth | Status | Evidence | +|---|-------|--------|----------| +| 1 | Documents uploaded via API are saved to MinIO and a KbDocument row is created with status=processing | VERIFIED | `kb.py` L150-157: inserts `KnowledgeBaseDocument(status='processing')`, `L162-176`: uploads bytes to MinIO via boto3 | +| 2 | The Celery ingestion task extracts text from PDF, DOCX, PPTX, XLSX, CSV, TXT, and MD files | VERIFIED | `extractors.py`: real implementations for all 7 formats using pypdf, python-docx, python-pptx, pandas, UTF-8 decode | +| 3 | Extracted text is chunked (500 chars, 50 overlap) and embedded via all-MiniLM-L6-v2 into kb_chunks with tenant_id | VERIFIED | `ingest.py` L56-92: `chunk_text` sliding window; L174: `embed_texts(chunks)`; L186-202: raw SQL INSERT into kb_chunks with CAST vector | +| 4 | kb_search tool receives tenant_id injection from executor and returns matching chunks | VERIFIED | `executor.py` L126-127: `args["tenant_id"] = str(tenant_id)`; `kb_search.py` L24: accepts `tenant_id` kwarg, runs pgvector cosine similarity query | +| 5 | BRAVE_API_KEY and FIRECRAWL_API_KEY are platform-wide settings in shared config | VERIFIED | `config.py` L223-227: `brave_api_key` and `firecrawl_api_key` as Field entries | +| 6 | Tool executor injects tenant_id and agent_id into tool handler kwargs for context-aware tools | VERIFIED | `executor.py` L126-127: injection occurs after schema validation (L98-103), before handler call (L134) | + +#### Plan 10-02 Truths + +| # | Truth | Status | Evidence | +|---|-------|--------|----------| +| 7 | Tenant admin can initiate Google Calendar OAuth from the portal and authorize calendar access | VERIFIED | `calendar_auth.py` L104-130: `GET /install` endpoint returns Google OAuth URL with HMAC-signed state, offline access, and consent prompt | +| 8 | Calendar OAuth callback exchanges code for tokens and stores them encrypted per tenant | VERIFIED | `calendar_auth.py` L175-235: httpx POST to Google token endpoint, Fernet encrypt, upsert ChannelConnection(channel_type=GOOGLE_CALENDAR) | +| 9 | Calendar tool reads per-tenant OAuth tokens from channel_connections and calls Google Calendar API | VERIFIED | `calendar_lookup.py` L137-147: SELECT ChannelConnection WHERE channel_type=GOOGLE_CALENDAR; L178: builds Google Credentials; L194-207: run_in_executor for API call | +| 10 | Calendar tool supports list events, check availability, and create event actions | VERIFIED | `calendar_lookup.py` L267-273: dispatches to `_action_list`, `_action_check_availability`, `_action_create`; all three fully implemented | +| 11 | Token auto-refresh works — expired access tokens are refreshed via stored refresh_token and written back to DB | VERIFIED | `calendar_lookup.py` L190: records `token_before`; L210-225: if `creds.token != token_before`, encrypts and commits updated token to DB | +| 12 | Tool results are formatted as natural language (no raw JSON) | VERIFIED | `builder.py` L180-181: system prompt appends "Never show raw data or JSON to the user"; all `calendar_lookup` actions return formatted strings, not dicts | + +#### Plan 10-03 Truths + +| # | Truth | Status | Evidence | +|---|-------|--------|----------| +| 13 | Operators can see a Knowledge Base page in the portal navigation | VERIFIED | `nav.tsx` L49: `{ href: "/knowledge-base", label: t("knowledgeBase"), icon: BookOpen }`; i18n key present in en/es/pt message files | +| 14 | Operators can upload files via drag-and-drop or file picker dialog | VERIFIED | `upload-dialog.tsx` 249 lines: drag-and-drop zone, file picker input, sequential upload via `uploadKbDocument`; `api.ts` uses `new FormData()` | +| 15 | Uploaded documents show processing status with live polling | VERIFIED | `queries.ts` L518-521: `refetchInterval` returns 5000 when any doc has `status === "processing"`, false otherwise | + +**Score:** 15/15 truths verified + +--- + +### Required Artifacts + +| Artifact | Status | Details | +|----------|--------|---------| +| `migrations/versions/014_kb_status.py` | VERIFIED | Adds status, error_message, chunk_count to kb_documents; makes agent_id nullable | +| `migrations/versions/013_google_calendar_channel.py` | VERIFIED | Adds google_calendar to channel_connections CHECK constraint | +| `packages/orchestrator/orchestrator/tools/extractors.py` | VERIFIED | 142 lines; real implementations for all 7 format families; exports `extract_text` | +| `packages/orchestrator/orchestrator/tools/ingest.py` | VERIFIED | 323 lines; exports `chunk_text` and `ingest_document_pipeline`; full pipeline with MinIO, YouTube, Firecrawl | +| `packages/shared/shared/api/kb.py` | VERIFIED | 377 lines; 5 endpoints; exports `kb_router` | +| `packages/orchestrator/orchestrator/tasks.py` | VERIFIED | `ingest_document` Celery task at L1008-1036; calls `asyncio.run(ingest_document_pipeline(...))` | +| `packages/orchestrator/orchestrator/tools/executor.py` | VERIFIED | Tenant/agent injection at L126-127, after schema validation, before handler call | +| `packages/shared/shared/api/calendar_auth.py` | VERIFIED | Full OAuth flow; exports `calendar_auth_router`; 3 endpoints | +| `packages/orchestrator/orchestrator/tools/builtins/calendar_lookup.py` | VERIFIED | Service account stub replaced; per-tenant OAuth; list/create/check_availability; token refresh write-back | +| `packages/orchestrator/orchestrator/tools/registry.py` | VERIFIED | All 4 tools in registry; calendar_lookup schema updated with action enum, event_summary, event_start, event_end | +| `packages/gateway/gateway/main.py` | VERIFIED | `kb_router` and `calendar_auth_router` mounted at L174-175 | +| `packages/portal/app/(dashboard)/knowledge-base/page.tsx` | VERIFIED | 88 lines; RBAC-conditional buttons; uses session for tenantId | +| `packages/portal/components/kb/document-list.tsx` | VERIFIED | 259 lines; status badges; delete confirm dialog; re-index; polling via `useKbDocuments` | +| `packages/portal/components/kb/upload-dialog.tsx` | VERIFIED | 249 lines; drag-and-drop; file picker; sequential upload with per-file progress | +| `packages/portal/components/kb/url-ingest-dialog.tsx` | VERIFIED | 162 lines; URL input; auto-YouTube detection; radio source type | +| `tests/unit/test_extractors.py` | VERIFIED | Exists on disk | +| `tests/unit/test_kb_upload.py` | VERIFIED | Exists on disk | +| `tests/unit/test_ingestion.py` | VERIFIED | Exists on disk | +| `tests/unit/test_executor_injection.py` | VERIFIED | Exists on disk | +| `tests/unit/test_calendar_lookup.py` | VERIFIED | Exists on disk | +| `tests/unit/test_calendar_auth.py` | VERIFIED | Exists on disk | + +--- + +### Key Link Verification + +| From | To | Via | Status | Details | +|------|----|-----|--------|---------| +| `shared/api/kb.py` | `orchestrator/tasks.py` | `ingest_document.delay(document_id, tenant_id)` | WIRED | L185-187 in kb.py: `_get_ingest_task().delay(str(doc_id), str(tenant_id))`; lazy import avoids circular dep | +| `orchestrator/tools/executor.py` | `tool.handler` | `tenant_id/agent_id` injected into kwargs | WIRED | L126-127: `args["tenant_id"] = str(tenant_id); args["agent_id"] = str(agent_id)` after schema validation | +| `shared/api/calendar_auth.py` | `channel_connections` table | Upsert with `channel_type='google_calendar'` and encrypted token | WIRED | L213-233: `enc_svc.encrypt(token_json)`, upsert `ChannelConnection(channel_type=GOOGLE_CALENDAR, config={"token": encrypted_token})` | +| `orchestrator/tools/builtins/calendar_lookup.py` | `channel_connections` table | Load encrypted token, decrypt, build Credentials | WIRED | L137-147: SELECT ChannelConnection; L167-172: `enc_svc.decrypt(encrypted_token)`; L76-83: `Credentials(refresh_token=...)` | +| `portal/components/kb/knowledge-base/page.tsx` | `/api/portal/kb/{tenant_id}/documents` | TanStack Query fetch + polling | WIRED | `document-list.tsx` L30: imports `useKbDocuments`; L111: `const { data } = useKbDocuments(tenantId)`; `queries.ts` L518-521: conditional `refetchInterval` | +| `portal/components/kb/upload-dialog.tsx` | `/api/portal/kb/{tenant_id}/documents` | FormData multipart POST | WIRED | L109: `await uploadKbDocument(tenantId, files[i].file, authHeaders)`; `api.ts` L378: `const formData = new FormData()` | +| `gateway/gateway/main.py` | `kb_router` + `calendar_auth_router` | `app.include_router(...)` | WIRED | L174-175: both routers mounted | + +--- + +### Requirements Coverage + +| Requirement | Source Plan | Description | Status | Evidence | +|-------------|-------------|-------------|--------|----------| +| CAP-01 | 10-01 | Web search tool returns real results from Brave Search | SATISFIED | `web_search.py` L23: `_BRAVE_API_URL = "https://api.search.brave.com/res/v1/web/search"`; L40: `settings.brave_api_key`; full httpx call with error handling | +| CAP-02 | 10-01 | KB tool searches tenant-scoped documents chunked and embedded in pgvector | SATISFIED | `kb_search.py`: pgvector cosine similarity query on kb_chunks; executor injects tenant_id; `ingest.py`: embed_texts + INSERT with CAST vector | +| CAP-03 | 10-01, 10-03 | Operators can upload documents (PDF, DOCX, TXT) via portal | SATISFIED | `kb.py`: upload endpoint + Celery dispatch; portal KB page with upload dialog, URL ingest, status polling, delete, reindex | +| CAP-04 | 10-02 (confirmed) | HTTP request tool can call operator-configured URLs with timeout | SATISFIED | `http_request.py`: full httpx implementation, 30s timeout, 1MB cap, in registry | +| CAP-05 | 10-02 | Calendar tool can check Google Calendar availability and create events | SATISFIED | `calendar_lookup.py`: per-tenant OAuth, list/check_availability/create actions; full Google Calendar API integration | +| CAP-06 | 10-02 | Tool results incorporated naturally — no raw JSON shown to users | SATISFIED | `builder.py` L180-181: system prompt instruction; all tool handlers return formatted strings | +| CAP-07 | 10-02 (confirmed) | All tool invocations logged in audit trail | SATISFIED | `executor.py` L137-145: `audit_logger.log_tool_call(...)` on every success; L153-161: logged on every error; L192: logged on validation failure | + +**All 7 requirements satisfied. No orphaned requirements.** + +--- + +### Anti-Patterns Found + +None detected. Scanned all key backend and portal files for TODO, FIXME, placeholder, `return null`, `return {}`, `console.log` — none found. + +--- + +### Human Verification Required + +**1. Google Calendar OAuth end-to-end flow** +**Test:** With GOOGLE_CLIENT_ID/SECRET configured, navigate to portal settings, click "Connect Google Calendar", complete Google consent, verify redirect back with `?calendar=connected` +**Expected:** Token stored in channel_connections; subsequent agent messages can list/create calendar events +**Why human:** External OAuth redirect flow cannot be verified programmatically without real Google credentials and a live browser session + +**2. Knowledge Base document ingestion end-to-end** +**Test:** Upload a PDF or DOCX via the portal KB page, wait for status to change from "Processing" to "Ready", then send a message to an agent with kb_search assigned that references the document content +**Expected:** Agent correctly cites information from the uploaded document +**Why human:** Requires live MinIO, Celery worker, pgvector DB, and LLM inference stack to be running + +**3. Portal RBAC enforcement on KB page** +**Test:** Log in as a customer_operator user, navigate to /knowledge-base +**Expected:** Document list is visible; "Upload Files" and "Add URL" buttons are hidden; Delete and Re-index action buttons are hidden +**Why human:** RBAC conditional rendering requires live portal with a real operator session + +**4. Web search returns real results** +**Test:** With BRAVE_API_KEY set, trigger an agent tool call to `web_search` with a current events query +**Expected:** Agent receives and summarizes real search results, not cached or static data +**Why human:** Requires live Brave API key and working agent inference loop + +--- + +### Gaps Summary + +No gaps. All 15 must-have truths verified, all 7 requirements satisfied (CAP-01 through CAP-07), all key links wired, no anti-patterns found, all artifacts are substantive implementations (not stubs). + +Notable: The portal KB implementation (Plan 10-03) is in a git submodule at `packages/portal`. The commit `c525c02` exists in the submodule log but is not surfaced in the parent repo's git log — this is expected submodule behavior. The files exist on disk and are substantive. + +--- + +_Verified: 2026-03-25_ +_Verifier: Claude (gsd-verifier)_