13 KiB
phase, verified, status, score, re_verification
| phase | verified | status | score | re_verification |
|---|---|---|---|---|
| 10-agent-capabilities | 2026-03-25T22:00:00Z | passed | 15/15 must-haves verified | false |
Phase 10: Agent Capabilities Verification Report
Phase Goal: Connect the 4 built-in agent tools to real external services so AI Employees can actually search the web, query a knowledge base of uploaded documents, make HTTP API calls, and check calendar availability Verified: 2026-03-25 Status: PASSED Re-verification: No — initial verification
Goal Achievement
Observable Truths
All must-haves are drawn from plan frontmatter across plans 10-01, 10-02, and 10-03.
Plan 10-01 Truths
| # | Truth | Status | Evidence |
|---|---|---|---|
| 1 | Documents uploaded via API are saved to MinIO and a KbDocument row is created with status=processing | VERIFIED | kb.py L150-157: inserts KnowledgeBaseDocument(status='processing'), L162-176: uploads bytes to MinIO via boto3 |
| 2 | The Celery ingestion task extracts text from PDF, DOCX, PPTX, XLSX, CSV, TXT, and MD files | VERIFIED | extractors.py: real implementations for all 7 formats using pypdf, python-docx, python-pptx, pandas, UTF-8 decode |
| 3 | Extracted text is chunked (500 chars, 50 overlap) and embedded via all-MiniLM-L6-v2 into kb_chunks with tenant_id | VERIFIED | ingest.py L56-92: chunk_text sliding window; L174: embed_texts(chunks); L186-202: raw SQL INSERT into kb_chunks with CAST vector |
| 4 | kb_search tool receives tenant_id injection from executor and returns matching chunks | VERIFIED | executor.py L126-127: args["tenant_id"] = str(tenant_id); kb_search.py L24: accepts tenant_id kwarg, runs pgvector cosine similarity query |
| 5 | BRAVE_API_KEY and FIRECRAWL_API_KEY are platform-wide settings in shared config | VERIFIED | config.py L223-227: brave_api_key and firecrawl_api_key as Field entries |
| 6 | Tool executor injects tenant_id and agent_id into tool handler kwargs for context-aware tools | VERIFIED | executor.py L126-127: injection occurs after schema validation (L98-103), before handler call (L134) |
Plan 10-02 Truths
| # | Truth | Status | Evidence |
|---|---|---|---|
| 7 | Tenant admin can initiate Google Calendar OAuth from the portal and authorize calendar access | VERIFIED | calendar_auth.py L104-130: GET /install endpoint returns Google OAuth URL with HMAC-signed state, offline access, and consent prompt |
| 8 | Calendar OAuth callback exchanges code for tokens and stores them encrypted per tenant | VERIFIED | calendar_auth.py L175-235: httpx POST to Google token endpoint, Fernet encrypt, upsert ChannelConnection(channel_type=GOOGLE_CALENDAR) |
| 9 | Calendar tool reads per-tenant OAuth tokens from channel_connections and calls Google Calendar API | VERIFIED | calendar_lookup.py L137-147: SELECT ChannelConnection WHERE channel_type=GOOGLE_CALENDAR; L178: builds Google Credentials; L194-207: run_in_executor for API call |
| 10 | Calendar tool supports list events, check availability, and create event actions | VERIFIED | calendar_lookup.py L267-273: dispatches to _action_list, _action_check_availability, _action_create; all three fully implemented |
| 11 | Token auto-refresh works — expired access tokens are refreshed via stored refresh_token and written back to DB | VERIFIED | calendar_lookup.py L190: records token_before; L210-225: if creds.token != token_before, encrypts and commits updated token to DB |
| 12 | Tool results are formatted as natural language (no raw JSON) | VERIFIED | builder.py L180-181: system prompt appends "Never show raw data or JSON to the user"; all calendar_lookup actions return formatted strings, not dicts |
Plan 10-03 Truths
| # | Truth | Status | Evidence |
|---|---|---|---|
| 13 | Operators can see a Knowledge Base page in the portal navigation | VERIFIED | nav.tsx L49: { href: "/knowledge-base", label: t("knowledgeBase"), icon: BookOpen }; i18n key present in en/es/pt message files |
| 14 | Operators can upload files via drag-and-drop or file picker dialog | VERIFIED | upload-dialog.tsx 249 lines: drag-and-drop zone, file picker input, sequential upload via uploadKbDocument; api.ts uses new FormData() |
| 15 | Uploaded documents show processing status with live polling | VERIFIED | queries.ts L518-521: refetchInterval returns 5000 when any doc has status === "processing", false otherwise |
Score: 15/15 truths verified
Required Artifacts
| Artifact | Status | Details |
|---|---|---|
migrations/versions/014_kb_status.py |
VERIFIED | Adds status, error_message, chunk_count to kb_documents; makes agent_id nullable |
migrations/versions/013_google_calendar_channel.py |
VERIFIED | Adds google_calendar to channel_connections CHECK constraint |
packages/orchestrator/orchestrator/tools/extractors.py |
VERIFIED | 142 lines; real implementations for all 7 format families; exports extract_text |
packages/orchestrator/orchestrator/tools/ingest.py |
VERIFIED | 323 lines; exports chunk_text and ingest_document_pipeline; full pipeline with MinIO, YouTube, Firecrawl |
packages/shared/shared/api/kb.py |
VERIFIED | 377 lines; 5 endpoints; exports kb_router |
packages/orchestrator/orchestrator/tasks.py |
VERIFIED | ingest_document Celery task at L1008-1036; calls asyncio.run(ingest_document_pipeline(...)) |
packages/orchestrator/orchestrator/tools/executor.py |
VERIFIED | Tenant/agent injection at L126-127, after schema validation, before handler call |
packages/shared/shared/api/calendar_auth.py |
VERIFIED | Full OAuth flow; exports calendar_auth_router; 3 endpoints |
packages/orchestrator/orchestrator/tools/builtins/calendar_lookup.py |
VERIFIED | Service account stub replaced; per-tenant OAuth; list/create/check_availability; token refresh write-back |
packages/orchestrator/orchestrator/tools/registry.py |
VERIFIED | All 4 tools in registry; calendar_lookup schema updated with action enum, event_summary, event_start, event_end |
packages/gateway/gateway/main.py |
VERIFIED | kb_router and calendar_auth_router mounted at L174-175 |
packages/portal/app/(dashboard)/knowledge-base/page.tsx |
VERIFIED | 88 lines; RBAC-conditional buttons; uses session for tenantId |
packages/portal/components/kb/document-list.tsx |
VERIFIED | 259 lines; status badges; delete confirm dialog; re-index; polling via useKbDocuments |
packages/portal/components/kb/upload-dialog.tsx |
VERIFIED | 249 lines; drag-and-drop; file picker; sequential upload with per-file progress |
packages/portal/components/kb/url-ingest-dialog.tsx |
VERIFIED | 162 lines; URL input; auto-YouTube detection; radio source type |
tests/unit/test_extractors.py |
VERIFIED | Exists on disk |
tests/unit/test_kb_upload.py |
VERIFIED | Exists on disk |
tests/unit/test_ingestion.py |
VERIFIED | Exists on disk |
tests/unit/test_executor_injection.py |
VERIFIED | Exists on disk |
tests/unit/test_calendar_lookup.py |
VERIFIED | Exists on disk |
tests/unit/test_calendar_auth.py |
VERIFIED | Exists on disk |
Key Link Verification
| From | To | Via | Status | Details |
|---|---|---|---|---|
shared/api/kb.py |
orchestrator/tasks.py |
ingest_document.delay(document_id, tenant_id) |
WIRED | L185-187 in kb.py: _get_ingest_task().delay(str(doc_id), str(tenant_id)); lazy import avoids circular dep |
orchestrator/tools/executor.py |
tool.handler |
tenant_id/agent_id injected into kwargs |
WIRED | L126-127: args["tenant_id"] = str(tenant_id); args["agent_id"] = str(agent_id) after schema validation |
shared/api/calendar_auth.py |
channel_connections table |
Upsert with channel_type='google_calendar' and encrypted token |
WIRED | L213-233: enc_svc.encrypt(token_json), upsert ChannelConnection(channel_type=GOOGLE_CALENDAR, config={"token": encrypted_token}) |
orchestrator/tools/builtins/calendar_lookup.py |
channel_connections table |
Load encrypted token, decrypt, build Credentials | WIRED | L137-147: SELECT ChannelConnection; L167-172: enc_svc.decrypt(encrypted_token); L76-83: Credentials(refresh_token=...) |
portal/components/kb/knowledge-base/page.tsx |
/api/portal/kb/{tenant_id}/documents |
TanStack Query fetch + polling | WIRED | document-list.tsx L30: imports useKbDocuments; L111: const { data } = useKbDocuments(tenantId); queries.ts L518-521: conditional refetchInterval |
portal/components/kb/upload-dialog.tsx |
/api/portal/kb/{tenant_id}/documents |
FormData multipart POST | WIRED | L109: await uploadKbDocument(tenantId, files[i].file, authHeaders); api.ts L378: const formData = new FormData() |
gateway/gateway/main.py |
kb_router + calendar_auth_router |
app.include_router(...) |
WIRED | L174-175: both routers mounted |
Requirements Coverage
| Requirement | Source Plan | Description | Status | Evidence |
|---|---|---|---|---|
| CAP-01 | 10-01 | Web search tool returns real results from Brave Search | SATISFIED | web_search.py L23: _BRAVE_API_URL = "https://api.search.brave.com/res/v1/web/search"; L40: settings.brave_api_key; full httpx call with error handling |
| CAP-02 | 10-01 | KB tool searches tenant-scoped documents chunked and embedded in pgvector | SATISFIED | kb_search.py: pgvector cosine similarity query on kb_chunks; executor injects tenant_id; ingest.py: embed_texts + INSERT with CAST vector |
| CAP-03 | 10-01, 10-03 | Operators can upload documents (PDF, DOCX, TXT) via portal | SATISFIED | kb.py: upload endpoint + Celery dispatch; portal KB page with upload dialog, URL ingest, status polling, delete, reindex |
| CAP-04 | 10-02 (confirmed) | HTTP request tool can call operator-configured URLs with timeout | SATISFIED | http_request.py: full httpx implementation, 30s timeout, 1MB cap, in registry |
| CAP-05 | 10-02 | Calendar tool can check Google Calendar availability and create events | SATISFIED | calendar_lookup.py: per-tenant OAuth, list/check_availability/create actions; full Google Calendar API integration |
| CAP-06 | 10-02 | Tool results incorporated naturally — no raw JSON shown to users | SATISFIED | builder.py L180-181: system prompt instruction; all tool handlers return formatted strings |
| CAP-07 | 10-02 (confirmed) | All tool invocations logged in audit trail | SATISFIED | executor.py L137-145: audit_logger.log_tool_call(...) on every success; L153-161: logged on every error; L192: logged on validation failure |
All 7 requirements satisfied. No orphaned requirements.
Anti-Patterns Found
None detected. Scanned all key backend and portal files for TODO, FIXME, placeholder, return null, return {}, console.log — none found.
Human Verification Required
1. Google Calendar OAuth end-to-end flow
Test: With GOOGLE_CLIENT_ID/SECRET configured, navigate to portal settings, click "Connect Google Calendar", complete Google consent, verify redirect back with ?calendar=connected
Expected: Token stored in channel_connections; subsequent agent messages can list/create calendar events
Why human: External OAuth redirect flow cannot be verified programmatically without real Google credentials and a live browser session
2. Knowledge Base document ingestion end-to-end Test: Upload a PDF or DOCX via the portal KB page, wait for status to change from "Processing" to "Ready", then send a message to an agent with kb_search assigned that references the document content Expected: Agent correctly cites information from the uploaded document Why human: Requires live MinIO, Celery worker, pgvector DB, and LLM inference stack to be running
3. Portal RBAC enforcement on KB page Test: Log in as a customer_operator user, navigate to /knowledge-base Expected: Document list is visible; "Upload Files" and "Add URL" buttons are hidden; Delete and Re-index action buttons are hidden Why human: RBAC conditional rendering requires live portal with a real operator session
4. Web search returns real results
Test: With BRAVE_API_KEY set, trigger an agent tool call to web_search with a current events query
Expected: Agent receives and summarizes real search results, not cached or static data
Why human: Requires live Brave API key and working agent inference loop
Gaps Summary
No gaps. All 15 must-have truths verified, all 7 requirements satisfied (CAP-01 through CAP-07), all key links wired, no anti-patterns found, all artifacts are substantive implementations (not stubs).
Notable: The portal KB implementation (Plan 10-03) is in a git submodule at packages/portal. The commit c525c02 exists in the submodule log but is not surfaced in the parent repo's git log — this is expected submodule behavior. The files exist on disk and are substantive.
Verified: 2026-03-25 Verifier: Claude (gsd-verifier)