108 lines
4.8 KiB
Markdown
108 lines
4.8 KiB
Markdown
# Phase 10: Agent Capabilities - Context
|
|
|
|
**Gathered:** 2026-03-26
|
|
**Status:** Ready for planning
|
|
|
|
<domain>
|
|
## Phase Boundary
|
|
|
|
Connect the 4 built-in agent tools to real external services. The biggest deliverable is the knowledge base document pipeline (upload → chunk → embed → search). Web search and HTTP request tools already have working implementations that need API keys configured. Calendar tool needs Google Calendar OAuth integration with full CRUD (not just read-only).
|
|
|
|
</domain>
|
|
|
|
<decisions>
|
|
## Implementation Decisions
|
|
|
|
### Knowledge Base & Document Upload
|
|
- **Supported formats:**
|
|
- Files: PDF, DOCX/Word, TXT, Markdown, CSV/Excel, PPT/PowerPoint
|
|
- URLs: Web page scraping/crawling via Firecrawl
|
|
- YouTube: Transcriptions (use existing transcripts when available, OpenWhisper for transcription when not)
|
|
- KB is **per-tenant** — all agents in a tenant share the same knowledge base
|
|
- Dedicated **KB management page** in the portal (not inline in Agent Designer)
|
|
- Upload files (drag-and-drop + file picker)
|
|
- Add URLs for scraping
|
|
- Add YouTube URLs for transcription
|
|
- View ingested documents with status (processing, ready, error)
|
|
- Delete documents (removes chunks from pgvector)
|
|
- Re-index option
|
|
- Document processing is **async/background** — upload returns immediately, Celery task handles chunking + embedding
|
|
- Processing status visible in portal (progress indicator per document)
|
|
|
|
### Web Search
|
|
- Brave Search API (already implemented in `web_search.py`)
|
|
- Configuration: Claude's discretion (platform-wide key recommended for simplicity, BYO optional)
|
|
- `BRAVE_API_KEY` added to `.env`
|
|
|
|
### HTTP Request Tool
|
|
- Already implemented in `http_request.py` with timeout and size limits
|
|
- Operator configures allowed URLs in Agent Designer tool_assignments
|
|
- No changes needed — tool is functional
|
|
|
|
### Calendar Integration
|
|
- Google Calendar OAuth per tenant — tenant admin authorizes in portal
|
|
- Full CRUD for v1: check availability, list upcoming events, **create events** (not read-only)
|
|
- OAuth callback handled in portal (similar pattern to Slack OAuth)
|
|
- Calendar credentials stored encrypted per tenant (reuse Fernet encryption from Phase 3)
|
|
|
|
### Claude's Discretion
|
|
- Web search: platform-wide vs per-tenant API key (recommend platform-wide)
|
|
- Chunking strategy (chunk size, overlap)
|
|
- Embedding model for KB (reuse all-MiniLM-L6-v2 or upgrade)
|
|
- Firecrawl integration approach (self-hosted vs cloud API)
|
|
- YouTube transcription: when to use existing captions vs OpenWhisper
|
|
- Document size limits
|
|
- KB chunk deduplication strategy
|
|
|
|
</decisions>
|
|
|
|
<specifics>
|
|
## Specific Ideas
|
|
|
|
- The KB page should show document processing status live — operators need to know when their docs are ready for agents to search
|
|
- YouTube transcription is a killer feature for SMBs — they can feed training videos, product demos, and meeting recordings into the agent's knowledge base
|
|
- URL scraping via Firecrawl means agents can learn from the company's website, help docs, and blog posts automatically
|
|
- Calendar event creation makes the Sales Assistant and Office Manager templates immediately valuable — they can actually book meetings
|
|
|
|
</specifics>
|
|
|
|
<code_context>
|
|
## Existing Code Insights
|
|
|
|
### Reusable Assets
|
|
- `packages/orchestrator/orchestrator/tools/builtins/web_search.py` — Brave Search API integration (working, needs key)
|
|
- `packages/orchestrator/orchestrator/tools/builtins/kb_search.py` — pgvector similarity search (needs chunk data)
|
|
- `packages/orchestrator/orchestrator/tools/builtins/http_request.py` — HTTP client with limits (working)
|
|
- `packages/orchestrator/orchestrator/tools/builtins/calendar_lookup.py` — Placeholder stub (needs Google Calendar)
|
|
- `packages/orchestrator/orchestrator/memory/embedder.py` — SentenceTransformer singleton (reuse for KB embedding)
|
|
- `packages/shared/shared/models/kb.py` — KbDocument and KbChunk ORM models (created in Phase 2 migration)
|
|
- `packages/shared/shared/crypto.py` — Fernet encryption (reuse for Google Calendar tokens)
|
|
- `packages/shared/shared/api/channels.py` — OAuth pattern (reuse for Google Calendar OAuth)
|
|
|
|
### Established Patterns
|
|
- Celery tasks for background processing (fire-and-forget with `embed_and_store.delay()`)
|
|
- pgvector HNSW cosine similarity with tenant_id pre-filter
|
|
- MinIO/S3 for file storage (configured but not used for KB yet)
|
|
- Fernet encrypted credential storage per tenant
|
|
|
|
### Integration Points
|
|
- Portal needs new `/knowledge-base` page (similar to `/settings/api-keys`)
|
|
- Gateway needs document upload endpoint (multipart file upload)
|
|
- Gateway needs Google Calendar OAuth callback route
|
|
- Agent Designer needs Google Calendar connection status display
|
|
- Nav needs KB link added for customer_admin + platform_admin
|
|
|
|
</code_context>
|
|
|
|
<deferred>
|
|
## Deferred Ideas
|
|
|
|
None — discussion stayed within phase scope
|
|
|
|
</deferred>
|
|
|
|
---
|
|
|
|
*Phase: 10-agent-capabilities*
|
|
*Context gathered: 2026-03-26*
|