adelorenzo/konstruct
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(10): create phase plan
Some checks failed
CI / Backend Tests (push) Has been cancelled
Details
CI / Portal E2E (push) Has been cancelled
Details

Browse Source
This commit is contained in:
Adolfo Delorenzo
2026-03-25 23:33:27 -06:00
parent 95d05f5f88
commit eae4b0324d
4 changed files with 805 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

14
.planning/ROADMAP.md
View File

@@ -131,7 +131,7 @@ Plans:
## Progress
**Execution Order:**
Phases execute in numeric order: 1 -> 2 -> 3 -> 4 -> 5 -> 6 -> 7 -> 8 -> 9
Phases execute in numeric order: 1 -> 2 -> 3 -> 4 -> 5 -> 6 -> 7 -> 8 -> 9 -> 10
| Phase | Plans Complete | Status | Completed |
|-------|----------------|--------|-----------|
@@ -144,7 +144,7 @@ Phases execute in numeric order: 1 -> 2 -> 3 -> 4 -> 5 -> 6 -> 7 -> 8 -> 9
| 7. Multilanguage | 4/4 | Complete | 2026-03-25 |
| 8. Mobile + PWA | 4/4 | Complete | 2026-03-26 |
| 9. Testing & QA | 3/3 | Complete | 2026-03-26 |
| 10. Agent Capabilities | 0/0 | Not started | - |
| 10. Agent Capabilities | 0/3 | In progress | - |
---
@@ -210,7 +210,7 @@ Plans:
- [ ] 09-03-PLAN.md — Gitea Actions CI pipeline (backend lint+pytest, portal build+E2E+Lighthouse) + human verification
### Phase 10: Agent Capabilities
**Goal**: Connect the 4 built-in agent tools to real external services so AI Employees can actually search the web, query a knowledge base of uploaded documents, make HTTP API calls, and check calendar availability
**Goal**: Connect the 4 built-in agent tools to real external services so AI Employees can actually search the web, query a knowledge base of uploaded documents, make HTTP API calls, and check calendar availability — with full CRUD Google Calendar integration and a dedicated KB management portal page
**Depends on**: Phase 9
**Requirements**: CAP-01, CAP-02, CAP-03, CAP-04, CAP-05, CAP-06, CAP-07
**Success Criteria** (what must be TRUE):
@@ -221,11 +221,13 @@ Plans:
5. Calendar tool can check availability on Google Calendar (read-only for v1)
6. Tool results are incorporated naturally into agent responses (no raw JSON dumps)
7. All tool invocations are logged in the audit trail with input/output
**Plans**: 0 plans
**Plans**: 3 plans
Plans:
- [ ] TBD (run /gsd:plan-phase 10 to break down)
- [ ] 10-01-PLAN.md — KB ingestion pipeline backend: migration 013, text extractors (PDF/DOCX/PPTX/XLSX/CSV/TXT/MD), chunking + embedding Celery task, KB API router (upload/list/delete/reindex/URL), executor tenant_id injection, web search config
- [ ] 10-02-PLAN.md — Google Calendar OAuth per tenant: install/callback endpoints, calendar_lookup replacement with list/create/check_availability, encrypted token storage, router mounting, tool response formatting
- [ ] 10-03-PLAN.md — Portal KB management page: document list with status polling, file upload (drag-and-drop), URL/YouTube ingestion, delete/reindex, RBAC, human verification
---
*Roadmap created: 2026-03-23*
*Coverage: 25/25 v1 requirements + 6 RBAC requirements + 5 Employee Design requirements + 5 Web Chat requirements + 6 Multilanguage requirements + 6 Mobile+PWA requirements + 7 Testing & QA requirements mapped*
*Coverage: 25/25 v1 requirements + 6 RBAC requirements + 5 Employee Design requirements + 5 Web Chat requirements + 6 Multilanguage requirements + 6 Mobile+PWA requirements + 7 Testing & QA requirements + 7 Agent Capabilities requirements mapped*

338
.planning/phases/10-agent-capabilities/10-01-PLAN.md Normal file
View File

@@ -0,0 +1,338 @@
---
phase: 10-agent-capabilities
plan: 01
type: execute
wave: 1
depends_on: []
files_modified:
- migrations/versions/013_kb_status_and_calendar.py
- packages/shared/shared/models/kb.py
- packages/shared/shared/models/tenant.py
- packages/shared/shared/config.py
- packages/shared/shared/api/kb.py
- packages/orchestrator/orchestrator/tools/ingest.py
- packages/orchestrator/orchestrator/tools/extractors.py
- packages/orchestrator/orchestrator/tasks.py
- packages/orchestrator/orchestrator/tools/executor.py
- packages/orchestrator/orchestrator/tools/builtins/kb_search.py
- packages/orchestrator/pyproject.toml
- .env.example
- tests/unit/test_extractors.py
- tests/unit/test_kb_upload.py
autonomous: true
requirements:
- CAP-01
- CAP-02
- CAP-03
- CAP-04
- CAP-07
must_haves:
truths:
- "Documents uploaded via API are saved to MinIO and a KbDocument row is created with status=processing"
- "The Celery ingestion task extracts text from PDF, DOCX, PPTX, XLSX, CSV, TXT, and MD files"
- "Extracted text is chunked (500 chars, 50 overlap) and embedded via all-MiniLM-L6-v2 into kb_chunks with tenant_id"
- "kb_search tool receives tenant_id injection from executor and returns matching chunks"
- "BRAVE_API_KEY and FIRECRAWL_API_KEY are platform-wide settings in shared config"
- "Tool executor injects tenant_id and agent_id into tool handler kwargs for context-aware tools"
artifacts:
- path: "migrations/versions/013_kb_status_and_calendar.py"
provides: "DB migration: kb_documents status/error_message/chunk_count columns, agent_id nullable, channel_type CHECK update for google_calendar"
contains: "status"
- path: "packages/orchestrator/orchestrator/tools/extractors.py"
provides: "Text extraction functions for all supported document formats"
exports: ["extract_text"]
- path: "packages/orchestrator/orchestrator/tools/ingest.py"
provides: "Document chunking and ingestion pipeline logic"
exports: ["chunk_text", "ingest_document_pipeline"]
- path: "packages/shared/shared/api/kb.py"
provides: "KB management API router (upload, list, delete, re-index)"
exports: ["kb_router"]
- path: "tests/unit/test_extractors.py"
provides: "Unit tests for text extraction functions"
key_links:
- from: "packages/shared/shared/api/kb.py"
to: "packages/orchestrator/orchestrator/tasks.py"
via: "ingest_document.delay(document_id, tenant_id)"
pattern: "ingest_document\\.delay"
- from: "packages/orchestrator/orchestrator/tools/executor.py"
to: "tool.handler"
via: "tenant_id/agent_id injection into kwargs"
pattern: "tenant_id.*agent_id.*handler"
---
<objective>
Build the knowledge base document ingestion pipeline backend and activate web search/HTTP tools.
Purpose: This is the core backend for CAP-02/CAP-03 -- the document upload, text extraction, chunking, embedding, and storage pipeline that makes the KB search tool functional with real data. Also fixes the tool executor to inject tenant context into tool handlers, activates web search via BRAVE_API_KEY config, and confirms HTTP request tool needs no changes (CAP-04).
Output: Working KB upload API, Celery ingestion task, text extractors for all formats, migration 013, executor tenant_id injection, updated config with new env vars.
</objective>
<execution_context>
@/home/adelorenzo/.claude/get-shit-done/workflows/execute-plan.md
@/home/adelorenzo/.claude/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/10-agent-capabilities/10-CONTEXT.md
@.planning/phases/10-agent-capabilities/10-RESEARCH.md
<interfaces>
<!-- Key types and contracts the executor needs -->
From packages/shared/shared/models/kb.py:
```python
class KnowledgeBaseDocument(KBBase):
__tablename__ = "kb_documents"
id: Mapped[uuid.UUID]
tenant_id: Mapped[uuid.UUID]
agent_id: Mapped[uuid.UUID] # Currently NOT NULL — migration 013 makes nullable
filename: Mapped[str | None]
source_url: Mapped[str | None]
content_type: Mapped[str | None]
created_at: Mapped[datetime]
chunks: Mapped[list[KBChunk]]
class KBChunk(KBBase):
__tablename__ = "kb_chunks"
id: Mapped[uuid.UUID]
tenant_id: Mapped[uuid.UUID]
document_id: Mapped[uuid.UUID]
content: Mapped[str]
chunk_index: Mapped[int | None]
created_at: Mapped[datetime]
```
From packages/orchestrator/orchestrator/tools/executor.py:
```python
async def execute_tool(
tool_call: dict[str, Any],
registry: dict[str, "ToolDefinition"],
tenant_id: uuid.UUID,
agent_id: uuid.UUID,
audit_logger: "AuditLogger",
) -> str:
# Line 126: result = await tool.handler(**args)
# PROBLEM: only LLM-provided args are passed, tenant_id/agent_id NOT injected
```
From packages/orchestrator/orchestrator/memory/embedder.py:
```python
def embed_text(text: str) -> list[float]: # Returns 384-dim vector
def embed_texts(texts: list[str]) -> list[list[float]]: # Batch embedding
```
From packages/shared/shared/config.py:
```python
class Settings(BaseSettings):
minio_endpoint: str
minio_access_key: str
minio_secret_key: str
minio_media_bucket: str
```
From packages/shared/shared/api/channels.py:
```python
channels_router = APIRouter(prefix="/api/portal/channels", tags=["channels"])
# Uses: require_tenant_admin, get_session, KeyEncryptionService
# OAuth state: generate_oauth_state() / verify_oauth_state() with HMAC-SHA256
```
From packages/shared/shared/api/rbac.py:
```python
class PortalCaller: ...
async def require_tenant_admin(...) -> PortalCaller: ...
async def require_tenant_member(...) -> PortalCaller: ...
```
</interfaces>
</context>
<tasks>
<task type="auto" tdd="true">
<name>Task 1: Migration 013, ORM updates, config settings, text extractors, KB API router</name>
<files>
migrations/versions/013_kb_status_and_calendar.py,
packages/shared/shared/models/kb.py,
packages/shared/shared/models/tenant.py,
packages/shared/shared/config.py,
packages/shared/shared/api/kb.py,
packages/orchestrator/orchestrator/tools/extractors.py,
packages/orchestrator/pyproject.toml,
.env.example,
tests/unit/test_extractors.py,
tests/unit/test_kb_upload.py
</files>
<behavior>
- extract_text("hello.pdf", pdf_bytes) returns extracted text from PDF pages
- extract_text("doc.docx", docx_bytes) returns paragraph text from DOCX
- extract_text("slides.pptx", pptx_bytes) returns slide text from PPTX
- extract_text("data.xlsx", xlsx_bytes) returns CSV-formatted cell data
- extract_text("data.csv", csv_bytes) returns decoded UTF-8 text
- extract_text("notes.txt", txt_bytes) returns decoded text
- extract_text("notes.md", md_bytes) returns decoded text
- extract_text("file.exe", bytes) raises ValueError("Unsupported file extension")
- KB upload endpoint returns 201 with document_id for valid file
- KB list endpoint returns documents with status field
- KB delete endpoint removes document and chunks
</behavior>
<action>
1. **Migration 013** (`migrations/versions/013_kb_status_and_calendar.py`):
- ALTER TABLE kb_documents ADD COLUMN status TEXT NOT NULL DEFAULT 'processing'
- ALTER TABLE kb_documents ADD COLUMN error_message TEXT
- ALTER TABLE kb_documents ADD COLUMN chunk_count INTEGER
- ALTER TABLE kb_documents ALTER COLUMN agent_id DROP NOT NULL (KB is per-tenant per locked decision)
- DROP + re-ADD channel_connections CHECK constraint to include 'google_calendar' (same pattern as migration 008)
- New channel types tuple: slack, whatsapp, mattermost, rocketchat, teams, telegram, signal, web, google_calendar
- Add CHECK constraint on kb_documents.status: CHECK (status IN ('processing', 'ready', 'error'))
2. **ORM updates**:
- `packages/shared/shared/models/kb.py`: Add status (str, server_default='processing'), error_message (str | None), chunk_count (int | None) mapped columns to KnowledgeBaseDocument. Change agent_id to nullable=True.
- `packages/shared/shared/models/tenant.py`: Add GOOGLE_CALENDAR = "google_calendar" to ChannelTypeEnum
3. **Config** (`packages/shared/shared/config.py`):
- Add brave_api_key: str = Field(default="", description="Brave Search API key")
- Add firecrawl_api_key: str = Field(default="", description="Firecrawl API key for URL scraping")
- Add google_client_id: str = Field(default="", description="Google OAuth client ID")
- Add google_client_secret: str = Field(default="", description="Google OAuth client secret")
- Add minio_kb_bucket: str = Field(default="kb-documents", description="MinIO bucket for KB documents")
- Update .env.example with all new env vars
4. **Install dependencies** on orchestrator:
```bash
uv add --project packages/orchestrator pypdf python-docx python-pptx openpyxl pandas firecrawl-py youtube-transcript-api google-api-python-client google-auth-oauthlib
```
5. **Text extractors** (`packages/orchestrator/orchestrator/tools/extractors.py`):
- Create extract_text(filename: str, file_bytes: bytes) -> str function
- PDF: pypdf PdfReader on BytesIO, join page text with newlines
- DOCX: python-docx Document on BytesIO, join paragraph text
- PPTX: python-pptx Presentation on BytesIO, iterate slides/shapes for text
- XLSX/XLS: pandas read_excel on BytesIO, to_csv(index=False)
- CSV: decode UTF-8 with errors="replace"
- TXT/MD: decode UTF-8 with errors="replace"
- Raise ValueError for unsupported extensions
- After extraction, check if len(text.strip()) < 100 chars for PDF — return error message about OCR not supported
6. **KB API router** (`packages/shared/shared/api/kb.py`):
- kb_router = APIRouter(prefix="/api/portal/kb", tags=["knowledge-base"])
- POST /{tenant_id}/documents — multipart file upload (UploadFile + File)
- Validate file extension against supported list
- Read file bytes, upload to MinIO kb-documents bucket with key: {tenant_id}/{doc_id}/{filename}
- Insert KnowledgeBaseDocument(tenant_id, filename, content_type, status='processing', agent_id=None)
- Call ingest_document.delay(str(doc.id), str(tenant_id)) — import from orchestrator.tasks
- Return 201 with {"id": str(doc.id), "filename": filename, "status": "processing"}
- Guard with require_tenant_admin
- POST /{tenant_id}/documents/url — JSON body {url: str, source_type: "web" | "youtube"}
- Insert KnowledgeBaseDocument(tenant_id, source_url=url, status='processing', agent_id=None)
- Call ingest_document.delay(str(doc.id), str(tenant_id))
- Return 201
- Guard with require_tenant_admin
- GET /{tenant_id}/documents — list KbDocuments for tenant with status, chunk_count, created_at
- Guard with require_tenant_member (operators can view)
- DELETE /{tenant_id}/documents/{document_id} — delete document (CASCADE deletes chunks)
- Also delete file from MinIO if filename present
- Guard with require_tenant_admin
- POST /{tenant_id}/documents/{document_id}/reindex — delete existing chunks, re-dispatch ingest_document.delay
- Guard with require_tenant_admin
7. **Tests** (write BEFORE implementation per tdd=true):
- test_extractors.py: test each format extraction with minimal valid files (create in-memory test fixtures using the libraries)
- test_kb_upload.py: test upload endpoint with mocked MinIO and mocked Celery task dispatch
</action>
<verify>
<automated>cd /home/adelorenzo/repos/konstruct && python -m pytest tests/unit/test_extractors.py tests/unit/test_kb_upload.py -x -q</automated>
</verify>
<done>Migration 013 exists with all schema changes. Text extractors handle all 7 format families. KB API router has upload, list, delete, URL ingest, and reindex endpoints. All unit tests pass.</done>
</task>
<task type="auto" tdd="true">
<name>Task 2: Celery ingestion task, executor tenant_id injection, KB search wiring</name>
<files>
packages/orchestrator/orchestrator/tasks.py,
packages/orchestrator/orchestrator/tools/ingest.py,
packages/orchestrator/orchestrator/tools/executor.py,
packages/orchestrator/orchestrator/tools/builtins/kb_search.py,
packages/orchestrator/orchestrator/tools/builtins/web_search.py,
tests/unit/test_ingestion.py,
tests/unit/test_executor_injection.py
</files>
<behavior>
- chunk_text("hello world " * 100, chunk_size=500, overlap=50) returns overlapping chunks of correct size
- ingest_document_pipeline fetches file from MinIO, extracts text, chunks, embeds, inserts kb_chunks rows, updates status to 'ready'
- ingest_document_pipeline sets status='error' with error_message on failure
- execute_tool injects tenant_id and agent_id into handler kwargs before calling handler
- web_search reads BRAVE_API_KEY from settings (not os.getenv) for consistency
- kb_search receives injected tenant_id from executor
</behavior>
<action>
1. **Chunking + ingestion logic** (`packages/orchestrator/orchestrator/tools/ingest.py`):
- chunk_text(text: str, chunk_size: int = 500, overlap: int = 50) -> list[str]
- Simple sliding window chunker, strip empty chunks
- async ingest_document_pipeline(document_id: str, tenant_id: str) -> None:
- Load KnowledgeBaseDocument from DB by ID (use RLS with tenant_id)
- If filename: download file bytes from MinIO (boto3 client, kb-documents bucket, key: {tenant_id}/{document_id}/{filename})
- If source_url and source_url contains "youtube.com" or "youtu.be": use youtube_transcript_api to fetch transcript
- If source_url and not YouTube: use firecrawl-py to scrape URL to markdown (graceful error if FIRECRAWL_API_KEY not set)
- Call extract_text(filename, file_bytes) for file uploads
- Call chunk_text(text) on extracted text
- Batch embed chunks using embed_texts() from embedder.py
- INSERT kb_chunks rows with embedding vectors (use raw SQL text() with CAST(:embedding AS vector) pattern from kb_search.py)
- UPDATE kb_documents SET status='ready', chunk_count=len(chunks)
- On any error: UPDATE kb_documents SET status='error', error_message=str(exc)
2. **Celery task** in `packages/orchestrator/orchestrator/tasks.py`:
- Add ingest_document Celery task (sync def with asyncio.run per hard architectural constraint)
- @celery_app.task(bind=True, max_retries=2, ignore_result=True)
- def ingest_document(self, document_id: str, tenant_id: str) -> None
- Calls asyncio.run(ingest_document_pipeline(document_id, tenant_id))
- On exception: asyncio.run to mark document as error, then self.retry(countdown=60)
3. **Executor tenant_id injection** (`packages/orchestrator/orchestrator/tools/executor.py`):
- Before calling tool.handler(**args), inject tenant_id and agent_id as string kwargs:
args["tenant_id"] = str(tenant_id)
args["agent_id"] = str(agent_id)
- This makes kb_search, calendar_lookup, and future context-aware tools work without LLM needing to know tenant context
- Place injection AFTER schema validation (line ~126) so the injected keys don't fail validation
4. **Update web_search.py**: Change `os.getenv("BRAVE_API_KEY", "")` to import settings from shared.config and use `settings.brave_api_key` for consistency with platform-wide config pattern.
5. **Tests** (write BEFORE implementation):
- test_ingestion.py: test chunk_text with various inputs, test ingest_document_pipeline with mocked MinIO/DB/embedder
- test_executor_injection.py: test that execute_tool injects tenant_id/agent_id into handler kwargs
</action>
<verify>
<automated>cd /home/adelorenzo/repos/konstruct && python -m pytest tests/unit/test_ingestion.py tests/unit/test_executor_injection.py -x -q</automated>
</verify>
<done>Celery ingest_document task dispatches async ingestion pipeline. Pipeline downloads files from MinIO, extracts text, chunks, embeds, and stores in kb_chunks. Executor injects tenant_id/agent_id into all tool handlers. web_search uses shared config. All tests pass.</done>
</task>
</tasks>
<verification>
- Migration 013 applies cleanly: `cd /home/adelorenzo/repos/konstruct && alembic upgrade head`
- All unit tests pass: `pytest tests/unit/test_extractors.py tests/unit/test_kb_upload.py tests/unit/test_ingestion.py tests/unit/test_executor_injection.py -x -q`
- KB API router mounts and serves: import kb_router without errors
- Executor properly injects tenant context into tool handlers
</verification>
<success_criteria>
- KnowledgeBaseDocument has status, error_message, chunk_count columns; agent_id is nullable
- channel_connections CHECK constraint includes 'google_calendar'
- Text extraction works for PDF, DOCX, PPTX, XLSX, CSV, TXT, MD
- KB upload endpoint accepts files and dispatches Celery task
- KB list/delete/reindex endpoints work
- URL and YouTube ingestion endpoints dispatch Celery tasks
- Celery ingestion pipeline: extract -> chunk -> embed -> store
- Tool executor injects tenant_id and agent_id into handler kwargs
- BRAVE_API_KEY and FIRECRAWL_API_KEY in shared config
- All unit tests pass
</success_criteria>
<output>
After completion, create `.planning/phases/10-agent-capabilities/10-01-SUMMARY.md`
</output>

262
.planning/phases/10-agent-capabilities/10-02-PLAN.md Normal file
View File

@@ -0,0 +1,262 @@
---
phase: 10-agent-capabilities
plan: 02
type: execute
wave: 1
depends_on: []
files_modified:
- packages/shared/shared/api/calendar_auth.py
- packages/orchestrator/orchestrator/tools/builtins/calendar_lookup.py
- packages/orchestrator/orchestrator/tools/registry.py
- tests/unit/test_calendar_lookup.py
- tests/unit/test_calendar_auth.py
autonomous: true
requirements:
- CAP-05
- CAP-06
user_setup:
- service: google-cloud
why: "Google Calendar OAuth for per-tenant calendar access"
env_vars:
- name: GOOGLE_CLIENT_ID
source: "Google Cloud Console -> APIs & Services -> Credentials -> OAuth 2.0 Client ID (Web application)"
- name: GOOGLE_CLIENT_SECRET
source: "Google Cloud Console -> APIs & Services -> Credentials -> OAuth 2.0 Client ID secret"
dashboard_config:
- task: "Create OAuth 2.0 Client ID (Web application type)"
location: "Google Cloud Console -> APIs & Services -> Credentials"
- task: "Add authorized redirect URI: {PORTAL_URL}/api/portal/calendar/callback"
location: "Google Cloud Console -> Credentials -> OAuth client -> Authorized redirect URIs"
- task: "Enable Google Calendar API"
location: "Google Cloud Console -> APIs & Services -> Library -> Google Calendar API"
must_haves:
truths:
- "Tenant admin can initiate Google Calendar OAuth from the portal and authorize calendar access"
- "Calendar OAuth callback exchanges code for tokens and stores them encrypted per tenant"
- "Calendar tool reads per-tenant OAuth tokens from channel_connections and calls Google Calendar API"
- "Calendar tool supports list events, check availability, and create event actions"
- "Token auto-refresh works — expired access tokens are refreshed via stored refresh_token and written back to DB"
- "Tool results are formatted as natural language (no raw JSON)"
artifacts:
- path: "packages/shared/shared/api/calendar_auth.py"
provides: "Google Calendar OAuth install + callback endpoints"
exports: ["calendar_auth_router"]
- path: "packages/orchestrator/orchestrator/tools/builtins/calendar_lookup.py"
provides: "Per-tenant OAuth calendar tool with list/create/check_availability"
exports: ["calendar_lookup"]
- path: "tests/unit/test_calendar_lookup.py"
provides: "Unit tests for calendar tool with mocked Google API"
key_links:
- from: "packages/shared/shared/api/calendar_auth.py"
to: "channel_connections table"
via: "Upsert ChannelConnection(channel_type='google_calendar') with encrypted token"
pattern: "google_calendar.*encrypt"
- from: "packages/orchestrator/orchestrator/tools/builtins/calendar_lookup.py"
to: "channel_connections table"
via: "Load encrypted token, decrypt, build Credentials, call Google API"
pattern: "Credentials.*refresh_token"
---
<objective>
Build Google Calendar OAuth per-tenant integration and replace the service-account stub with full CRUD calendar tool.
Purpose: Enables CAP-05 (calendar availability checking + event creation) by replacing the service account stub in calendar_lookup.py with per-tenant OAuth token lookup. Also addresses CAP-06 (natural language tool results) by ensuring calendar and all tool outputs are formatted as readable text.
Output: Google Calendar OAuth install/callback endpoints, fully functional calendar_lookup tool with list/create/check_availability actions, encrypted per-tenant token storage, token auto-refresh with write-back.
</objective>
<execution_context>
@/home/adelorenzo/.claude/get-shit-done/workflows/execute-plan.md
@/home/adelorenzo/.claude/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/phases/10-agent-capabilities/10-CONTEXT.md
@.planning/phases/10-agent-capabilities/10-RESEARCH.md
<interfaces>
<!-- Existing OAuth pattern from Slack to reuse -->
From packages/shared/shared/api/channels.py:
```python
channels_router = APIRouter(prefix="/api/portal/channels", tags=["channels"])
def _generate_oauth_state(tenant_id: uuid.UUID) -> str:
"""HMAC-SHA256 signed state with embedded tenant_id + nonce."""
...
def _verify_oauth_state(state: str) -> uuid.UUID:
"""Verify HMAC signature, return tenant_id. Raises HTTPException on failure."""
...
```
From packages/shared/shared/crypto.py:
```python
class KeyEncryptionService:
def encrypt(self, plaintext: str) -> str: ...
def decrypt(self, ciphertext: str) -> str: ...
```
From packages/shared/shared/models/tenant.py:
```python
class ChannelConnection(Base):
__tablename__ = "channel_connections"
id: Mapped[uuid.UUID]
tenant_id: Mapped[uuid.UUID]
channel_type: Mapped[ChannelTypeEnum] # TEXT + CHECK in DB
workspace_id: Mapped[str]
config: Mapped[dict] # JSON — stores encrypted token
created_at: Mapped[datetime]
```
From packages/shared/shared/config.py (after Plan 01):
```python
class Settings(BaseSettings):
google_client_id: str = ""
google_client_secret: str = ""
```
</interfaces>
</context>
<tasks>
<task type="auto" tdd="true">
<name>Task 1: Google Calendar OAuth endpoints and calendar tool replacement</name>
<files>
packages/shared/shared/api/calendar_auth.py,
packages/orchestrator/orchestrator/tools/builtins/calendar_lookup.py,
tests/unit/test_calendar_lookup.py,
tests/unit/test_calendar_auth.py
</files>
<behavior>
- OAuth install endpoint returns redirect URL with HMAC-signed state containing tenant_id
- OAuth callback verifies HMAC state, exchanges code for tokens, encrypts and stores in channel_connections as google_calendar type
- OAuth callback redirects to portal settings page with connected=true param
- calendar_lookup(date, action="list", tenant_id=...) loads encrypted token from DB, decrypts, calls Google Calendar API, returns formatted event list
- calendar_lookup(date, action="create", event_summary=..., event_start=..., event_end=..., tenant_id=...) creates a Google Calendar event and returns confirmation
- calendar_lookup(date, action="check_availability", tenant_id=...) returns free/busy summary
- calendar_lookup returns informative message when no Google Calendar is connected for tenant
- Token refresh: if access_token expired, google-auth auto-refreshes, updated token written back to DB
- All results are natural language strings, not raw JSON
</behavior>
<action>
1. **Calendar OAuth router** (`packages/shared/shared/api/calendar_auth.py`):
- calendar_auth_router = APIRouter(prefix="/api/portal/calendar", tags=["calendar"])
- Import and reuse _generate_oauth_state / _verify_oauth_state from channels.py (or extract to shared utility if private)
- If they are private (_prefix), create equivalent functions in this module using the same HMAC pattern
- GET /install?tenant_id={id}:
- Guard with require_tenant_admin
- Generate HMAC-signed state with tenant_id
- Build Google OAuth URL: https://accounts.google.com/o/oauth2/v2/auth with:
- client_id from settings
- redirect_uri = settings.portal_url + "/api/portal/calendar/callback"
- scope = "https://www.googleapis.com/auth/calendar" (full read+write per locked decision)
- state = hmac_state
- access_type = "offline" (to get refresh_token)
- prompt = "consent" (force consent to always get refresh_token)
- Return {"url": oauth_url}
- GET /callback?code={code}&state={state}:
- NO auth guard (external redirect from Google — no session cookie)
- Verify HMAC state to recover tenant_id
- Exchange code for tokens using google_auth_oauthlib or httpx POST to https://oauth2.googleapis.com/token
- Encrypt token JSON with KeyEncryptionService (Fernet)
- Upsert ChannelConnection(tenant_id=tenant_id, channel_type="google_calendar", workspace_id=str(tenant_id), config={"token": encrypted_token})
- Redirect to portal /settings?calendar=connected
- GET /{tenant_id}/status:
- Guard with require_tenant_member
- Check if ChannelConnection with channel_type='google_calendar' exists for tenant
- Return {"connected": true/false}
2. **Replace calendar_lookup.py** entirely:
- Remove all service account code
- New signature: async def calendar_lookup(date: str, action: str = "list", event_summary: str | None = None, event_start: str | None = None, event_end: str | None = None, calendar_id: str = "primary", tenant_id: str | None = None, **kwargs) -> str
- If no tenant_id: return "Calendar not available: missing tenant context."
- Load ChannelConnection(channel_type='google_calendar', tenant_id=tenant_uuid) from DB
- If not found: return "Google Calendar is not connected for this tenant. Ask an admin to connect it in Settings."
- Decrypt token JSON, build google.oauth2.credentials.Credentials
- Build Calendar service: build("calendar", "v3", credentials=creds, cache_discovery=False)
- Run API call in thread executor (same pattern as original — avoid blocking event loop)
- action="list": list events for date, format as "Calendar events for {date}:\n- {time}: {summary}\n..."
- action="check_availability": list events, format as "Busy slots on {date}:\n..." or "No events — the entire day is free."
- action="create": insert event with summary, start, end, return "Event created: {summary} from {start} to {end}"
- After API call: check if credentials.token changed (refresh occurred) — if so, encrypt and UPDATE channel_connections.config with new token
- All errors return human-readable messages, never raw exceptions
3. **Update tool registry** if needed — ensure calendar_lookup parameters schema includes action, event_summary, event_start, event_end fields so LLM knows about CRUD capabilities. Check packages/orchestrator/orchestrator/tools/registry.py for the calendar_lookup entry and update its parameters JSON schema.
4. **Tests** (write BEFORE implementation):
- test_calendar_lookup.py: mock Google Calendar API (googleapiclient.discovery.build), mock DB session to return encrypted token, test list/create/check_availability actions, test "not connected" path, test token refresh write-back
- test_calendar_auth.py: mock httpx for token exchange, test HMAC state generation/verification, test callback stores encrypted token
</action>
<verify>
<automated>cd /home/adelorenzo/repos/konstruct && python -m pytest tests/unit/test_calendar_lookup.py tests/unit/test_calendar_auth.py -x -q</automated>
</verify>
<done>Google Calendar OAuth install/callback endpoints work. Calendar tool loads per-tenant tokens, supports list/create/check_availability, formats results as natural language. Token refresh writes back to DB. Service account stub completely removed. All tests pass.</done>
</task>
<task type="auto">
<name>Task 2: Mount new API routers on gateway and update tool response formatting</name>
<files>
packages/gateway/gateway/main.py,
packages/orchestrator/orchestrator/tools/registry.py,
packages/orchestrator/orchestrator/agents/prompt.py
</files>
<action>
1. **Mount routers on gateway** (`packages/gateway/gateway/main.py`):
- Import kb_router from shared.api.kb and include it on the FastAPI app (same pattern as channels_router, billing_router, etc.)
- Import calendar_auth_router from shared.api.calendar_auth and include it on the app
- Verify both are accessible via curl or import
2. **Update tool registry** (`packages/orchestrator/orchestrator/tools/registry.py`):
- Update calendar_lookup tool definition's parameters schema to include:
- action: enum ["list", "check_availability", "create"] (required)
- event_summary: string (optional, for create)
- event_start: string (optional, ISO 8601 with timezone, for create)
- event_end: string (optional, ISO 8601 with timezone, for create)
- date: string (required, YYYY-MM-DD format)
- Update description to mention CRUD capabilities: "Look up, check availability, or create calendar events"
3. **Tool result formatting check** (CAP-06):
- Review agent runner prompt — the LLM already receives tool results as 'tool' role messages and formulates a response. Verify the system prompt does NOT contain instructions to dump raw JSON.
- If the system prompt builder (`packages/orchestrator/orchestrator/agents/prompt.py` or similar) has tool-related instructions, ensure it says: "When using tool results, incorporate the information naturally into your response. Never show raw data or JSON to the user."
- If no such instruction exists, add it as a tool usage instruction appended to the system prompt when tools are assigned.
4. **Verify CAP-04 (HTTP request tool)**: Confirm http_request.py needs no changes — it already works. Just verify it's in the tool registry and functions correctly.
5. **Verify CAP-07 (audit logging)**: Confirm executor.py already calls audit_logger.log_tool_call() on every invocation (it does — verified in code review). No changes needed.
</action>
<verify>
<automated>cd /home/adelorenzo/repos/konstruct && python -c "from shared.api.kb import kb_router; from shared.api.calendar_auth import calendar_auth_router; print('Routers import OK')" && python -c "from orchestrator.tools.registry import TOOL_REGISTRY; print(f'Registry has {len(TOOL_REGISTRY)} tools')"</automated>
</verify>
<done>KB and Calendar Auth routers mounted on gateway. Calendar tool registry updated with CRUD parameters. System prompt includes tool result formatting instruction. CAP-04 (HTTP) confirmed working. CAP-07 (audit) confirmed working. All routers importable.</done>
</task>
</tasks>
<verification>
- Calendar OAuth endpoints accessible: GET /api/portal/calendar/install, GET /api/portal/calendar/callback
- KB API endpoints accessible: POST/GET/DELETE /api/portal/kb/{tenant_id}/documents
- Calendar tool supports list, create, check_availability actions
- All unit tests pass: `pytest tests/unit/test_calendar_lookup.py tests/unit/test_calendar_auth.py -x -q`
- Tool registry has updated calendar_lookup schema with CRUD params
</verification>
<success_criteria>
- Google Calendar OAuth flow: install -> Google consent -> callback -> encrypted token stored in channel_connections
- Calendar tool reads per-tenant tokens and calls Google Calendar API for list, create, and availability check
- Token auto-refresh works with write-back to DB
- Natural language formatting on all tool results (no raw JSON)
- All new routers mounted on gateway
- CAP-04 and CAP-07 confirmed already working
- All unit tests pass
</success_criteria>
<output>
After completion, create `.planning/phases/10-agent-capabilities/10-02-SUMMARY.md`
</output>

197
.planning/phases/10-agent-capabilities/10-03-PLAN.md Normal file
View File

@@ -0,0 +1,197 @@
---
phase: 10-agent-capabilities
plan: 03
type: execute
wave: 2
depends_on: ["10-01"]
files_modified:
- packages/portal/app/(dashboard)/knowledge-base/page.tsx
- packages/portal/components/kb/document-list.tsx
- packages/portal/components/kb/upload-dialog.tsx
- packages/portal/components/kb/url-ingest-dialog.tsx
- packages/portal/components/nav/sidebar.tsx
- packages/portal/lib/api.ts
autonomous: false
requirements:
- CAP-03
must_haves:
truths:
- "Operators can see a Knowledge Base page in the portal navigation"
- "Operators can upload files via drag-and-drop or file picker dialog"
- "Operators can add URLs (web pages) and YouTube URLs for ingestion"
- "Uploaded documents show processing status (processing, ready, error) with live polling"
- "Operators can delete documents from the knowledge base"
- "Operators can re-index a document"
- "Customer operators can view the KB but not upload or delete (RBAC)"
artifacts:
- path: "packages/portal/app/(dashboard)/knowledge-base/page.tsx"
provides: "KB management page with document list, upload, and URL ingestion"
min_lines: 50
- path: "packages/portal/components/kb/document-list.tsx"
provides: "Document list component with status badges and action buttons"
- path: "packages/portal/components/kb/upload-dialog.tsx"
provides: "File upload dialog with drag-and-drop and file picker"
key_links:
- from: "packages/portal/app/(dashboard)/knowledge-base/page.tsx"
to: "/api/portal/kb/{tenant_id}/documents"
via: "TanStack Query fetch + polling"
pattern: "useQuery.*kb.*documents"
- from: "packages/portal/components/kb/upload-dialog.tsx"
to: "/api/portal/kb/{tenant_id}/documents"
via: "FormData multipart POST"
pattern: "FormData.*upload"
---
<objective>
Build the Knowledge Base management page in the portal where operators can upload documents, add URLs, view processing status, and manage their tenant's knowledge base.
Purpose: Completes CAP-03 by providing the user-facing interface for document management. Operators need to see what's in their KB, upload new content, and monitor ingestion status.
Output: Fully functional /knowledge-base portal page with file upload, URL/YouTube ingestion, document list with status polling, delete, and re-index.
</objective>
<execution_context>
@/home/adelorenzo/.claude/get-shit-done/workflows/execute-plan.md
@/home/adelorenzo/.claude/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/phases/10-agent-capabilities/10-CONTEXT.md
@.planning/phases/10-agent-capabilities/10-01-SUMMARY.md
<interfaces>
<!-- KB API endpoints from Plan 01 -->
POST /api/portal/kb/{tenant_id}/documents — multipart file upload, returns 201 {id, filename, status}
POST /api/portal/kb/{tenant_id}/documents/url — JSON {url, source_type}, returns 201 {id, source_url, status}
GET /api/portal/kb/{tenant_id}/documents — returns [{id, filename, source_url, content_type, status, error_message, chunk_count, created_at}]
DELETE /api/portal/kb/{tenant_id}/documents/{document_id} — returns 204
POST /api/portal/kb/{tenant_id}/documents/{document_id}/reindex — returns 200
<!-- Portal patterns -->
- TanStack Query for data fetching (useQuery, useMutation)
- shadcn/ui components (Button, Dialog, Badge, Table, etc.)
- Tailwind CSS for styling
- next-intl useTranslations() for i18n
- RBAC: session.user.role determines admin vs operator capabilities
</interfaces>
</context>
<tasks>
<task type="auto">
<name>Task 1: Knowledge Base page with document list, upload, and URL ingestion</name>
<files>
packages/portal/app/(dashboard)/knowledge-base/page.tsx,
packages/portal/components/kb/document-list.tsx,
packages/portal/components/kb/upload-dialog.tsx,
packages/portal/components/kb/url-ingest-dialog.tsx,
packages/portal/lib/api.ts,
packages/portal/components/nav/sidebar.tsx
</files>
<action>
1. **Add KB link to navigation** (`sidebar.tsx` or equivalent nav component):
- Add "Knowledge Base" link to sidebar nav, visible for platform_admin and customer_admin roles
- customer_operator can view (read-only) — add to nav but upload/delete buttons hidden
- Icon: use a document/book icon from lucide-react
2. **KB page** (`packages/portal/app/(dashboard)/knowledge-base/page.tsx`):
- Server Component wrapper that renders the client KB content
- Page title: "Knowledge Base" with subtitle showing tenant context
- Two action buttons for admins: "Upload Files" (opens upload dialog), "Add URL" (opens URL dialog)
- Document list component below actions
- Use tenant_id from session/route context (same pattern as other dashboard pages)
3. **Document list** (`packages/portal/components/kb/document-list.tsx`):
- Client component using useQuery to fetch GET /api/portal/kb/{tenant_id}/documents
- Poll every 5 seconds while any document has status='processing' (refetchInterval: 5000 conditional)
- Table with columns: Name (filename or source_url), Type (file/url/youtube), Status (badge), Chunks, Date, Actions
- Status badges: "Processing" (amber/spinning), "Ready" (green), "Error" (red with tooltip showing error_message)
- Actions per row (admin only): Delete button, Re-index button
- Empty state: "No documents in knowledge base yet. Upload files or add URLs to get started."
- Delete: useMutation calling DELETE endpoint, invalidate query on success, confirm dialog before delete
- Re-index: useMutation calling POST reindex endpoint, invalidate query on success
4. **Upload dialog** (`packages/portal/components/kb/upload-dialog.tsx`):
- shadcn/ui Dialog component
- Drag-and-drop zone (onDragOver, onDrop handlers) with visual feedback
- File picker button (input type="file" with accept for supported extensions: .pdf,.docx,.pptx,.xlsx,.csv,.txt,.md)
- Support multiple file selection
- Show selected files list before upload
- Upload button: for each file, POST FormData to /api/portal/kb/{tenant_id}/documents
- Show upload progress (file-by-file)
- Close dialog and invalidate document list query on success
- Error handling: show toast on failure
5. **URL ingest dialog** (`packages/portal/components/kb/url-ingest-dialog.tsx`):
- shadcn/ui Dialog component
- Input field for URL
- Radio or select for source type: "Web Page" or "YouTube Video"
- Auto-detect: if URL contains youtube.com or youtu.be, default to YouTube
- Submit: POST to /api/portal/kb/{tenant_id}/documents/url
- Close dialog and invalidate document list query on success
6. **API client updates** (`packages/portal/lib/api.ts`):
- Add KB API functions: fetchKbDocuments, uploadKbDocument, addKbUrl, deleteKbDocument, reindexKbDocument
- Use the same fetch wrapper pattern as existing API calls
7. **i18n**: Add English, Spanish, and Portuguese translations for KB page strings (following existing i18n pattern with next-intl message files). Add keys like: kb.title, kb.upload, kb.addUrl, kb.empty, kb.status.processing, kb.status.ready, kb.status.error, kb.delete.confirm, etc.
</action>
<verify>
<automated>cd /home/adelorenzo/repos/konstruct/packages/portal && npx next build 2>&1 | tail -5</automated>
</verify>
<done>Knowledge Base page exists at /knowledge-base with document list, file upload dialog (drag-and-drop + picker), URL/YouTube ingest dialog, status polling, delete, and re-index. Navigation updated. i18n strings added for all three languages. Portal builds successfully.</done>
</task>
<task type="checkpoint:human-verify" gate="blocking">
<name>Task 2: Human verification of Knowledge Base portal page</name>
<files>packages/portal/app/(dashboard)/knowledge-base/page.tsx</files>
<action>
Verify the Knowledge Base management page in the portal:
- File upload via drag-and-drop and file picker (PDF, DOCX, PPTX, XLSX, CSV, TXT, MD)
- URL ingestion (web pages via Firecrawl, YouTube transcripts)
- Document list with live processing status (processing/ready/error)
- Delete and re-index actions
- RBAC: admins can upload/delete, operators can only view
Steps:
1. Navigate to the portal and confirm "Knowledge Base" appears in the sidebar navigation
2. Click Knowledge Base — verify the page loads with empty state message
3. Click "Upload Files" — verify drag-and-drop zone and file picker appear
4. Upload a small PDF or TXT file — verify it appears in the document list with "Processing" status
5. Wait for processing to complete — verify status changes to "Ready" with chunk count
6. Click "Add URL" — verify URL input dialog with web/YouTube type selector
7. Add a URL — verify it appears in the list and processes
8. Click delete on a document — verify confirmation dialog, then document removed
9. If logged in as customer_operator — verify upload/delete buttons are hidden but document list is visible
</action>
<verify>Human verification of KB page functionality and RBAC</verify>
<done>KB page approved by human testing — upload, URL ingest, status polling, delete, re-index, and RBAC all working</done>
</task>
</tasks>
<verification>
- Portal builds: `cd packages/portal && npx next build`
- KB page renders at /knowledge-base
- Document upload triggers backend ingestion
- Status polling shows processing -> ready transition
- RBAC enforced on upload/delete actions
</verification>
<success_criteria>
- Knowledge Base page accessible in portal navigation
- File upload works with drag-and-drop and file picker
- URL and YouTube ingestion works
- Document list shows live processing status with polling
- Delete and re-index work
- RBAC enforced (admin: full access, operator: view only)
- All three languages have KB translations
- Human verification approved
</success_criteria>
<output>
After completion, create `.planning/phases/10-agent-capabilities/10-03-SUMMARY.md`
</output>