---
phase: 10-agent-capabilities
plan: 01
subsystem: api
tags: [knowledge-base, celery, minio, pgvector, pdf, docx, pptx, embeddings, text-extraction]

# Dependency graph
requires:
  - phase: 02-agent-features
    provides: pgvector kb_chunks table, embed_texts, kb_search tool, executor framework
  - phase: 01-foundation
    provides: Celery task infrastructure, MinIO, asyncio.run pattern, RLS session factory

provides:
  - Migration 014: kb_documents status/error_message/chunk_count columns, agent_id nullable
  - Text extractors for PDF, DOCX, PPTX, XLSX/XLS, CSV, TXT, MD
  - KB management API: upload file, ingest URL/YouTube, list, delete, reindex endpoints
  - Celery ingest_document task: download → extract → chunk → embed → store pipeline
  - Executor tenant_id/agent_id injection into all tool handlers
  - brave_api_key + firecrawl_api_key + google_client_id/secret + minio_kb_bucket in shared config

affects: [10-02, 10-03, 10-04, kb-search, agent-tools]

# Tech tracking
tech-stack:
  added:
    - pypdf (PDF text extraction)
    - python-docx (DOCX paragraph extraction)
    - python-pptx (PPTX slide text extraction)
    - openpyxl (XLSX/XLS reading via pandas)
    - pandas (spreadsheet to CSV conversion)
    - firecrawl-py (URL scraping for KB ingestion)
    - youtube-transcript-api (YouTube video transcripts)
    - google-api-python-client (Google API client)
    - google-auth-oauthlib (Google OAuth)
  patterns:
    - Lazy Celery task import in kb.py to avoid circular dependencies
    - Executor context injection pattern (tenant_id/agent_id injected after schema validation)
    - chunk_text sliding window chunker (default 500 chars, 50 overlap)
    - ingest_document_pipeline: fetch → extract → chunk → embed → store in single async transaction

key-files:
  created:
    - migrations/versions/014_kb_status.py
    - packages/orchestrator/orchestrator/tools/extractors.py
    - packages/orchestrator/orchestrator/tools/ingest.py
    - packages/shared/shared/api/kb.py
    - tests/unit/test_extractors.py
    - tests/unit/test_kb_upload.py
    - tests/unit/test_ingestion.py
    - tests/unit/test_executor_injection.py
  modified:
    - packages/shared/shared/models/kb.py (status/error_message/chunk_count columns, agent_id nullable)
    - packages/shared/shared/models/tenant.py (GOOGLE_CALENDAR added to ChannelTypeEnum)
    - packages/shared/shared/config.py (brave_api_key, firecrawl_api_key, google_client_id/secret, minio_kb_bucket)
    - packages/orchestrator/orchestrator/tools/executor.py (tenant_id/agent_id injection)
    - packages/orchestrator/orchestrator/tools/builtins/web_search.py (use settings.brave_api_key)
    - packages/orchestrator/orchestrator/tasks.py (ingest_document Celery task added)
    - packages/orchestrator/pyproject.toml (new dependencies)
    - .env.example (BRAVE_API_KEY, FIRECRAWL_API_KEY, GOOGLE_CLIENT_ID/SECRET, MINIO_KB_BUCKET)

key-decisions:
  - "Migration numbered 014 (not 013) — 013 was already used by google_calendar channel type migration from prior session"
  - "KB is per-tenant not per-agent — agent_id made nullable in kb_documents"
  - "Executor injects tenant_id/agent_id as strings after schema validation to avoid schema rejections"
  - "Lazy import of ingest_document task in kb.py router via _get_ingest_task() — avoids shared→orchestrator circular dependency"
  - "ingest_document_pipeline uses ORM select for document fetch (testable) and raw SQL for chunk inserts (pgvector CAST pattern)"
  - "web_search migrated from os.getenv to settings.brave_api_key — consistent with platform-wide config pattern"
  - "chunk_text returns empty list for empty/whitespace text, not error — silent skip is safer in async pipeline"
  - "PDF extraction returns warning message (not exception) for image-only PDFs with < 100 chars extracted"

patterns-established:
  - "Context injection pattern: executor injects tenant_id/agent_id as str kwargs after schema validation, before handler call"
  - "KB ingestion pipeline: try/except updates doc.status to error with error_message on any failure"
  - "Lazy circular dep avoidance: _get_ingest_task() function returns task at call time, imported inside function"

requirements-completed: [CAP-01, CAP-02, CAP-03, CAP-04, CAP-07]

# Metrics
duration: 11min
completed: 2026-03-26
---

# Phase 10 Plan 01: KB Ingestion Pipeline Summary

**Document ingestion pipeline for KB search: text extractors (PDF/DOCX/PPTX/XLSX/CSV/TXT/MD), Celery async ingest task, executor tenant context injection, and KB management REST API**

## Performance

- **Duration:** 11 min
- **Started:** 2026-03-26T14:59:19Z
- **Completed:** 2026-03-26T15:10:06Z
- **Tasks:** 2
- **Files modified:** 16

## Accomplishments

- Full document text extraction for 7 format families using pypdf, python-docx, python-pptx, pandas, plus CSV/TXT/MD decode
- KB management REST API with file upload, URL/YouTube ingest, list, delete, and reindex endpoints
- Celery `ingest_document` task runs async pipeline: MinIO download → extract → chunk (500 char sliding window) → embed (all-MiniLM-L6-v2) → store kb_chunks
- Tool executor now injects `tenant_id` and `agent_id` as string kwargs into every tool handler before invocation
- 31 unit tests pass across all 4 test files

## Task Commits

1. **Task 1: Migration 013, ORM updates, config settings, text extractors, KB API router** - `e8d3e8a` (feat)
2. **Task 2: Celery ingestion task, executor tenant_id injection, KB search wiring** - `9c7686a` (feat)

## Files Created/Modified

- `migrations/versions/014_kb_status.py` - Migration: add status/error_message/chunk_count to kb_documents, make agent_id nullable
- `packages/shared/shared/models/kb.py` - Added status/error_message/chunk_count mapped columns, agent_id nullable
- `packages/shared/shared/models/tenant.py` - Added GOOGLE_CALENDAR and WEB to ChannelTypeEnum
- `packages/shared/shared/config.py` - Added brave_api_key, firecrawl_api_key, google_client_id, google_client_secret, minio_kb_bucket
- `packages/shared/shared/api/kb.py` - New KB management API router (5 endpoints)
- `packages/orchestrator/orchestrator/tools/extractors.py` - Text extraction for all 7 formats
- `packages/orchestrator/orchestrator/tools/ingest.py` - chunk_text + ingest_document_pipeline
- `packages/orchestrator/orchestrator/tasks.py` - Added ingest_document Celery task
- `packages/orchestrator/orchestrator/tools/executor.py` - tenant_id/agent_id injection after schema validation
- `packages/orchestrator/orchestrator/tools/builtins/web_search.py` - Migrated to settings.brave_api_key
- `packages/orchestrator/pyproject.toml` - Added 8 new dependencies
- `.env.example` - Added BRAVE_API_KEY, FIRECRAWL_API_KEY, GOOGLE_CLIENT_ID/SECRET, MINIO_KB_BUCKET

## Decisions Made

- Migration numbered 014 (not 013) — 013 was already used by a google_calendar channel type migration from a prior session
- KB is per-tenant not per-agent — agent_id made nullable in kb_documents
- Executor injects tenant_id/agent_id as strings after schema validation to avoid triggering schema rejections
- Lazy import of ingest_document task in kb.py via `_get_ingest_task()` function — avoids shared→orchestrator circular dependency at module load time
- `ingest_document_pipeline` uses ORM `select(KnowledgeBaseDocument)` for document fetch (testable via mock) and raw SQL for chunk INSERTs (pgvector CAST pattern)

## Deviations from Plan

### Auto-fixed Issues

**1. [Rule 3 - Blocking] Migration renumbered from 013 to 014**
- **Found during:** Task 1 (Migration creation)
- **Issue:** Migration 013 already existed (`013_google_calendar_channel.py`) from a prior phase session
- **Fix:** Renamed migration file to `014_kb_status.py` with revision=014, down_revision=013
- **Files modified:** migrations/versions/014_kb_status.py
- **Verification:** File renamed, revision chain intact
- **Committed in:** e8d3e8a (Task 1 commit)

**2. [Rule 2 - Missing Critical] Added WEB to ChannelTypeEnum alongside GOOGLE_CALENDAR**
- **Found during:** Task 1 (tenant.py update)
- **Issue:** WEB channel type was missing from the enum (google_calendar was not the only new type)
- **Fix:** Added both `WEB = "web"` and `GOOGLE_CALENDAR = "google_calendar"` to ChannelTypeEnum
- **Files modified:** packages/shared/shared/models/tenant.py
- **Committed in:** e8d3e8a (Task 1 commit)

**3. [Rule 1 - Bug] FastAPI Depends overrides required for KB upload tests**
- **Found during:** Task 1 (test_kb_upload.py)
- **Issue:** Initial test approach used `patch()` to mock auth deps but FastAPI calls Depends directly — 422 returned
- **Fix:** Updated test to use `app.dependency_overrides` (correct FastAPI testing pattern)
- **Files modified:** tests/unit/test_kb_upload.py
- **Committed in:** e8d3e8a (Task 1 commit)

---

**Total deviations:** 3 auto-fixed (1 blocking, 1 missing critical, 1 bug)
**Impact on plan:** All fixes necessary for correctness. No scope creep.

## Issues Encountered

None beyond the deviations documented above.

## User Setup Required

New environment variables needed:
- `BRAVE_API_KEY` — Brave Search API key (https://brave.com/search/api/)
- `FIRECRAWL_API_KEY` — Firecrawl API key for URL scraping (https://firecrawl.dev)
- `GOOGLE_CLIENT_ID` / `GOOGLE_CLIENT_SECRET` — Google OAuth credentials
- `MINIO_KB_BUCKET` — MinIO bucket for KB documents (default: `kb-documents`)

## Next Phase Readiness

- KB ingestion pipeline is fully functional and tested
- kb_search tool already wired to query kb_chunks via pgvector (existing from Phase 2)
- Executor now injects tenant context — all context-aware tools (kb_search, calendar) will work correctly
- Ready for 10-02 (calendar tool) and 10-03 (any remaining agent capability work)

## Self-Check: PASSED

All files found on disk. All commits verified in git log.

---
*Phase: 10-agent-capabilities*
*Completed: 2026-03-26*