diff --git a/.planning/REQUIREMENTS.md b/.planning/REQUIREMENTS.md index a86aa34..faae98c 100644 --- a/.planning/REQUIREMENTS.md +++ b/.planning/REQUIREMENTS.md @@ -27,8 +27,8 @@ Requirements for beta-ready release. Each maps to roadmap phases. ### LLM Backend -- [ ] **LLM-01**: LiteLLM router abstracts LLM provider selection with fallback routing -- [ ] **LLM-02**: Platform supports Ollama (local) and commercial APIs (Anthropic, OpenAI) as LLM providers +- [x] **LLM-01**: LiteLLM router abstracts LLM provider selection with fallback routing +- [x] **LLM-02**: Platform supports Ollama (local) and commercial APIs (Anthropic, OpenAI) as LLM providers - [ ] **LLM-03**: Tenant can provide their own API keys for supported LLM providers (BYO keys, encrypted at rest) ⚠️ CONFLICT: listed as v1 here but out-of-scope in PROJECT.md — resolve before Phase 3 planning ### Multi-Tenancy & Security @@ -107,8 +107,8 @@ Which phases cover which requirements. Updated during roadmap creation. | AGNT-05 | Phase 2 | Pending | | AGNT-06 | Phase 2 | Pending | | AGNT-07 | Phase 3 | Pending | -| LLM-01 | Phase 1 | Pending | -| LLM-02 | Phase 1 | Pending | +| LLM-01 | Phase 1 | Complete | +| LLM-02 | Phase 1 | Complete | | LLM-03 | Phase 3 | Pending | | TNNT-01 | Phase 1 | Complete | | TNNT-02 | Phase 1 | Complete | diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index ec95158..3b577dc 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -77,7 +77,7 @@ Phases execute in numeric order: 1 → 2 → 3 | Phase | Plans Complete | Status | Completed | |-------|----------------|--------|-----------| -| 1. Foundation | 1/4 | In Progress| | +| 1. Foundation | 2/4 | In Progress| | | 2. Agent Features | 0/4 | Not started | - | | 3. Operator Experience | 0/2 | Not started | - | diff --git a/.planning/STATE.md b/.planning/STATE.md index 4d0ddf0..fff6279 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -3,14 +3,14 @@ gsd_state_version: 1.0 milestone: v1.0 milestone_name: milestone status: planning -stopped_at: Completed 01-foundation 01-01-PLAN.md -last_updated: "2026-03-23T15:59:38.482Z" +stopped_at: Completed 01-foundation 01-02-PLAN.md +last_updated: "2026-03-23T16:08:44.982Z" last_activity: 2026-03-23 — Roadmap created, ready for Phase 1 planning progress: total_phases: 3 completed_phases: 0 total_plans: 4 - completed_plans: 1 + completed_plans: 2 percent: 0 --- @@ -51,6 +51,7 @@ Progress: [░░░░░░░░░░] 0% *Updated after each plan completion* | Phase 01-foundation P01 | 12 | 2 tasks | 32 files | +| Phase 01-foundation P02 | 6 | 2 tasks | 15 files | ## Accumulated Context @@ -65,6 +66,10 @@ Recent decisions affecting current work: - [Phase 01-foundation]: PostgreSQL RLS with FORCE ROW LEVEL SECURITY chosen for tenant isolation; app connects as konstruct_app role (not superuser) - [Phase 01-foundation]: SET LOCAL app.current_tenant uses UUID-sanitized f-string (not parameterized) — asyncpg does not support prepared statement placeholders for SET LOCAL - [Phase 01-foundation]: channel_type stored as TEXT with CHECK constraint — native sa.Enum caused duplicate CREATE TYPE DDL in Alembic migrations +- [Phase 01-foundation]: LiteLLM pinned to ==1.82.5, not latest — September 2025 OOM regression in later versions +- [Phase 01-foundation]: Celery tasks are always sync def with asyncio.run() — hard architectural constraint, never async def +- [Phase 01-foundation]: AI transparency clause is unconditional in system prompt — agents must disclose AI identity when directly asked +- [Phase 01-foundation]: llm-pool port 8004 (consistent with shared/config.py llm_pool_url default, not plan-stated 8002) ### Pending Todos @@ -76,6 +81,6 @@ None yet. ## Session Continuity -Last session: 2026-03-23T15:59:38.480Z -Stopped at: Completed 01-foundation 01-01-PLAN.md +Last session: 2026-03-23T16:08:44.980Z +Stopped at: Completed 01-foundation 01-02-PLAN.md Resume file: None diff --git a/.planning/phases/01-foundation/01-02-SUMMARY.md b/.planning/phases/01-foundation/01-02-SUMMARY.md new file mode 100644 index 0000000..5a956bc --- /dev/null +++ b/.planning/phases/01-foundation/01-02-SUMMARY.md @@ -0,0 +1,162 @@ +--- +phase: 01-foundation +plan: 02 +subsystem: llm +tags: [litellm, celery, redis, ollama, anthropic, openai, fastapi, httpx, pytest] + +# Dependency graph +requires: + - phase: 01-foundation plan 01 + provides: "Shared models (KonstructMessage, Agent), shared config (settings), shared db (get_session, engine), shared rls (configure_rls_hook, current_tenant_id)" + +provides: + - "LLM Backend Pool FastAPI service (port 8004) with /complete and /health endpoints" + - "LiteLLM Router with fast (Ollama qwen3:8b) and quality (Anthropic claude-sonnet-4 + OpenAI gpt-4o) model groups" + - "Automatic fallback chain: quality providers -> fast group" + - "Celery app with Redis broker/backend (orchestrator.main)" + - "handle_message Celery task (sync def, asyncio.run pattern)" + - "System prompt builder: assembles system_prompt + identity + persona + AI transparency clause" + - "Agent runner: httpx POST to llm-pool /complete with polite fallback on error" + - "19 integration tests: 7 fallback routing tests (LLM-01), 12 provider config tests (LLM-02)" + +affects: + - "01-foundation plan 03 (Channel Gateway — dispatches handle_message tasks to Celery)" + - "All future orchestrator plans (must maintain sync-def Celery task pattern)" + - "Phase 2 memory and tool plans (extend _process_message pipeline)" + +# Tech tracking +tech-stack: + added: + - "litellm==1.82.5 (pinned — September 2025 OOM regression in later versions)" + - "celery[redis]>=5.4.0" + - "fastapi[standard] (added to llm-pool package)" + patterns: + - "Celery sync-def + asyncio.run() pattern for async work in tasks" + - "LiteLLM Router model groups (fast/quality) as abstraction over provider selection" + - "httpx.AsyncClient for service-to-service calls (orchestrator -> llm-pool)" + - "ContextVar (current_tenant_id) for RLS scope — set/reset around DB block" + +key-files: + created: + - packages/llm-pool/llm_pool/router.py + - packages/llm-pool/llm_pool/main.py + - packages/llm-pool/llm_pool/__init__.py + - packages/llm-pool/llm_pool/providers/__init__.py + - packages/orchestrator/orchestrator/main.py + - packages/orchestrator/orchestrator/tasks.py + - packages/orchestrator/orchestrator/agents/builder.py + - packages/orchestrator/orchestrator/agents/runner.py + - packages/orchestrator/orchestrator/__init__.py + - packages/orchestrator/orchestrator/agents/__init__.py + - tests/integration/test_llm_fallback.py + - tests/integration/test_llm_providers.py + modified: + - packages/llm-pool/pyproject.toml + - packages/orchestrator/pyproject.toml + - docker-compose.yml + +key-decisions: + - "LiteLLM pinned to ==1.82.5, not latest — September 2025 OOM regression in later versions; do not upgrade without testing" + - "llm-pool runs on port 8004, consistent with shared/config.py llm_pool_url default (plan originally stated 8002 but shared config established 8004 in Plan 01)" + - "Celery tasks are always sync def with asyncio.run() — this is a hard architectural constraint, never async def" + - "AI transparency clause is unconditional in system prompt — agents must always disclose AI identity when directly asked" + - "LiteLLM Router fallback: quality -> fast (not quality -> 503) gives graceful degradation to local inference" + +patterns-established: + - "Celery sync-def pattern: All @app.task functions must be def (not async def). Use asyncio.run() for async sub-calls." + - "LLM pool abstraction: callers use model group names ('quality', 'fast') not provider-specific model IDs" + - "Runner fallback: non-200 from llm-pool returns polite fallback string, never raises to caller" + - "RLS context: configure_rls_hook(engine) once, set current_tenant_id ContextVar around DB operations, always reset in finally block" + +requirements-completed: [LLM-01, LLM-02] + +# Metrics +duration: 6min +completed: 2026-03-23 +--- + +# Phase 1 Plan 2: LLM Backend Pool + Celery Orchestrator Summary + +**LiteLLM Router service (port 8004) with Ollama/Anthropic/OpenAI fallback chain and Celery handle_message task using sync-def + asyncio.run pattern, verified by 19 integration tests** + +## Performance + +- **Duration:** 6 min +- **Started:** 2026-03-23T16:01:17Z +- **Completed:** 2026-03-23T16:07:10Z +- **Tasks:** 2 +- **Files modified:** 15 (12 created, 3 modified) + +## Accomplishments + +- LLM Backend Pool FastAPI service with LiteLLM Router: fast (Ollama qwen3:8b) and quality (Anthropic claude-sonnet-4 + OpenAI gpt-4o) model groups, automatic cross-group fallback, HTTP 503 on total exhaustion +- Celery orchestrator skeleton: handle_message task (sync def), system prompt builder (name + role + persona + AI transparency clause), runner (httpx to llm-pool with polite fallback) +- 19 green integration tests covering fallback routing (LLM-01) and provider configuration (LLM-02) +- Docker Compose updated with llm-pool (port 8004, healthcheck) and celery-worker services + +## Task Commits + +Each task was committed atomically: + +1. **Task 1: LLM Backend Pool service with LiteLLM Router and fallback** - `ee2f88e` (feat) +2. **Task 2: Celery orchestrator with system prompt builder and integration tests** - `8257c55` (feat) + +**Plan metadata:** _(docs commit follows self-check)_ + +## Files Created/Modified + +- `packages/llm-pool/llm_pool/router.py` — LiteLLM Router: 3-entry model_list, fallbacks, latency routing, `complete()` async function +- `packages/llm-pool/llm_pool/main.py` — FastAPI app on port 8004: POST /complete, GET /health, 503 error handling +- `packages/llm-pool/llm_pool/__init__.py` — Package exports (complete, llm_router) +- `packages/llm-pool/llm_pool/providers/__init__.py` — Empty placeholder for future provider customization +- `packages/llm-pool/pyproject.toml` — Pinned litellm==1.82.5, added fastapi[standard] +- `packages/orchestrator/orchestrator/main.py` — Celery app: Redis broker/backend, task discovery, task_acks_late=True +- `packages/orchestrator/orchestrator/tasks.py` — handle_message (sync def!), _process_message (async), RLS context setup +- `packages/orchestrator/orchestrator/agents/builder.py` — build_system_prompt + build_messages with AI transparency clause +- `packages/orchestrator/orchestrator/agents/runner.py` — run_agent: httpx POST to llm-pool, 120s timeout, polite fallback on error +- `packages/orchestrator/pyproject.toml` — Added celery[redis]>=5.4.0 +- `docker-compose.yml` — Added llm-pool and celery-worker services +- `tests/integration/test_llm_fallback.py` — 7 tests: success paths, 503 on total failure, metadata forwarding (LLM-01) +- `tests/integration/test_llm_providers.py` — 12 tests: model_list structure, provider routing, fallback config (LLM-02) + +## Decisions Made + +- **LiteLLM pinned to ==1.82.5**: Explicitly not latest — a September 2025 OOM regression exists in later releases. Warning comment added to pyproject.toml and router.py. +- **Port 8004, not 8002**: The plan stated port 8002, but Plan 01's shared/config.py already defined `llm_pool_url = "http://localhost:8004"`. Used 8004 to maintain config consistency. +- **AI transparency clause is unconditional**: Added without configuration option — per product design, agents must never deny being AIs when directly asked. +- **Celery sync-def is a hard rule**: Enforced with prominent comment block in tasks.py; `_process_message` is a private async function called only via `asyncio.run()`. + +## Deviations from Plan + +### Auto-fixed Issues + +**1. [Rule 1 - Bug] Fixed test for LiteLLM fallback behavior at Router boundary** +- **Found during:** Task 2 (integration test verification) +- **Issue:** Initial test `test_fallback_invoked_when_primary_raises` mocked `Router.acompletion` to raise on first call then succeed on second, expecting a 200. But our code's exception handler catches the first raise and immediately returns 503 — LiteLLM's internal retry/fallback happens *inside* `acompletion`, not across multiple calls to it from our code. +- **Fix:** Renamed test to `test_fallback_succeeds_when_router_returns_response` and updated to correctly test the boundary: if `acompletion` succeeds (router resolved fallback internally), endpoint returns 200; if it raises (all exhausted), returns 503. +- **Files modified:** `tests/integration/test_llm_fallback.py` +- **Verification:** All 19 tests pass +- **Committed in:** `8257c55` (Task 2 commit) + +--- + +**Total deviations:** 1 auto-fixed (Rule 1 — incorrect test assumption) +**Impact on plan:** Test correction only — no production code changed. 19 tests accurately verify the specified behavior. + +## Issues Encountered + +- `python` binary not in PATH for uv project — all test/import verification commands use `uv run python` and `uv run pytest` + +## User Setup Required + +None - no external service configuration required. LLM API keys are read from `.env` at runtime; empty strings are the safe default for local development with Ollama only. + +## Next Phase Readiness + +- LLM pool and Celery orchestrator are ready for the Channel Gateway (Plan 03) to dispatch `handle_message` tasks +- Docker Compose llm-pool and celery-worker services defined (not yet built/tested in container — deferred to integration phase) +- `handle_message` task interface is stable: accepts `KonstructMessage.model_dump()`, returns `{message_id, response, tenant_id}` + +--- +*Phase: 01-foundation* +*Completed: 2026-03-23*