docs(09): research phase Testing & QA
This commit is contained in:
764
.planning/phases/09-testing-qa/09-RESEARCH.md
Normal file
764
.planning/phases/09-testing-qa/09-RESEARCH.md
Normal file
@@ -0,0 +1,764 @@
|
|||||||
|
# Phase 9: Testing & QA - Research
|
||||||
|
|
||||||
|
**Researched:** 2026-03-25
|
||||||
|
**Domain:** Playwright E2E, Lighthouse CI, visual regression, axe-core accessibility, Gitea Actions CI
|
||||||
|
**Confidence:** HIGH
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
Phase 9 is a greenfield testing layer added on top of a fully-built portal (Next.js 16 standalone, FastAPI gateway, Celery worker). No Playwright config exists yet — the Playwright MCP plugin is installed for manual use but there is no `playwright.config.ts`, no `tests/e2e/` content, and no `.gitea/workflows/` CI file. Everything must be created from scratch.
|
||||||
|
|
||||||
|
The core challenges are: (1) Auth.js v5 JWT sessions that Playwright must obtain and reuse across multiple role fixtures (platform_admin, customer_admin, customer_operator); (2) the WebSocket chat flow at `/chat/ws/{conversation_id}` that needs mocking via `page.routeWebSocket()`; (3) Lighthouse CI that requires a running Next.js server (standalone output complicates `startServerCommand`); and (4) a sub-5-minute pipeline on Gitea Actions that is nearly syntax-identical to GitHub Actions.
|
||||||
|
|
||||||
|
**Primary recommendation:** Place Playwright config and tests inside `packages/portal/` (Next.js co-location pattern), use `storageState` with three saved auth fixtures for roles, mock the WebSocket endpoint with `page.routeWebSocket()` for the chat flow, and run `@lhci/cli` in a separate post-build CI stage.
|
||||||
|
|
||||||
|
<user_constraints>
|
||||||
|
## User Constraints (from CONTEXT.md)
|
||||||
|
|
||||||
|
### Locked Decisions
|
||||||
|
All decisions at Claude's discretion — user trusts judgment.
|
||||||
|
|
||||||
|
- Playwright for all E2E tests (cross-browser built-in, official Next.js recommendation)
|
||||||
|
- Critical flows to test (priority order):
|
||||||
|
1. Login → dashboard loads → session persists
|
||||||
|
2. Create tenant → tenant appears in list
|
||||||
|
3. Deploy template agent → agent appears in employees list
|
||||||
|
4. Chat: open conversation → send message → receive streaming response (mock LLM)
|
||||||
|
5. RBAC: operator cannot access /agents/new, /billing, /users
|
||||||
|
6. Language switcher → UI updates to selected language
|
||||||
|
7. Mobile viewport: bottom tab bar renders, sidebar hidden
|
||||||
|
- LLM responses mocked in E2E tests (no real Ollama/API calls)
|
||||||
|
- Test data: seed a test tenant + test user via API calls in test setup, clean up after
|
||||||
|
- Lighthouse targets: >= 90 (fail at 80, warn at 85)
|
||||||
|
- Pages: login, dashboard, chat, agents/new
|
||||||
|
- Visual regression at 3 viewports: desktop 1280x800, tablet 768x1024, mobile 375x812
|
||||||
|
- Key pages: login, dashboard, agents list, agents/new (3-card entry), chat (empty state), templates gallery
|
||||||
|
- Baseline snapshots committed to repo
|
||||||
|
- axe-core via @axe-core/playwright, zero critical violations required
|
||||||
|
- "serious" violations logged as warnings (not blockers for beta)
|
||||||
|
- Keyboard navigation test: Tab through login form, chat input, nav items
|
||||||
|
- Cross-browser: chromium, firefox, webkit
|
||||||
|
- Visual regression: chromium only
|
||||||
|
- Gitea Actions, triggers: push to main, PR to main
|
||||||
|
- Pipeline stages: lint → type-check → unit tests (pytest) → build portal → E2E tests → Lighthouse
|
||||||
|
- Docker Compose for CI infra
|
||||||
|
- JUnit XML + HTML trace viewer reports
|
||||||
|
- Fail-fast: lint/type errors block everything; unit test failures block E2E
|
||||||
|
- Target: < 5 min pipeline
|
||||||
|
|
||||||
|
### Claude's Discretion
|
||||||
|
- Playwright config details (timeouts, retries, parallelism)
|
||||||
|
- Test file organization (by feature vs by page)
|
||||||
|
- Fixture/helper patterns for auth, tenant setup, API mocking
|
||||||
|
- Lighthouse CI tool (lighthouse-ci vs @lhci/cli)
|
||||||
|
- Whether to include a smoke test for the WebSocket chat connection
|
||||||
|
- Visual regression threshold (pixel diff tolerance)
|
||||||
|
|
||||||
|
### Deferred Ideas (OUT OF SCOPE)
|
||||||
|
None — discussion stayed within phase scope
|
||||||
|
</user_constraints>
|
||||||
|
|
||||||
|
<phase_requirements>
|
||||||
|
## Phase Requirements
|
||||||
|
|
||||||
|
| ID | Description | Research Support |
|
||||||
|
|----|-------------|-----------------|
|
||||||
|
| QA-01 | Playwright E2E tests cover all critical user flows (login, tenant CRUD, agent deploy, chat, billing, RBAC) | Playwright storageState auth fixtures + routeWebSocket for chat mock |
|
||||||
|
| QA-02 | Lighthouse scores >= 90 for performance, accessibility, best practices, SEO on key pages | @lhci/cli with minScore assertions per category |
|
||||||
|
| QA-03 | Visual regression snapshots at desktop/tablet/mobile for all key pages | toHaveScreenshot with maxDiffPixelRatio, viewports per project |
|
||||||
|
| QA-04 | axe-core accessibility audit passes with zero critical violations across all pages | @axe-core/playwright AxeBuilder with impact filter |
|
||||||
|
| QA-05 | E2E tests pass on Chrome, Firefox, Safari (WebKit) | Playwright projects array with three browser engines |
|
||||||
|
| QA-06 | Empty states, error states, loading states tested and rendered correctly | Dedicated test cases + API mocking for empty/error responses |
|
||||||
|
| QA-07 | CI-ready test suite runnable in Gitea Actions pipeline | .gitea/workflows/ci.yml with Docker Compose service containers |
|
||||||
|
</phase_requirements>
|
||||||
|
|
||||||
|
## Standard Stack
|
||||||
|
|
||||||
|
### Core
|
||||||
|
| Library | Version | Purpose | Why Standard |
|
||||||
|
|---------|---------|---------|--------------|
|
||||||
|
| @playwright/test | ^1.51 | E2E + visual regression + accessibility runner | Official Next.js recommendation, cross-browser built-in, no extra dependencies |
|
||||||
|
| @axe-core/playwright | ^4.10 | Accessibility scanning within Playwright tests | Official Deque package, integrates directly with Playwright page objects |
|
||||||
|
| @lhci/cli | ^0.15 | Lighthouse CI score assertions | Google-maintained, headless Lighthouse, assertion config via lighthouserc |
|
||||||
|
|
||||||
|
### Supporting
|
||||||
|
| Library | Version | Purpose | When to Use |
|
||||||
|
|---------|---------|---------|-------------|
|
||||||
|
| axe-html-reporter | ^2.2 | HTML accessibility reports | When you want human-readable a11y reports attached to CI artifacts |
|
||||||
|
|
||||||
|
### Alternatives Considered
|
||||||
|
| Instead of | Could Use | Tradeoff |
|
||||||
|
|------------|-----------|----------|
|
||||||
|
| @lhci/cli | lighthouse npm module directly | @lhci/cli handles multi-run averaging, assertions, and CI upload; raw lighthouse requires custom scripting |
|
||||||
|
| @axe-core/playwright | axe-playwright (third-party) | @axe-core/playwright is the official Deque package; axe-playwright is a community wrapper with same API but extra dep |
|
||||||
|
|
||||||
|
**Installation (portal):**
|
||||||
|
```bash
|
||||||
|
cd packages/portal
|
||||||
|
npm install --save-dev @playwright/test @axe-core/playwright @lhci/cli
|
||||||
|
npx playwright install --with-deps chromium firefox webkit
|
||||||
|
```
|
||||||
|
|
||||||
|
## Architecture Patterns
|
||||||
|
|
||||||
|
### Recommended Project Structure
|
||||||
|
```
|
||||||
|
packages/portal/
|
||||||
|
├── playwright.config.ts # Main config: projects, webServer, globalSetup
|
||||||
|
├── e2e/
|
||||||
|
│ ├── auth.setup.ts # Global setup: save storageState per role
|
||||||
|
│ ├── fixtures.ts # Extended test: auth fixtures, axe builder, API helpers
|
||||||
|
│ ├── helpers/
|
||||||
|
│ │ ├── seed.ts # Seed test tenant + user via API, return IDs
|
||||||
|
│ │ └── cleanup.ts # Delete seeded data after test suite
|
||||||
|
│ ├── flows/
|
||||||
|
│ │ ├── login.spec.ts # Flow 1: login → dashboard loads → session persists
|
||||||
|
│ │ ├── tenant-crud.spec.ts # Flow 2: create tenant → appears in list
|
||||||
|
│ │ ├── agent-deploy.spec.ts # Flow 3: deploy template → appears in employees
|
||||||
|
│ │ ├── chat.spec.ts # Flow 4: open chat → send msg → streaming response (mocked WS)
|
||||||
|
│ │ ├── rbac.spec.ts # Flow 5: operator access denied to restricted pages
|
||||||
|
│ │ ├── i18n.spec.ts # Flow 6: language switcher → UI updates
|
||||||
|
│ │ └── mobile.spec.ts # Flow 7: mobile viewport → bottom tab bar, sidebar hidden
|
||||||
|
│ ├── accessibility/
|
||||||
|
│ │ └── a11y.spec.ts # axe-core scan on every key page, keyboard nav test
|
||||||
|
│ ├── visual/
|
||||||
|
│ │ └── snapshots.spec.ts # Visual regression at 3 viewports (chromium only)
|
||||||
|
│ └── lighthouse/
|
||||||
|
│ └── lighthouserc.json # @lhci/cli config: URLs, score thresholds
|
||||||
|
├── playwright/.auth/ # gitignored — saved storageState files
|
||||||
|
│ ├── platform-admin.json
|
||||||
|
│ ├── customer-admin.json
|
||||||
|
│ └── customer-operator.json
|
||||||
|
└── __snapshots__/ # Committed baseline screenshots
|
||||||
|
.gitea/
|
||||||
|
└── workflows/
|
||||||
|
└── ci.yml # Pipeline: lint → typecheck → pytest → build → E2E → lhci
|
||||||
|
```
|
||||||
|
|
||||||
|
### Pattern 1: Auth.js v5 storageState with Multiple Roles
|
||||||
|
|
||||||
|
**What:** Authenticate each role once in a global setup project, save to JSON. All E2E tests consume the saved state — no repeated login UI interactions.
|
||||||
|
|
||||||
|
**When to use:** Any test that requires a logged-in user. Each spec declares which role it needs via `test.use({ storageState })`.
|
||||||
|
|
||||||
|
**Key insight for Auth.js v5:** The credentials provider calls the FastAPI `/api/portal/auth/verify` endpoint. Playwright must fill the login form (not call the API directly) because `next-auth` sets `HttpOnly` session cookies that only the browser can hold. The storageState captures those cookies.
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// Source: https://playwright.dev/docs/auth
|
||||||
|
// e2e/auth.setup.ts
|
||||||
|
import { test as setup, expect } from "@playwright/test";
|
||||||
|
import path from "path";
|
||||||
|
|
||||||
|
const PLATFORM_ADMIN_AUTH = path.resolve(__dirname, "../playwright/.auth/platform-admin.json");
|
||||||
|
const CUSTOMER_ADMIN_AUTH = path.resolve(__dirname, "../playwright/.auth/customer-admin.json");
|
||||||
|
const OPERATOR_AUTH = path.resolve(__dirname, "../playwright/.auth/customer-operator.json");
|
||||||
|
|
||||||
|
setup("authenticate as platform admin", async ({ page }) => {
|
||||||
|
await page.goto("/login");
|
||||||
|
await page.getByLabel("Email").fill(process.env.E2E_ADMIN_EMAIL!);
|
||||||
|
await page.getByLabel("Password").fill(process.env.E2E_ADMIN_PASSWORD!);
|
||||||
|
await page.getByRole("button", { name: /sign in/i }).click();
|
||||||
|
await page.waitForURL("/dashboard");
|
||||||
|
await page.context().storageState({ path: PLATFORM_ADMIN_AUTH });
|
||||||
|
});
|
||||||
|
|
||||||
|
setup("authenticate as customer admin", async ({ page }) => {
|
||||||
|
// seed returns { email, password } for a fresh customer_admin user
|
||||||
|
await page.goto("/login");
|
||||||
|
await page.getByLabel("Email").fill(process.env.E2E_CADMIN_EMAIL!);
|
||||||
|
await page.getByLabel("Password").fill(process.env.E2E_CADMIN_PASSWORD!);
|
||||||
|
await page.getByRole("button", { name: /sign in/i }).click();
|
||||||
|
await page.waitForURL("/dashboard");
|
||||||
|
await page.context().storageState({ path: CUSTOMER_ADMIN_AUTH });
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
### Pattern 2: WebSocket Mocking for Chat Flow
|
||||||
|
|
||||||
|
**What:** Intercept the `/chat/ws/{conversationId}` WebSocket before the gateway is contacted. Respond to the auth message, then simulate streaming tokens on a user message.
|
||||||
|
|
||||||
|
**When to use:** Flow 4 (chat E2E test). The gateway WebSocket endpoint at `ws://localhost:8001/chat/ws/{id}` is routed via the Next.js API proxy — intercept at the browser level.
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// Source: https://playwright.dev/docs/api/class-websocketroute
|
||||||
|
// e2e/flows/chat.spec.ts
|
||||||
|
test("chat: send message → receive streaming response", async ({ page }) => {
|
||||||
|
await page.routeWebSocket(/\/chat\/ws\//, (ws) => {
|
||||||
|
ws.onMessage((msg) => {
|
||||||
|
const data = JSON.parse(msg as string);
|
||||||
|
|
||||||
|
if (data.type === "auth") {
|
||||||
|
// Acknowledge auth — no response needed, gateway just proceeds
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (data.type === "message") {
|
||||||
|
// Simulate typing indicator
|
||||||
|
ws.send(JSON.stringify({ type: "typing" }));
|
||||||
|
// Simulate streaming tokens
|
||||||
|
const tokens = ["Hello", " from", " your", " AI", " assistant!"];
|
||||||
|
tokens.forEach((token, i) => {
|
||||||
|
setTimeout(() => {
|
||||||
|
ws.send(JSON.stringify({ type: "chunk", token }));
|
||||||
|
}, i * 50);
|
||||||
|
});
|
||||||
|
setTimeout(() => {
|
||||||
|
ws.send(JSON.stringify({
|
||||||
|
type: "response",
|
||||||
|
text: tokens.join(""),
|
||||||
|
conversation_id: data.conversation_id,
|
||||||
|
}));
|
||||||
|
ws.send(JSON.stringify({ type: "done", text: tokens.join("") }));
|
||||||
|
}, tokens.length * 50 + 100);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
await page.goto("/chat?agentId=test-agent");
|
||||||
|
await page.getByPlaceholder(/type a message/i).fill("Hello!");
|
||||||
|
await page.keyboard.press("Enter");
|
||||||
|
await expect(page.getByText("Hello from your AI assistant!")).toBeVisible({ timeout: 5000 });
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
### Pattern 3: Visual Regression at Multiple Viewports
|
||||||
|
|
||||||
|
**What:** Configure separate Playwright projects for each viewport, run snapshots only on chromium to avoid cross-browser rendering diffs.
|
||||||
|
|
||||||
|
**When to use:** QA-03. Visual regression baseline committed to repo; CI fails on diff.
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// Source: https://playwright.dev/docs/test-snapshots
|
||||||
|
// playwright.config.ts (visual projects section)
|
||||||
|
{
|
||||||
|
name: "visual-desktop",
|
||||||
|
use: {
|
||||||
|
...devices["Desktop Chrome"],
|
||||||
|
viewport: { width: 1280, height: 800 },
|
||||||
|
},
|
||||||
|
testMatch: "e2e/visual/**",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "visual-tablet",
|
||||||
|
use: {
|
||||||
|
browserName: "chromium",
|
||||||
|
viewport: { width: 768, height: 1024 },
|
||||||
|
},
|
||||||
|
testMatch: "e2e/visual/**",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "visual-mobile",
|
||||||
|
use: {
|
||||||
|
...devices["iPhone 12"],
|
||||||
|
viewport: { width: 375, height: 812 },
|
||||||
|
},
|
||||||
|
testMatch: "e2e/visual/**",
|
||||||
|
},
|
||||||
|
```
|
||||||
|
|
||||||
|
Global threshold:
|
||||||
|
```typescript
|
||||||
|
// playwright.config.ts
|
||||||
|
expect: {
|
||||||
|
toHaveScreenshot: {
|
||||||
|
maxDiffPixelRatio: 0.02, // 2% tolerance — accounts for antialiasing
|
||||||
|
threshold: 0.2, // pixel color threshold (0–1)
|
||||||
|
},
|
||||||
|
},
|
||||||
|
```
|
||||||
|
|
||||||
|
### Pattern 4: axe-core Fixture
|
||||||
|
|
||||||
|
**What:** Shared fixture that creates an AxeBuilder for each page, scoped to WCAG 2.1 AA, filtering results by impact level.
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// Source: https://playwright.dev/docs/accessibility-testing
|
||||||
|
// e2e/fixtures.ts
|
||||||
|
import { test as base, expect } from "@playwright/test";
|
||||||
|
import AxeBuilder from "@axe-core/playwright";
|
||||||
|
|
||||||
|
export const test = base.extend<{ axe: () => AxeBuilder }>({
|
||||||
|
axe: async ({ page }, use) => {
|
||||||
|
const makeBuilder = () =>
|
||||||
|
new AxeBuilder({ page })
|
||||||
|
.withTags(["wcag2a", "wcag2aa", "wcag21aa"]);
|
||||||
|
await use(makeBuilder);
|
||||||
|
},
|
||||||
|
});
|
||||||
|
|
||||||
|
// In a test:
|
||||||
|
const results = await axe().analyze();
|
||||||
|
const criticalViolations = results.violations.filter(v => v.impact === "critical");
|
||||||
|
const seriousViolations = results.violations.filter(v => v.impact === "serious");
|
||||||
|
|
||||||
|
expect(criticalViolations, "Critical a11y violations found").toHaveLength(0);
|
||||||
|
if (seriousViolations.length > 0) {
|
||||||
|
console.warn("Serious a11y violations (non-blocking):", seriousViolations);
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Pattern 5: Lighthouse CI Config
|
||||||
|
|
||||||
|
**What:** `lighthouserc.json` drives `@lhci/cli autorun` in CI. Pages run headlessly against the built portal.
|
||||||
|
|
||||||
|
```json
|
||||||
|
// Source: https://googlechrome.github.io/lighthouse-ci/docs/configuration.html
|
||||||
|
// e2e/lighthouse/lighthouserc.json
|
||||||
|
{
|
||||||
|
"ci": {
|
||||||
|
"collect": {
|
||||||
|
"url": [
|
||||||
|
"http://localhost:3000/login",
|
||||||
|
"http://localhost:3000/dashboard",
|
||||||
|
"http://localhost:3000/chat",
|
||||||
|
"http://localhost:3000/agents/new"
|
||||||
|
],
|
||||||
|
"numberOfRuns": 1,
|
||||||
|
"settings": {
|
||||||
|
"preset": "desktop",
|
||||||
|
"chromeFlags": "--no-sandbox --disable-dev-shm-usage"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"assert": {
|
||||||
|
"assertions": {
|
||||||
|
"categories:performance": ["error", {"minScore": 0.80}],
|
||||||
|
"categories:accessibility": ["error", {"minScore": 0.80}],
|
||||||
|
"categories:best-practices": ["error", {"minScore": 0.80}],
|
||||||
|
"categories:seo": ["error", {"minScore": 0.80}]
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"upload": {
|
||||||
|
"target": "filesystem",
|
||||||
|
"outputDir": ".lighthouseci"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Note: `error` at 0.80 means CI fails below 80; the 90 target is aspirational. Set warn at 0.85 for soft alerts.
|
||||||
|
|
||||||
|
### Pattern 6: Playwright Config (Full)
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// packages/portal/playwright.config.ts
|
||||||
|
import { defineConfig, devices } from "@playwright/test";
|
||||||
|
|
||||||
|
export default defineConfig({
|
||||||
|
testDir: "./e2e",
|
||||||
|
fullyParallel: false, // Stability in CI with shared DB state
|
||||||
|
forbidOnly: !!process.env.CI,
|
||||||
|
retries: process.env.CI ? 1 : 0,
|
||||||
|
workers: process.env.CI ? 1 : undefined,
|
||||||
|
timeout: 30_000,
|
||||||
|
|
||||||
|
reporter: [
|
||||||
|
["html", { outputFolder: "playwright-report" }],
|
||||||
|
["junit", { outputFile: "playwright-results.xml" }],
|
||||||
|
["list"],
|
||||||
|
],
|
||||||
|
|
||||||
|
use: {
|
||||||
|
baseURL: process.env.PLAYWRIGHT_BASE_URL ?? "http://localhost:3000",
|
||||||
|
trace: "on-first-retry",
|
||||||
|
screenshot: "only-on-failure",
|
||||||
|
serviceWorkers: "block", // Prevents Serwist from intercepting test requests
|
||||||
|
},
|
||||||
|
|
||||||
|
expect: {
|
||||||
|
toHaveScreenshot: {
|
||||||
|
maxDiffPixelRatio: 0.02,
|
||||||
|
threshold: 0.2,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
|
||||||
|
projects: [
|
||||||
|
// Auth setup runs first for all browser projects
|
||||||
|
{ name: "setup", testMatch: /auth\.setup\.ts/ },
|
||||||
|
|
||||||
|
// E2E flows — all 3 browsers
|
||||||
|
{
|
||||||
|
name: "chromium",
|
||||||
|
use: { ...devices["Desktop Chrome"], storageState: "playwright/.auth/platform-admin.json" },
|
||||||
|
dependencies: ["setup"],
|
||||||
|
testMatch: "e2e/flows/**",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "firefox",
|
||||||
|
use: { ...devices["Desktop Firefox"], storageState: "playwright/.auth/platform-admin.json" },
|
||||||
|
dependencies: ["setup"],
|
||||||
|
testMatch: "e2e/flows/**",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "webkit",
|
||||||
|
use: { ...devices["Desktop Safari"], storageState: "playwright/.auth/platform-admin.json" },
|
||||||
|
dependencies: ["setup"],
|
||||||
|
testMatch: "e2e/flows/**",
|
||||||
|
},
|
||||||
|
|
||||||
|
// Visual regression — chromium only, 3 viewports
|
||||||
|
{ name: "visual-desktop", use: { browserName: "chromium", viewport: { width: 1280, height: 800 } }, testMatch: "e2e/visual/**", dependencies: ["setup"] },
|
||||||
|
{ name: "visual-tablet", use: { browserName: "chromium", viewport: { width: 768, height: 1024 } }, testMatch: "e2e/visual/**", dependencies: ["setup"] },
|
||||||
|
{ name: "visual-mobile", use: { ...devices["iPhone 12"] }, testMatch: "e2e/visual/**", dependencies: ["setup"] },
|
||||||
|
|
||||||
|
// Accessibility — chromium only
|
||||||
|
{
|
||||||
|
name: "a11y",
|
||||||
|
use: { ...devices["Desktop Chrome"] },
|
||||||
|
dependencies: ["setup"],
|
||||||
|
testMatch: "e2e/accessibility/**",
|
||||||
|
},
|
||||||
|
],
|
||||||
|
|
||||||
|
webServer: {
|
||||||
|
command: "node .next/standalone/server.js",
|
||||||
|
url: "http://localhost:3000",
|
||||||
|
reuseExistingServer: !process.env.CI,
|
||||||
|
env: {
|
||||||
|
PORT: "3000",
|
||||||
|
API_URL: process.env.API_URL ?? "http://localhost:8001",
|
||||||
|
AUTH_SECRET: process.env.AUTH_SECRET ?? "test-secret-32-chars-minimum-len",
|
||||||
|
AUTH_URL: "http://localhost:3000",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
**Critical:** `serviceWorkers: "block"` is required because Serwist (PWA service worker) intercepts network requests and makes them invisible to `page.route()` / `page.routeWebSocket()`.
|
||||||
|
|
||||||
|
### Pattern 7: Gitea Actions CI Pipeline
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
# .gitea/workflows/ci.yml
|
||||||
|
name: CI
|
||||||
|
|
||||||
|
on:
|
||||||
|
push:
|
||||||
|
branches: [main]
|
||||||
|
pull_request:
|
||||||
|
branches: [main]
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
backend:
|
||||||
|
name: Backend Tests
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
services:
|
||||||
|
postgres:
|
||||||
|
image: pgvector/pgvector:pg16
|
||||||
|
env:
|
||||||
|
POSTGRES_DB: konstruct
|
||||||
|
POSTGRES_USER: postgres
|
||||||
|
POSTGRES_PASSWORD: postgres_dev
|
||||||
|
options: >-
|
||||||
|
--health-cmd pg_isready
|
||||||
|
--health-interval 5s
|
||||||
|
--health-timeout 5s
|
||||||
|
--health-retries 10
|
||||||
|
redis:
|
||||||
|
image: redis:7-alpine
|
||||||
|
options: >-
|
||||||
|
--health-cmd "redis-cli ping"
|
||||||
|
--health-interval 5s
|
||||||
|
env:
|
||||||
|
DATABASE_URL: postgresql+asyncpg://konstruct_app:konstruct_dev@localhost:5432/konstruct
|
||||||
|
DATABASE_ADMIN_URL: postgresql+asyncpg://postgres:postgres_dev@localhost:5432/konstruct
|
||||||
|
REDIS_URL: redis://localhost:6379/0
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v4
|
||||||
|
- uses: actions/setup-python@v5
|
||||||
|
with: { python-version: "3.12" }
|
||||||
|
- run: pip install uv
|
||||||
|
- run: uv sync
|
||||||
|
- run: uv run ruff check packages/ tests/
|
||||||
|
- run: uv run mypy --strict packages/
|
||||||
|
- run: uv run pytest tests/ -x --tb=short
|
||||||
|
|
||||||
|
portal:
|
||||||
|
name: Portal E2E
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
needs: backend # E2E blocked until backend passes
|
||||||
|
services:
|
||||||
|
postgres:
|
||||||
|
image: pgvector/pgvector:pg16
|
||||||
|
env:
|
||||||
|
POSTGRES_DB: konstruct
|
||||||
|
POSTGRES_USER: postgres
|
||||||
|
POSTGRES_PASSWORD: postgres_dev
|
||||||
|
options: --health-cmd pg_isready --health-interval 5s --health-retries 10
|
||||||
|
redis:
|
||||||
|
image: redis:7-alpine
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v4
|
||||||
|
- uses: actions/setup-node@v4
|
||||||
|
with: { node-version: "22" }
|
||||||
|
- name: Install portal deps
|
||||||
|
working-directory: packages/portal
|
||||||
|
run: npm ci
|
||||||
|
- name: Build portal
|
||||||
|
working-directory: packages/portal
|
||||||
|
run: npm run build
|
||||||
|
env:
|
||||||
|
NEXT_PUBLIC_API_URL: http://localhost:8001
|
||||||
|
- name: Install Playwright browsers
|
||||||
|
working-directory: packages/portal
|
||||||
|
run: npx playwright install --with-deps chromium firefox webkit
|
||||||
|
- name: Start gateway (background)
|
||||||
|
run: |
|
||||||
|
pip install uv && uv sync
|
||||||
|
uv run alembic upgrade head
|
||||||
|
uv run uvicorn gateway.main:app --host 0.0.0.0 --port 8001 &
|
||||||
|
env:
|
||||||
|
DATABASE_URL: postgresql+asyncpg://konstruct_app:konstruct_dev@localhost:5432/konstruct
|
||||||
|
DATABASE_ADMIN_URL: postgresql+asyncpg://postgres:postgres_dev@localhost:5432/konstruct
|
||||||
|
REDIS_URL: redis://localhost:6379/0
|
||||||
|
LLM_POOL_URL: http://localhost:8004 # not running — mocked in E2E
|
||||||
|
- name: Wait for gateway
|
||||||
|
run: timeout 30 bash -c 'until curl -sf http://localhost:8001/health; do sleep 1; done'
|
||||||
|
- name: Run E2E tests
|
||||||
|
working-directory: packages/portal
|
||||||
|
run: npx playwright test e2e/flows/ e2e/accessibility/
|
||||||
|
env:
|
||||||
|
CI: "true"
|
||||||
|
PLAYWRIGHT_BASE_URL: http://localhost:3000
|
||||||
|
API_URL: http://localhost:8001
|
||||||
|
AUTH_SECRET: ${{ secrets.AUTH_SECRET }}
|
||||||
|
E2E_ADMIN_EMAIL: ${{ secrets.E2E_ADMIN_EMAIL }}
|
||||||
|
E2E_ADMIN_PASSWORD: ${{ secrets.E2E_ADMIN_PASSWORD }}
|
||||||
|
- name: Run Lighthouse CI
|
||||||
|
working-directory: packages/portal
|
||||||
|
run: |
|
||||||
|
npx lhci autorun --config=e2e/lighthouse/lighthouserc.json
|
||||||
|
env:
|
||||||
|
LHCI_BUILD_CONTEXT__CURRENT_HASH: ${{ github.sha }}
|
||||||
|
- name: Upload Playwright report
|
||||||
|
if: always()
|
||||||
|
uses: actions/upload-artifact@v4
|
||||||
|
with:
|
||||||
|
name: playwright-report
|
||||||
|
path: packages/portal/playwright-report/
|
||||||
|
- name: Upload Lighthouse report
|
||||||
|
if: always()
|
||||||
|
uses: actions/upload-artifact@v4
|
||||||
|
with:
|
||||||
|
name: lighthouse-report
|
||||||
|
path: packages/portal/.lighthouseci/
|
||||||
|
```
|
||||||
|
|
||||||
|
### Anti-Patterns to Avoid
|
||||||
|
|
||||||
|
- **Hardcoded IDs in selectors:** Use `getByRole`, `getByLabel`, `getByText` — never CSS `#id` or `[data-testid]` unless semantic selectors are unavailable. Semantic selectors are more resilient and double as accessibility checks.
|
||||||
|
- **Real LLM calls in E2E:** Never let E2E tests reach Ollama/OpenAI. Mock the WebSocket and gateway LLM calls. Real calls introduce flakiness and cost.
|
||||||
|
- **Superuser DB connections in test seeds:** The existing conftest uses `konstruct_app` role to preserve RLS. E2E seeds should call the FastAPI admin API endpoints, not connect directly to the DB.
|
||||||
|
- **Enabling service workers in tests:** Serwist intercepts all requests. Always set `serviceWorkers: "block"` in Playwright config.
|
||||||
|
- **Parallel workers with shared DB state:** Set `workers: 1` in CI. Tenant/agent mutations are not thread-safe across workers without per-worker isolation.
|
||||||
|
- **Running visual regression on all browsers:** Browser rendering engines produce expected pixel diffs. Visual regression on chromium only; cross-browser covered by functional E2E.
|
||||||
|
|
||||||
|
## Don't Hand-Roll
|
||||||
|
|
||||||
|
| Problem | Don't Build | Use Instead | Why |
|
||||||
|
|---------|-------------|-------------|-----|
|
||||||
|
| Screenshot diffs | Custom pixel comparator | `toHaveScreenshot()` built into Playwright | Handles baseline storage, update workflow, CI reporting |
|
||||||
|
| Accessibility scanning | Custom ARIA traversal | `@axe-core/playwright` | Covers 57 WCAG rules including ones humans miss |
|
||||||
|
| Performance score gating | Parsing Lighthouse JSON manually | `@lhci/cli assert` | Handles multi-run averaging, threshold config, exit codes |
|
||||||
|
| Auth state reuse | Logging in before every test | Playwright `storageState` | Session reuse makes the suite 10x faster |
|
||||||
|
| WS mock server | Running a real mock websocket server | `page.routeWebSocket()` | In-process, no port conflicts, no flakiness |
|
||||||
|
|
||||||
|
## Common Pitfalls
|
||||||
|
|
||||||
|
### Pitfall 1: Auth.js HttpOnly Cookies
|
||||||
|
**What goes wrong:** Trying to authenticate by calling `/api/portal/auth/verify` directly with Playwright `request` — this bypasses Auth.js cookie-setting, so the browser session never exists.
|
||||||
|
**Why it happens:** Auth.js v5 JWT is set as `HttpOnly` secure cookie by the Next.js server, not by the FastAPI backend.
|
||||||
|
**How to avoid:** Always use Playwright's UI login flow (fill form → submit → wait for redirect) to let Next.js set the cookie. Then save with `storageState`.
|
||||||
|
**Warning signs:** Tests pass the login assertion but fail immediately after on authenticated pages.
|
||||||
|
|
||||||
|
### Pitfall 2: Serwist Service Worker Intercepting Test Traffic
|
||||||
|
**What goes wrong:** `page.route()` and `page.routeWebSocket()` handlers never fire because the PWA service worker handles requests first.
|
||||||
|
**Why it happens:** Serwist registers a service worker that intercepts all requests matching the scope. Playwright's routing operates at the network level before the service worker, but only if service workers are blocked.
|
||||||
|
**How to avoid:** Set `serviceWorkers: "block"` in `playwright.config.ts` under `use`.
|
||||||
|
**Warning signs:** Mock routes never called; tests see real responses or network errors.
|
||||||
|
|
||||||
|
### Pitfall 3: Next.js Standalone Output Path for webServer
|
||||||
|
**What goes wrong:** `command: "npm run start"` fails in CI because `next start` requires the dev server setup, not standalone output.
|
||||||
|
**Why it happens:** The portal uses `output: "standalone"` in `next.config.ts`. The build produces `.next/standalone/server.js`, not the standard Next.js CLI server.
|
||||||
|
**How to avoid:** Use `command: "node .next/standalone/server.js"` in Playwright's `webServer` config. Copy static files if needed: the build step must run `cp -r .next/static .next/standalone/.next/static && cp -r public .next/standalone/public`.
|
||||||
|
**Warning signs:** `webServer` process exits immediately; Playwright reports "server did not start".
|
||||||
|
|
||||||
|
### Pitfall 4: Visual Regression Baseline Committed Without CI Environment Lock
|
||||||
|
**What goes wrong:** Baselines created on a developer's Mac differ from Linux CI renderings (font rendering, subpixel AA, etc.).
|
||||||
|
**Why it happens:** Screenshot comparisons are pixel-exact. OS-level rendering differences cause 1–5% false failures.
|
||||||
|
**How to avoid:** Generate baselines inside the same Docker/Linux environment as CI. Run `npx playwright test --update-snapshots` on Linux (or in the Playwright Docker image) to commit initial baselines. Use `maxDiffPixelRatio: 0.02` to absorb minor remaining differences.
|
||||||
|
**Warning signs:** Visual tests pass locally but always fail in CI.
|
||||||
|
|
||||||
|
### Pitfall 5: Lighthouse Pages Behind Auth
|
||||||
|
**What goes wrong:** Lighthouse visits `/dashboard` and gets redirected to `/login` — scores an empty page.
|
||||||
|
**Why it happens:** Lighthouse runs as an unauthenticated browser session. LHCI doesn't support Auth.js cookie injection.
|
||||||
|
**How to avoid:** For authenticated pages, either (a) test only public pages with Lighthouse (login, landing), or (b) use LHCI's `basicAuth` option for pages behind HTTP auth (not applicable here), or (c) create a special unauthenticated preview mode. **For this project:** Run Lighthouse on `/login` only, plus any public-accessible marketing pages. Skip `/dashboard` and `/chat` for Lighthouse.
|
||||||
|
**Warning signs:** Lighthouse scores 100 for accessibility on dashboard — suspiciously perfect because it's measuring an empty redirect.
|
||||||
|
|
||||||
|
### Pitfall 6: WebSocket URL Resolution in Tests
|
||||||
|
**What goes wrong:** `page.routeWebSocket("/chat/ws/")` doesn't match because the portal derives the WS URL from `NEXT_PUBLIC_API_URL` (baked at build time), which points to `ws://localhost:8001`, not a relative path.
|
||||||
|
**Why it happens:** `use-chat-socket.ts` computes `WS_BASE` from `process.env.NEXT_PUBLIC_API_URL` and builds `ws://localhost:8001/chat/ws/{id}`.
|
||||||
|
**How to avoid:** Use a regex pattern: `page.routeWebSocket(/\/chat\/ws\//, handler)` — this matches the full absolute URL.
|
||||||
|
**Warning signs:** Chat mock never fires; test times out waiting for WS message.
|
||||||
|
|
||||||
|
### Pitfall 7: Gitea Actions Runner Needs Docker
|
||||||
|
**What goes wrong:** Service containers fail to start because the Gitea runner is not configured with Docker access.
|
||||||
|
**Why it happens:** Gitea Actions service containers require Docker socket access on the runner.
|
||||||
|
**How to avoid:** Ensure the `act_runner` is added to the `docker` group on the host. Alternative: use `docker compose` in a setup step instead of service containers.
|
||||||
|
**Warning signs:** Job fails immediately with "Cannot connect to Docker daemon".
|
||||||
|
|
||||||
|
## Code Examples
|
||||||
|
|
||||||
|
### Seed Helper via API
|
||||||
|
```typescript
|
||||||
|
// e2e/helpers/seed.ts
|
||||||
|
// Uses Playwright APIRequestContext to create test data via FastAPI endpoints.
|
||||||
|
// Must run BEFORE storageState setup (needs platform_admin creds via env).
|
||||||
|
export async function seedTestTenant(request: APIRequestContext): Promise<{ tenantId: string; tenantSlug: string }> {
|
||||||
|
const suffix = Math.random().toString(36).slice(2, 8);
|
||||||
|
const res = await request.post("http://localhost:8001/api/portal/tenants", {
|
||||||
|
headers: {
|
||||||
|
"X-User-Id": process.env.E2E_ADMIN_ID!,
|
||||||
|
"X-User-Role": "platform_admin",
|
||||||
|
"X-Active-Tenant": "",
|
||||||
|
},
|
||||||
|
data: { name: `E2E Tenant ${suffix}`, slug: `e2e-tenant-${suffix}` },
|
||||||
|
});
|
||||||
|
const body = await res.json() as { id: string; slug: string };
|
||||||
|
return { tenantId: body.id, tenantSlug: body.slug };
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### RBAC Test Pattern
|
||||||
|
```typescript
|
||||||
|
// e2e/flows/rbac.spec.ts
|
||||||
|
// Tests that operator role is silently redirected, not 403-paged
|
||||||
|
test.describe("RBAC enforcement", () => {
|
||||||
|
test.use({ storageState: "playwright/.auth/customer-operator.json" });
|
||||||
|
|
||||||
|
const restrictedPaths = ["/agents/new", "/billing", "/users"];
|
||||||
|
|
||||||
|
for (const path of restrictedPaths) {
|
||||||
|
test(`operator cannot access ${path}`, async ({ page }) => {
|
||||||
|
await page.goto(path);
|
||||||
|
// proxy.ts does silent redirect — operator ends up on /dashboard
|
||||||
|
await expect(page).not.toHaveURL(path);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
### Mobile Viewport Behavioral Test
|
||||||
|
```typescript
|
||||||
|
// e2e/flows/mobile.spec.ts
|
||||||
|
test("mobile: bottom tab bar renders, sidebar hidden", async ({ page }) => {
|
||||||
|
await page.setViewportSize({ width: 375, height: 812 });
|
||||||
|
await page.goto("/dashboard");
|
||||||
|
// Bottom tab bar visible
|
||||||
|
await expect(page.getByRole("navigation", { name: /mobile/i })).toBeVisible();
|
||||||
|
// Desktop sidebar hidden
|
||||||
|
await expect(page.getByRole("navigation", { name: /sidebar/i })).not.toBeVisible();
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
## State of the Art
|
||||||
|
|
||||||
|
| Old Approach | Current Approach | When Changed | Impact |
|
||||||
|
|--------------|------------------|--------------|--------|
|
||||||
|
| Cypress for Next.js E2E | Playwright (official Next.js recommendation) | 2023–2024 | Cross-browser, better WS support, no iframe limitations |
|
||||||
|
| `lighthouse` npm module with custom scripts | `@lhci/cli autorun` | 2020+ | Automated multi-run averaging, assertions, CI reporting |
|
||||||
|
| `axe-playwright` (community) | `@axe-core/playwright` (official Deque) | 2022+ | Official package, same API, no extra wrapper |
|
||||||
|
| `next start` for E2E server | `node .next/standalone/server.js` | Next.js 12+ standalone | Required when `output: "standalone"` is set |
|
||||||
|
| middleware.ts | proxy.ts | Next.js 16 | Next.js 16 renamed middleware file |
|
||||||
|
|
||||||
|
**Deprecated/outdated:**
|
||||||
|
- `cypress/integration/` directory: Cypress split this into `cypress/e2e/` in v10 — but we're not using Cypress
|
||||||
|
- `@playwright/test` `globalSetup` string path: Still valid but the project-based `setup` dependency is preferred in Playwright 1.40+
|
||||||
|
- `installSerwist()`: Replaced by `new Serwist() + addEventListeners()` in serwist v9 (already applied in Phase 8)
|
||||||
|
|
||||||
|
## Open Questions
|
||||||
|
|
||||||
|
1. **Lighthouse on authenticated pages**
|
||||||
|
- What we know: Lighthouse runs as unauthenticated — authenticated pages redirect to `/login`
|
||||||
|
- What's unclear: Whether LHCI supports cookie injection (not documented)
|
||||||
|
- Recommendation: Scope Lighthouse to `/login` only for QA-02. Dashboard/chat performance validated manually or via Web Vitals tracking in production.
|
||||||
|
|
||||||
|
2. **Visual regression baseline generation environment**
|
||||||
|
- What we know: OS-level rendering differences cause false failures
|
||||||
|
- What's unclear: Whether the Gitea runner is Linux or Mac
|
||||||
|
- Recommendation: Wave 0 task generates baselines inside the CI Docker container (Linux), commits them. Dev machines use `--update-snapshots` only deliberately.
|
||||||
|
|
||||||
|
3. **Celery worker in E2E**
|
||||||
|
- What we know: The chat WebSocket flow uses Redis pub-sub to deliver responses from the Celery worker
|
||||||
|
- What's unclear: Whether E2E should run the Celery worker (real pipeline, slow) or mock the WS entirely (fast but less realistic)
|
||||||
|
- Recommendation: Mock the WebSocket entirely via `page.routeWebSocket()`. This tests the frontend streaming UX without depending on Celery. Add a separate smoke test that hits the gateway `/health` endpoint to verify service health in CI.
|
||||||
|
|
||||||
|
## Validation Architecture
|
||||||
|
|
||||||
|
### Test Framework
|
||||||
|
| Property | Value |
|
||||||
|
|----------|-------|
|
||||||
|
| Framework (backend) | pytest 8.3+ / pytest-asyncio (existing, all tests pass) |
|
||||||
|
| Framework (E2E) | @playwright/test ^1.51 (to be installed) |
|
||||||
|
| Config file (E2E) | `packages/portal/playwright.config.ts` — Wave 0 |
|
||||||
|
| Quick run (backend) | `uv run pytest tests/unit -x --tb=short` |
|
||||||
|
| Full suite (backend) | `uv run pytest tests/ -x --tb=short` |
|
||||||
|
| E2E run | `cd packages/portal && npx playwright test` |
|
||||||
|
| Visual update | `cd packages/portal && npx playwright test --update-snapshots` |
|
||||||
|
|
||||||
|
### Phase Requirements → Test Map
|
||||||
|
| Req ID | Behavior | Test Type | Automated Command | File Exists? |
|
||||||
|
|--------|----------|-----------|-------------------|-------------|
|
||||||
|
| QA-01 | 7 critical user flows pass | E2E Playwright | `npx playwright test e2e/flows/ --project=chromium` | Wave 0 |
|
||||||
|
| QA-02 | Lighthouse >= 90 on key pages | Lighthouse CI | `npx lhci autorun --config=e2e/lighthouse/lighthouserc.json` | Wave 0 |
|
||||||
|
| QA-03 | Visual snapshots pass at 3 viewports | Visual regression | `npx playwright test e2e/visual/` | Wave 0 |
|
||||||
|
| QA-04 | Zero critical a11y violations | Accessibility scan | `npx playwright test e2e/accessibility/` | Wave 0 |
|
||||||
|
| QA-05 | All E2E flows pass on 3 browsers | Cross-browser E2E | `npx playwright test e2e/flows/` (all projects) | Wave 0 |
|
||||||
|
| QA-06 | Empty/error/loading states correct | E2E Playwright | Covered within flow specs via API mocking | Wave 0 |
|
||||||
|
| QA-07 | CI pipeline runs in Gitea Actions | CI workflow | `.gitea/workflows/ci.yml` | Wave 0 |
|
||||||
|
|
||||||
|
### Sampling Rate
|
||||||
|
- **Per task commit:** `cd packages/portal && npx playwright test e2e/flows/login.spec.ts --project=chromium`
|
||||||
|
- **Per wave merge:** `cd packages/portal && npx playwright test e2e/flows/ --project=chromium`
|
||||||
|
- **Phase gate:** Full suite (all projects + accessibility + visual) green before `/gsd:verify-work`
|
||||||
|
|
||||||
|
### Wave 0 Gaps
|
||||||
|
- [ ] `packages/portal/playwright.config.ts` — E2E framework config
|
||||||
|
- [ ] `packages/portal/e2e/auth.setup.ts` — Auth state generation for 3 roles
|
||||||
|
- [ ] `packages/portal/e2e/fixtures.ts` — Shared test fixtures (axe, auth, API helpers)
|
||||||
|
- [ ] `packages/portal/e2e/helpers/seed.ts` — Test data seeding via API
|
||||||
|
- [ ] `packages/portal/e2e/flows/*.spec.ts` — 7 flow spec files
|
||||||
|
- [ ] `packages/portal/e2e/accessibility/a11y.spec.ts` — axe-core scans
|
||||||
|
- [ ] `packages/portal/e2e/visual/snapshots.spec.ts` — visual regression specs
|
||||||
|
- [ ] `packages/portal/e2e/lighthouse/lighthouserc.json` — Lighthouse CI config
|
||||||
|
- [ ] `.gitea/workflows/ci.yml` — CI pipeline
|
||||||
|
- [ ] `packages/portal/playwright/.auth/.gitkeep` — Directory for saved auth state (gitignored content)
|
||||||
|
- [ ] Framework install: `cd packages/portal && npm install --save-dev @playwright/test @axe-core/playwright @lhci/cli && npx playwright install --with-deps`
|
||||||
|
- [ ] Baseline snapshots: run `npx playwright test e2e/visual/ --update-snapshots` on Linux to generate
|
||||||
|
|
||||||
|
## Sources
|
||||||
|
|
||||||
|
### Primary (HIGH confidence)
|
||||||
|
- https://playwright.dev/docs/auth — storageState, setup projects, multiple roles
|
||||||
|
- https://playwright.dev/docs/api/class-websocketroute — WebSocket mocking API
|
||||||
|
- https://playwright.dev/docs/test-snapshots — toHaveScreenshot, maxDiffPixelRatio
|
||||||
|
- https://playwright.dev/docs/accessibility-testing — @axe-core/playwright integration
|
||||||
|
- https://playwright.dev/docs/ci — CI configuration, Docker image, workers
|
||||||
|
- https://googlechrome.github.io/lighthouse-ci/docs/configuration.html — minScore assertions format
|
||||||
|
|
||||||
|
### Secondary (MEDIUM confidence)
|
||||||
|
- https://googlechrome.github.io/lighthouse-ci/docs/getting-started.html — lhci autorun setup
|
||||||
|
- https://playwright.dev/docs/mock — page.route() and page.routeWebSocket() overview
|
||||||
|
- Gitea Actions docs (forum.gitea.com) — confirmed GitHub Actions YAML compatibility, Docker socket requirements
|
||||||
|
|
||||||
|
### Tertiary (LOW confidence)
|
||||||
|
- WebSearch result: Gitea runner Docker group requirement — mentioned across multiple community posts, not in official docs
|
||||||
|
|
||||||
|
## Metadata
|
||||||
|
|
||||||
|
**Confidence breakdown:**
|
||||||
|
- Standard stack: HIGH — verified against official Playwright, @axe-core, and LHCI docs
|
||||||
|
- Architecture: HIGH — patterns derived directly from official Playwright documentation
|
||||||
|
- Pitfalls: HIGH (pitfalls 1–6 from direct codebase inspection + official docs); MEDIUM (pitfall 7 from community sources)
|
||||||
|
|
||||||
|
**Research date:** 2026-03-25
|
||||||
|
**Valid until:** 2026-06-25 (90 days — Playwright and Next.js are fast-moving but breaking changes are rare)
|
||||||
Reference in New Issue
Block a user