2 Commits

Author	SHA1	Message	Date
Adolfo Delorenzo	9fb894c5af	feat(update): pre-flight gates + deeper healthcheck + auto-rollback ... Phase 8 of v0.3. Tightens the update lifecycle on both ends. Pre-flight (apply.go, before any download): - Free-space check on the passive partition: image size + 10% headroom must be available. Uses statfs(2) via the new pkg/partition.FreeBytes / HasFreeSpaceFor helpers (tests cover happy path, tiny request, huge request, missing path). Catches corrupted-FS and shrunk-partition cases before we destroy the existing slot data. - Node-block-label check: refuses if the local K8s node carries the updates.kubesolo.io/block=true label. New pkg/health.CheckNodeBlocked shells out to kubectl per the project's zero-deps stance. Silently bypassed when no kubeconfig is reachable (air-gap case). Skipped by --force. Healthcheck (extended via new pkg/health/extended.go + preflight.go): - CheckKubeSystemReady waits until every kube-system pod has held the Running phase for >= N seconds (default 30). Catches "started ok, will crash-loop" bugs that a single-shot phase check misses. - CheckProbeURL fetches an operator-supplied URL; 200 = pass. Wired through update.conf as healthcheck_url= and cloud-init updates.healthcheck_url. - CheckDiskWritable writes/fsyncs/reads a 1-KiB probe under /var/lib/kubesolo. Always runs in healthcheck so a wedged data partition fails fast. - pkg/health.Status grows KubeSystemReady, ProbeURL, DiskWritable booleans. Optional checks default to true in RunAll() so they don't block when unconfigured. health_test.go updated to the new 6-field shape. Auto-rollback (healthcheck.go): - state.UpdateState gains HealthCheckFailures (consecutive post-Activated failures). Reset on a clean pass. - --auto-rollback-after N (also auto_rollback_after= in update.conf) triggers env.ForceRollback() when the failure count reaches the threshold. State transitions to RolledBack with a descriptive LastError. The command still exits with the healthcheck error; the operator/init is expected to reboot. - Only fires while Phase == Activated. Doesn't second-guess a long-stable system that happens to fail one healthcheck. config / opts / cloud-init plumbing: - update.conf gains healthcheck_url= and auto_rollback_after= keys. - New CLI flags: --healthcheck-url, --auto-rollback-after, --kube-system-settle. - cloud-init full-config.yaml documents the new updates: subfields. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-14 19:08:30 -06:00
Adolfo Delorenzo	bce565e2f7	feat(update): persistent state machine + lifecycle metrics ... Phase 5 of v0.3. Adds an explicit, on-disk state machine to the update agent so the lifecycle of an attempt is observable end-to-end, instead of being inferred from logs and side effects. New package update/pkg/state: - Phase enum (idle, checking, downloading, staged, activated, verifying, success, rolled_back, failed) - UpdateState struct persisted to /var/lib/kubesolo/update/state.json (overridable via --state). Atomic write (.tmp + rename). Survives reboots and slot switches because the file lives on the data partition. - Transition helper that bumps AttemptCount when an attempt starts, resets it when the target version changes, sets/clears LastError on failed/success transitions, and stamps StartedAt + UpdatedAt. - 13 unit tests cover the lifecycle, atomic write, version-change reset, error recording, idempotent SetFromVersion, garbage-file handling. Wired into the existing commands: - apply.go transitions Idle -> Checking -> Downloading -> Staged, with RecordError on any step failure. Reads the active slot's version file to populate FromVersion. - activate.go transitions to Activated. - healthcheck.go transitions Activated -> Verifying -> Success on pass, or to Failed on fail. Skips transitions if state isn't post-activation (manual healthcheck on a stable system shouldn't churn the state). - rollback.go transitions to RolledBack with LastError="manual rollback". - check.go intentionally untouched — checks are passive queries, not attempts; they shouldn't reset AttemptCount. status.go gains a --json mode that emits the full state report (A/B slots, boot counter, full UpdateState) for orchestration tooling. Human-readable mode also prints an Update Lifecycle section when state.phase != idle. pkg/metrics gains three new series, derived from state.json at scrape time: - kubesolo_update_phase{phase="..."} — 1 for current, 0 for all others; all nine phase values always emitted so dashboards see complete series - kubesolo_update_attempts_total - kubesolo_update_last_attempt_timestamp_seconds Server.SetStatePath() configures the file location; defaults to absent which emits Idle defaults. Three new tests cover the absent / active / all-phases-emitted cases. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-14 18:11:47 -06:00