Files
kubesolo-os/update/main.go
Adolfo Delorenzo 9fb894c5af
Some checks failed
ARM64 Build / Build generic ARM64 disk image (push) Failing after 4s
CI / Go Tests (push) Successful in 1m29s
CI / Shellcheck (push) Successful in 48s
CI / Build Go Binaries (amd64, linux, linux-amd64) (push) Successful in 1m12s
CI / Build Go Binaries (arm64, linux, linux-arm64) (push) Has been cancelled
feat(update): pre-flight gates + deeper healthcheck + auto-rollback
Phase 8 of v0.3. Tightens the update lifecycle on both ends.

Pre-flight (apply.go, before any download):
- Free-space check on the passive partition: image size + 10% headroom must
  be available. Uses statfs(2) via the new pkg/partition.FreeBytes /
  HasFreeSpaceFor helpers (tests cover happy path, tiny request, huge
  request, missing path). Catches corrupted-FS and shrunk-partition cases
  before we destroy the existing slot data.
- Node-block-label check: refuses if the local K8s node carries the
  updates.kubesolo.io/block=true label. New pkg/health.CheckNodeBlocked
  shells out to kubectl per the project's zero-deps stance. Silently bypassed
  when no kubeconfig is reachable (air-gap case). Skipped by --force.

Healthcheck (extended via new pkg/health/extended.go + preflight.go):
- CheckKubeSystemReady waits until every kube-system pod has held the Running
  phase for >= N seconds (default 30). Catches "started ok, will crash-loop"
  bugs that a single-shot phase check misses.
- CheckProbeURL fetches an operator-supplied URL; 200 = pass. Wired through
  update.conf as healthcheck_url= and cloud-init updates.healthcheck_url.
- CheckDiskWritable writes/fsyncs/reads a 1-KiB probe under /var/lib/kubesolo.
  Always runs in healthcheck so a wedged data partition fails fast.
- pkg/health.Status grows KubeSystemReady, ProbeURL, DiskWritable booleans.
  Optional checks default to true in RunAll() so they don't block when
  unconfigured. health_test.go updated to the new 6-field shape.

Auto-rollback (healthcheck.go):
- state.UpdateState gains HealthCheckFailures (consecutive post-Activated
  failures). Reset on a clean pass.
- --auto-rollback-after N (also auto_rollback_after= in update.conf) triggers
  env.ForceRollback() when the failure count reaches the threshold. State
  transitions to RolledBack with a descriptive LastError. The command still
  exits with the healthcheck error; the operator/init is expected to reboot.
- Only fires while Phase == Activated. Doesn't second-guess a long-stable
  system that happens to fail one healthcheck.

config / opts / cloud-init plumbing:
- update.conf gains healthcheck_url= and auto_rollback_after= keys.
- New CLI flags: --healthcheck-url, --auto-rollback-after, --kube-system-settle.
- cloud-init full-config.yaml documents the new updates: subfields.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-14 19:08:30 -06:00

106 lines
3.8 KiB
Go

// kubesolo-update is the atomic update agent for KubeSolo OS.
//
// It manages A/B partition updates with automatic rollback:
//
// kubesolo-update check Check for available updates
// kubesolo-update apply Download + write update to passive partition
// kubesolo-update activate Set passive partition as next boot target
// kubesolo-update rollback Force rollback to other partition
// kubesolo-update healthcheck Post-boot health verification
// kubesolo-update status Show current A/B slot and boot status
// kubesolo-update sign Sign update artifacts with Ed25519 key
// kubesolo-update genkey Generate new Ed25519 signing key pair
// kubesolo-update metrics Start Prometheus-compatible metrics server
package main
import (
"fmt"
"log/slog"
"os"
"github.com/portainer/kubesolo-os/update/cmd"
)
func main() {
slog.SetDefault(slog.New(slog.NewTextHandler(os.Stderr, &slog.HandlerOptions{
Level: slog.LevelInfo,
})))
if len(os.Args) < 2 {
usage()
os.Exit(1)
}
var err error
switch os.Args[1] {
case "check":
err = cmd.Check(os.Args[2:])
case "apply":
err = cmd.Apply(os.Args[2:])
case "activate":
err = cmd.Activate(os.Args[2:])
case "rollback":
err = cmd.Rollback(os.Args[2:])
case "healthcheck":
err = cmd.Healthcheck(os.Args[2:])
case "status":
err = cmd.Status(os.Args[2:])
case "sign":
err = cmd.Sign(os.Args[2:])
case "genkey":
err = cmd.GenKey(os.Args[2:])
case "metrics":
err = cmd.Metrics(os.Args[2:])
default:
fmt.Fprintf(os.Stderr, "unknown command: %s\n\n", os.Args[1])
usage()
os.Exit(1)
}
if err != nil {
slog.Error("command failed", "command", os.Args[1], "error", err)
os.Exit(1)
}
}
func usage() {
fmt.Fprintf(os.Stderr, `Usage: kubesolo-update <command> [options]
Commands:
check Check for available updates
apply Download and write update to passive partition
activate Set passive partition as next boot target
rollback Force rollback to other partition
healthcheck Post-boot health verification (marks boot successful)
status Show current A/B slot and boot status
sign Sign artifacts with Ed25519 private key (build system)
genkey Generate new Ed25519 signing key pair
metrics Start Prometheus-compatible metrics HTTP server
Options:
--server URL HTTP update server (mutually exclusive with --registry)
--registry REPO OCI registry repository, e.g. ghcr.io/portainer/kubesolo-os
(mutually exclusive with --server)
--tag TAG OCI tag to pull (default: channel name, then "stable")
--conf PATH update.conf path (default: /etc/kubesolo/update.conf)
--state PATH Update state file (default: /var/lib/kubesolo/update/state.json)
--channel NAME Update channel (default: "stable", or value from update.conf)
--maintenance-window HH:MM-HH:MM local time window; apply refuses outside it
--force Bypass maintenance-window check
--grubenv PATH Path to grubenv file (default: /boot/grub/grubenv)
--timeout SECS Health check timeout in seconds (default: 120)
--pubkey PATH Ed25519 public key for signature verification (optional)
--healthcheck-url URL Optional GET probe in healthcheck; 200 = pass
--auto-rollback-after N healthcheck: rollback after N consecutive failures
--kube-system-settle N healthcheck: require kube-system pods Running ≥ N seconds
--json For 'status': emit JSON instead of human-readable output
Examples:
kubesolo-update apply --server https://updates.example.com
kubesolo-update apply --registry ghcr.io/portainer/kubesolo-os --tag stable
kubesolo-update apply --force # uses /etc/kubesolo/update.conf
kubesolo-update healthcheck
kubesolo-update status --json
`)
}