feat(update): pre-flight gates + deeper healthcheck + auto-rollback
Some checks failed
ARM64 Build / Build generic ARM64 disk image (push) Failing after 4s
CI / Go Tests (push) Successful in 1m29s
CI / Shellcheck (push) Successful in 48s
CI / Build Go Binaries (amd64, linux, linux-amd64) (push) Successful in 1m12s
CI / Build Go Binaries (arm64, linux, linux-arm64) (push) Has been cancelled
Some checks failed
ARM64 Build / Build generic ARM64 disk image (push) Failing after 4s
CI / Go Tests (push) Successful in 1m29s
CI / Shellcheck (push) Successful in 48s
CI / Build Go Binaries (amd64, linux, linux-amd64) (push) Successful in 1m12s
CI / Build Go Binaries (arm64, linux, linux-arm64) (push) Has been cancelled
Phase 8 of v0.3. Tightens the update lifecycle on both ends. Pre-flight (apply.go, before any download): - Free-space check on the passive partition: image size + 10% headroom must be available. Uses statfs(2) via the new pkg/partition.FreeBytes / HasFreeSpaceFor helpers (tests cover happy path, tiny request, huge request, missing path). Catches corrupted-FS and shrunk-partition cases before we destroy the existing slot data. - Node-block-label check: refuses if the local K8s node carries the updates.kubesolo.io/block=true label. New pkg/health.CheckNodeBlocked shells out to kubectl per the project's zero-deps stance. Silently bypassed when no kubeconfig is reachable (air-gap case). Skipped by --force. Healthcheck (extended via new pkg/health/extended.go + preflight.go): - CheckKubeSystemReady waits until every kube-system pod has held the Running phase for >= N seconds (default 30). Catches "started ok, will crash-loop" bugs that a single-shot phase check misses. - CheckProbeURL fetches an operator-supplied URL; 200 = pass. Wired through update.conf as healthcheck_url= and cloud-init updates.healthcheck_url. - CheckDiskWritable writes/fsyncs/reads a 1-KiB probe under /var/lib/kubesolo. Always runs in healthcheck so a wedged data partition fails fast. - pkg/health.Status grows KubeSystemReady, ProbeURL, DiskWritable booleans. Optional checks default to true in RunAll() so they don't block when unconfigured. health_test.go updated to the new 6-field shape. Auto-rollback (healthcheck.go): - state.UpdateState gains HealthCheckFailures (consecutive post-Activated failures). Reset on a clean pass. - --auto-rollback-after N (also auto_rollback_after= in update.conf) triggers env.ForceRollback() when the failure count reaches the threshold. State transitions to RolledBack with a descriptive LastError. The command still exits with the healthcheck error; the operator/init is expected to reboot. - Only fires while Phase == Activated. Doesn't second-guess a long-stable system that happens to fail one healthcheck. config / opts / cloud-init plumbing: - update.conf gains healthcheck_url= and auto_rollback_after= keys. - New CLI flags: --healthcheck-url, --auto-rollback-after, --kube-system-settle. - cloud-init full-config.yaml documents the new updates: subfields. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -24,15 +24,20 @@ import (
|
||||
|
||||
// Status represents the result of a health check.
|
||||
type Status struct {
|
||||
Containerd bool
|
||||
APIServer bool
|
||||
NodeReady bool
|
||||
Message string
|
||||
Containerd bool
|
||||
APIServer bool
|
||||
NodeReady bool
|
||||
KubeSystemReady bool // optional — true unless KubeSystemSettle is non-zero
|
||||
ProbeURL bool // optional — true unless ProbeURL is set
|
||||
DiskWritable bool // optional — true unless DataDir is set
|
||||
Message string
|
||||
}
|
||||
|
||||
// IsHealthy returns true if all checks passed.
|
||||
// IsHealthy returns true if all required checks passed. Optional checks
|
||||
// default to true when not configured, so they don't block the result.
|
||||
func (s *Status) IsHealthy() bool {
|
||||
return s.Containerd && s.APIServer && s.NodeReady
|
||||
return s.Containerd && s.APIServer && s.NodeReady &&
|
||||
s.KubeSystemReady && s.ProbeURL && s.DiskWritable
|
||||
}
|
||||
|
||||
// Checker performs health checks against the local KubeSolo instance.
|
||||
@@ -40,6 +45,11 @@ type Checker struct {
|
||||
kubeconfigPath string
|
||||
apiServerAddr string
|
||||
timeout time.Duration
|
||||
|
||||
// Optional gates. Zero values disable the check (it reports true).
|
||||
KubeSystemSettle time.Duration
|
||||
ProbeURL string
|
||||
DataDir string
|
||||
}
|
||||
|
||||
// NewChecker creates a health checker.
|
||||
@@ -149,12 +159,37 @@ func (c *Checker) CheckNodeReady() bool {
|
||||
}
|
||||
|
||||
// RunAll performs all health checks and returns the combined status.
|
||||
//
|
||||
// Optional checks (kube-system settle, user probe URL, disk writability) are
|
||||
// only run if the corresponding Checker fields are set; otherwise they
|
||||
// report true so as not to block the result.
|
||||
func (c *Checker) RunAll() *Status {
|
||||
return &Status{
|
||||
Containerd: c.CheckContainerd(),
|
||||
APIServer: c.CheckAPIServer(),
|
||||
NodeReady: c.CheckNodeReady(),
|
||||
s := &Status{
|
||||
Containerd: c.CheckContainerd(),
|
||||
APIServer: c.CheckAPIServer(),
|
||||
NodeReady: c.CheckNodeReady(),
|
||||
KubeSystemReady: true,
|
||||
ProbeURL: true,
|
||||
DiskWritable: true,
|
||||
}
|
||||
if c.KubeSystemSettle > 0 {
|
||||
s.KubeSystemReady = c.CheckKubeSystemReady(c.KubeSystemSettle)
|
||||
}
|
||||
if c.ProbeURL != "" {
|
||||
ok, err := CheckProbeURL(c.ProbeURL)
|
||||
if err != nil {
|
||||
slog.Warn("probe URL check failed", "url", c.ProbeURL, "error", err)
|
||||
}
|
||||
s.ProbeURL = ok
|
||||
}
|
||||
if c.DataDir != "" {
|
||||
ok, err := CheckDiskWritable(c.DataDir)
|
||||
if err != nil {
|
||||
slog.Warn("disk writability check failed", "dir", c.DataDir, "error", err)
|
||||
}
|
||||
s.DiskWritable = ok
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// WaitForHealthy polls health checks until all pass or timeout expires.
|
||||
|
||||
Reference in New Issue
Block a user