kubesolo-os/docs/release-notes-0.3.0.md

KubeSolo OS v0.3.0 — Release Notes

Released: 2026-05-14

v0.3.0 is the second feature release after v0.2.0 and the first release that ships a generic ARM64 build alongside x86_64. The update agent grew up: it now has an explicit on-disk lifecycle, OCI registry distribution, and a fleet-friendly set of policy gates (channels, maintenance windows, version-stepping-stones, pre-flight checks, auto-rollback).

This document is the operator-facing summary. The full per-phase changelog lives in CHANGELOG.md.

What's new

Generic ARM64 build

The image you build with make disk-image-arm64 now targets any UEFI-capable ARM64 host: AWS Graviton, Oracle Ampere, generic ARM64 servers, future SBCs with UEFI-compatible firmware. The kernel comes from kernel.org mainline LTS (6.12.10 by default, configurable via MAINLINE_KERNEL_VERSION in build/config/versions.env).

This is distinct from the Raspberry Pi build path. RPi keeps its specialised kernel from raspberrypi/linux with bcm-defconfig + custom DTBs; the generic ARM64 path uses mainline + arm64-defconfig + UEFI/virtio. See docs/arm64-architecture.md for the file-by-file split.

KubeSolo bumped to v1.1.5 (was v1.1.0). New flags surfaced via cloud-init:

• kubesolo.full — disable edge-optimised k8s overrides
• kubesolo.disable-ipv6 — disable IPv6 cluster-wide
• kubesolo.db-wal-repair — recover from unclean shutdowns

Update lifecycle is now observable

The update agent writes a state.json at /var/lib/kubesolo/update/state.json recording where the current attempt is in the lifecycle:

idle → checking → downloading → staged → activated → verifying → success
                                                              ↘ rolled_back
                                                              ↘ failed

kubesolo-update status --json emits the full state for orchestration tooling. The Prometheus metrics endpoint gains three new series:

• kubesolo_update_phase{phase="..."} — 1 for current phase, 0 for others (all 9 always emitted)
• kubesolo_update_attempts_total
• kubesolo_update_last_attempt_timestamp_seconds

OCI registry distribution

Update artifacts can now be pulled from any OCI-compliant registry alongside the existing HTTP latest.json protocol:

# HTTP, unchanged from v0.2:
kubesolo-update apply --server https://updates.example.com

# New: OCI from ghcr.io (or quay.io, harbor, zot, ...)
kubesolo-update apply --registry ghcr.io/yourorg/kubesolo-os --tag stable

Multi-arch is handled transparently — the same stable tag points at a manifest index, the agent picks the manifest matching its runtime.GOARCH.

Publish your own artifacts with build/scripts/push-oci-artifact.sh. See the script's header comment for the full publishing flow.

Policy gates

apply now enforces five gates before destroying the passive slot:

1. Maintenance window (configurable, e.g. 03:00-05:00; wrapping midnight supported)
2. Node-block-label — refuses if the K8s node carries updates.kubesolo.io/block=true (workload-author kill switch)
3. Channel — stable / beta / edge must match between the artifact metadata and the local channel
4. Architecture — refuses cross-arch artifacts via runtime.GOARCH check
5. Min compatible version — stepping-stone enforcement; refuses an upgrade that bypasses a required intermediate version

--force bypasses the maintenance window and node-block label (channel / arch / min-version are non-negotiable). Failures are recorded in state.json with a clear LastError field.

Healthcheck deepening + auto-rollback

kubesolo-update healthcheck grew three optional probes:

• Kube-system pods must hold Running for ≥ N seconds before passing
• Operator probe URL — GET an operator-supplied endpoint; 200 = pass
• Disk smoke test — write/fsync/read/delete a probe file under /var/lib/kubesolo to catch a wedged data partition

Plus auto-rollback: with --auto-rollback-after N (or auto_rollback_after= in update.conf), after N consecutive post-activation failures, the agent calls ForceRollback() and the operator/init is expected to reboot. The counter resets on a clean pass.

Persistent configuration via /etc/kubesolo/update.conf

Cloud-init writes this file on first boot from a new updates: block; you can also hand-edit it. Recognised keys:

server = https://updates.example.com         # or omit if using registry
registry =                                   # OCI registry ref (alt to server)
channel = stable
maintenance_window = 03:00-05:00
pubkey = /etc/kubesolo/update-pubkey.hex
healthcheck_url = http://localhost:8000/ready
auto_rollback_after = 3

Cloud-init full reference at cloud-init/examples/full-config.yaml.

Migration from v0.2.x

This is a non-breaking release for live systems. v0.2.x → v0.3.0 changes:

• state.json will appear at /var/lib/kubesolo/update/state.json the first time a v0.3 agent runs apply. Pre-existing v0.2 deployments without this file are fine — the agent treats a missing file as fresh Idle state.
• update.conf is optional. v0.2 deployments that pass everything via CLI flags keep working unchanged.
• HTTP latest.json protocol unchanged. Existing update servers don't need a rebuild.
• GRUB env (boot counter, active slot) unchanged. The bootloader's rollback behaviour is the same.
• No new mandatory kernel command-line parameters.

To opt into the new lifecycle, transports, and gates, drop in an update.conf (or update cloud-init) and switch to --registry if you want OCI distribution.

Known limitations

These shipped intentionally with v0.3.0 and are explicitly tracked for v0.3.1+:

• OCI signature verification — the OCI transport is digest-verified end-to-end via oras-go, but does not yet consume cosign-style referrer attestations. The HTTP transport still honours --pubkey for .sig files.
• ARM64 LABEL=KSOLODATA resolution doesn't work yet — piCore's blkid/findfs crash on QEMU virt under our mainline kernel; the static busybox-static we ship doesn't include those applets. build/grub/grub-arm64.cfg hardcodes kubesolo.data=/dev/vda4 as a workaround. On real ARM64 hardware the device path may differ.
• Real-hardware ARM64 validation is pending. The image builds and boots end-to-end under QEMU virt; production certification waits on a Graviton / Ampere run.
• AppArmor profile load fails on ARM64 (apparmor_parser ABI mismatch). Init reports the failure; boot continues without AppArmor enforcement.
• QEMU TCG performance can trigger KubeSolo's first-boot image-import deadline. Not an OS defect; real hardware and KVM-accelerated QEMU complete the import in seconds.

How to upgrade your build host

git pull
make distclean   # optional — drops the build cache; full rebuild takes ~30 min
make iso         # or disk-image, or disk-image-arm64

The Docker-based builder (make docker-build) regenerates its own image from build/Dockerfile.builder on next invocation; oras 1.2.3 and busybox-static are now included.

Acknowledgements

v0.3.0 work was driven by a single multi-week pair-programming session working through Phases 0–9 of the v0.3 roadmap. The Odroid self-hosted Gitea Actions runner (odroid.local, arm64-linux) carried every ARM64 build during development.