kubesolo-os/CHANGELOG.md

# Changelog

All notable changes to KubeSolo OS are documented in this file.

Format based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
versioning follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [0.3.0-dev] - unreleased

### Added

- Generic ARM64 build track distinct from Raspberry Pi:
  - `make kernel-arm64` builds a mainline kernel.org LTS kernel (6.12.10 by
    default) from `arm64 defconfig` + shared `kernel-container.fragment` +
    arm64 virt-host enables (VIRTIO_*, EFI_STUB, NVMe).
  - `make disk-image-arm64` produces a UEFI-bootable raw GPT image with A/B
    system partitions and GRUB-EFI ARM64. Targets QEMU virt, Graviton, Ampere,
    or any UEFI ARM64 host.
  - `hack/dev-vm-arm64.sh --disk` boots the built image through QEMU UEFI for
    end-to-end testing.
  - `test/qemu/test-boot-arm64-disk.sh` automated boot smoke test.
- Bumped KubeSolo to v1.1.5 (was v1.1.0). New cloud-init flags surfaced:
  - `kubesolo.full` (v1.1.4+) — disable edge-optimised overrides
  - `kubesolo.disable-ipv6` (v1.1.5+)
  - `kubesolo.db-wal-repair` (v1.1.5+) — recover from unclean shutdowns
- Per-arch supply-chain verification: `KUBESOLO_SHA256_AMD64` and
  `KUBESOLO_SHA256_ARM64` in `versions.env`, applied to the tarball before
  extract.
- `docs/arm64-architecture.md` — defines the generic-vs-RPi two-track layout.
- `docs/arm64-status.md` — Phase 3 status snapshot, known limitations, what's
  needed to ship.
- `docs/ci-runners.md` — Gitea Actions runner setup (Odroid arm64-linux).

### Changed

- `build/scripts/build-kernel-arm64.sh` is now the **generic ARM64** kernel
  build (mainline kernel.org LTS, generic UEFI/virtio).
- Renamed `build/scripts/build-kernel-rpi.sh` (was `build-kernel-arm64.sh`).
  RPi kernel build (raspberrypi/linux fork, bcm2711_defconfig) lives here now.
- Renamed `build/config/kernel-container.fragment` (was
  `rpi-kernel-config.fragment`). Misnomer: contents are arch-agnostic and now
  shared across x86, ARM64-generic, and RPi kernels.
- `build/scripts/build-kernel.sh` (x86) refactored to consume the shared
  fragment via a generic `apply_fragment` function. ~50 lines of duplication
  killed.
- `KUBESOLO_VERSION` moved out of `fetch-components.sh` defaults into
  `versions.env`. Bumping is now a one-line PR.

### Fixed

- Native ARM64 build hosts (e.g. an Odroid runner) no longer require the x86
  cross-compiler. Both `build-kernel-arm64.sh` and `build-kernel-rpi.sh` detect
  `uname -m` and use the host's gcc directly when arch matches.
- ARM64 grub.cfg console ordering: `ttyAMA0` is now the primary console
  (`console=ttyS0,... console=ttyAMA0,...`). Init output is now visible on
  QEMU virt and most ARM64 SBCs without further configuration.
- ARM64 boot: replaced piCore64's `/init` with our staged init at `/init` and
  `/sbin/init`. Previously the kernel ran piCore's TCE handler which
  segfaulted in our environment.
- ARM64 boot: replaced piCore64's broken dynamic BusyBox with the build
  host's `busybox-static`. piCore's binary triggered EL0 instruction-abort
  panics on QEMU virt under both `-cpu cortex-a72` and `-cpu max`.
- POSIX-character-class portability: `tr -d '[:space:]'` in
  `30-kernel-modules.sh` and `40-sysctl.sh` replaced with explicit
  `' \t\r\n'`. Ubuntu's busybox-static 1.30.1 doesn't parse `[:space:]` and
  instead deletes the literal characters `[ : s p a c e ]`, which truncated
  module names (`virtio_net` → `virtio_nt`, etc.) and sysctl keys.
- `inject-kubesolo.sh` no longer copies `init/lib/functions.sh` into
  `init.d/`. Previously the main init loop tried to run it as a stage after
  stage 90 and panicked with "Init completed without exec'ing KubeSolo".
- ARM64 disk image: `TARGET_ARCH=arm64 create-disk-image.sh` produces
  `BOOTAA64.EFI` via `grub-mkimage -O arm64-efi` (not `bootx64.efi`). Skips
  the BIOS-only `grub-install --target=i386-pc` step.
- `build/Dockerfile.builder`: added `grub-efi-amd64-bin`, `grub-efi-arm64-bin`,
  `grub-pc-bin`, `grub-common`, `grub2-common`, and `busybox-static` so the
  Docker-based build flow can produce ARM64 disk images and gets the same
  BusyBox swap behaviour as native builds.

### Known limitations (deferred to follow-up)

- ARM64 `kubesolo.data=LABEL=KSOLODATA` resolution doesn't work yet —
  piCore's `blkid`/`findfs` crash in QEMU and our static busybox lacks the
  applets. Hardcoded `/dev/vda4` as a workaround. Production fix: ship
  static `blkid`/`findfs` or replace LABEL resolution with a sysfs walk.
- AppArmor profile load fails on ARM64 (apparmor_parser ABI mismatch).
- KubeSolo's image-import deadline can fire under QEMU TCG (software
  emulation). On real hardware (or with KVM) the import finishes in seconds.

## [0.2.0] - 2026-02-12

### Added
- Cloud-init: support all documented KubeSolo CLI flags (`--local-storage-shared-path`, `--debug`, `--pprof-server`, `--portainer-edge-id`, `--portainer-edge-key`, `--portainer-edge-async`)
- Cloud-init: `full-config.yaml` example showing all supported parameters
- Cloud-init: KubeSolo configuration reference table in docs/cloud-init.md
- Security hardening: mount hardening, sysctl, kernel module lock, AppArmor profiles
- ARM64 Raspberry Pi support with A/B boot via tryboot
- BootEnv abstraction for GRUB and RPi boot environments
- Go 1.25.5 installed on host for native builds

## [0.1.0] - 2026-02-12

First release with all 5 design-doc phases complete. ISO boots and runs K8s pods.

### Added

#### Custom Kernel
- Custom kernel build (6.18.2-tinycore64) with container-critical configs
- Added CONFIG_CGROUP_BPF, CONFIG_DEVTMPFS, CONFIG_DEVTMPFS_MOUNT, CONFIG_MEMCG, CONFIG_CFS_BANDWIDTH
- Stripped unnecessary subsystems (sound, GPU, wireless, Bluetooth, etc.)
- Selective kernel module install — only modules.list + transitive deps in initramfs

#### Init System (Phase 1)
- POSIX sh init system with staged boot (00-early-mount through 90-kubesolo)
- switch_root from initramfs to SquashFS root
- Persistent data partition mount with bind-mounts for K8s state
- Kernel module loading, sysctl tuning, network, hostname, NTP
- Emergency shell fallback on boot failure
- Device node creation via mknod fallback from sysfs

#### Cloud-Init (Phase 2)
- Go-based cloud-init parser (~2.7 MB static binary)
- Network configuration: DHCP and static IP modes
- Hostname and machine-id generation
- KubeSolo configuration (node-name, extra flags)
- Portainer Edge Agent integration via K8s manifest injection
- Persistent config saved to /mnt/data/ for next-boot fast path
- 22 Go tests

#### A/B Atomic Updates (Phase 3)
- 4-partition GPT disk image: EFI + System A + System B + Data
- GRUB 2 bootloader with A/B slot selection and boot counter rollback
- Go update agent (~6.0 MB static binary) with check, apply, activate, rollback commands
- Health check: containerd + K8s API + node Ready verification
- Update server protocol: HTTP serving latest.json + image files
- K8s CronJob for automated update checks (every 6 hours)
- Zero external Go dependencies — uses kubectl/ctr exec commands

#### Production Hardening (Phase 4)
- Ed25519 image signing with pure Go stdlib (zero external deps)
- Key generation, signing, and verification CLI commands
- Portainer Edge Agent deployment via cloud-init
- SSH extension injection for debugging (hack/inject-ssh.sh)
- Boot time and resource usage benchmarks
- Deployment guide documentation

#### Distribution & Fleet Management (Phase 5)
- Gitea Actions CI/CD (test + build + shellcheck on push, release on tags)
- OCI container image packaging (scratch-based)
- Prometheus metrics endpoint (zero-dependency text exposition format)
- USB provisioning script with cloud-init injection
- ARM64 cross-compilation support

#### Build System
- Makefile with full build orchestration
- Dockerized reproducible builds (build/Dockerfile.builder)
- Component fetching with version pinning
- ISO and raw disk image creation
- Fast rebuild path (`make quick`)

#### Documentation
- Architecture design document
- Boot flow reference
- A/B update flow reference
- Cloud-init configuration reference
- Deployment and operations guide

### Fixed
- Replaced `grep -oP` with POSIX-safe `sed` in functions.sh (BusyBox compatibility)
- Replaced `grep -qiE` with `grep -qi -e` pattern (POSIX compliance)
- Fixed KVM flag handling in dev-vm.sh (bash array context)
- Added iptables table pre-initialization before kube-proxy start (nf_tables issue)
- Added /dev/kmsg and /etc/machine-id creation for kubelet
- Added CA certificates bundle to initramfs (containerd TLS verification for Docker Hub)
- Added DNS fallback (10.0.2.3 + 8.8.8.8) when DHCP client doesn't populate resolv.conf
- Added headless Service to Portainer Edge Agent manifest (agent peer discovery DNS)
- Added kubesolo.edge_id/edge_key kernel boot parameters for Portainer Edge
- Added auto-format of unformatted data disks on first boot
- Rewrote dev-vm.sh for macOS: bsdtar ISO extraction, Homebrew mkfs.ext4 detection, direct kernel boot, TCG acceleration, port 8080 forwarding
- Kubeconfig now served via HTTP on port 8080 (serial console truncates base64 lines)
- Added 127.0.0.1 and 10.0.2.15 to API server SANs for QEMU port forwarding
- dev-vm.sh now works on Linux: fallback ISO extraction via isoinfo or loop mount, KVM auto-detection, platform-aware error messages