21 Commits

Author SHA1 Message Date
80aca5e372 feat: ARM64 generic UEFI disk image (GPT + GRUB A/B)
Some checks failed
CI / Go Tests (push) Successful in 2m38s
CI / Shellcheck (push) Failing after 37s
CI / Build Go Binaries (amd64, linux, linux-amd64) (push) Failing after 1m22s
CI / Build Go Binaries (arm64, linux, linux-arm64) (push) Failing after 1m11s
Produces a UEFI-bootable raw disk image for generic ARM64 hosts (QEMU virt,
Ampere/Graviton cloud, ARM64 SBCs with UEFI). Reuses the existing 4-partition
A/B layout from x86 (EFI 256 MB FAT32 + System A 512 MB ext4 + System B 512 MB
ext4 + Data ext4 remainder).

Changes:
- build/scripts/create-disk-image.sh: TARGET_ARCH env var (amd64 default,
  arm64). Selects kernel source path, grub-mkimage target (x86_64-efi vs
  arm64-efi), EFI binary name (bootx64.efi vs BOOTAA64.EFI), grub.cfg variant,
  and whether to also install BIOS GRUB (x86 only).
- build/grub/grub-arm64.cfg: ARM64 variant of grub.cfg. Identical A/B logic;
  console=ttyAMA0+ttyS0 to cover QEMU virt PL011, Ampere PL011, and Graviton
  16550-compat.
- build/Dockerfile.builder: add grub-efi-amd64-bin, grub-efi-arm64-bin,
  grub-pc-bin, grub-common, grub2-common so the builder container can produce
  EFI images for both architectures.
- hack/dev-vm-arm64.sh: split into kernel mode (direct -kernel/-initrd, fast
  iteration) and --disk mode (UEFI firmware + GRUB + disk image, full
  integration test). Probes common UEFI firmware paths on Ubuntu/Fedora/macOS.
  Default kernel path now points at kernel-arm64-generic/Image with fallback
  to the renamed custom-kernel-rpi/Image.
- test/qemu/test-boot-arm64-disk.sh: new CI test for the full UEFI -> GRUB ->
  kernel -> stage-90 boot chain. Uses a scratch copy of the disk so grubenv
  writes don't mutate the source artifact.
- Makefile: new disk-image-arm64 target (depends on rootfs-arm64 + kernel-arm64),
  new test-boot-arm64-disk target, .PHONY + help updates.

Phase 3 scaffold is in place. First real end-to-end ARM64 build runs in the
next step on the Odroid runner — that's where we find out what's actually
broken.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-14 10:36:08 -06:00
d51618badb build: separate generic ARM64 from Raspberry Pi kernel builds
Splits the ARM64 build into two tracks per docs/arm64-architecture.md:

Generic ARM64 (mainline kernel.org, UEFI, virtio, GRUB):
- New build/scripts/build-kernel-arm64.sh builds mainline LTS (6.12.x by default)
  from arm64 defconfig + shared container fragment + arm64-virt enables
  (VIRTIO_*, EFI_STUB, NVMe). Output: build/cache/kernel-arm64-generic/.
- New Makefile targets: kernel-arm64, rootfs-arm64 (now consumes the mainline
  kernel modules via TARGET_VARIANT=generic).
- versions.env: pin MAINLINE_KERNEL_VERSION=6.12.10, declare cdn.kernel.org URL
  and SHA256 placeholder.

Raspberry Pi (raspberrypi/linux fork, custom DTBs, autoboot.txt):
- build-kernel-arm64.sh (RPi-flavoured) renamed to build-kernel-rpi.sh; cache
  dir renamed from custom-kernel-arm64 to custom-kernel-rpi.
- New Makefile targets: kernel-rpi, rootfs-arm64-rpi (uses TARGET_VARIANT=rpi).
- rpi-image now depends on rootfs-arm64-rpi + kernel-rpi instead of the generic
  rootfs-arm64.
- create-rpi-image.sh + inject-kubesolo.sh updated to reference the new cache
  path. inject-kubesolo.sh now takes a TARGET_VARIANT env var (rpi|generic) to
  select which ARM64 kernel modules to consume.

Shared substrate:
- rpi-kernel-config.fragment renamed to kernel-container.fragment. The contents
  were never RPi-specific (cgroup, namespaces, AppArmor, netfilter) — just
  misnamed. Extended with extra subsystem disables (KVM, WLAN, CFG80211,
  INFINIBAND, PCMCIA, HAMRADIO, ISDN, ATM, INPUT_JOYSTICK, INPUT_TABLET, FPGA)
  and CONFIG_LSM=lockdown,yama,apparmor.
- build-kernel.sh (x86) refactored to apply the shared fragment via a generic
  apply_fragment function (two-pass for the TC stock config security dance),
  killing ~50 lines of inline config duplication.

Note: rename detection shows build-kernel-arm64.sh as 'modified' because the
new file at that path is the mainline build, while the old RPi-flavoured
content lives in build-kernel-rpi.sh (which appears as a new file). The git
log for build-kernel-rpi.sh is empty; the RPi history is preserved at the
original path until this commit.

No actual kernel build runs in this commit — that's Phase 3 work.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-14 10:30:11 -06:00
19b99cf101 docs: define generic ARM64 vs RPi build-track architecture
Phase 1 audit finding: existing ARM64 build code is mostly already generic.
Only build-kernel-arm64.sh and rpi-kernel-config.fragment are misnamed (the
former is RPi-only, the latter is actually arch-agnostic). The QEMU virt
harness, modules-arm64.list, extract-core arm64 branch, and inject-kubesolo
arm64 branch are all generic.

This document records the target two-track layout for v0.3.0:
- Generic ARM64: mainline kernel, UEFI, GRUB, virtio, GPT 4-part image
- Raspberry Pi: raspberrypi/linux fork, autoboot.txt, MBR 4-part image
- Shared: init, cloud-init, update agent, modules list, kernel-container fragment

Phases 2 and 3 will execute the migration (rename build-kernel-arm64.sh ->
build-kernel-rpi.sh, write a new mainline build-kernel-arm64.sh, etc.).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-14 10:02:29 -06:00
059ec7955f chore: housekeeping for v0.3 prep
- Pin KUBESOLO_VERSION in versions.env (was soft-defaulted in fetch-components.sh)
- Gitignore screenshots, macOS resource forks, and common image extensions
- Update README roadmap: x86_64 stable, ARM64 generic in progress (v0.3),
  ARM64 RPi paused pending hardware
- Add docs/ci-runners.md documenting the Odroid arm64-linux Gitea runner

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-14 09:44:01 -06:00
a6c5d56ade rpi: drop to interactive shell on boot failure, add initcall_debug
Some checks failed
CI / Go Tests (push) Has been cancelled
CI / Shellcheck (push) Has been cancelled
CI / Build Go Binaries (amd64, linux, linux-amd64) (push) Has been cancelled
CI / Build Go Binaries (arm64, linux, linux-arm64) (push) Has been cancelled
Instead of returning 1 (which triggers kernel panic via set -e before
emergency_shell runs), exec an interactive shell on /dev/console so
the user can run dmesg and debug interactively. Add initcall_debug
and loglevel=7 to cmdline.txt to show every driver probe during boot.
Also dump last 60 lines of dmesg before dropping to shell.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-12 20:50:20 -06:00
6c6940afac rpi: add boot diagnostics and remove quiet for debugging
Some checks failed
CI / Go Tests (push) Has been cancelled
CI / Build Go Binaries (amd64, linux, linux-amd64) (push) Has been cancelled
CI / Build Go Binaries (arm64, linux, linux-arm64) (push) Has been cancelled
CI / Shellcheck (push) Has been cancelled
Remove 'quiet' from RPi cmdline.txt so kernel probe messages are
visible on HDMI. Add comprehensive diagnostics to the data device
error path: dmesg for MMC/SDHCI/regulators/firmware, /sys/class/block
listing, and error message scanning. This will reveal why zero block
devices appear despite all kernel configs being correct.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-12 20:12:26 -06:00
4e3f1d6cf0 fix: use kernel-built DTBs for RPi SD card driver probe
Some checks failed
CI / Go Tests (push) Has been cancelled
CI / Build Go Binaries (amd64, linux, linux-amd64) (push) Has been cancelled
CI / Build Go Binaries (arm64, linux, linux-arm64) (push) Has been cancelled
CI / Shellcheck (push) Has been cancelled
Release / Test (push) Has been cancelled
Release / Build Binaries (amd64, linux, linux-amd64) (push) Has been cancelled
Release / Build Binaries (arm64, linux, linux-arm64) (push) Has been cancelled
Release / Build ISO (amd64) (push) Has been cancelled
Release / Create Release (push) Has been cancelled
The sdhci-iproc driver (RPi 4 SD card controller) probes via Device
Tree matching. Using DTBs from the firmware repo instead of the
kernel build caused a mismatch — the driver silently failed to probe,
resulting in zero block devices after boot.

Changes:
- Use DTBs from custom-kernel-arm64/dtbs/ (matches the kernel)
- Firmware blobs (start4.elf, fixup4.dat) still from firmware repo
- Also includes prior fix for LABEL= resolution in persistent mount

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-12 19:27:54 -06:00
6ff77c4482 fix: resolve LABEL= syntax for RPi data partition
Some checks failed
CI / Go Tests (push) Has been cancelled
CI / Build Go Binaries (amd64, linux, linux-amd64) (push) Has been cancelled
CI / Build Go Binaries (arm64, linux, linux-arm64) (push) Has been cancelled
Release / Test (push) Has been cancelled
CI / Shellcheck (push) Has been cancelled
Release / Build Binaries (amd64, linux, linux-amd64) (push) Has been cancelled
Release / Build Binaries (arm64, linux, linux-arm64) (push) Has been cancelled
Release / Build ISO (amd64) (push) Has been cancelled
Release / Create Release (push) Has been cancelled
The cmdline uses kubesolo.data=LABEL=KSOLODATA, but the wait loop
in 20-persistent-mount.sh checked [ -b "LABEL=KSOLODATA" ] which
is always false — it's a label reference, not a block device path.

Fix by detecting LABEL= prefix and resolving it to a block device
path via blkid -L in the wait loop. Also loads mmc_block module as
fallback for platforms where it's not built-in.

Adds debug output listing available block devices on failure.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-12 19:05:10 -06:00
a2764218fc fix: make RPi partition 1 self-sufficient boot fallback
Some checks failed
CI / Go Tests (push) Has been cancelled
CI / Build Go Binaries (amd64, linux, linux-amd64) (push) Has been cancelled
CI / Build Go Binaries (arm64, linux, linux-arm64) (push) Has been cancelled
CI / Shellcheck (push) Has been cancelled
Release / Test (push) Has been cancelled
Release / Build Binaries (amd64, linux, linux-amd64) (push) Has been cancelled
Release / Build Binaries (arm64, linux, linux-arm64) (push) Has been cancelled
Release / Build ISO (amd64) (push) Has been cancelled
Release / Create Release (push) Has been cancelled
The autoboot.txt A/B redirect requires newer RPi EEPROM firmware.
On older EEPROMs, autoboot.txt is silently ignored and the firmware
tries to boot from partition 1 directly — failing with a rainbow
screen because partition 1 had no kernel or initramfs.

Changes:
- Increase partition 1 from 32 MB to 384 MB
- Populate partition 1 with full boot files (kernel, initramfs,
  config.txt with kernel= directive, DTBs, overlays)
- Keep autoboot.txt for A/B redirect on supported EEPROMs
- When autoboot.txt works: boots from partition 2 (A/B scheme)
- When autoboot.txt is unsupported: boots from partition 1 (fallback)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-12 18:52:21 -06:00
2ba816bf6e fix: add config.txt and DTBs to RPi boot control partition
Some checks failed
CI / Go Tests (push) Has been cancelled
CI / Build Go Binaries (amd64, linux, linux-amd64) (push) Has been cancelled
CI / Build Go Binaries (arm64, linux, linux-arm64) (push) Has been cancelled
CI / Shellcheck (push) Has been cancelled
Release / Test (push) Has been cancelled
Release / Build Binaries (amd64, linux, linux-amd64) (push) Has been cancelled
Release / Build Binaries (arm64, linux, linux-arm64) (push) Has been cancelled
Release / Build ISO (amd64) (push) Has been cancelled
Release / Create Release (push) Has been cancelled
The Raspberry Pi firmware reads config.txt from partition 1 BEFORE
processing autoboot.txt. Without arm_64bit=1 on the boot control
partition, the firmware defaults to 32-bit mode and shows only a
rainbow square. Add minimal config.txt, device tree blobs, and
overlays to partition 1 so the firmware can initialize correctly
before redirecting to the A/B boot partitions.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-12 18:29:28 -06:00
65dcddb47e fix: RPi image uses MBR and firmware on boot partition
Some checks failed
CI / Go Tests (push) Has been cancelled
CI / Build Go Binaries (amd64, linux, linux-amd64) (push) Has been cancelled
CI / Build Go Binaries (arm64, linux, linux-arm64) (push) Has been cancelled
CI / Shellcheck (push) Has been cancelled
Release / Test (push) Has been cancelled
Release / Build Binaries (amd64, linux, linux-amd64) (push) Has been cancelled
Release / Build Binaries (arm64, linux, linux-arm64) (push) Has been cancelled
Release / Build ISO (amd64) (push) Has been cancelled
Release / Create Release (push) Has been cancelled
- Switch from GPT to MBR (dos) partition table — GPT + autoboot.txt
  fails on many Pi 4 EEPROM versions
- Copy firmware blobs (start*.elf, fixup*.dat) to partition 1 (KSOLOCTL)
  so the EEPROM can find and load them
- Increase boot control partition from 16 MB to 32 MB to fit firmware
- Mark partition 1 as bootable

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-12 18:16:34 -06:00
ba4812f637 fix: complete ARM64 RPi build pipeline
Some checks failed
CI / Go Tests (push) Has been cancelled
CI / Build Go Binaries (amd64, linux, linux-amd64) (push) Has been cancelled
CI / Build Go Binaries (arm64, linux, linux-arm64) (push) Has been cancelled
CI / Shellcheck (push) Has been cancelled
Release / Test (push) Has been cancelled
Release / Build Binaries (amd64, linux, linux-amd64) (push) Has been cancelled
Release / Build Binaries (arm64, linux, linux-arm64) (push) Has been cancelled
Release / Build ISO (amd64) (push) Has been cancelled
Release / Create Release (push) Has been cancelled
- fetch-components.sh: download ARM64 KubeSolo binary (kubesolo-arm64)
- inject-kubesolo.sh: use arch-specific binaries for KubeSolo, cloud-init,
  and update agent; detect KVER from custom kernel when rootfs has none;
  cross-arch module resolution via find fallback when modprobe fails
- create-rpi-image.sh: kpartx support for Docker container builds
- Makefile: rootfs-arm64 depends on build-cross, includes pack-initramfs

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-12 17:20:04 -06:00
09dcea84ef fix: disk image build, piCore64 URL, license
Some checks failed
CI / Go Tests (push) Has been cancelled
CI / Build Go Binaries (amd64, linux, linux-amd64) (push) Has been cancelled
CI / Build Go Binaries (arm64, linux, linux-arm64) (push) Has been cancelled
CI / Shellcheck (push) Has been cancelled
Release / Test (push) Has been cancelled
Release / Build Binaries (amd64, linux, linux-amd64) (push) Has been cancelled
Release / Build Binaries (arm64, linux, linux-arm64) (push) Has been cancelled
Release / Build ISO (amd64) (push) Has been cancelled
Release / Create Release (push) Has been cancelled
- Add kpartx for reliable loop partition mapping in Docker containers
- Fix piCore64 download URL (changed from .img.gz to .zip format)
- Fix piCore64 boot partition mount (initramfs on p1, not p2)
- Fix tar --wildcards for RPi firmware extraction
- Add MIT license (same as KubeSolo)
- Add kpartx and unzip to Docker builder image

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-12 17:05:03 -06:00
a4e719ba0e chore: bump version to 0.2.0
Some checks failed
CI / Go Tests (push) Has been cancelled
CI / Build Go Binaries (amd64, linux, linux-amd64) (push) Has been cancelled
CI / Build Go Binaries (arm64, linux, linux-arm64) (push) Has been cancelled
CI / Shellcheck (push) Has been cancelled
Release / Test (push) Has been cancelled
Release / Build Binaries (amd64, linux, linux-amd64) (push) Has been cancelled
Release / Build Binaries (arm64, linux, linux-arm64) (push) Has been cancelled
Release / Build ISO (amd64) (push) Has been cancelled
Release / Create Release (push) Has been cancelled
Includes cloud-init full flag support, security hardening, AppArmor,
and ARM64 Raspberry Pi support.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-12 16:36:05 -06:00
61bd28c692 feat: cloud-init supports all documented KubeSolo CLI flags
Some checks failed
CI / Go Tests (push) Has been cancelled
CI / Build Go Binaries (amd64, linux, linux-amd64) (push) Has been cancelled
CI / Build Go Binaries (arm64, linux, linux-arm64) (push) Has been cancelled
CI / Shellcheck (push) Has been cancelled
Add missing flags (--local-storage-shared-path, --debug, --pprof-server,
--portainer-edge-id, --portainer-edge-key, --portainer-edge-async) so all
10 documented KubeSolo parameters can be configured via cloud-init YAML.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-12 15:49:31 -06:00
4fc078f7a3 fix: kubeconfig server accessible via port forwarding, integration tests use proper auth
Some checks failed
CI / Go Tests (push) Has been cancelled
CI / Build Go Binaries (amd64, linux, linux-amd64) (push) Has been cancelled
CI / Build Go Binaries (arm64, linux, linux-arm64) (push) Has been cancelled
CI / Shellcheck (push) Has been cancelled
Bind kubeconfig HTTP server to 0.0.0.0:8080 (was 127.0.0.1) so integration
tests can reach it via QEMU SLIRP port forwarding. Add shared wait_for_boot
and fetch_kubeconfig helpers to qemu-helpers.sh. Update all 5 integration
tests to fetch kubeconfig via HTTP and use it for kubectl authentication.

All 6 tests pass on Linux with KVM: boot (18s), security (7/7), K8s ready
(15s), workload deploy, local storage, network policy.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-12 15:25:32 -06:00
6c15ba7776 fix: kernel AppArmor 2-pass olddefconfig and QEMU test direct kernel boot
Some checks failed
CI / Go Tests (push) Has been cancelled
CI / Build Go Binaries (amd64, linux, linux-amd64) (push) Has been cancelled
CI / Build Go Binaries (arm64, linux, linux-arm64) (push) Has been cancelled
CI / Shellcheck (push) Has been cancelled
The stock TinyCore kernel config has "# CONFIG_SECURITY is not set" which
caused make olddefconfig to silently revert all security configs in a single
pass. Fix by applying security configs (AppArmor, Audit, LSM) after the
first olddefconfig resolves base dependencies, then running a second pass.
Added mandatory verification that exits on missing critical configs.

All QEMU test scripts converted from broken -cdrom + -append pattern to
direct kernel boot (-kernel + -initrd) via shared test/lib/qemu-helpers.sh
helper library. The -append flag only works with -kernel, not -cdrom.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-12 14:11:38 -06:00
958524e6d8 fix: Go version, test scripts, and shellcheck warnings from validation
Some checks failed
CI / Go Tests (push) Has been cancelled
CI / Build Go Binaries (amd64, linux, linux-amd64) (push) Has been cancelled
CI / Build Go Binaries (arm64, linux, linux-arm64) (push) Has been cancelled
CI / Shellcheck (push) Has been cancelled
- Dockerfile.builder: Go 1.24.0 → 1.25.5 (go.mod requires it)
- test-boot.sh: use direct kernel boot via ISO extraction instead of
  broken -cdrom + -append; fix boot marker to "KubeSolo is running"
  (Stage 90 blocks on wait, never emits "complete")
- test-security-hardening.sh: same direct kernel boot and marker fixes
- run-vm.sh, dev-vm.sh, dev-vm-arm64.sh: quote QEMU -net args to
  silence shellcheck SC2054
- fetch-components.sh, fetch-rpi-firmware.sh, dev-vm-arm64.sh: fix
  trap quoting (SC2064)

Validated: full Docker build, 94 Go tests pass, QEMU boot (73s),
security hardening test (6/6 pass, 1 AppArmor skip pending kernel
rebuild).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-12 13:30:55 -06:00
efc7f80b65 feat: add security hardening, AppArmor, and ARM64 Raspberry Pi support (Phase 6)
Some checks failed
CI / Go Tests (push) Has been cancelled
CI / Build Go Binaries (amd64, linux, linux-amd64) (push) Has been cancelled
CI / Build Go Binaries (arm64, linux, linux-arm64) (push) Has been cancelled
CI / Shellcheck (push) Has been cancelled
Security hardening: bind kubeconfig server to localhost, mount hardening
(noexec/nosuid/nodev on tmpfs), sysctl network hardening, kernel module
loading lock after boot, SHA256 checksum verification for downloads,
kernel AppArmor + Audit support, complain-mode AppArmor profiles for
containerd and kubelet, and security integration test.

ARM64 Raspberry Pi support: piCore64 base extraction, RPi kernel build
from raspberrypi/linux fork, RPi firmware fetch, SD card image with 4-
partition GPT and tryboot A/B mechanism, BootEnv Go interface abstracting
GRUB vs RPi boot environments, architecture-aware build scripts, QEMU
aarch64 dev VM and boot test.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-12 13:08:17 -06:00
7abf0e0c04 build: add TINYCORE-MODIFICATIONS.md to .gitignore
Some checks failed
CI / Go Tests (push) Has been cancelled
CI / Build Go Binaries (amd64, linux, linux-amd64) (push) Has been cancelled
CI / Build Go Binaries (arm64, linux, linux-arm64) (push) Has been cancelled
CI / Shellcheck (push) Has been cancelled
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-12 11:38:01 -06:00
60d0edaf84 docs: update README with kubeconfig retrieval and Portainer Edge usage
Some checks failed
CI / Go Tests (push) Has been cancelled
CI / Build Go Binaries (amd64, linux, linux-amd64) (push) Has been cancelled
CI / Build Go Binaries (arm64, linux, linux-arm64) (push) Has been cancelled
CI / Shellcheck (push) Has been cancelled
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-12 10:50:44 -06:00
63 changed files with 4442 additions and 336 deletions

14
.gitignore vendored
View File

@@ -18,8 +18,22 @@ build/rootfs-work/
# OS # OS
.DS_Store .DS_Store
._*
Thumbs.db Thumbs.db
# Photos / screenshots — keep documentation images under docs/ instead
*.PNG
*.png
*.JPG
*.jpg
*.JPEG
*.jpeg
*.HEIC
*.heic
# Go # Go
update/update-agent update/update-agent
cloud-init/cloud-init-parser cloud-init/cloud-init-parser
# Local docs (not tracked)
TINYCORE-MODIFICATIONS.md

View File

@@ -5,6 +5,17 @@ All notable changes to KubeSolo OS are documented in this file.
Format based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), Format based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
versioning follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html). versioning follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [0.2.0] - 2026-02-12
### Added
- Cloud-init: support all documented KubeSolo CLI flags (`--local-storage-shared-path`, `--debug`, `--pprof-server`, `--portainer-edge-id`, `--portainer-edge-key`, `--portainer-edge-async`)
- Cloud-init: `full-config.yaml` example showing all supported parameters
- Cloud-init: KubeSolo configuration reference table in docs/cloud-init.md
- Security hardening: mount hardening, sysctl, kernel module lock, AppArmor profiles
- ARM64 Raspberry Pi support with A/B boot via tryboot
- BootEnv abstraction for GRUB and RPi boot environments
- Go 1.25.5 installed on host for native builds
## [0.1.0] - 2026-02-12 ## [0.1.0] - 2026-02-12
First release with all 5 design-doc phases complete. ISO boots and runs K8s pods. First release with all 5 design-doc phases complete. ISO boots and runs K8s pods.
@@ -86,3 +97,4 @@ First release with all 5 design-doc phases complete. ISO boots and runs K8s pods
- Rewrote dev-vm.sh for macOS: bsdtar ISO extraction, Homebrew mkfs.ext4 detection, direct kernel boot, TCG acceleration, port 8080 forwarding - Rewrote dev-vm.sh for macOS: bsdtar ISO extraction, Homebrew mkfs.ext4 detection, direct kernel boot, TCG acceleration, port 8080 forwarding
- Kubeconfig now served via HTTP on port 8080 (serial console truncates base64 lines) - Kubeconfig now served via HTTP on port 8080 (serial console truncates base64 lines)
- Added 127.0.0.1 and 10.0.2.15 to API server SANs for QEMU port forwarding - Added 127.0.0.1 and 10.0.2.15 to API server SANs for QEMU port forwarding
- dev-vm.sh now works on Linux: fallback ISO extraction via isoinfo or loop mount, KVM auto-detection, platform-aware error messages

21
LICENSE Normal file
View File

@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2025 Anthony De Lorenzo
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

View File

@@ -1,9 +1,10 @@
.PHONY: all fetch kernel build-cloudinit build-update-agent build-cross rootfs initramfs \ .PHONY: all fetch kernel build-cloudinit build-update-agent build-cross rootfs initramfs \
iso disk-image oci-image \ iso disk-image disk-image-arm64 oci-image rpi-image \
test-boot test-k8s test-persistence test-deploy test-storage test-all \ kernel-arm64 kernel-rpi rootfs-arm64 rootfs-arm64-rpi \
test-cloudinit test-update-agent \ test-boot test-k8s test-persistence test-deploy test-storage test-security test-all \
test-boot-arm64 test-boot-arm64-disk test-cloudinit test-update-agent \
bench-boot bench-resources \ bench-boot bench-resources \
dev-vm dev-vm-shell quick docker-build shellcheck \ dev-vm dev-vm-shell dev-vm-arm64 quick docker-build shellcheck \
kernel-audit clean distclean help kernel-audit clean distclean help
SHELL := /bin/bash SHELL := /bin/bash
@@ -71,6 +72,48 @@ build-cross:
@echo "==> Cross-compiling for amd64 + arm64..." @echo "==> Cross-compiling for amd64 + arm64..."
$(BUILD_DIR)/scripts/build-cross.sh $(BUILD_DIR)/scripts/build-cross.sh
# =============================================================================
# ARM64 generic targets (mainline kernel, UEFI, virtio — for cloud / SBCs)
# =============================================================================
kernel-arm64:
@echo "==> Building generic ARM64 kernel (mainline LTS)..."
$(BUILD_DIR)/scripts/build-kernel-arm64.sh
# Generic ARM64 rootfs consumes the mainline kernel modules.
rootfs-arm64: build-cross
@echo "==> Preparing generic ARM64 rootfs..."
TARGET_ARCH=arm64 $(BUILD_DIR)/scripts/fetch-components.sh
TARGET_ARCH=arm64 $(BUILD_DIR)/scripts/extract-core.sh
TARGET_ARCH=arm64 TARGET_VARIANT=generic $(BUILD_DIR)/scripts/inject-kubesolo.sh
@echo "==> Packing generic ARM64 initramfs..."
$(BUILD_DIR)/scripts/pack-initramfs.sh
disk-image-arm64: rootfs-arm64 kernel-arm64
@echo "==> Creating generic ARM64 disk image (UEFI + GRUB A/B)..."
TARGET_ARCH=arm64 $(BUILD_DIR)/scripts/create-disk-image.sh
@echo "==> Built: $(OUTPUT_DIR)/$(OS_NAME)-$(VERSION).arm64.img"
# =============================================================================
# ARM64 Raspberry Pi targets (RPi-patched kernel, firmware blobs, SD card)
# =============================================================================
kernel-rpi:
@echo "==> Building RPi kernel (raspberrypi/linux)..."
$(BUILD_DIR)/scripts/build-kernel-rpi.sh
# RPi-flavoured rootfs consumes the RPi kernel modules.
rootfs-arm64-rpi: build-cross
@echo "==> Preparing RPi ARM64 rootfs..."
TARGET_ARCH=arm64 $(BUILD_DIR)/scripts/fetch-components.sh
TARGET_ARCH=arm64 $(BUILD_DIR)/scripts/extract-core.sh
TARGET_ARCH=arm64 TARGET_VARIANT=rpi $(BUILD_DIR)/scripts/inject-kubesolo.sh
@echo "==> Packing RPi ARM64 initramfs..."
$(BUILD_DIR)/scripts/pack-initramfs.sh
rpi-image: rootfs-arm64-rpi kernel-rpi
@echo "==> Creating Raspberry Pi SD card image..."
$(BUILD_DIR)/scripts/create-rpi-image.sh
@echo "==> Built: $(OUTPUT_DIR)/$(OS_NAME)-$(VERSION).rpi.img"
# ============================================================================= # =============================================================================
# Kernel validation # Kernel validation
# ============================================================================= # =============================================================================
@@ -101,6 +144,18 @@ test-storage: iso
@echo "==> Testing local storage provisioning..." @echo "==> Testing local storage provisioning..."
test/integration/test-local-storage.sh $(OUTPUT_DIR)/$(OS_NAME)-$(VERSION).iso test/integration/test-local-storage.sh $(OUTPUT_DIR)/$(OS_NAME)-$(VERSION).iso
test-security: iso
@echo "==> Testing security hardening..."
test/integration/test-security-hardening.sh $(OUTPUT_DIR)/$(OS_NAME)-$(VERSION).iso
test-boot-arm64:
@echo "==> Testing ARM64 boot in QEMU (direct kernel)..."
test/qemu/test-boot-arm64.sh
test-boot-arm64-disk: disk-image-arm64
@echo "==> Testing ARM64 UEFI disk boot in QEMU..."
test/qemu/test-boot-arm64-disk.sh $(OUTPUT_DIR)/$(OS_NAME)-$(VERSION).arm64.img
test-all: test-boot test-k8s test-persistence test-all: test-boot test-k8s test-persistence
# Cloud-init Go tests # Cloud-init Go tests
@@ -163,6 +218,10 @@ dev-vm-debug: iso
@echo "==> Launching dev VM (debug mode)..." @echo "==> Launching dev VM (debug mode)..."
hack/dev-vm.sh $(OUTPUT_DIR)/$(OS_NAME)-$(VERSION).iso --debug hack/dev-vm.sh $(OUTPUT_DIR)/$(OS_NAME)-$(VERSION).iso --debug
dev-vm-arm64:
@echo "==> Launching ARM64 dev VM..."
hack/dev-vm-arm64.sh
# Fast rebuild: only repack initramfs + ISO (skip fetch/extract) # Fast rebuild: only repack initramfs + ISO (skip fetch/extract)
quick: quick:
@echo "==> Quick rebuild (repack + ISO only)..." @echo "==> Quick rebuild (repack + ISO only)..."
@@ -199,7 +258,7 @@ distclean: clean
help: help:
@echo "KubeSolo OS Build System (v$(VERSION))" @echo "KubeSolo OS Build System (v$(VERSION))"
@echo "" @echo ""
@echo "Build targets:" @echo "Build targets (x86_64):"
@echo " make fetch Download Tiny Core ISO, KubeSolo, dependencies" @echo " make fetch Download Tiny Core ISO, KubeSolo, dependencies"
@echo " make kernel Build custom kernel with CONFIG_CGROUP_BPF=y" @echo " make kernel Build custom kernel with CONFIG_CGROUP_BPF=y"
@echo " make build-cloudinit Build cloud-init Go binary" @echo " make build-cloudinit Build cloud-init Go binary"
@@ -213,25 +272,39 @@ help:
@echo " make quick Fast rebuild (re-inject + repack + ISO only)" @echo " make quick Fast rebuild (re-inject + repack + ISO only)"
@echo " make docker-build Reproducible build inside Docker" @echo " make docker-build Reproducible build inside Docker"
@echo "" @echo ""
@echo "Build targets (ARM64 generic — UEFI / cloud / SBCs):"
@echo " make kernel-arm64 Build mainline ARM64 kernel from kernel.org LTS"
@echo " make rootfs-arm64 Prepare generic ARM64 rootfs (mainline kernel modules)"
@echo " make disk-image-arm64 Create UEFI-bootable A/B GPT disk image (.arm64.img)"
@echo ""
@echo "Build targets (ARM64 Raspberry Pi):"
@echo " make kernel-rpi Build RPi kernel from raspberrypi/linux"
@echo " make rootfs-arm64-rpi Prepare RPi-flavoured rootfs (RPi kernel modules)"
@echo " make rpi-image Create Raspberry Pi SD card image with A/B autoboot"
@echo ""
@echo "Test targets:" @echo "Test targets:"
@echo " make test-boot Boot ISO in QEMU, verify boot success" @echo " make test-boot Boot ISO in QEMU, verify boot success"
@echo " make test-k8s Boot + verify K8s node reaches Ready" @echo " make test-k8s Boot + verify K8s node reaches Ready"
@echo " make test-persist Reboot disk image, verify state persists" @echo " make test-persist Reboot disk image, verify state persists"
@echo " make test-deploy Deploy nginx pod, verify Running" @echo " make test-deploy Deploy nginx pod, verify Running"
@echo " make test-storage Test PVC with local-path provisioner" @echo " make test-storage Test PVC with local-path provisioner"
@echo " make test-security Verify security hardening (AppArmor, sysctl, mounts)"
@echo " make test-cloudinit Run cloud-init Go unit tests" @echo " make test-cloudinit Run cloud-init Go unit tests"
@echo " make test-update-agent Run update agent Go unit tests" @echo " make test-update-agent Run update agent Go unit tests"
@echo " make test-update A/B update cycle integration test" @echo " make test-update A/B update cycle integration test"
@echo " make test-rollback Forced rollback integration test" @echo " make test-rollback Forced rollback integration test"
@echo " make test-boot-arm64 ARM64 boot test (direct kernel, fast)"
@echo " make test-boot-arm64-disk ARM64 full UEFI disk-boot test"
@echo " make test-all Run core tests (boot + k8s + persistence)" @echo " make test-all Run core tests (boot + k8s + persistence)"
@echo " make test-integ Run full integration suite" @echo " make test-integ Run full integration suite"
@echo " make bench-boot Benchmark boot performance (3 runs)" @echo " make bench-boot Benchmark boot performance (3 runs)"
@echo " make bench-resources Benchmark resource usage (requires running VM)" @echo " make bench-resources Benchmark resource usage (requires running VM)"
@echo "" @echo ""
@echo "Dev targets:" @echo "Dev targets:"
@echo " make dev-vm Launch interactive QEMU VM" @echo " make dev-vm Launch interactive QEMU VM (x86_64)"
@echo " make dev-vm-shell Launch QEMU VM -> emergency shell" @echo " make dev-vm-shell Launch QEMU VM -> emergency shell"
@echo " make dev-vm-debug Launch QEMU VM with debug logging" @echo " make dev-vm-debug Launch QEMU VM with debug logging"
@echo " make dev-vm-arm64 Launch ARM64 QEMU VM"
@echo " make kernel-audit Check kernel config against requirements" @echo " make kernel-audit Check kernel config against requirements"
@echo " make shellcheck Lint all shell scripts" @echo " make shellcheck Lint all shell scripts"
@echo "" @echo ""

View File

@@ -2,7 +2,7 @@
An immutable, bootable Linux distribution purpose-built for [KubeSolo](https://github.com/portainer/kubesolo) — Portainer's ultra-lightweight single-node Kubernetes. An immutable, bootable Linux distribution purpose-built for [KubeSolo](https://github.com/portainer/kubesolo) — Portainer's ultra-lightweight single-node Kubernetes.
> **Status:** All 5 phases complete. Boots and runs K8s workloads. > **Status:** x86_64 is stable — boots and runs K8s workloads, Portainer Edge Agent tested and connected. ARM64 generic UEFI is the active focus for v0.3.0; ARM64 Raspberry Pi support is paused pending physical hardware testing.
## What is this? ## What is this?
@@ -47,6 +47,24 @@ Or build everything at once inside Docker:
make docker-build make docker-build
``` ```
After boot, retrieve the kubeconfig and manage your cluster from the host:
```bash
curl -s http://localhost:8080 > ~/.kube/kubesolo-config
export KUBECONFIG=~/.kube/kubesolo-config
kubectl get nodes
```
### Portainer Edge Agent
Pass Edge credentials via boot parameters:
```bash
./hack/dev-vm.sh --edge-id=YOUR_EDGE_ID --edge-key=YOUR_EDGE_KEY
```
Or configure via [cloud-init YAML](cloud-init/examples/portainer-edge.yaml).
## Requirements ## Requirements
**Build host:** **Build host:**
@@ -104,7 +122,7 @@ Unnecessary subsystems (sound, GPU, wireless, Bluetooth, etc.) are stripped to k
## Cloud-Init ## Cloud-Init
First-boot configuration via a simple YAML schema: First-boot configuration via a simple YAML schema. All [documented KubeSolo flags](https://www.kubesolo.io/documentation#install) are supported:
```yaml ```yaml
hostname: edge-node-01 hostname: edge-node-01
@@ -115,10 +133,15 @@ network:
dns: dns:
- 8.8.8.8 - 8.8.8.8
kubesolo: kubesolo:
node-name: edge-node-01 local-storage: true
portainer: local-storage-shared-path: "/mnt/shared"
edge_id: "your-edge-id" apiserver-extra-sans:
edge_key: "your-edge-key" - edge-node-01.local
debug: false
pprof-server: false
portainer-edge-id: "your-edge-id"
portainer-edge-key: "your-edge-key"
portainer-edge-async: true
``` ```
See [docs/cloud-init.md](docs/cloud-init.md) and the [examples](cloud-init/examples/). See [docs/cloud-init.md](docs/cloud-init.md) and the [examples](cloud-init/examples/).
@@ -189,7 +212,7 @@ Metrics include: `kubesolo_os_info`, `boot_success`, `boot_counter`, `uptime_sec
| `make build-cross` | Cross-compile for amd64 + arm64 | | `make build-cross` | Cross-compile for amd64 + arm64 |
| `make docker-build` | Build everything in Docker | | `make docker-build` | Build everything in Docker |
| `make quick` | Fast rebuild (re-inject + repack + ISO) | | `make quick` | Fast rebuild (re-inject + repack + ISO) |
| `make dev-vm` | Launch QEMU dev VM | | `make dev-vm` | Launch QEMU dev VM (Linux + macOS) |
| `make test-all` | Run all tests | | `make test-all` | Run all tests |
## Documentation ## Documentation
@@ -204,13 +227,17 @@ Metrics include: `kubesolo_os_info`, `boot_success`, `boot_counter`, `uptime_sec
| Phase | Scope | Status | | Phase | Scope | Status |
|-------|-------|--------| |-------|-------|--------|
| 1 | PoC: boot Tiny Core + KubeSolo, verify K8s | Complete | | 1 | PoC: boot Tiny Core + KubeSolo, verify K8s | Complete (x86_64) |
| 2 | Cloud-init Go parser, network, hostname | Complete | | 2 | Cloud-init Go parser, network, hostname | Complete |
| 3 | A/B atomic updates, GRUB, rollback agent | Complete | | 3 | A/B atomic updates, GRUB, rollback agent | Complete (x86_64) |
| 4 | Ed25519 signing, Portainer Edge, SSH extension | Complete | | 4 | Ed25519 signing, Portainer Edge, SSH extension | Complete |
| 5 | CI/CD, OCI distribution, Prometheus metrics, ARM64 | Complete | | 5 | CI/CD, OCI distribution, Prometheus metrics, ARM64 cross-compile | Complete |
| - | Custom kernel build for container runtime fixes | Complete | | 6 | Security hardening, AppArmor | Complete |
| - | Custom kernel build for container runtime fixes | Complete (x86_64) |
| 7 | ARM64 generic (mainline kernel, UEFI, virtio) | In progress (v0.3.0) |
| 8 | Update engine v2 (state machine, OCI distribution, channels) | In progress (v0.3.0) |
| - | ARM64 Raspberry Pi (custom kernel, firmware, SD card image) | Paused — needs hardware |
## License ## License
TBD MIT License — see [LICENSE](LICENSE) for details.

View File

@@ -1 +1 @@
0.1.0 0.2.0

View File

@@ -18,6 +18,11 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
file \ file \
flex \ flex \
genisoimage \ genisoimage \
grub-common \
grub-efi-amd64-bin \
grub-efi-arm64-bin \
grub-pc-bin \
grub2-common \
gzip \ gzip \
isolinux \ isolinux \
iptables \ iptables \
@@ -31,13 +36,20 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
syslinux \ syslinux \
syslinux-common \ syslinux-common \
syslinux-utils \ syslinux-utils \
apparmor \
apparmor-utils \
gcc-aarch64-linux-gnu \
binutils-aarch64-linux-gnu \
git \
kpartx \
unzip \
wget \ wget \
xorriso \ xorriso \
xz-utils \ xz-utils \
&& rm -rf /var/lib/apt/lists/* && rm -rf /var/lib/apt/lists/*
# Install Go (for building cloud-init and update agent) # Install Go (for building cloud-init and update agent)
ARG GO_VERSION=1.24.0 ARG GO_VERSION=1.25.5
RUN curl -fsSL "https://go.dev/dl/go${GO_VERSION}.linux-amd64.tar.gz" \ RUN curl -fsSL "https://go.dev/dl/go${GO_VERSION}.linux-amd64.tar.gz" \
| tar -C /usr/local -xzf - | tar -C /usr/local -xzf -
ENV PATH="/usr/local/go/bin:${PATH}" ENV PATH="/usr/local/go/bin:${PATH}"

View File

@@ -128,7 +128,12 @@ echo "Security:"
check_config CONFIG_SECCOMP recommended "Seccomp (container security)" check_config CONFIG_SECCOMP recommended "Seccomp (container security)"
check_config CONFIG_SECCOMP_FILTER recommended "Seccomp BPF filter" check_config CONFIG_SECCOMP_FILTER recommended "Seccomp BPF filter"
check_config CONFIG_BPF_SYSCALL recommended "BPF syscall" check_config CONFIG_BPF_SYSCALL recommended "BPF syscall"
check_config CONFIG_AUDIT recommended "Audit framework" check_config CONFIG_AUDIT mandatory "Audit framework"
check_config CONFIG_AUDITSYSCALL mandatory "Audit system call events"
check_config CONFIG_SECURITY mandatory "Security framework"
check_config CONFIG_SECURITYFS mandatory "Security filesystem"
check_config CONFIG_SECURITY_APPARMOR mandatory "AppArmor LSM"
check_config CONFIG_SECURITY_NETWORK recommended "Network security hooks"
echo "" echo ""
# --- Crypto --- # --- Crypto ---

View File

@@ -0,0 +1,90 @@
# KubeSolo OS — Shared kernel config fragment for container workloads
#
# Applied on top of:
# - Tiny Core stock config (x86_64) via build-kernel.sh
# - mainline kernel.org arm64 defconfig via build-kernel-arm64.sh
# - bcm2711_defconfig / bcm2712_defconfig via build-kernel-rpi.sh
#
# All entries here are architecture-agnostic.
# Apply this fragment twice with `make olddefconfig` between passes — TC's stock
# config has CONFIG_SECURITY disabled, which causes a single-pass olddefconfig
# to strip the security subtree before its dependencies (SYSFS, MULTIUSER) are
# resolved.
# cgroup v2 (mandatory for containerd/runc)
CONFIG_CGROUPS=y
CONFIG_CGROUP_CPUACCT=y
CONFIG_CGROUP_DEVICE=y
CONFIG_CGROUP_FREEZER=y
CONFIG_CGROUP_SCHED=y
CONFIG_CGROUP_PIDS=y
CONFIG_MEMCG=y
CONFIG_CGROUP_BPF=y
CONFIG_CFS_BANDWIDTH=y
# BPF (required for cgroup v2 device control)
CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
# Namespaces (mandatory for containers)
CONFIG_NAMESPACES=y
CONFIG_NET_NS=y
CONFIG_PID_NS=y
CONFIG_USER_NS=y
CONFIG_UTS_NS=y
CONFIG_IPC_NS=y
# Device management
CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y
# Filesystem
CONFIG_OVERLAY_FS=y
CONFIG_SQUASHFS=y
CONFIG_EXT4_FS=y
CONFIG_VFAT_FS=y
# Networking
CONFIG_BRIDGE=m
CONFIG_NETFILTER=y
CONFIG_NF_CONNTRACK=m
CONFIG_NF_NAT=m
CONFIG_NF_TABLES=m
CONFIG_VETH=m
CONFIG_VXLAN=m
# Security: AppArmor + Audit
CONFIG_AUDIT=y
CONFIG_AUDITSYSCALL=y
CONFIG_SECURITY=y
CONFIG_SECURITYFS=y
CONFIG_SECURITY_NETWORK=y
CONFIG_SECURITY_APPARMOR=y
CONFIG_DEFAULT_SECURITY_APPARMOR=y
CONFIG_LSM=lockdown,yama,apparmor
# Security: seccomp
CONFIG_SECCOMP=y
CONFIG_SECCOMP_FILTER=y
# Crypto (image verification)
CONFIG_CRYPTO_SHA256=y
# Disable unnecessary subsystems for headless edge appliance
# CONFIG_SOUND is not set
# CONFIG_DRM is not set
# CONFIG_KVM is not set
# CONFIG_MEDIA_SUPPORT is not set
# CONFIG_WIRELESS is not set
# CONFIG_WLAN is not set
# CONFIG_CFG80211 is not set
# CONFIG_BT is not set
# CONFIG_NFC is not set
# CONFIG_INFINIBAND is not set
# CONFIG_PCMCIA is not set
# CONFIG_HAMRADIO is not set
# CONFIG_ISDN is not set
# CONFIG_ATM is not set
# CONFIG_INPUT_JOYSTICK is not set
# CONFIG_INPUT_TABLET is not set
# CONFIG_FPGA is not set

View File

@@ -0,0 +1,81 @@
# Kernel modules loaded at boot by init (ARM64 / Raspberry Pi)
# One module per line. Lines starting with # are ignored.
# Modules are loaded in order listed — dependencies must come first.
# Network device drivers (loaded early so interfaces are available)
# Note: no e1000/e1000e on ARM64 — those are x86 Intel NIC drivers
virtio_net
# Virtio support (for QEMU VMs — block, entropy)
virtio_blk
virtio_rng
# Raspberry Pi specific (USB Ethernet on Pi 4 is built-in, no module needed)
# Pi 5 uses PCIe ethernet, also typically built-in
# Filesystem — overlay (required for containerd)
overlay
# Netfilter dependencies (must load before conntrack)
nf_defrag_ipv4
nf_defrag_ipv6
# Netfilter / connection tracking (required for kube-proxy)
nf_conntrack
nf_nat
nf_conntrack_netlink
# nftables (modern iptables backend)
nf_tables
nft_compat
nft_chain_nat
nft_ct
nft_masq
nft_nat
nft_redir
# Netfilter xt match/target modules (used by kube-proxy iptables rules via nft_compat)
xt_conntrack
xt_MASQUERADE
xt_mark
xt_comment
xt_multiport
xt_nat
xt_addrtype
xt_connmark
xt_REDIRECT
xt_recent
xt_statistic
xt_set
# nft extras (reject, fib — used by kube-proxy nf_tables rules)
nft_reject
nft_reject_ipv4
nft_reject_ipv6
nft_fib
nft_fib_ipv4
nft_fib_ipv6
# Reject targets (used by kube-proxy iptables-restore rules)
nf_reject_ipv4
nf_reject_ipv6
ipt_REJECT
ip6t_REJECT
# nfacct extension (kube-proxy probes for it)
xt_nfacct
# Networking — bridge and netfilter (required for K8s pod networking)
# Load order: llc → stp → bridge → br_netfilter
llc
stp
bridge
br_netfilter
veth
vxlan
# IPVS — useful for kube-proxy IPVS mode and CNI plugins
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh

View File

@@ -9,11 +9,45 @@ TINYCORE_ISO=CorePure64-${TINYCORE_VERSION}.iso
TINYCORE_ISO_URL=${TINYCORE_MIRROR}/${TINYCORE_VERSION%%.*}.x/${TINYCORE_ARCH}/release/${TINYCORE_ISO} TINYCORE_ISO_URL=${TINYCORE_MIRROR}/${TINYCORE_VERSION%%.*}.x/${TINYCORE_ARCH}/release/${TINYCORE_ISO}
# KubeSolo # KubeSolo
# Pinned release tag from https://github.com/portainer/kubesolo/releases.
# Bump here and re-run `make fetch` to pull a new version.
KUBESOLO_VERSION=v1.1.0
KUBESOLO_INSTALL_URL=https://get.kubesolo.io KUBESOLO_INSTALL_URL=https://get.kubesolo.io
# Build tools (used inside builder container) # Build tools (used inside builder container)
GRUB_VERSION=2.12 GRUB_VERSION=2.12
SYSLINUX_VERSION=6.03 SYSLINUX_VERSION=6.03
# SHA256 checksums for supply chain verification
# Populate by running: sha256sum build/cache/<file>
# Leave empty to skip verification (useful for first fetch)
TINYCORE_ISO_SHA256=""
KUBESOLO_SHA256=""
NETFILTER_TCZ_SHA256=""
NET_BRIDGING_TCZ_SHA256=""
IPTABLES_TCZ_SHA256=""
# piCore64 (ARM64 — Raspberry Pi)
PICORE_VERSION=15.0.0
PICORE_ARCH=aarch64
PICORE_IMAGE=piCore64-${PICORE_VERSION}.zip
PICORE_IMAGE_URL=http://www.tinycorelinux.net/${PICORE_VERSION%%.*}.x/${PICORE_ARCH}/releases/RPi/${PICORE_IMAGE}
# Raspberry Pi firmware (boot blobs, DTBs)
RPI_FIRMWARE_TAG=1.20240529
RPI_FIRMWARE_URL=https://github.com/raspberrypi/firmware/archive/refs/tags/${RPI_FIRMWARE_TAG}.tar.gz
# Raspberry Pi kernel source
RPI_KERNEL_BRANCH=rpi-6.6.y
RPI_KERNEL_REPO=https://github.com/raspberrypi/linux
# Mainline Linux kernel (for generic ARM64 — kernel.org LTS)
# Bump within the 6.12 LTS series as patch levels release.
# 6.12 LTS is supported until Dec 2029.
MAINLINE_KERNEL_VERSION=6.12.10
MAINLINE_KERNEL_MAJOR=v6.x
MAINLINE_KERNEL_URL=https://cdn.kernel.org/pub/linux/kernel/${MAINLINE_KERNEL_MAJOR}/linux-${MAINLINE_KERNEL_VERSION}.tar.xz
MAINLINE_KERNEL_SHA256=""
# Output naming # Output naming
OS_NAME=kubesolo-os OS_NAME=kubesolo-os

86
build/grub/grub-arm64.cfg Normal file
View File

@@ -0,0 +1,86 @@
# KubeSolo OS — GRUB Configuration (ARM64)
# A/B partition boot with automatic rollback.
#
# Same A/B logic as build/grub/grub.cfg; only the console parameters differ
# (ARM64 PL011 / 16550-compat UART rather than x86 ttyS0).
#
# Partition layout:
# (hd0,gpt1) — EFI/Boot (256 MB, FAT32) — contains GRUB + grubenv
# (hd0,gpt2) — System A (512 MB, ext4) — vmlinuz + kubesolo-os.gz
# (hd0,gpt3) — System B (512 MB, ext4) — vmlinuz + kubesolo-os.gz
# (hd0,gpt4) — Data (remaining, ext4) — persistent K8s state
set default=0
set timeout=3
load_env
# --- A/B Rollback Logic (identical to amd64 grub.cfg) ---
if [ "${boot_success}" != "1" ]; then
if [ "${boot_counter}" = "0" ]; then
if [ "${active_slot}" = "A" ]; then
set active_slot=B
else
set active_slot=A
fi
save_env active_slot
set boot_counter=3
save_env boot_counter
else
if [ "${boot_counter}" = "3" ]; then
set boot_counter=2
elif [ "${boot_counter}" = "2" ]; then
set boot_counter=1
elif [ "${boot_counter}" = "1" ]; then
set boot_counter=0
fi
save_env boot_counter
fi
fi
set boot_success=0
save_env boot_success
if [ "${active_slot}" = "A" ]; then
set root='(hd0,gpt2)'
set slot_label="System A"
else
set root='(hd0,gpt3)'
set slot_label="System B"
fi
# --- ARM64 console string ---
# Covers QEMU virt (ttyAMA0), Ampere/RPi-equivalent PL011 (ttyAMA0), and
# Graviton/16550-compat (ttyS0). Last `console=` becomes the system console.
menuentry "KubeSolo OS (${slot_label})" {
echo "Booting KubeSolo OS from ${slot_label}..."
echo "Boot counter: ${boot_counter}, Boot success: ${boot_success}"
linux /vmlinuz kubesolo.data=LABEL=KSOLODATA console=ttyAMA0,115200 console=ttyS0,115200 quiet
initrd /kubesolo-os.gz
}
menuentry "KubeSolo OS (${slot_label}) — Debug Mode" {
echo "Booting KubeSolo OS (debug) from ${slot_label}..."
linux /vmlinuz kubesolo.data=LABEL=KSOLODATA kubesolo.debug console=ttyAMA0,115200 console=ttyS0,115200
initrd /kubesolo-os.gz
}
menuentry "KubeSolo OS — Emergency Shell" {
echo "Booting to emergency shell..."
linux /vmlinuz kubesolo.shell console=ttyAMA0,115200 console=ttyS0,115200
initrd /kubesolo-os.gz
}
menuentry "KubeSolo OS — Boot Other Slot" {
if [ "${active_slot}" = "A" ]; then
set root='(hd0,gpt3)'
echo "Booting from System B (passive)..."
else
set root='(hd0,gpt2)'
echo "Booting from System A (passive)..."
fi
linux /vmlinuz kubesolo.data=LABEL=KSOLODATA kubesolo.debug console=ttyAMA0,115200 console=ttyS0,115200
initrd /kubesolo-os.gz
}

View File

@@ -0,0 +1,52 @@
# AppArmor profile for containerd
# Start in complain mode to log without blocking
#include <tunables/global>
profile containerd /usr/bin/containerd flags=(complain) {
#include <abstractions/base>
# Binary and shared libraries
/usr/bin/containerd mr,
/usr/lib/** mr,
/lib/** mr,
# Containerd runtime state
/var/lib/containerd/** rw,
/run/containerd/** rw,
# Container image layers and snapshots
/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/** rw,
# CNI networking
/etc/cni/** r,
/opt/cni/bin/** ix,
# Proc and sys access for containers
@{PROC}/** r,
/sys/** r,
# Device access for containers
/dev/** rw,
# Network access
network,
# Container runtime needs broad capabilities
capability,
# Allow executing container runtimes
/usr/bin/containerd-shim-runc-v2 ix,
/usr/bin/runc ix,
/usr/sbin/runc ix,
# Temp files
/tmp/** rw,
# Log files
/var/log/** rw,
# Signal handling for child processes
signal,
ptrace,
}

View File

@@ -0,0 +1,56 @@
# AppArmor profile for kubesolo (kubelet + control plane)
# Start in complain mode to log without blocking
#include <tunables/global>
profile kubesolo /usr/bin/kubesolo flags=(complain) {
#include <abstractions/base>
# Binary and shared libraries
/usr/bin/kubesolo mr,
/usr/lib/** mr,
/lib/** mr,
# KubeSolo state (etcd/SQLite, certificates, manifests)
/var/lib/kubesolo/** rw,
# KubeSolo configuration
/etc/kubesolo/** r,
# Containerd socket
/run/containerd/** rw,
# CNI networking
/etc/cni/** r,
/opt/cni/bin/** ix,
# Proc and sys access
@{PROC}/** r,
/sys/** r,
# Device access
/dev/** rw,
# Network access (API server, kubelet, etcd)
network,
# Control plane needs broad capabilities
capability,
# Kubectl and other tools
/usr/bin/kubectl ix,
/usr/local/bin/** ix,
# Temp files
/tmp/** rw,
# Log files
/var/log/** rw,
# Kubelet needs to manage pods
/var/lib/kubelet/** rw,
# Signal handling
signal,
ptrace,
}

View File

@@ -0,0 +1,27 @@
# Security hardening — applied automatically by 40-sysctl.sh
# Network: anti-spoofing
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
# Network: SYN flood protection
net.ipv4.tcp_syncookies = 1
# Network: ICMP hardening
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.icmp_ignore_bogus_error_responses = 1
net.ipv4.conf.all.log_martians = 1
# Network: IPv6 hardening
net.ipv6.conf.all.accept_redirects = 0
net.ipv6.conf.default.accept_redirects = 0
net.ipv6.conf.all.accept_ra = 0
# Network: source routing
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0
# Kernel: information disclosure
kernel.kptr_restrict = 2
kernel.dmesg_restrict = 1
kernel.perf_event_paranoid = 3
# Kernel: core dump safety
fs.suid_dumpable = 0

View File

@@ -0,0 +1,204 @@
#!/bin/bash
# build-kernel-arm64.sh — Build generic ARM64 kernel (mainline LTS)
#
# Builds a Linux kernel from kernel.org mainline LTS source, suitable for:
# - qemu-system-aarch64 -machine virt
# - UEFI ARM64 hosts (Ampere, Graviton, generic ARM64 servers)
# - Future ARM64 SBCs with UEFI/u-boot generic-distro support
#
# This is the GENERIC ARM64 build track. For Raspberry Pi specifically
# (raspberrypi/linux fork, RPi firmware boot path, custom DTBs), see
# build/scripts/build-kernel-rpi.sh.
#
# Output is cached in $CACHE_DIR/kernel-arm64-generic/ and reused across builds.
#
# Requirements:
# - gcc-aarch64-linux-gnu (cross-compiler)
# - Standard kernel build deps (bc, bison, flex, libelf-dev, libssl-dev)
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
CACHE_DIR="${CACHE_DIR:-$PROJECT_ROOT/build/cache}"
# shellcheck source=../config/versions.env
. "$SCRIPT_DIR/../config/versions.env"
KVER="$MAINLINE_KERNEL_VERSION"
CUSTOM_KERNEL_DIR="$CACHE_DIR/kernel-arm64-generic"
CUSTOM_IMAGE="$CUSTOM_KERNEL_DIR/Image"
CUSTOM_MODULES="$CUSTOM_KERNEL_DIR/modules"
mkdir -p "$CACHE_DIR" "$CUSTOM_KERNEL_DIR"
# --- Skip if already built ---
if [ -f "$CUSTOM_IMAGE" ] && [ -d "$CUSTOM_MODULES/lib/modules/$KVER" ]; then
echo "==> Generic ARM64 kernel already built (cached)"
echo " Image: $CUSTOM_IMAGE ($(du -h "$CUSTOM_IMAGE" | cut -f1))"
echo " Kernel: $KVER"
exit 0
fi
# --- Verify cross-compiler ---
if ! command -v aarch64-linux-gnu-gcc >/dev/null 2>&1; then
echo "ERROR: aarch64-linux-gnu-gcc not found"
echo "Install: apt-get install gcc-aarch64-linux-gnu"
exit 1
fi
echo "==> Building generic ARM64 kernel (mainline $KVER)..."
echo " Source: $MAINLINE_KERNEL_URL"
# --- Download mainline kernel source ---
KERNEL_SRC_ARCHIVE="$CACHE_DIR/linux-${KVER}.tar.xz"
if [ ! -f "$KERNEL_SRC_ARCHIVE" ]; then
echo "==> Downloading mainline kernel source (~140 MB)..."
wget -q --show-progress -O "$KERNEL_SRC_ARCHIVE" "$MAINLINE_KERNEL_URL" 2>/dev/null || \
curl -fSL "$MAINLINE_KERNEL_URL" -o "$KERNEL_SRC_ARCHIVE"
echo " Downloaded: $(du -h "$KERNEL_SRC_ARCHIVE" | cut -f1)"
else
echo "==> Kernel source already cached: $(du -h "$KERNEL_SRC_ARCHIVE" | cut -f1)"
fi
# --- Verify checksum if pinned ---
if [ -n "${MAINLINE_KERNEL_SHA256:-}" ]; then
actual=$(sha256sum "$KERNEL_SRC_ARCHIVE" | awk '{print $1}')
if [ "$actual" != "$MAINLINE_KERNEL_SHA256" ]; then
echo "ERROR: Kernel source checksum mismatch"
echo " Expected: $MAINLINE_KERNEL_SHA256"
echo " Got: $actual"
exit 1
fi
echo " Checksum OK"
fi
# --- Extract to case-sensitive fs ---
# The kernel source has files differing only by case (xt_mark.h vs xt_MARK.h).
# Build in /tmp (ext4 on Linux runners, case-sensitive).
KERNEL_BUILD_DIR="/tmp/kernel-build-arm64-generic"
rm -rf "$KERNEL_BUILD_DIR"
mkdir -p "$KERNEL_BUILD_DIR"
echo "==> Extracting kernel source..."
tar -xf "$KERNEL_SRC_ARCHIVE" -C "$KERNEL_BUILD_DIR"
KERNEL_SRC_DIR=$(find "$KERNEL_BUILD_DIR" -maxdepth 1 -type d -name 'linux-*' | head -1)
if [ -z "$KERNEL_SRC_DIR" ]; then
echo "ERROR: Could not find extracted source directory"
ls -la "$KERNEL_BUILD_DIR"/
exit 1
fi
cd "$KERNEL_SRC_DIR"
# --- Base config: arm64 defconfig (generic ARMv8) ---
echo "==> Applying arm64 defconfig..."
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- defconfig
# --- Apply shared container fragment ---
CONFIG_FRAGMENT="$PROJECT_ROOT/build/config/kernel-container.fragment"
if [ ! -f "$CONFIG_FRAGMENT" ]; then
echo "ERROR: Config fragment not found: $CONFIG_FRAGMENT"
exit 1
fi
apply_fragment() {
local fragment="$1"
while IFS= read -r line; do
case "$line" in
"# CONFIG_"*" is not set")
key=$(echo "$line" | sed -n 's/^# \(CONFIG_[A-Z0-9_]*\) is not set$/\1/p')
[ -n "$key" ] && ./scripts/config --disable "${key#CONFIG_}"
continue
;;
\#*|"") continue ;;
esac
key="${line%%=*}"
value="${line#*=}"
case "$value" in
y) ./scripts/config --enable "$key" ;;
m) ./scripts/config --module "$key" ;;
n) ./scripts/config --disable "${key#CONFIG_}" ;;
*) ./scripts/config --set-str "$key" "$value" ;;
esac
done < "$fragment"
}
echo "==> Applying kernel-container.fragment (pass 1)..."
apply_fragment "$CONFIG_FRAGMENT"
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- olddefconfig
echo "==> Applying kernel-container.fragment (pass 2)..."
apply_fragment "$CONFIG_FRAGMENT"
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- olddefconfig
# --- ARM64 virt-host specific enables ---
# These are needed for the generic UEFI/virtio boot path but are arch-specific
# so they live in this script rather than the shared fragment.
echo "==> Enabling ARM64 virt-host configs..."
./scripts/config --enable CONFIG_EFI
./scripts/config --enable CONFIG_EFI_STUB
./scripts/config --enable CONFIG_VIRTIO
./scripts/config --enable CONFIG_VIRTIO_PCI
./scripts/config --enable CONFIG_VIRTIO_BLK
./scripts/config --enable CONFIG_VIRTIO_NET
./scripts/config --enable CONFIG_VIRTIO_CONSOLE
./scripts/config --enable CONFIG_VIRTIO_MMIO
./scripts/config --enable CONFIG_HW_RANDOM_VIRTIO
# NVMe for cloud / bare-metal ARM64 hosts that don't use virtio
./scripts/config --enable CONFIG_BLK_DEV_NVME
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- olddefconfig
# --- Verify critical configs ---
echo "==> Verifying critical configs..."
for cfg in CGROUP_BPF SECURITY_APPARMOR AUDIT VIRTIO_BLK EFI_STUB; do
if ! grep -q "CONFIG_${cfg}=y" .config; then
echo "ERROR: CONFIG_${cfg} not set after olddefconfig"
grep "CONFIG_${cfg}" .config || echo " (not found)"
exit 1
fi
echo " CONFIG_${cfg}=y confirmed"
done
# --- Build kernel + modules (no DTBs — UEFI hosts use ACPI/virtio) ---
NPROC=$(nproc 2>/dev/null || echo 4)
echo ""
echo "==> Building ARM64 kernel (${NPROC} parallel jobs)..."
echo " This may take 20-40 minutes on a 6-core Odroid..."
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- -j"$NPROC" Image modules 2>&1
echo "==> Kernel build complete"
# --- Install to staging ---
echo "==> Installing Image..."
cp arch/arm64/boot/Image "$CUSTOM_IMAGE"
echo "==> Installing modules (stripped)..."
rm -rf "$CUSTOM_MODULES"
mkdir -p "$CUSTOM_MODULES"
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- \
INSTALL_MOD_STRIP=1 modules_install INSTALL_MOD_PATH="$CUSTOM_MODULES"
# Pick up actual kernel version (e.g. 6.12.10 if KVER differs from package suffix)
ACTUAL_KVER=$(ls "$CUSTOM_MODULES/lib/modules/" | head -1)
rm -f "$CUSTOM_MODULES/lib/modules/$ACTUAL_KVER/build"
rm -f "$CUSTOM_MODULES/lib/modules/$ACTUAL_KVER/source"
depmod -a -b "$CUSTOM_MODULES" "$ACTUAL_KVER" 2>/dev/null || true
cp .config "$CUSTOM_KERNEL_DIR/.config"
# --- Clean up ---
echo "==> Cleaning kernel build directory..."
cd /
rm -rf "$KERNEL_BUILD_DIR"
# --- Summary ---
echo ""
echo "==> Generic ARM64 kernel build complete:"
echo " Image: $CUSTOM_IMAGE ($(du -h "$CUSTOM_IMAGE" | cut -f1))"
echo " Kernel ver: $ACTUAL_KVER"
MOD_COUNT=$(find "$CUSTOM_MODULES/lib/modules/$ACTUAL_KVER" -name '*.ko*' 2>/dev/null | wc -l)
echo " Modules: $MOD_COUNT"
echo " Modules size: $(du -sh "$CUSTOM_MODULES/lib/modules/$ACTUAL_KVER" 2>/dev/null | cut -f1)"
echo ""

161
build/scripts/build-kernel-rpi.sh Executable file
View File

@@ -0,0 +1,161 @@
#!/bin/bash
# build-kernel-rpi.sh — Build kernel for Raspberry Pi 4/5 (ARM64)
#
# Uses the official raspberrypi/linux kernel fork with bcm2711_defconfig as the
# base, overlaid with the shared container-config fragment.
#
# This is the RPi-specific build track. For generic ARM64 (UEFI / virtio /
# kernel.org mainline) see build/scripts/build-kernel-arm64.sh.
#
# Output is cached in $CACHE_DIR/custom-kernel-rpi/ and reused across builds.
#
# Requirements:
# - gcc-aarch64-linux-gnu (cross-compiler)
# - Standard kernel build deps (bc, bison, flex, etc.)
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
CACHE_DIR="${CACHE_DIR:-$PROJECT_ROOT/build/cache}"
# shellcheck source=../config/versions.env
. "$SCRIPT_DIR/../config/versions.env"
CUSTOM_KERNEL_DIR="$CACHE_DIR/custom-kernel-rpi"
CUSTOM_IMAGE="$CUSTOM_KERNEL_DIR/Image"
CUSTOM_MODULES="$CUSTOM_KERNEL_DIR/modules"
CUSTOM_DTBS="$CUSTOM_KERNEL_DIR/dtbs"
mkdir -p "$CACHE_DIR" "$CUSTOM_KERNEL_DIR"
# --- Skip if already built ---
if [ -f "$CUSTOM_IMAGE" ] && [ -d "$CUSTOM_MODULES" ]; then
echo "==> RPi kernel already built (cached)"
echo " Image: $CUSTOM_IMAGE ($(du -h "$CUSTOM_IMAGE" | cut -f1))"
exit 0
fi
# --- Verify cross-compiler ---
if ! command -v aarch64-linux-gnu-gcc >/dev/null 2>&1; then
echo "ERROR: aarch64-linux-gnu-gcc not found"
echo "Install: apt-get install gcc-aarch64-linux-gnu"
exit 1
fi
echo "==> Building RPi kernel (raspberrypi/linux)..."
echo " Branch: $RPI_KERNEL_BRANCH"
echo " Repo: $RPI_KERNEL_REPO"
# --- Download kernel source ---
KERNEL_SRC_DIR="$CACHE_DIR/rpi-linux-${RPI_KERNEL_BRANCH}"
if [ ! -d "$KERNEL_SRC_DIR" ]; then
echo "==> Downloading RPi kernel source (shallow clone)..."
git clone --depth 1 --branch "$RPI_KERNEL_BRANCH" \
"$RPI_KERNEL_REPO" "$KERNEL_SRC_DIR"
else
echo "==> Kernel source already cached"
fi
# --- Build in /tmp for case-sensitivity ---
KERNEL_BUILD_DIR="/tmp/kernel-build-arm64"
rm -rf "$KERNEL_BUILD_DIR"
cp -a "$KERNEL_SRC_DIR" "$KERNEL_BUILD_DIR"
cd "$KERNEL_BUILD_DIR"
# --- Apply base config (Pi 4 = bcm2711) ---
echo "==> Applying bcm2711_defconfig..."
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- bcm2711_defconfig
# --- Apply container config overrides ---
CONFIG_FRAGMENT="$PROJECT_ROOT/build/config/kernel-container.fragment"
if [ -f "$CONFIG_FRAGMENT" ]; then
echo "==> Applying KubeSolo config overrides..."
while IFS= read -r line; do
# Skip comments and empty lines
case "$line" in \#*|"") continue ;; esac
key="${line%%=*}"
value="${line#*=}"
case "$value" in
y) ./scripts/config --enable "$key" ;;
m) ./scripts/config --module "$key" ;;
n) ./scripts/config --disable "${key#CONFIG_}" ;;
*) ./scripts/config --set-str "$key" "$value" ;;
esac
done < "$CONFIG_FRAGMENT"
fi
# Handle "is not set" comments as disables
if [ -f "$CONFIG_FRAGMENT" ]; then
while IFS= read -r line; do
case "$line" in
"# CONFIG_"*" is not set")
key=$(echo "$line" | sed -n 's/^# \(CONFIG_[A-Z_]*\) is not set$/\1/p')
[ -n "$key" ] && ./scripts/config --disable "${key#CONFIG_}"
;;
esac
done < "$CONFIG_FRAGMENT"
fi
# Resolve dependencies
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- olddefconfig
# --- Build kernel + modules + DTBs ---
NPROC=$(nproc 2>/dev/null || echo 4)
echo ""
echo "==> Building RPi kernel (${NPROC} parallel jobs)..."
echo " This may take 20-30 minutes..."
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- -j"$NPROC" Image modules dtbs 2>&1
echo "==> RPi kernel build complete"
# --- Install to staging ---
echo "==> Installing Image..."
cp arch/arm64/boot/Image "$CUSTOM_IMAGE"
echo "==> Installing modules (stripped)..."
rm -rf "$CUSTOM_MODULES"
mkdir -p "$CUSTOM_MODULES"
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- \
INSTALL_MOD_STRIP=1 modules_install INSTALL_MOD_PATH="$CUSTOM_MODULES"
# Remove build/source symlinks
KVER=$(ls "$CUSTOM_MODULES/lib/modules/" | head -1)
rm -f "$CUSTOM_MODULES/lib/modules/$KVER/build"
rm -f "$CUSTOM_MODULES/lib/modules/$KVER/source"
# Run depmod
depmod -a -b "$CUSTOM_MODULES" "$KVER" 2>/dev/null || true
echo "==> Installing Device Tree Blobs..."
rm -rf "$CUSTOM_DTBS"
mkdir -p "$CUSTOM_DTBS/overlays"
# Pi 4 DTBs
cp arch/arm64/boot/dts/broadcom/bcm2711*.dtb "$CUSTOM_DTBS/" 2>/dev/null || true
# Pi 5 DTBs
cp arch/arm64/boot/dts/broadcom/bcm2712*.dtb "$CUSTOM_DTBS/" 2>/dev/null || true
# Overlays we need
for overlay in disable-wifi disable-bt; do
[ -f "arch/arm64/boot/dts/overlays/${overlay}.dtbo" ] && \
cp "arch/arm64/boot/dts/overlays/${overlay}.dtbo" "$CUSTOM_DTBS/overlays/"
done
# Save config for reference
cp .config "$CUSTOM_KERNEL_DIR/.config"
# --- Clean up ---
echo "==> Cleaning kernel build directory..."
cd /
rm -rf "$KERNEL_BUILD_DIR"
# --- Summary ---
echo ""
echo "==> RPi kernel build complete:"
echo " Image: $CUSTOM_IMAGE ($(du -h "$CUSTOM_IMAGE" | cut -f1))"
echo " Kernel ver: $KVER"
MOD_COUNT=$(find "$CUSTOM_MODULES/lib/modules/$KVER" -name '*.ko*' 2>/dev/null | wc -l)
echo " Modules: $MOD_COUNT"
echo " Modules size: $(du -sh "$CUSTOM_MODULES/lib/modules/$KVER" 2>/dev/null | cut -f1)"
echo " DTBs: $(ls "$CUSTOM_DTBS"/*.dtb 2>/dev/null | wc -l)"
echo ""

View File

@@ -85,73 +85,53 @@ echo " Source dir: $(basename "$KERNEL_SRC_DIR")"
cd "$KERNEL_SRC_DIR" cd "$KERNEL_SRC_DIR"
# --- Apply stock config + enable CONFIG_CGROUP_BPF --- # --- Apply stock config + shared container-config fragment ---
echo "==> Applying stock Tiny Core config..." echo "==> Applying stock Tiny Core config..."
cp "$KERNEL_CFG" .config cp "$KERNEL_CFG" .config
echo "==> Enabling required kernel configs..." CONFIG_FRAGMENT="$PROJECT_ROOT/build/config/kernel-container.fragment"
./scripts/config --enable CONFIG_CGROUP_BPF if [ ! -f "$CONFIG_FRAGMENT" ]; then
./scripts/config --enable CONFIG_DEVTMPFS echo "ERROR: Config fragment not found: $CONFIG_FRAGMENT"
./scripts/config --enable CONFIG_DEVTMPFS_MOUNT exit 1
./scripts/config --enable CONFIG_MEMCG fi
./scripts/config --enable CONFIG_CFS_BANDWIDTH
# --- Strip unnecessary subsystems for smallest footprint --- # Apply the fragment: each "CONFIG_X=v" line becomes the right scripts/config
# This is a headless K8s edge appliance — no sound, GPU, wireless, etc. # invocation; "# CONFIG_X is not set" comments become --disable.
echo "==> Disabling unnecessary subsystems for minimal footprint..." apply_fragment() {
local fragment="$1"
while IFS= read -r line; do
case "$line" in
"# CONFIG_"*" is not set")
key=$(echo "$line" | sed -n 's/^# \(CONFIG_[A-Z0-9_]*\) is not set$/\1/p')
[ -n "$key" ] && ./scripts/config --disable "${key#CONFIG_}"
continue
;;
\#*|"") continue ;;
esac
key="${line%%=*}"
value="${line#*=}"
case "$value" in
y) ./scripts/config --enable "$key" ;;
m) ./scripts/config --module "$key" ;;
n) ./scripts/config --disable "${key#CONFIG_}" ;;
*) ./scripts/config --set-str "$key" "$value" ;;
esac
done < "$fragment"
}
# Sound subsystem (not needed on headless appliance) # Two-pass apply: TC's stock config has CONFIG_SECURITY disabled, so olddefconfig
./scripts/config --disable SOUND # strips the security subtree before its dependencies resolve. Re-applying the
# fragment after the first olddefconfig restores those entries.
# GPU/DRM (serial console only, no display) echo "==> Applying kernel-container.fragment (pass 1)..."
./scripts/config --disable DRM apply_fragment "$CONFIG_FRAGMENT"
# KVM hypervisor (this IS the guest/bare metal, not a hypervisor)
./scripts/config --disable KVM
# Media/camera/TV/radio (not needed)
./scripts/config --disable MEDIA_SUPPORT
# Wireless networking (wired edge device)
./scripts/config --disable WIRELESS
./scripts/config --disable WLAN
./scripts/config --disable CFG80211
# Bluetooth (not needed)
./scripts/config --disable BT
# NFC (not needed)
./scripts/config --disable NFC
# Infiniband (not needed on edge)
./scripts/config --disable INFINIBAND
# PCMCIA (legacy, not needed)
./scripts/config --disable PCMCIA
# Amateur radio (not needed)
./scripts/config --disable HAMRADIO
# ISDN (not needed)
./scripts/config --disable ISDN
# ATM networking (not needed)
./scripts/config --disable ATM
# Joystick/gamepad (not needed)
./scripts/config --disable INPUT_JOYSTICK
./scripts/config --disable INPUT_TABLET
# FPGA (not needed)
./scripts/config --disable FPGA
# Resolve dependencies (olddefconfig accepts defaults for new options)
make olddefconfig make olddefconfig
# Verify CONFIG_CGROUP_BPF is set echo "==> Applying kernel-container.fragment (pass 2)..."
if grep -q 'CONFIG_CGROUP_BPF=y' .config; then apply_fragment "$CONFIG_FRAGMENT"
echo " CONFIG_CGROUP_BPF=y confirmed in .config" make olddefconfig
else
# Verify critical configs are set
if ! grep -q 'CONFIG_CGROUP_BPF=y' .config; then
echo "ERROR: CONFIG_CGROUP_BPF not set after olddefconfig" echo "ERROR: CONFIG_CGROUP_BPF not set after olddefconfig"
grep 'CGROUP_BPF' .config || echo " (CGROUP_BPF not found in .config)" grep 'CGROUP_BPF' .config || echo " (CGROUP_BPF not found in .config)"
echo "" echo ""
@@ -159,10 +139,25 @@ else
grep -E 'CONFIG_BPF=|CONFIG_BPF_SYSCALL=' .config || echo " BPF not found" grep -E 'CONFIG_BPF=|CONFIG_BPF_SYSCALL=' .config || echo " BPF not found"
exit 1 exit 1
fi fi
echo " CONFIG_CGROUP_BPF=y confirmed"
# Show what changed if ! grep -q 'CONFIG_SECURITY_APPARMOR=y' .config; then
echo " Config diff from stock:" echo "ERROR: CONFIG_SECURITY_APPARMOR not set after olddefconfig"
diff "$KERNEL_CFG" .config | grep '^[<>]' | head -20 || echo " (no differences beyond CGROUP_BPF)" echo " Security-related configs:"
grep -E 'CONFIG_SECURITY=|CONFIG_SECURITYFS=|CONFIG_SECURITY_APPARMOR=' .config
exit 1
fi
echo " CONFIG_SECURITY_APPARMOR=y confirmed"
if ! grep -q 'CONFIG_AUDIT=y' .config; then
echo "ERROR: CONFIG_AUDIT not set after olddefconfig"
exit 1
fi
echo " CONFIG_AUDIT=y confirmed"
# Show what changed (security-related)
echo " Key config values:"
grep -E 'CONFIG_SECURITY=|CONFIG_SECURITY_APPARMOR=|CONFIG_AUDIT=|CONFIG_LSM=|CONFIG_CGROUP_BPF=' .config | sed 's/^/ /'
# --- Build kernel + modules --- # --- Build kernel + modules ---
NPROC=$(nproc 2>/dev/null || echo 4) NPROC=$(nproc 2>/dev/null || echo 4)

View File

@@ -6,28 +6,61 @@
# Part 2: System A (512 MB, ext4) — vmlinuz + kubesolo-os.gz (active) # Part 2: System A (512 MB, ext4) — vmlinuz + kubesolo-os.gz (active)
# Part 3: System B (512 MB, ext4) — vmlinuz + kubesolo-os.gz (passive) # Part 3: System B (512 MB, ext4) — vmlinuz + kubesolo-os.gz (passive)
# Part 4: Data (remaining, ext4) — persistent K8s state # Part 4: Data (remaining, ext4) — persistent K8s state
#
# Supports both x86_64 (default) and ARM64 generic UEFI targets. ARM64 RPi
# uses a different image format — see build/scripts/create-rpi-image.sh.
#
# Environment:
# TARGET_ARCH amd64 (default) or arm64
# IMG_SIZE_MB Image size in MB (default 4096)
# CACHE_DIR Build cache (default <project>/build/cache)
# ROOTFS_DIR Rootfs work dir (default <project>/build/rootfs-work)
# OUTPUT_DIR Output dir (default <project>/output)
set -euo pipefail set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
ROOTFS_DIR="${ROOTFS_DIR:-$PROJECT_ROOT/build/rootfs-work}" ROOTFS_DIR="${ROOTFS_DIR:-$PROJECT_ROOT/build/rootfs-work}"
CACHE_DIR="${CACHE_DIR:-$PROJECT_ROOT/build/cache}"
OUTPUT_DIR="${OUTPUT_DIR:-$PROJECT_ROOT/output}" OUTPUT_DIR="${OUTPUT_DIR:-$PROJECT_ROOT/output}"
VERSION="$(cat "$PROJECT_ROOT/VERSION")" VERSION="$(cat "$PROJECT_ROOT/VERSION")"
OS_NAME="kubesolo-os" OS_NAME="kubesolo-os"
TARGET_ARCH="${TARGET_ARCH:-amd64}"
IMG_OUTPUT="$OUTPUT_DIR/${OS_NAME}-${VERSION}.img"
IMG_SIZE_MB="${IMG_SIZE_MB:-4096}" # 4 GB default (larger for A/B) IMG_SIZE_MB="${IMG_SIZE_MB:-4096}" # 4 GB default (larger for A/B)
# --- Arch-specific paths ---
case "$TARGET_ARCH" in
amd64)
IMG_OUTPUT="$OUTPUT_DIR/${OS_NAME}-${VERSION}.img"
VMLINUZ="$ROOTFS_DIR/vmlinuz" VMLINUZ="$ROOTFS_DIR/vmlinuz"
INITRAMFS="$ROOTFS_DIR/kubesolo-os.gz"
GRUB_CFG="$PROJECT_ROOT/build/grub/grub.cfg" GRUB_CFG="$PROJECT_ROOT/build/grub/grub.cfg"
GRUB_TARGET="x86_64-efi"
GRUB_EFI_BIN="bootx64.efi"
GRUB_INSTALL_BIOS=true
;;
arm64)
IMG_OUTPUT="$OUTPUT_DIR/${OS_NAME}-${VERSION}.arm64.img"
VMLINUZ="$CACHE_DIR/kernel-arm64-generic/Image"
GRUB_CFG="$PROJECT_ROOT/build/grub/grub-arm64.cfg"
GRUB_TARGET="arm64-efi"
GRUB_EFI_BIN="BOOTAA64.EFI"
GRUB_INSTALL_BIOS=false
;;
*)
echo "ERROR: TARGET_ARCH must be 'amd64' or 'arm64' (got: $TARGET_ARCH)"
exit 1
;;
esac
INITRAMFS="$ROOTFS_DIR/kubesolo-os.gz"
GRUB_ENV_DEFAULTS="$PROJECT_ROOT/build/grub/grub-env-defaults" GRUB_ENV_DEFAULTS="$PROJECT_ROOT/build/grub/grub-env-defaults"
for f in "$VMLINUZ" "$INITRAMFS" "$GRUB_CFG" "$GRUB_ENV_DEFAULTS"; do for f in "$VMLINUZ" "$INITRAMFS" "$GRUB_CFG" "$GRUB_ENV_DEFAULTS"; do
[ -f "$f" ] || { echo "ERROR: Missing $f"; exit 1; } [ -f "$f" ] || { echo "ERROR: Missing $f"; exit 1; }
done done
echo "==> Creating ${IMG_SIZE_MB}MB disk image with A/B partitions..." echo "==> Creating ${IMG_SIZE_MB}MB ${TARGET_ARCH} disk image with A/B partitions..."
mkdir -p "$OUTPUT_DIR" mkdir -p "$OUTPUT_DIR"
# Create sparse image # Create sparse image
@@ -51,10 +84,39 @@ size=1048576, type=0FC63DAF-8483-4772-8E79-3D69D8477DE4, name="SystemB"
type=0FC63DAF-8483-4772-8E79-3D69D8477DE4, name="Data" type=0FC63DAF-8483-4772-8E79-3D69D8477DE4, name="Data"
EOF EOF
# Set up loop device # Set up loop device with partition mappings
LOOP=$(losetup --show -fP "$IMG_OUTPUT") LOOP=$(losetup --show -f "$IMG_OUTPUT")
echo "==> Loop device: $LOOP" echo "==> Loop device: $LOOP"
# Use kpartx for reliable partition device nodes (works in Docker/containers)
USE_KPARTX=false
if [ ! -b "${LOOP}p1" ]; then
if command -v kpartx >/dev/null 2>&1; then
kpartx -a "$LOOP"
USE_KPARTX=true
sleep 1
LOOP_NAME=$(basename "$LOOP")
P1="/dev/mapper/${LOOP_NAME}p1"
P2="/dev/mapper/${LOOP_NAME}p2"
P3="/dev/mapper/${LOOP_NAME}p3"
P4="/dev/mapper/${LOOP_NAME}p4"
else
# Retry with -P flag
losetup -d "$LOOP"
LOOP=$(losetup --show -fP "$IMG_OUTPUT")
sleep 1
P1="${LOOP}p1"
P2="${LOOP}p2"
P3="${LOOP}p3"
P4="${LOOP}p4"
fi
else
P1="${LOOP}p1"
P2="${LOOP}p2"
P3="${LOOP}p3"
P4="${LOOP}p4"
fi
MNT_EFI=$(mktemp -d) MNT_EFI=$(mktemp -d)
MNT_SYSA=$(mktemp -d) MNT_SYSA=$(mktemp -d)
MNT_SYSB=$(mktemp -d) MNT_SYSB=$(mktemp -d)
@@ -65,22 +127,25 @@ cleanup() {
umount "$MNT_SYSA" 2>/dev/null || true umount "$MNT_SYSA" 2>/dev/null || true
umount "$MNT_SYSB" 2>/dev/null || true umount "$MNT_SYSB" 2>/dev/null || true
umount "$MNT_DATA" 2>/dev/null || true umount "$MNT_DATA" 2>/dev/null || true
if [ "$USE_KPARTX" = true ]; then
kpartx -d "$LOOP" 2>/dev/null || true
fi
losetup -d "$LOOP" 2>/dev/null || true losetup -d "$LOOP" 2>/dev/null || true
rm -rf "$MNT_EFI" "$MNT_SYSA" "$MNT_SYSB" "$MNT_DATA" 2>/dev/null || true rm -rf "$MNT_EFI" "$MNT_SYSA" "$MNT_SYSB" "$MNT_DATA" 2>/dev/null || true
} }
trap cleanup EXIT trap cleanup EXIT
# Format partitions # Format partitions
mkfs.vfat -F 32 -n KSOLOEFI "${LOOP}p1" mkfs.vfat -F 32 -n KSOLOEFI "$P1"
mkfs.ext4 -q -L KSOLOA "${LOOP}p2" mkfs.ext4 -q -L KSOLOA "$P2"
mkfs.ext4 -q -L KSOLOB "${LOOP}p3" mkfs.ext4 -q -L KSOLOB "$P3"
mkfs.ext4 -q -L KSOLODATA "${LOOP}p4" mkfs.ext4 -q -L KSOLODATA "$P4"
# Mount all partitions # Mount all partitions
mount "${LOOP}p1" "$MNT_EFI" mount "$P1" "$MNT_EFI"
mount "${LOOP}p2" "$MNT_SYSA" mount "$P2" "$MNT_SYSA"
mount "${LOOP}p3" "$MNT_SYSB" mount "$P3" "$MNT_SYSB"
mount "${LOOP}p4" "$MNT_DATA" mount "$P4" "$MNT_DATA"
# --- EFI/Boot Partition --- # --- EFI/Boot Partition ---
echo " Installing GRUB..." echo " Installing GRUB..."
@@ -129,25 +194,33 @@ else
mv "$GRUBENV_FILE.tmp" "$GRUBENV_FILE" mv "$GRUBENV_FILE.tmp" "$GRUBENV_FILE"
fi fi
# Install GRUB EFI binary if available # Install GRUB EFI binary
if command -v grub-mkimage >/dev/null 2>&1; then # Modules required: part_gpt + fat (boot partition), ext2 (system A/B),
grub-mkimage -O x86_64-efi -o "$MNT_EFI/EFI/BOOT/bootx64.efi" \ # normal + linux + echo + configfile + loadenv (boot menu + grubenv),
-p /boot/grub \ # search_* (locate partitions by label).
part_gpt ext2 fat normal linux echo all_video test search \ # all_video + test are x86-specific (DRM init); leave them out on arm64.
search_fs_uuid search_label configfile loadenv \ if [ "$TARGET_ARCH" = "arm64" ]; then
2>/dev/null || echo " WARN: grub-mkimage failed — use QEMU -bios flag" GRUB_MODULES="part_gpt ext2 fat normal linux echo test search search_fs_uuid search_label configfile loadenv"
elif command -v grub2-mkimage >/dev/null 2>&1; then
grub2-mkimage -O x86_64-efi -o "$MNT_EFI/EFI/BOOT/bootx64.efi" \
-p /boot/grub \
part_gpt ext2 fat normal linux echo all_video test search \
search_fs_uuid search_label configfile loadenv \
2>/dev/null || echo " WARN: grub2-mkimage failed — use QEMU -bios flag"
else else
echo " WARN: grub-mkimage not found — EFI boot image not created" GRUB_MODULES="part_gpt ext2 fat normal linux echo all_video test search search_fs_uuid search_label configfile loadenv"
echo " Install grub2-tools or use QEMU -kernel/-initrd flags"
fi fi
# For BIOS boot: install GRUB i386-pc modules if available # shellcheck disable=SC2086 # GRUB_MODULES is intentionally word-split
if command -v grub-mkimage >/dev/null 2>&1; then
grub-mkimage -O "$GRUB_TARGET" -o "$MNT_EFI/EFI/BOOT/$GRUB_EFI_BIN" \
-p /boot/grub $GRUB_MODULES \
|| echo " WARN: grub-mkimage failed — use QEMU -bios flag"
elif command -v grub2-mkimage >/dev/null 2>&1; then
grub2-mkimage -O "$GRUB_TARGET" -o "$MNT_EFI/EFI/BOOT/$GRUB_EFI_BIN" \
-p /boot/grub $GRUB_MODULES \
|| echo " WARN: grub2-mkimage failed — use QEMU -bios flag"
else
echo " WARN: grub-mkimage not found — EFI boot image not created"
echo " Install grub-efi-${TARGET_ARCH}-bin or use QEMU -kernel/-initrd flags"
fi
# For BIOS boot: install GRUB i386-pc modules (x86 only — ARM64 is UEFI-only).
if [ "$GRUB_INSTALL_BIOS" = "true" ]; then
if command -v grub-install >/dev/null 2>&1; then if command -v grub-install >/dev/null 2>&1; then
grub-install --target=i386-pc --boot-directory="$MNT_EFI/boot" \ grub-install --target=i386-pc --boot-directory="$MNT_EFI/boot" \
--no-floppy "$LOOP" 2>/dev/null || { --no-floppy "$LOOP" 2>/dev/null || {
@@ -159,6 +232,7 @@ elif command -v grub2-install >/dev/null 2>&1; then
echo " WARN: BIOS GRUB install failed — EFI-only or use QEMU -kernel" echo " WARN: BIOS GRUB install failed — EFI-only or use QEMU -kernel"
} }
fi fi
fi
# --- System A Partition (active) --- # --- System A Partition (active) ---
echo " Populating System A (active)..." echo " Populating System A (active)..."
@@ -181,9 +255,9 @@ done
sync sync
echo "" echo ""
echo "==> Disk image created: $IMG_OUTPUT" echo "==> ${TARGET_ARCH} disk image created: $IMG_OUTPUT"
echo " Size: $(du -h "$IMG_OUTPUT" | cut -f1)" echo " Size: $(du -h "$IMG_OUTPUT" | cut -f1)"
echo " Part 1 (KSOLOEFI): GRUB + A/B boot config" echo " Part 1 (KSOLOEFI): GRUB ($GRUB_TARGET) + A/B boot config"
echo " Part 2 (KSOLOA): System A — kernel + initramfs (active)" echo " Part 2 (KSOLOA): System A — kernel + initramfs (active)"
echo " Part 3 (KSOLOB): System B — kernel + initramfs (passive)" echo " Part 3 (KSOLOB): System B — kernel + initramfs (passive)"
echo " Part 4 (KSOLODATA): Persistent K8s state" echo " Part 4 (KSOLODATA): Persistent K8s state"

256
build/scripts/create-rpi-image.sh Executable file
View File

@@ -0,0 +1,256 @@
#!/bin/bash
# create-rpi-image.sh — Create a raw disk image for Raspberry Pi SD card
#
# Partition layout (MBR):
# Part 1: Boot/Control (384 MB, FAT32, label KSOLOCTL) — firmware + kernel + initramfs + autoboot.txt
# Part 2: Boot A (256 MB, FAT32, label KSOLOA) — kernel + DTBs + initramfs
# Part 3: Boot B (256 MB, FAT32, label KSOLOB) — same as Boot A (initially identical)
# Part 4: Data (remaining of 2GB, ext4, label KSOLODATA)
#
# The RPi EEPROM loads start4.elf from partition 1.
# If autoboot.txt is supported (newer EEPROM), firmware redirects to partition 2/3 for A/B boot.
# If autoboot.txt is NOT supported (older EEPROM), partition 1 has full boot files as fallback.
#
# MBR is required — GPT + autoboot.txt is not reliably supported on Pi 4.
#
# Usage: build/scripts/create-rpi-image.sh
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
# shellcheck source=../config/versions.env
. "$SCRIPT_DIR/../config/versions.env"
ROOTFS_DIR="${ROOTFS_DIR:-$PROJECT_ROOT/build/rootfs-work}"
OUTPUT_DIR="${OUTPUT_DIR:-$PROJECT_ROOT/output}"
CACHE_DIR="${CACHE_DIR:-$PROJECT_ROOT/build/cache}"
VERSION="$(cat "$PROJECT_ROOT/VERSION")"
IMG_OUTPUT="$OUTPUT_DIR/${OS_NAME}-${VERSION}.rpi.img"
IMG_SIZE_MB="${IMG_SIZE_MB:-2048}" # 2 GB default
# ARM64 kernel (Image format, not bzImage)
KERNEL="${CACHE_DIR}/custom-kernel-rpi/Image"
INITRAMFS="${ROOTFS_DIR}/kubesolo-os.gz"
RPI_FIRMWARE_DIR="${CACHE_DIR}/rpi-firmware"
# DTBs MUST come from the kernel build (not firmware repo) to match the kernel.
# A DTB mismatch causes sdhci-iproc to silently fail — zero block devices.
KERNEL_DTBS_DIR="${CACHE_DIR}/custom-kernel-rpi/dtbs"
echo "==> Creating ${IMG_SIZE_MB}MB Raspberry Pi disk image..."
# --- Verify required files ---
MISSING=0
for f in "$KERNEL" "$INITRAMFS"; do
if [ ! -f "$f" ]; then
echo "ERROR: Missing $f"
MISSING=1
fi
done
if [ ! -d "$RPI_FIRMWARE_DIR" ]; then
echo "ERROR: Missing RPi firmware directory: $RPI_FIRMWARE_DIR"
echo " Run 'make fetch' to download firmware blobs."
MISSING=1
fi
if [ "$MISSING" = "1" ]; then
echo ""
echo "Required files:"
echo " Kernel: $KERNEL (run 'make kernel-arm64')"
echo " Initramfs: $INITRAMFS (run 'make initramfs')"
echo " Firmware: $RPI_FIRMWARE_DIR/ (run 'make fetch')"
exit 1
fi
mkdir -p "$OUTPUT_DIR"
# --- Create sparse image ---
dd if=/dev/zero of="$IMG_OUTPUT" bs=1M count=0 seek="$IMG_SIZE_MB" 2>/dev/null
# --- Partition table (MBR) ---
# MBR is required for reliable RPi boot with autoboot.txt.
# GPT + autoboot.txt fails on many Pi 4 EEPROM versions.
# Part 1: Boot/Control 384 MB FAT32 (firmware + kernel + initramfs + autoboot.txt)
# Part 2: Boot A 256 MB FAT32 (kernel + initramfs + DTBs)
# Part 3: Boot B 256 MB FAT32 (kernel + initramfs + DTBs)
# Part 4: Data remaining ext4
sfdisk "$IMG_OUTPUT" << EOF
label: dos
# Boot/Control partition: 384 MB, FAT32 (type 0c = W95 FAT32 LBA)
# Contains firmware + autoboot.txt for A/B redirect, PLUS full boot files as fallback
start=2048, size=786432, type=c, bootable
# Boot A partition: 256 MB, FAT32
size=524288, type=c
# Boot B partition: 256 MB, FAT32
size=524288, type=c
# Data partition: remaining, Linux
type=83
EOF
# --- Set up loop device ---
LOOP=$(losetup --show -f "$IMG_OUTPUT")
echo "==> Loop device: $LOOP"
# Use kpartx for reliable partition device nodes (works in Docker/containers)
USE_KPARTX=false
if [ ! -b "${LOOP}p1" ]; then
if command -v kpartx >/dev/null 2>&1; then
kpartx -a "$LOOP"
USE_KPARTX=true
sleep 1
LOOP_NAME=$(basename "$LOOP")
P1="/dev/mapper/${LOOP_NAME}p1"
P2="/dev/mapper/${LOOP_NAME}p2"
P3="/dev/mapper/${LOOP_NAME}p3"
P4="/dev/mapper/${LOOP_NAME}p4"
else
# Retry with -P flag
losetup -d "$LOOP"
LOOP=$(losetup --show -fP "$IMG_OUTPUT")
sleep 1
P1="${LOOP}p1"
P2="${LOOP}p2"
P3="${LOOP}p3"
P4="${LOOP}p4"
fi
else
P1="${LOOP}p1"
P2="${LOOP}p2"
P3="${LOOP}p3"
P4="${LOOP}p4"
fi
MNT_CTL=$(mktemp -d)
MNT_BOOTA=$(mktemp -d)
MNT_BOOTB=$(mktemp -d)
MNT_DATA=$(mktemp -d)
cleanup() {
umount "$MNT_CTL" 2>/dev/null || true
umount "$MNT_BOOTA" 2>/dev/null || true
umount "$MNT_BOOTB" 2>/dev/null || true
umount "$MNT_DATA" 2>/dev/null || true
if [ "$USE_KPARTX" = true ]; then
kpartx -d "$LOOP" 2>/dev/null || true
fi
losetup -d "$LOOP" 2>/dev/null || true
rm -rf "$MNT_CTL" "$MNT_BOOTA" "$MNT_BOOTB" "$MNT_DATA" 2>/dev/null || true
}
trap cleanup EXIT
# --- Format partitions ---
mkfs.vfat -F 32 -n KSOLOCTL "$P1"
mkfs.vfat -F 32 -n KSOLOA "$P2"
mkfs.vfat -F 32 -n KSOLOB "$P3"
mkfs.ext4 -q -L KSOLODATA "$P4"
# --- Mount all partitions ---
mount "$P1" "$MNT_CTL"
mount "$P2" "$MNT_BOOTA"
mount "$P3" "$MNT_BOOTB"
mount "$P4" "$MNT_DATA"
# --- Helper: populate a boot partition ---
populate_boot_partition() {
local MNT="$1"
local LABEL="$2"
echo " Populating $LABEL..."
# config.txt — Raspberry Pi boot configuration
cat > "$MNT/config.txt" << 'CFGTXT'
arm_64bit=1
kernel=kernel8.img
initramfs kubesolo-os.gz followkernel
enable_uart=1
gpu_mem=16
dtoverlay=disable-wifi
dtoverlay=disable-bt
CFGTXT
# cmdline.txt — kernel command line
# Note: must be a single line
echo "console=serial0,115200 console=tty1 kubesolo.data=LABEL=KSOLODATA initcall_debug loglevel=7" > "$MNT/cmdline.txt"
# Copy kernel as kernel8.img (RPi 3/4/5 ARM64 convention)
cp "$KERNEL" "$MNT/kernel8.img"
# Copy initramfs
cp "$INITRAMFS" "$MNT/kubesolo-os.gz"
# Copy DTBs from kernel build (MUST match kernel to avoid driver probe failures)
if ls "$KERNEL_DTBS_DIR"/bcm27*.dtb 1>/dev/null 2>&1; then
cp "$KERNEL_DTBS_DIR"/bcm27*.dtb "$MNT/"
fi
# Copy overlays — prefer kernel-built, fall back to firmware repo
if [ -d "$KERNEL_DTBS_DIR/overlays" ]; then
cp -r "$KERNEL_DTBS_DIR/overlays" "$MNT/"
elif [ -d "$RPI_FIRMWARE_DIR/overlays" ]; then
cp -r "$RPI_FIRMWARE_DIR/overlays" "$MNT/"
fi
# Write version marker
echo "$VERSION" > "$MNT/version.txt"
}
# --- Boot Control Partition (KSOLOCTL) ---
# Partition 1 serves dual purpose:
# 1. Contains firmware + autoboot.txt for A/B redirect (if EEPROM supports it)
# 2. Contains full boot files (kernel + initramfs) as fallback if autoboot.txt isn't supported
echo " Writing firmware + autoboot.txt + boot files to partition 1..."
# autoboot.txt — tells firmware which partition to boot from (A/B switching)
# If the EEPROM doesn't support this, it's silently ignored and the firmware
# falls back to booting from partition 1 using config.txt below.
cat > "$MNT_CTL/autoboot.txt" << 'AUTOBOOT'
[all]
tryboot_a_b=1
boot_partition=2
[tryboot]
boot_partition=3
AUTOBOOT
# Copy firmware blobs — REQUIRED on partition 1 for EEPROM to boot
if ls "$RPI_FIRMWARE_DIR"/start*.elf 1>/dev/null 2>&1; then
cp "$RPI_FIRMWARE_DIR"/start*.elf "$MNT_CTL/"
fi
if ls "$RPI_FIRMWARE_DIR"/fixup*.dat 1>/dev/null 2>&1; then
cp "$RPI_FIRMWARE_DIR"/fixup*.dat "$MNT_CTL/"
fi
if [ -f "$RPI_FIRMWARE_DIR/bootcode.bin" ]; then
cp "$RPI_FIRMWARE_DIR/bootcode.bin" "$MNT_CTL/"
fi
# Full boot files on partition 1 — fallback if autoboot.txt redirect doesn't work.
# When autoboot.txt works, firmware switches to partition 2 and reads config.txt there.
# When autoboot.txt is unsupported, firmware reads THIS config.txt and boots from here.
populate_boot_partition "$MNT_CTL" "Boot Control (KSOLOCTL)"
# --- Boot A Partition (KSOLOA) ---
populate_boot_partition "$MNT_BOOTA" "Boot A (KSOLOA)"
# --- Boot B Partition (KSOLOB, initially identical) ---
populate_boot_partition "$MNT_BOOTB" "Boot B (KSOLOB)"
# --- Data Partition (KSOLODATA) ---
echo " Preparing data partition..."
for dir in kubesolo containerd etc-kubesolo log usr-local network images; do
mkdir -p "$MNT_DATA/$dir"
done
sync
echo ""
echo "==> Raspberry Pi disk image created: $IMG_OUTPUT"
echo " Size: $(du -h "$IMG_OUTPUT" | cut -f1)"
echo " Part 1 (KSOLOCTL): Firmware + kernel + initramfs + autoboot.txt (boot/control)"
echo " Part 2 (KSOLOA): Boot A — kernel + initramfs + DTBs"
echo " Part 3 (KSOLOB): Boot B — kernel + initramfs + DTBs"
echo " Part 4 (KSOLODATA): Persistent K8s state"
echo ""
echo "Write to SD card with:"
echo " sudo dd if=$IMG_OUTPUT of=/dev/sdX bs=4M status=progress"
echo ""

View File

@@ -10,6 +10,111 @@ ROOTFS_DIR="${ROOTFS_DIR:-$PROJECT_ROOT/build/rootfs-work}"
# shellcheck source=../config/versions.env # shellcheck source=../config/versions.env
. "$SCRIPT_DIR/../config/versions.env" . "$SCRIPT_DIR/../config/versions.env"
EXTRACT_ARCH="${TARGET_ARCH:-amd64}"
# Clean previous rootfs
rm -rf "$ROOTFS_DIR"
mkdir -p "$ROOTFS_DIR"
# =========================================================================
# ARM64: piCore64 .img.gz extraction (SD card image, not ISO)
# =========================================================================
if [ "$EXTRACT_ARCH" = "arm64" ]; then
PICORE_IMG="$CACHE_DIR/$PICORE_IMAGE"
if [ ! -f "$PICORE_IMG" ]; then
echo "ERROR: piCore64 image not found: $PICORE_IMG"
echo "Run 'TARGET_ARCH=arm64 make fetch' first."
exit 1
fi
echo "==> Extracting piCore64 image: $PICORE_IMG"
# Decompress to raw image (.img.gz or .zip)
PICORE_RAW="$CACHE_DIR/piCore-${PICORE_VERSION}.img"
if [ ! -f "$PICORE_RAW" ]; then
echo " Decompressing..."
case "$PICORE_IMG" in
*.zip)
unzip -o -j "$PICORE_IMG" '*.img' -d "$CACHE_DIR" 2>/dev/null || \
unzip -o "$PICORE_IMG" -d "$CACHE_DIR"
# Find the extracted .img file
EXTRACTED_IMG=$(find "$CACHE_DIR" -maxdepth 1 -name '*.img' -newer "$PICORE_IMG" | head -1)
if [ -n "$EXTRACTED_IMG" ] && [ "$EXTRACTED_IMG" != "$PICORE_RAW" ]; then
mv "$EXTRACTED_IMG" "$PICORE_RAW"
fi
;;
*.img.gz)
gunzip -k "$PICORE_IMG" 2>/dev/null || \
zcat "$PICORE_IMG" > "$PICORE_RAW"
;;
*)
echo "ERROR: Unknown piCore image format: $PICORE_IMG"
exit 1
;;
esac
fi
# Mount the piCore boot partition (partition 1) to find kernel/initramfs
# piCore layout: p1=boot (FAT32, has kernel+initramfs), p2=rootfs (ext4, has tce/)
IMG_MNT=$(mktemp -d)
echo " Mounting piCore boot partition..."
# Get partition 1 offset (boot/FAT partition with kernel+initramfs)
OFFSET=$(fdisk -l "$PICORE_RAW" 2>/dev/null | awk '/^.*img1/{print $2}')
if [ -z "$OFFSET" ]; then
# Fallback: try sfdisk (first partition)
OFFSET=$(sfdisk -d "$PICORE_RAW" 2>/dev/null | awk -F'[=,]' '/start=/{print $2; exit}' | tr -d ' ')
fi
if [ -z "$OFFSET" ]; then
echo "ERROR: Could not determine partition offset in piCore image"
fdisk -l "$PICORE_RAW" || true
exit 1
fi
BYTE_OFFSET=$((OFFSET * 512))
mount -o loop,ro,offset="$BYTE_OFFSET" "$PICORE_RAW" "$IMG_MNT" || {
echo "ERROR: Failed to mount piCore boot partition (need root for losetup)"
exit 1
}
# Find initramfs in the piCore boot partition
COREGZ=""
for f in "$IMG_MNT"/rootfs-piCore64*.gz "$IMG_MNT"/boot/corepure64.gz "$IMG_MNT"/boot/core.gz "$IMG_MNT"/corepure64.gz "$IMG_MNT"/core.gz; do
[ -f "$f" ] && COREGZ="$f" && break
done
if [ -z "$COREGZ" ]; then
echo "ERROR: Could not find initramfs in piCore image"
echo "Contents:"
ls -la "$IMG_MNT"/
ls -la "$IMG_MNT"/boot/ 2>/dev/null || true
umount "$IMG_MNT" 2>/dev/null || true
exit 1
fi
echo "==> Found initramfs: $COREGZ"
# Extract initramfs
mkdir -p "$ROOTFS_DIR/rootfs"
cd "$ROOTFS_DIR/rootfs"
zcat "$COREGZ" | cpio -idm 2>/dev/null
# Note: ARM64 kernel comes from build-kernel-arm64.sh, not from piCore
# We only use piCore for the BusyBox userland
cd "$PROJECT_ROOT"
umount "$IMG_MNT" 2>/dev/null || true
rm -rf "$IMG_MNT"
echo "==> ARM64 rootfs extracted: $ROOTFS_DIR/rootfs"
echo " Size: $(du -sh "$ROOTFS_DIR/rootfs" | cut -f1)"
echo "==> Extract complete (ARM64). Kernel will come from build-kernel-arm64.sh"
exit 0
fi
# =========================================================================
# x86_64: Tiny Core ISO extraction
# =========================================================================
TC_ISO="$CACHE_DIR/$TINYCORE_ISO" TC_ISO="$CACHE_DIR/$TINYCORE_ISO"
ISO_MNT="$ROOTFS_DIR/iso-mount" ISO_MNT="$ROOTFS_DIR/iso-mount"
@@ -19,9 +124,7 @@ if [ ! -f "$TC_ISO" ]; then
exit 1 exit 1
fi fi
# Clean previous rootfs mkdir -p "$ISO_MNT"
rm -rf "$ROOTFS_DIR"
mkdir -p "$ROOTFS_DIR" "$ISO_MNT"
# --- Mount ISO and extract kernel + initramfs --- # --- Mount ISO and extract kernel + initramfs ---
echo "==> Mounting ISO: $TC_ISO" echo "==> Mounting ISO: $TC_ISO"

View File

@@ -10,9 +10,88 @@ CACHE_DIR="${CACHE_DIR:-$PROJECT_ROOT/build/cache}"
# shellcheck source=../config/versions.env # shellcheck source=../config/versions.env
. "$SCRIPT_DIR/../config/versions.env" . "$SCRIPT_DIR/../config/versions.env"
# Verify SHA256 checksum of a downloaded file
verify_checksum() {
local file="$1" expected="$2" name="$3"
# Skip if no expected checksum provided
[ -z "$expected" ] && return 0
local actual
actual=$(sha256sum "$file" | awk '{print $1}')
if [ "$actual" = "$expected" ]; then
echo " Checksum OK: $name"
return 0
else
echo "ERROR: Checksum mismatch for $name"
echo " Expected: $expected"
echo " Got: $actual"
rm -f "$file"
return 1
fi
}
mkdir -p "$CACHE_DIR" mkdir -p "$CACHE_DIR"
# --- Tiny Core Linux ISO --- # Detect target architecture
FETCH_ARCH="${TARGET_ARCH:-amd64}"
# --- ARM64: piCore64 image instead of x86_64 ISO ---
if [ "$FETCH_ARCH" = "arm64" ]; then
PICORE_IMG="$CACHE_DIR/$PICORE_IMAGE"
if [ -f "$PICORE_IMG" ]; then
echo "==> piCore64 image already cached: $PICORE_IMG"
else
echo "==> Downloading piCore64 ${PICORE_VERSION} (${PICORE_ARCH})..."
echo " URL: $PICORE_IMAGE_URL"
wget -q --show-progress -O "$PICORE_IMG" "$PICORE_IMAGE_URL" 2>/dev/null || \
curl -fSL "$PICORE_IMAGE_URL" -o "$PICORE_IMG"
echo "==> Downloaded: $PICORE_IMG ($(du -h "$PICORE_IMG" | cut -f1))"
fi
# Also fetch RPi firmware
echo "==> Fetching RPi firmware..."
"$SCRIPT_DIR/fetch-rpi-firmware.sh"
# Download ARM64 KubeSolo binary (KUBESOLO_VERSION set from versions.env)
KUBESOLO_BIN_ARM64="$CACHE_DIR/kubesolo-arm64"
if [ -f "$KUBESOLO_BIN_ARM64" ]; then
echo "==> KubeSolo ARM64 binary already cached: $KUBESOLO_BIN_ARM64"
else
echo "==> Downloading KubeSolo ${KUBESOLO_VERSION} (arm64)..."
BIN_URL="https://github.com/portainer/kubesolo/releases/download/${KUBESOLO_VERSION}/kubesolo-${KUBESOLO_VERSION}-linux-arm64-musl.tar.gz"
BIN_URL_FALLBACK="https://github.com/portainer/kubesolo/releases/download/${KUBESOLO_VERSION}/kubesolo-${KUBESOLO_VERSION}-linux-arm64.tar.gz"
TEMP_DIR=$(mktemp -d)
echo " URL: $BIN_URL"
if curl -fSL "$BIN_URL" -o "$TEMP_DIR/kubesolo.tar.gz" 2>/dev/null; then
echo " Downloaded musl variant (arm64)"
elif curl -fSL "$BIN_URL_FALLBACK" -o "$TEMP_DIR/kubesolo.tar.gz" 2>/dev/null; then
echo " Downloaded glibc variant (arm64 fallback)"
else
echo "ERROR: Failed to download KubeSolo ARM64 from GitHub."
rm -rf "$TEMP_DIR"
exit 1
fi
tar -xzf "$TEMP_DIR/kubesolo.tar.gz" -C "$TEMP_DIR"
FOUND_BIN=$(find "$TEMP_DIR" -name "kubesolo" -type f ! -name "*.tar.gz" | head -1)
if [ -z "$FOUND_BIN" ]; then
echo "ERROR: Could not find kubesolo binary in extracted archive"
rm -rf "$TEMP_DIR"
exit 1
fi
cp "$FOUND_BIN" "$KUBESOLO_BIN_ARM64"
chmod +x "$KUBESOLO_BIN_ARM64"
rm -rf "$TEMP_DIR"
echo "==> KubeSolo ARM64 binary: $KUBESOLO_BIN_ARM64 ($(du -h "$KUBESOLO_BIN_ARM64" | cut -f1))"
fi
# Skip x86_64 ISO and TCZ downloads for ARM64
echo ""
echo "==> ARM64 fetch complete."
echo "==> Component cache:"
ls -lh "$CACHE_DIR"/ 2>/dev/null || true
exit 0
fi
# --- x86_64: Tiny Core Linux ISO ---
TC_ISO="$CACHE_DIR/$TINYCORE_ISO" TC_ISO="$CACHE_DIR/$TINYCORE_ISO"
TC_URL="${TINYCORE_MIRROR}/${TINYCORE_VERSION%%.*}.x/${TINYCORE_ARCH}/release/${TINYCORE_ISO}" TC_URL="${TINYCORE_MIRROR}/${TINYCORE_VERSION%%.*}.x/${TINYCORE_ARCH}/release/${TINYCORE_ISO}"
@@ -28,10 +107,11 @@ else
wget -q --show-progress -O "$TC_ISO" "$TC_URL_ALT" wget -q --show-progress -O "$TC_ISO" "$TC_URL_ALT"
} }
echo "==> Downloaded: $TC_ISO ($(du -h "$TC_ISO" | cut -f1))" echo "==> Downloaded: $TC_ISO ($(du -h "$TC_ISO" | cut -f1))"
verify_checksum "$TC_ISO" "$TINYCORE_ISO_SHA256" "Tiny Core ISO"
fi fi
# --- KubeSolo --- # --- KubeSolo ---
KUBESOLO_VERSION="${KUBESOLO_VERSION:-v1.1.0}" # KUBESOLO_VERSION sourced from versions.env
KUBESOLO_BIN="$CACHE_DIR/kubesolo" KUBESOLO_BIN="$CACHE_DIR/kubesolo"
if [ -f "$KUBESOLO_BIN" ]; then if [ -f "$KUBESOLO_BIN" ]; then
@@ -50,7 +130,7 @@ else
BIN_URL_FALLBACK="https://github.com/portainer/kubesolo/releases/download/${KUBESOLO_VERSION}/kubesolo-${KUBESOLO_VERSION}-${OS}-${ARCH}.tar.gz" BIN_URL_FALLBACK="https://github.com/portainer/kubesolo/releases/download/${KUBESOLO_VERSION}/kubesolo-${KUBESOLO_VERSION}-${OS}-${ARCH}.tar.gz"
TEMP_DIR=$(mktemp -d) TEMP_DIR=$(mktemp -d)
trap "rm -rf '$TEMP_DIR'" EXIT trap 'rm -rf "$TEMP_DIR"' EXIT
echo " URL: $BIN_URL" echo " URL: $BIN_URL"
if curl -fSL "$BIN_URL" -o "$TEMP_DIR/kubesolo.tar.gz" 2>/dev/null; then if curl -fSL "$BIN_URL" -o "$TEMP_DIR/kubesolo.tar.gz" 2>/dev/null; then
@@ -88,6 +168,7 @@ else
rm -rf "$TEMP_DIR" rm -rf "$TEMP_DIR"
echo "==> KubeSolo binary: $KUBESOLO_BIN ($(du -h "$KUBESOLO_BIN" | cut -f1))" echo "==> KubeSolo binary: $KUBESOLO_BIN ($(du -h "$KUBESOLO_BIN" | cut -f1))"
verify_checksum "$KUBESOLO_BIN" "$KUBESOLO_SHA256" "KubeSolo binary"
fi fi
# --- Tiny Core kernel module extensions (netfilter, iptables) --- # --- Tiny Core kernel module extensions (netfilter, iptables) ---
@@ -114,6 +195,7 @@ else
if wget -q --show-progress -O "$NETFILTER_TCZ" "$NETFILTER_TCZ_URL" 2>/dev/null || \ if wget -q --show-progress -O "$NETFILTER_TCZ" "$NETFILTER_TCZ_URL" 2>/dev/null || \
curl -fSL "$NETFILTER_TCZ_URL" -o "$NETFILTER_TCZ" 2>/dev/null; then curl -fSL "$NETFILTER_TCZ_URL" -o "$NETFILTER_TCZ" 2>/dev/null; then
echo "==> Downloaded: $NETFILTER_TCZ ($(du -h "$NETFILTER_TCZ" | cut -f1))" echo "==> Downloaded: $NETFILTER_TCZ ($(du -h "$NETFILTER_TCZ" | cut -f1))"
verify_checksum "$NETFILTER_TCZ" "$NETFILTER_TCZ_SHA256" "netfilter TCZ"
else else
echo "WARN: Failed to download netfilter modules. kube-proxy may not work." echo "WARN: Failed to download netfilter modules. kube-proxy may not work."
rm -f "$NETFILTER_TCZ" rm -f "$NETFILTER_TCZ"
@@ -131,6 +213,7 @@ else
if wget -q --show-progress -O "$NET_BRIDGING_TCZ" "$NET_BRIDGING_TCZ_URL" 2>/dev/null || \ if wget -q --show-progress -O "$NET_BRIDGING_TCZ" "$NET_BRIDGING_TCZ_URL" 2>/dev/null || \
curl -fSL "$NET_BRIDGING_TCZ_URL" -o "$NET_BRIDGING_TCZ" 2>/dev/null; then curl -fSL "$NET_BRIDGING_TCZ_URL" -o "$NET_BRIDGING_TCZ" 2>/dev/null; then
echo "==> Downloaded: $NET_BRIDGING_TCZ ($(du -h "$NET_BRIDGING_TCZ" | cut -f1))" echo "==> Downloaded: $NET_BRIDGING_TCZ ($(du -h "$NET_BRIDGING_TCZ" | cut -f1))"
verify_checksum "$NET_BRIDGING_TCZ" "$NET_BRIDGING_TCZ_SHA256" "net-bridging TCZ"
else else
echo "WARN: Failed to download net-bridging modules. CNI bridge may not work." echo "WARN: Failed to download net-bridging modules. CNI bridge may not work."
rm -f "$NET_BRIDGING_TCZ" rm -f "$NET_BRIDGING_TCZ"
@@ -148,6 +231,7 @@ else
if wget -q --show-progress -O "$IPTABLES_TCZ" "$IPTABLES_TCZ_URL" 2>/dev/null || \ if wget -q --show-progress -O "$IPTABLES_TCZ" "$IPTABLES_TCZ_URL" 2>/dev/null || \
curl -fSL "$IPTABLES_TCZ_URL" -o "$IPTABLES_TCZ" 2>/dev/null; then curl -fSL "$IPTABLES_TCZ_URL" -o "$IPTABLES_TCZ" 2>/dev/null; then
echo "==> Downloaded: $IPTABLES_TCZ ($(du -h "$IPTABLES_TCZ" | cut -f1))" echo "==> Downloaded: $IPTABLES_TCZ ($(du -h "$IPTABLES_TCZ" | cut -f1))"
verify_checksum "$IPTABLES_TCZ" "$IPTABLES_TCZ_SHA256" "iptables TCZ"
else else
echo "WARN: Failed to download iptables. KubeSolo bundles its own but this is a fallback." echo "WARN: Failed to download iptables. KubeSolo bundles its own but this is a fallback."
rm -f "$IPTABLES_TCZ" rm -f "$IPTABLES_TCZ"

View File

@@ -0,0 +1,88 @@
#!/bin/bash
# fetch-rpi-firmware.sh — Download Raspberry Pi firmware blobs for boot
#
# Downloads firmware from the official raspberrypi/firmware GitHub repository.
# Extracts only the boot files needed: start*.elf, fixup*.dat, DTBs, bootcode.bin.
#
# Output: build/cache/rpi-firmware/ containing all required boot files.
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
CACHE_DIR="${CACHE_DIR:-$PROJECT_ROOT/build/cache}"
# shellcheck source=../config/versions.env
. "$SCRIPT_DIR/../config/versions.env"
RPI_FW_DIR="$CACHE_DIR/rpi-firmware"
RPI_FW_ARCHIVE="$CACHE_DIR/rpi-firmware-${RPI_FIRMWARE_TAG}.tar.gz"
# --- Skip if already fetched ---
if [ -d "$RPI_FW_DIR" ] && [ -f "$RPI_FW_DIR/start4.elf" ]; then
echo "==> RPi firmware already cached: $RPI_FW_DIR"
echo " Files: $(ls "$RPI_FW_DIR" | wc -l)"
exit 0
fi
echo "==> Downloading Raspberry Pi firmware (tag: ${RPI_FIRMWARE_TAG})..."
mkdir -p "$CACHE_DIR" "$RPI_FW_DIR"
# --- Download firmware archive ---
if [ ! -f "$RPI_FW_ARCHIVE" ]; then
echo " URL: $RPI_FIRMWARE_URL"
wget -q --show-progress -O "$RPI_FW_ARCHIVE" "$RPI_FIRMWARE_URL" 2>/dev/null || \
curl -fSL "$RPI_FIRMWARE_URL" -o "$RPI_FW_ARCHIVE"
echo " Downloaded: $(du -h "$RPI_FW_ARCHIVE" | cut -f1)"
else
echo " Archive already cached: $(du -h "$RPI_FW_ARCHIVE" | cut -f1)"
fi
# --- Extract boot files only ---
echo "==> Extracting boot files..."
TEMP_DIR=$(mktemp -d)
trap 'rm -rf "$TEMP_DIR"' EXIT
# Extract only the boot/ directory from the archive
# Archive structure: firmware-<tag>/boot/...
tar -xzf "$RPI_FW_ARCHIVE" -C "$TEMP_DIR" --strip-components=1 --wildcards '*/boot/'
BOOT_SRC="$TEMP_DIR/boot"
if [ ! -d "$BOOT_SRC" ]; then
echo "ERROR: boot/ directory not found in firmware archive"
ls -la "$TEMP_DIR"/
exit 1
fi
# Copy GPU firmware (required for boot)
for f in "$BOOT_SRC"/start*.elf "$BOOT_SRC"/fixup*.dat; do
[ -f "$f" ] && cp "$f" "$RPI_FW_DIR/"
done
# Copy bootcode.bin (first-stage boot for Pi 3 and older)
[ -f "$BOOT_SRC/bootcode.bin" ] && cp "$BOOT_SRC/bootcode.bin" "$RPI_FW_DIR/"
# Copy Device Tree Blobs for Pi 4 + Pi 5
for dtb in bcm2711-rpi-4-b.dtb bcm2711-rpi-400.dtb bcm2711-rpi-cm4.dtb \
bcm2712-rpi-5-b.dtb bcm2712d0-rpi-5-b.dtb; do
[ -f "$BOOT_SRC/$dtb" ] && cp "$BOOT_SRC/$dtb" "$RPI_FW_DIR/"
done
# Copy overlays directory (needed for config.txt dtoverlay= directives)
if [ -d "$BOOT_SRC/overlays" ]; then
mkdir -p "$RPI_FW_DIR/overlays"
# Only copy overlays we actually use (disable-wifi, disable-bt)
for overlay in disable-wifi.dtbo disable-bt.dtbo; do
[ -f "$BOOT_SRC/overlays/$overlay" ] && \
cp "$BOOT_SRC/overlays/$overlay" "$RPI_FW_DIR/overlays/"
done
fi
trap - EXIT
rm -rf "$TEMP_DIR"
# --- Summary ---
echo ""
echo "==> RPi firmware extracted to: $RPI_FW_DIR"
echo " Files:"
ls -1 "$RPI_FW_DIR" | head -20
echo " Total size: $(du -sh "$RPI_FW_DIR" | cut -f1)"

View File

@@ -8,6 +8,16 @@ CACHE_DIR="${CACHE_DIR:-$PROJECT_ROOT/build/cache}"
ROOTFS_DIR="${ROOTFS_DIR:-$PROJECT_ROOT/build/rootfs-work}" ROOTFS_DIR="${ROOTFS_DIR:-$PROJECT_ROOT/build/rootfs-work}"
ROOTFS="$ROOTFS_DIR/rootfs" ROOTFS="$ROOTFS_DIR/rootfs"
VERSION="$(cat "$PROJECT_ROOT/VERSION")" VERSION="$(cat "$PROJECT_ROOT/VERSION")"
INJECT_ARCH="${TARGET_ARCH:-amd64}"
# Architecture-specific paths
if [ "$INJECT_ARCH" = "arm64" ]; then
LIB_ARCH="aarch64-linux-gnu"
LD_SO="/lib/ld-linux-aarch64.so.1"
else
LIB_ARCH="x86_64-linux-gnu"
LD_SO="/lib64/ld-linux-x86-64.so.2"
fi
if [ ! -d "$ROOTFS" ]; then if [ ! -d "$ROOTFS" ]; then
echo "ERROR: Rootfs not found: $ROOTFS" echo "ERROR: Rootfs not found: $ROOTFS"
@@ -15,7 +25,11 @@ if [ ! -d "$ROOTFS" ]; then
exit 1 exit 1
fi fi
if [ "$INJECT_ARCH" = "arm64" ]; then
KUBESOLO_BIN="$CACHE_DIR/kubesolo-arm64"
else
KUBESOLO_BIN="$CACHE_DIR/kubesolo" KUBESOLO_BIN="$CACHE_DIR/kubesolo"
fi
if [ ! -f "$KUBESOLO_BIN" ]; then if [ ! -f "$KUBESOLO_BIN" ]; then
echo "ERROR: KubeSolo binary not found: $KUBESOLO_BIN" echo "ERROR: KubeSolo binary not found: $KUBESOLO_BIN"
echo "See fetch-components.sh output for instructions." echo "See fetch-components.sh output for instructions."
@@ -68,30 +82,51 @@ for lib in network.sh health.sh; do
done done
# Cloud-init binary (Go, built separately) # Cloud-init binary (Go, built separately)
CLOUDINIT_BIN="$CACHE_DIR/kubesolo-cloudinit" # Try arch-specific binary first, then fall back to generic
CLOUDINIT_BIN="$CACHE_DIR/kubesolo-cloudinit-linux-$INJECT_ARCH"
[ ! -f "$CLOUDINIT_BIN" ] && CLOUDINIT_BIN="$CACHE_DIR/kubesolo-cloudinit"
if [ -f "$CLOUDINIT_BIN" ]; then if [ -f "$CLOUDINIT_BIN" ]; then
cp "$CLOUDINIT_BIN" "$ROOTFS/usr/lib/kubesolo-os/kubesolo-cloudinit" cp "$CLOUDINIT_BIN" "$ROOTFS/usr/lib/kubesolo-os/kubesolo-cloudinit"
chmod +x "$ROOTFS/usr/lib/kubesolo-os/kubesolo-cloudinit" chmod +x "$ROOTFS/usr/lib/kubesolo-os/kubesolo-cloudinit"
echo " Installed cloud-init binary ($(du -h "$CLOUDINIT_BIN" | cut -f1))" echo " Installed cloud-init binary ($(du -h "$CLOUDINIT_BIN" | cut -f1))"
else else
echo " WARN: Cloud-init binary not found (run 'make build-cloudinit' to build)" echo " WARN: Cloud-init binary not found (run 'make build-cloudinit' or 'make build-cross' to build)"
fi fi
# Update agent binary (Go, built separately) # Update agent binary (Go, built separately)
UPDATE_BIN="$CACHE_DIR/kubesolo-update" # Try arch-specific binary first, then fall back to generic
UPDATE_BIN="$CACHE_DIR/kubesolo-update-linux-$INJECT_ARCH"
[ ! -f "$UPDATE_BIN" ] && UPDATE_BIN="$CACHE_DIR/kubesolo-update"
if [ -f "$UPDATE_BIN" ]; then if [ -f "$UPDATE_BIN" ]; then
cp "$UPDATE_BIN" "$ROOTFS/usr/lib/kubesolo-os/kubesolo-update" cp "$UPDATE_BIN" "$ROOTFS/usr/lib/kubesolo-os/kubesolo-update"
chmod +x "$ROOTFS/usr/lib/kubesolo-os/kubesolo-update" chmod +x "$ROOTFS/usr/lib/kubesolo-os/kubesolo-update"
echo " Installed update agent ($(du -h "$UPDATE_BIN" | cut -f1))" echo " Installed update agent ($(du -h "$UPDATE_BIN" | cut -f1))"
else else
echo " WARN: Update agent not found (run 'make build-update-agent' to build)" echo " WARN: Update agent not found (run 'make build-update-agent' or 'make build-cross' to build)"
fi fi
# --- 3. Custom kernel or TCZ kernel modules --- # --- 3. Custom kernel or TCZ kernel modules ---
# If a custom kernel was built (with CONFIG_CGROUP_BPF=y), use it. # If a custom kernel was built (with CONFIG_CGROUP_BPF=y), use it.
# Otherwise fall back to TCZ-extracted modules with manual modules.dep. # Otherwise fall back to TCZ-extracted modules with manual modules.dep.
if [ "$INJECT_ARCH" = "arm64" ]; then
# TARGET_VARIANT selects which ARM64 kernel to consume:
# rpi -> $CACHE_DIR/custom-kernel-rpi/ (raspberrypi/linux fork)
# generic -> $CACHE_DIR/kernel-arm64-generic/ (mainline kernel.org LTS)
# Default is rpi for backwards compatibility with existing rpi-image target.
TARGET_VARIANT="${TARGET_VARIANT:-rpi}"
case "$TARGET_VARIANT" in
generic) CUSTOM_KERNEL_DIR="$CACHE_DIR/kernel-arm64-generic" ;;
rpi) CUSTOM_KERNEL_DIR="$CACHE_DIR/custom-kernel-rpi" ;;
*)
echo "ERROR: TARGET_VARIANT must be 'rpi' or 'generic' (got: $TARGET_VARIANT)"
exit 1
;;
esac
CUSTOM_VMLINUZ="$CUSTOM_KERNEL_DIR/Image"
else
CUSTOM_KERNEL_DIR="$CACHE_DIR/custom-kernel" CUSTOM_KERNEL_DIR="$CACHE_DIR/custom-kernel"
CUSTOM_VMLINUZ="$CUSTOM_KERNEL_DIR/vmlinuz" CUSTOM_VMLINUZ="$CUSTOM_KERNEL_DIR/vmlinuz"
fi
CUSTOM_MODULES="$CUSTOM_KERNEL_DIR/modules" CUSTOM_MODULES="$CUSTOM_KERNEL_DIR/modules"
# Detect kernel version from rootfs # Detect kernel version from rootfs
@@ -100,8 +135,16 @@ for d in "$ROOTFS"/lib/modules/*/; do
[ -d "$d" ] && KVER="$(basename "$d")" && break [ -d "$d" ] && KVER="$(basename "$d")" && break
done done
# Fallback: detect from custom kernel modules directory
if [ -z "$KVER" ] && [ -d "$CUSTOM_MODULES/lib/modules" ]; then
for d in "$CUSTOM_MODULES"/lib/modules/*/; do
[ -d "$d" ] && KVER="$(basename "$d")" && break
done
echo " Detected kernel version from custom kernel: $KVER"
fi
if [ -z "$KVER" ]; then if [ -z "$KVER" ]; then
echo " WARN: Could not detect kernel version from rootfs" echo " WARN: Could not detect kernel version from rootfs or custom kernel"
fi fi
echo " Kernel version: $KVER" echo " Kernel version: $KVER"
@@ -130,20 +173,49 @@ if [ -f "$CUSTOM_VMLINUZ" ] && [ -d "$CUSTOM_MODULES/lib/modules/$KVER" ]; then
[ -f "$CUSTOM_MOD_DIR/$f" ] && cp "$CUSTOM_MOD_DIR/$f" "$ROOTFS/lib/modules/$KVER/" [ -f "$CUSTOM_MOD_DIR/$f" ] && cp "$CUSTOM_MOD_DIR/$f" "$ROOTFS/lib/modules/$KVER/"
done done
# Use modprobe --show-depends to resolve each module + its transitive deps # Resolve and install modules from modules.list + transitive deps
if [ "$INJECT_ARCH" = "arm64" ]; then
MODULES_LIST="$PROJECT_ROOT/build/config/modules-arm64.list"
else
MODULES_LIST="$PROJECT_ROOT/build/config/modules.list" MODULES_LIST="$PROJECT_ROOT/build/config/modules.list"
fi
NEEDED_MODS=$(mktemp) NEEDED_MODS=$(mktemp)
# Try modprobe first (works for same-arch builds)
MODPROBE_WORKS=true
FIRST_MOD=$(grep -v '^#' "$MODULES_LIST" | grep -v '^$' | head -1 | xargs)
if ! modprobe -S "$KVER" -d "$CUSTOM_MODULES" --show-depends "$FIRST_MOD" >/dev/null 2>&1; then
MODPROBE_WORKS=false
echo " modprobe cannot resolve modules (cross-arch build) — using find fallback"
fi
while IFS= read -r mod; do while IFS= read -r mod; do
# Skip comments and blank lines # Skip comments and blank lines
case "$mod" in \#*|"") continue ;; esac case "$mod" in \#*|"") continue ;; esac
mod=$(echo "$mod" | xargs) # trim whitespace mod=$(echo "$mod" | xargs) # trim whitespace
[ -z "$mod" ] && continue [ -z "$mod" ] && continue
if [ "$MODPROBE_WORKS" = true ]; then
# modprobe -S <ver> -d <root> --show-depends <module> lists all deps in load order # modprobe -S <ver> -d <root> --show-depends <module> lists all deps in load order
# Output format: "insmod /path/to/module.ko" — extract path with awk
modprobe -S "$KVER" -d "$CUSTOM_MODULES" --show-depends "$mod" 2>/dev/null \ modprobe -S "$KVER" -d "$CUSTOM_MODULES" --show-depends "$mod" 2>/dev/null \
| awk '/^insmod/{print $2}' >> "$NEEDED_MODS" \ | awk '/^insmod/{print $2}' >> "$NEEDED_MODS" \
|| echo " WARN: modprobe could not resolve: $mod" || echo " WARN: modprobe could not resolve: $mod"
else
# Cross-arch fallback: find module by name in kernel tree
found=$(find "$CUSTOM_MOD_DIR/kernel" -name "${mod}.ko" -o -name "${mod}.ko.xz" -o -name "${mod}.ko.gz" -o -name "${mod}.ko.zst" 2>/dev/null | head -1)
if [ -n "$found" ]; then
echo "$found" >> "$NEEDED_MODS"
else
# Try replacing hyphens with underscores and vice versa
mod_alt=$(echo "$mod" | tr '-' '_')
found=$(find "$CUSTOM_MOD_DIR/kernel" -name "${mod_alt}.ko" -o -name "${mod_alt}.ko.xz" -o -name "${mod_alt}.ko.gz" -o -name "${mod_alt}.ko.zst" 2>/dev/null | head -1)
if [ -n "$found" ]; then
echo "$found" >> "$NEEDED_MODS"
else
echo " WARN: could not find module: $mod"
fi
fi
fi
done < "$MODULES_LIST" done < "$MODULES_LIST"
# Deduplicate and copy each needed module # Deduplicate and copy each needed module
@@ -291,21 +363,22 @@ if [ -f /usr/sbin/xtables-nft-multi ]; then
ln -sf xtables-nft-multi "$ROOTFS/usr/sbin/$cmd" ln -sf xtables-nft-multi "$ROOTFS/usr/sbin/$cmd"
done done
# Copy required shared libraries # Copy required shared libraries (architecture-aware paths)
mkdir -p "$ROOTFS/usr/lib/x86_64-linux-gnu" "$ROOTFS/lib/x86_64-linux-gnu" "$ROOTFS/lib64" mkdir -p "$ROOTFS/usr/lib/$LIB_ARCH" "$ROOTFS/lib/$LIB_ARCH"
[ "$INJECT_ARCH" != "arm64" ] && mkdir -p "$ROOTFS/lib64"
for lib in \ for lib in \
/lib/x86_64-linux-gnu/libxtables.so.12* \ "/lib/$LIB_ARCH/libxtables.so.12"* \
/lib/x86_64-linux-gnu/libmnl.so.0* \ "/lib/$LIB_ARCH/libmnl.so.0"* \
/lib/x86_64-linux-gnu/libnftnl.so.11* \ "/lib/$LIB_ARCH/libnftnl.so.11"* \
/lib/x86_64-linux-gnu/libc.so.6 \ "/lib/$LIB_ARCH/libc.so.6" \
/lib64/ld-linux-x86-64.so.2; do "$LD_SO"; do
[ -e "$lib" ] && cp -aL "$lib" "$ROOTFS${lib}" 2>/dev/null || true [ -e "$lib" ] && cp -aL "$lib" "$ROOTFS${lib}" 2>/dev/null || true
done done
# Copy xtables modules directory (match extensions) # Copy xtables modules directory (match extensions)
if [ -d /usr/lib/x86_64-linux-gnu/xtables ]; then if [ -d "/usr/lib/$LIB_ARCH/xtables" ]; then
mkdir -p "$ROOTFS/usr/lib/x86_64-linux-gnu/xtables" mkdir -p "$ROOTFS/usr/lib/$LIB_ARCH/xtables"
cp -a /usr/lib/x86_64-linux-gnu/xtables/*.so "$ROOTFS/usr/lib/x86_64-linux-gnu/xtables/" 2>/dev/null || true cp -a "/usr/lib/$LIB_ARCH/xtables/"*.so "$ROOTFS/usr/lib/$LIB_ARCH/xtables/" 2>/dev/null || true
fi fi
echo " Installed iptables-nft (xtables-nft-multi) + shared libs" echo " Installed iptables-nft (xtables-nft-multi) + shared libs"
@@ -314,11 +387,16 @@ else
fi fi
# Kernel modules list (for init to load at boot) # Kernel modules list (for init to load at boot)
if [ "$INJECT_ARCH" = "arm64" ]; then
cp "$PROJECT_ROOT/build/config/modules-arm64.list" "$ROOTFS/usr/lib/kubesolo-os/modules.list"
else
cp "$PROJECT_ROOT/build/config/modules.list" "$ROOTFS/usr/lib/kubesolo-os/modules.list" cp "$PROJECT_ROOT/build/config/modules.list" "$ROOTFS/usr/lib/kubesolo-os/modules.list"
fi
# --- 4. Sysctl config --- # --- 4. Sysctl config ---
mkdir -p "$ROOTFS/etc/sysctl.d" mkdir -p "$ROOTFS/etc/sysctl.d"
cp "$PROJECT_ROOT/build/rootfs/etc/sysctl.d/k8s.conf" "$ROOTFS/etc/sysctl.d/k8s.conf" cp "$PROJECT_ROOT/build/rootfs/etc/sysctl.d/k8s.conf" "$ROOTFS/etc/sysctl.d/k8s.conf"
cp "$PROJECT_ROOT/build/rootfs/etc/sysctl.d/security.conf" "$ROOTFS/etc/sysctl.d/security.conf"
# --- 5. OS metadata --- # --- 5. OS metadata ---
echo "$VERSION" > "$ROOTFS/etc/kubesolo-os-version" echo "$VERSION" > "$ROOTFS/etc/kubesolo-os-version"
@@ -362,7 +440,35 @@ else
echo " WARN: No CA certificates found in builder — TLS verification will fail" echo " WARN: No CA certificates found in builder — TLS verification will fail"
fi fi
# --- 9. Ensure /etc/hosts and /etc/resolv.conf exist --- # --- 9. AppArmor parser + profiles ---
echo " Installing AppArmor..."
if [ -f /usr/sbin/apparmor_parser ]; then
mkdir -p "$ROOTFS/usr/sbin"
cp /usr/sbin/apparmor_parser "$ROOTFS/usr/sbin/apparmor_parser"
chmod +x "$ROOTFS/usr/sbin/apparmor_parser"
# Copy shared libraries required by apparmor_parser
for lib in "/lib/$LIB_ARCH/libapparmor.so.1"*; do
[ -e "$lib" ] && cp -aL "$lib" "$ROOTFS${lib}" 2>/dev/null || true
done
echo " Installed apparmor_parser + shared libs"
else
echo " WARN: apparmor_parser not found in builder (install apparmor package)"
fi
# Copy AppArmor profiles
APPARMOR_PROFILES="$PROJECT_ROOT/build/rootfs/etc/apparmor.d"
if [ -d "$APPARMOR_PROFILES" ]; then
mkdir -p "$ROOTFS/etc/apparmor.d"
cp "$APPARMOR_PROFILES"/* "$ROOTFS/etc/apparmor.d/" 2>/dev/null || true
PROFILE_COUNT=$(ls "$ROOTFS/etc/apparmor.d/" 2>/dev/null | wc -l)
echo " Installed $PROFILE_COUNT AppArmor profiles"
else
echo " WARN: No AppArmor profiles found at $APPARMOR_PROFILES"
fi
# --- 10. Ensure /etc/hosts and /etc/resolv.conf exist ---
if [ ! -f "$ROOTFS/etc/hosts" ]; then if [ ! -f "$ROOTFS/etc/hosts" ]; then
cat > "$ROOTFS/etc/hosts" << EOF cat > "$ROOTFS/etc/hosts" << EOF
127.0.0.1 localhost 127.0.0.1 localhost

View File

@@ -33,7 +33,13 @@ type NetworkConfig struct {
type KubeSoloConfig struct { type KubeSoloConfig struct {
ExtraFlags string `yaml:"extra-flags"` ExtraFlags string `yaml:"extra-flags"`
LocalStorage *bool `yaml:"local-storage"` LocalStorage *bool `yaml:"local-storage"`
LocalStorageSharedPath string `yaml:"local-storage-shared-path"`
ExtraSANs []string `yaml:"apiserver-extra-sans"` ExtraSANs []string `yaml:"apiserver-extra-sans"`
Debug bool `yaml:"debug"`
PprofServer bool `yaml:"pprof-server"`
PortainerEdgeID string `yaml:"portainer-edge-id"`
PortainerEdgeKey string `yaml:"portainer-edge-key"`
PortainerEdgeAsync bool `yaml:"portainer-edge-async"`
} }
// NTPConfig defines NTP settings. // NTPConfig defines NTP settings.

View File

@@ -0,0 +1,40 @@
# KubeSolo OS Cloud-Init — Full Configuration Reference
# Shows ALL supported KubeSolo parameters.
# Place at: /mnt/data/etc-kubesolo/cloud-init.yaml (on data partition)
# Or pass via boot param: kubesolo.cloudinit=/path/to/this.yaml
hostname: kubesolo-edge-01
network:
mode: dhcp
# interface: eth0 # Optional: specify interface (auto-detected if omitted)
# dns: # Optional: override DHCP-provided DNS
# - 8.8.8.8
kubesolo:
# Enable local-path-provisioner for persistent volumes (default: true)
local-storage: true
# Shared path for local-path-provisioner storage
local-storage-shared-path: "/mnt/shared"
# Extra SANs for API server TLS certificate
apiserver-extra-sans:
- kubesolo-edge-01.local
- 192.168.1.100
# Enable verbose debug logging
debug: false
# Enable Go pprof profiling server
pprof-server: false
# Portainer Edge Agent connection (alternative to portainer.edge-agent section)
# These generate --portainer-edge-id, --portainer-edge-key, --portainer-edge-async
# CLI flags for KubeSolo's built-in Edge Agent support.
portainer-edge-id: "your-edge-id"
portainer-edge-key: "your-edge-key"
portainer-edge-async: true
# Arbitrary extra flags passed directly to the KubeSolo binary
# extra-flags: "--disable traefik --disable servicelb"

View File

@@ -46,6 +46,30 @@ func buildExtraFlags(cfg *Config) string {
parts = append(parts, "--apiserver-extra-sans", san) parts = append(parts, "--apiserver-extra-sans", san)
} }
if cfg.KubeSolo.LocalStorageSharedPath != "" {
parts = append(parts, "--local-storage-shared-path", cfg.KubeSolo.LocalStorageSharedPath)
}
if cfg.KubeSolo.Debug {
parts = append(parts, "--debug")
}
if cfg.KubeSolo.PprofServer {
parts = append(parts, "--pprof-server")
}
if cfg.KubeSolo.PortainerEdgeID != "" {
parts = append(parts, "--portainer-edge-id", cfg.KubeSolo.PortainerEdgeID)
}
if cfg.KubeSolo.PortainerEdgeKey != "" {
parts = append(parts, "--portainer-edge-key", cfg.KubeSolo.PortainerEdgeKey)
}
if cfg.KubeSolo.PortainerEdgeAsync {
parts = append(parts, "--portainer-edge-async")
}
return strings.Join(parts, " ") return strings.Join(parts, " ")
} }

View File

@@ -44,6 +44,54 @@ func TestBuildExtraFlags(t *testing.T) {
}, },
want: "--disable servicelb --apiserver-extra-sans edge.local", want: "--disable servicelb --apiserver-extra-sans edge.local",
}, },
{
name: "debug flag",
cfg: Config{
KubeSolo: KubeSoloConfig{Debug: true},
},
want: "--debug",
},
{
name: "pprof-server flag",
cfg: Config{
KubeSolo: KubeSoloConfig{PprofServer: true},
},
want: "--pprof-server",
},
{
name: "local-storage-shared-path",
cfg: Config{
KubeSolo: KubeSoloConfig{LocalStorageSharedPath: "/mnt/shared"},
},
want: "--local-storage-shared-path /mnt/shared",
},
{
name: "portainer edge flags",
cfg: Config{
KubeSolo: KubeSoloConfig{
PortainerEdgeID: "test-id-123",
PortainerEdgeKey: "test-key-456",
PortainerEdgeAsync: true,
},
},
want: "--portainer-edge-id test-id-123 --portainer-edge-key test-key-456 --portainer-edge-async",
},
{
name: "all new flags",
cfg: Config{
KubeSolo: KubeSoloConfig{
ExtraFlags: "--disable traefik",
ExtraSANs: []string{"node.local"},
LocalStorageSharedPath: "/mnt/data/shared",
Debug: true,
PprofServer: true,
PortainerEdgeID: "eid",
PortainerEdgeKey: "ekey",
PortainerEdgeAsync: true,
},
},
want: "--disable traefik --apiserver-extra-sans node.local --local-storage-shared-path /mnt/data/shared --debug --pprof-server --portainer-edge-id eid --portainer-edge-key ekey --portainer-edge-async",
},
} }
for _, tt := range tests { for _, tt := range tests {
@@ -64,6 +112,11 @@ func TestApplyKubeSolo(t *testing.T) {
ExtraFlags: "--disable traefik", ExtraFlags: "--disable traefik",
LocalStorage: &tr, LocalStorage: &tr,
ExtraSANs: []string{"test.local"}, ExtraSANs: []string{"test.local"},
LocalStorageSharedPath: "/mnt/shared",
Debug: true,
PortainerEdgeID: "eid",
PortainerEdgeKey: "ekey",
PortainerEdgeAsync: true,
}, },
} }
@@ -83,6 +136,21 @@ func TestApplyKubeSolo(t *testing.T) {
if !strings.Contains(flags, "--apiserver-extra-sans test.local") { if !strings.Contains(flags, "--apiserver-extra-sans test.local") {
t.Errorf("extra-flags missing SANs: %q", flags) t.Errorf("extra-flags missing SANs: %q", flags)
} }
if !strings.Contains(flags, "--local-storage-shared-path /mnt/shared") {
t.Errorf("extra-flags missing local-storage-shared-path: %q", flags)
}
if !strings.Contains(flags, "--debug") {
t.Errorf("extra-flags missing --debug: %q", flags)
}
if !strings.Contains(flags, "--portainer-edge-id eid") {
t.Errorf("extra-flags missing --portainer-edge-id: %q", flags)
}
if !strings.Contains(flags, "--portainer-edge-key ekey") {
t.Errorf("extra-flags missing --portainer-edge-key: %q", flags)
}
if !strings.Contains(flags, "--portainer-edge-async") {
t.Errorf("extra-flags missing --portainer-edge-async: %q", flags)
}
// Check config.yaml // Check config.yaml
configData, err := os.ReadFile(filepath.Join(dir, "config.yaml")) configData, err := os.ReadFile(filepath.Join(dir, "config.yaml"))

View File

@@ -225,6 +225,7 @@ func TestParseExampleFiles(t *testing.T) {
"examples/static-ip.yaml", "examples/static-ip.yaml",
"examples/portainer-edge.yaml", "examples/portainer-edge.yaml",
"examples/airgapped.yaml", "examples/airgapped.yaml",
"examples/full-config.yaml",
} }
for _, path := range examples { for _, path := range examples {

124
docs/arm64-architecture.md Normal file
View File

@@ -0,0 +1,124 @@
# ARM64 Build Architecture
KubeSolo OS supports ARM64 via two distinct build tracks. This document defines the
split, lists which files belong to each track, and identifies the shared substrate.
## The two tracks
### Generic ARM64 (UEFI / virtio / GRUB)
**Target:** Any UEFI-compliant ARM64 host — Ampere/Graviton VMs, generic ARM64
servers, `qemu-system-aarch64 -machine virt`, future SBCs that boot via UEFI.
**Boot path:** UEFI firmware → GRUB-EFI → kernel + initramfs → KubeSolo init.
**Kernel:** Mainline Linux (kernel.org LTS), built from `defconfig` + shared
container-config fragment.
**Storage:** virtio-blk / NVMe / SATA — detected and probed by mainline drivers.
**Disk image format:** GPT, identical 4-partition layout to x86_64 (EFI + System A
+ System B + Data).
### Raspberry Pi ARM64
**Target:** Raspberry Pi 4 and 5 specifically.
**Boot path:** RPi EEPROM → VideoCore firmware (`start4.elf`) → `config.txt`
kernel + DTB + initramfs → KubeSolo init. (No UEFI, no GRUB — `autoboot.txt`
provides the A/B selection.)
**Kernel:** Built from `raspberrypi/linux` fork with `bcm2711_defconfig`
(Pi 4) or `bcm2712_defconfig` (Pi 5). RPi-patched, includes BCM-specific drivers
(sdhci-iproc, bcm2835-mmc, GPIO, mailbox).
**Storage:** SD card via `sdhci-iproc` driver — requires kernel-built DTBs to match
the kernel binary.
**Disk image format:** MBR with `autoboot.txt` A/B redirect:
- Part 1: Boot/Control (FAT32, firmware + fallback kernel)
- Part 2: Boot A (FAT32, kernel + DTBs + initramfs)
- Part 3: Boot B (FAT32, same as A initially)
- Part 4: Data (ext4)
## File-by-file ownership
### Shared substrate (used by both tracks)
| Path | Why shared |
|------|------------|
| `init/` (all of it) | Boot is identical post-kernel — same staged init, same persistent mount, same KubeSolo launch |
| `cloud-init/` | Arch-agnostic Go binary |
| `update/` | Arch-agnostic Go binary; bootenv abstraction handles GRUB vs RPi-autoboot variants |
| `build/scripts/inject-kubesolo.sh` | Single script; switches `LIB_ARCH` / `LD_SO` based on `TARGET_ARCH` |
| `build/scripts/extract-core.sh` | Single script; arm64 branch uses piCore64 userland (arch-agnostic BusyBox) |
| `build/config/modules-arm64.list` | Already generic — no BCM-specific modules; works in QEMU virt, AWS Graviton, and RPi |
| `build/config/rpi-kernel-config.fragment` | **Misnamed.** Contents (cgroup, namespaces, netfilter, AppArmor) are arch-agnostic. Will be renamed `kernel-container.fragment` in Phase 2 and applied to x86, generic-ARM64, and RPi kernels alike. |
| `hack/dev-vm-arm64.sh` | Uses `-machine virt` + virtio — generic, not RPi-specific |
| `test/qemu/test-boot-arm64.sh` | Same as above |
### Generic ARM64 only (to be created in Phases 23)
| Path | Purpose |
|------|---------|
| `build/scripts/build-kernel-arm64.sh` *(rewritten in Phase 2)* | Build mainline kernel.org LTS from `defconfig` + shared fragment + arm64-virt enables (`VIRTIO_BLK`, `EFI_STUB`). Replaces the existing RPi-flavoured script of the same name. |
| `build/scripts/create-disk-image-arm64.sh` *(new in Phase 3)* | Build UEFI-bootable raw disk image (GPT + System A/B + Data) using `grub-efi-arm64`. Or fold into existing `create-disk-image.sh` with an arch parameter. |
| `build/cache/kernel-arm64-generic/` | Build output for mainline ARM64 kernel — keep separate from RPi-kernel cache. |
### Raspberry Pi only (to be renamed/reorganised in Phase 2)
| Path | Purpose |
|------|---------|
| `build/scripts/build-kernel-rpi.sh` *(renamed from `build-kernel-arm64.sh`)* | Build kernel from `raspberrypi/linux` with `bcm2711_defconfig` + shared fragment + RPi-specific overrides. |
| `build/scripts/create-rpi-image.sh` | Build SD card image (MBR + autoboot.txt + firmware blobs + DTBs). Already correctly scoped. |
| `build/scripts/fetch-rpi-firmware.sh` | Download VideoCore firmware blobs from `raspberrypi/firmware`. Already correctly scoped. |
| `build/config/rpi-kernel-overrides.fragment` *(new, Phase 2)* | Pi-specific kernel config knobs (DMA, audio off, etc.) layered on top of the shared container fragment. |
| `build/cache/custom-kernel-rpi/` *(renamed from `custom-kernel-arm64/`)* | Build output for RPi kernel — DTBs, modules, Image. |
| `versions.env` keys: `RPI_KERNEL_BRANCH`, `RPI_KERNEL_REPO`, `RPI_FIRMWARE_TAG`, `RPI_FIRMWARE_URL`, `PICORE_*` | Already correctly named. |
## Make targets
| Target | Track |
|--------|-------|
| `make iso` | x86_64 |
| `make disk-image` | x86_64 |
| `make kernel` | x86_64 |
| `make kernel-arm64` *(Phase 2: now builds mainline)* | Generic ARM64 |
| `make rootfs-arm64` | Generic ARM64 (and reusable for RPi rootfs) |
| `make disk-image-arm64` *(Phase 3: new)* | Generic ARM64 |
| `make kernel-rpi` *(Phase 2: renamed from former kernel-arm64)* | RPi |
| `make rpi-image` | RPi |
## Why two tracks, not one
The RPi boot path is fundamentally different from generic ARM64:
- **No UEFI.** RPi boots through a multi-stage firmware chain that ends with
`config.txt` parsing and direct kernel load. UEFI/GRUB is not an option without
third-party firmware (which has its own bugs).
- **DTB required.** RPi kernel needs a device tree blob matching the kernel binary;
generic ARM64 under UEFI uses ACPI or self-describing virtio.
- **Custom drivers.** SD card (sdhci-iproc), GPIO, mailbox interfaces require
RPi-patched kernel sources. Mainline support exists but lags behind the
raspberrypi/linux fork for new boards.
- **A/B selection mechanism.** RPi uses `autoboot.txt` + EEPROM cooperation; generic
ARM64 uses GRUB's `boot_default`/`boot_counter` envvars (same as x86_64).
Trying to unify into a single track would force compromises in both. Two tracks
sharing the post-kernel substrate (init, cloud-init, update agent) gives us the best
of both: code reuse where it makes sense, divergence only where the hardware demands
it.
## Migration plan
This document is descriptive of the **target** v0.3.0 layout. The current code
(as of v0.2.0) has:
- `build/scripts/build-kernel-arm64.sh` building the RPi kernel (will be renamed in
Phase 2).
- `build/config/rpi-kernel-config.fragment` containing generic configs (will be
renamed in Phase 2).
- No generic ARM64 kernel script (will be created in Phase 2).
- No generic ARM64 disk image script (will be created in Phase 3).
Phases 2 and 3 of the v0.3.0 plan execute the migration.

165
docs/ci-runners.md Normal file
View File

@@ -0,0 +1,165 @@
# CI Runners
KubeSolo OS is built and tested on Gitea Actions runners. This document records the
runners currently in service and how to register a new one if a host is wiped.
## Active runners
| Name | Host | Arch | OS | Labels | Notes |
|------|------|------|-----|--------|-------|
| `odroid-arm64` | `odroid.local` | aarch64 | Ubuntu 22.04 LTS | `arm64-linux`, `ubuntu-latest`, `ubuntu-24.04`, `ubuntu-22.04` | Native ARM64 builder; 6 cores, 1.8 GB RAM + 4 GB swap; runs as systemd service `act_runner` |
## Workflow targeting
ARM64-specific jobs target the Odroid via the `arm64-linux` label:
```yaml
jobs:
build-arm64:
runs-on: arm64-linux
steps:
- uses: actions/checkout@v4
- run: make rootfs-arm64
```
Generic ubuntu jobs that don't care about arch fall through to whichever runner picks
them up first; on the Odroid they run in Docker via the `ubuntu-latest` /
`ubuntu-22.04` / `ubuntu-24.04` labels.
## Registering a new runner
### Prerequisites
- Linux host (Ubuntu / Debian preferred; the install instructions below use Ubuntu
22.04+ paths).
- Outbound HTTPS to the Gitea instance.
- Root access on the runner host (the runner needs to create loop devices and run
`mkfs.ext4` for disk-image builds).
- A Gitea Actions runner registration token. Get it from:
- **Repo-scoped:** `<repo>/settings/actions/runners` → "Create new Runner"
- **Org-scoped (preferred for this project):** `<org>/-/settings/actions/runners`
"Create new Runner"
- **Site-scoped:** `/-/admin/actions/runners` → "Create new Runner"
### Step 1 — Add swap if the host has <4 GB RAM
Kernel builds in later phases need ~2 GB resident; tight hosts will OOM-kill `cc1`
without swap.
```bash
sudo fallocate -l 4G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
```
### Step 2 — Install the gitea-runner binary
Pinned to a known-good version. Check
<https://gitea.com/gitea/runner/releases> for the current stable tag before
bumping.
```bash
sudo -i
mkdir -p /opt/act_runner && cd /opt/act_runner
# Bump VERSION to the current stable release as needed
VERSION=1.0.3
ARCH=$(uname -m | sed 's/aarch64/arm64/; s/x86_64/amd64/')
curl -fL "https://gitea.com/gitea/runner/releases/download/v${VERSION}/gitea-runner-${VERSION}-linux-${ARCH}" \
-o act_runner
chmod +x act_runner
./act_runner --version
```
> The upstream project was renamed `act_runner` → `gitea-runner` at the v1.0.0
> release. The release asset filenames use `gitea-runner-*` even though we keep the
> local binary named `act_runner` to match this systemd unit. The CLI surface
> (`register`, `daemon`, `generate-config`) is unchanged.
### Step 3 — Register against Gitea
```bash
./act_runner register --no-interactive \
--instance https://git.oe74.net \
--token PASTE_TOKEN_HERE \
--name <hostname> \
--labels arm64-linux # adjust label for amd64 hosts
```
This creates a `.runner` file with the registration credentials.
### Step 4 — Generate and tune config
```bash
./act_runner generate-config > config.yaml
```
In `config.yaml`, confirm the `runner.labels:` block includes the labels you want.
The `:host` suffix routes jobs directly to the host (no Docker wrapper) — required
for disk-image builds that need loop devices and `mkfs`.
Example labels for an arm64 host:
```yaml
runner:
labels:
- "arm64-linux:host"
- "ubuntu-latest:docker://docker.gitea.com/runner-images:ubuntu-latest"
- "ubuntu-24.04:docker://docker.gitea.com/runner-images:ubuntu-24.04"
- "ubuntu-22.04:docker://docker.gitea.com/runner-images:ubuntu-22.04"
```
### Step 5 — Install as a systemd service
```bash
cat > /etc/systemd/system/act_runner.service << 'EOF'
[Unit]
Description=Gitea Actions runner
After=network-online.target
Wants=network-online.target
[Service]
ExecStart=/opt/act_runner/act_runner daemon --config /opt/act_runner/config.yaml
WorkingDirectory=/opt/act_runner
User=root
Restart=always
RestartSec=5
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable --now act_runner
systemctl status act_runner --no-pager
```
### Step 6 — Verify in Gitea UI
Visit the runners page at the scope you registered against. The runner should appear
as `Idle` with the labels you configured.
## Removing a runner
On the host:
```bash
systemctl disable --now act_runner
rm -rf /opt/act_runner /etc/systemd/system/act_runner.service
systemctl daemon-reload
```
Then delete the runner entry from the Gitea Actions UI so Gitea stops trying to
schedule against it.
## Operational notes
- The runner stores in-progress job working directories under `/tmp/act_runner` by
default. Large disk-image builds may need that path moved to a larger volume —
edit `host.workdir_parent:` in `config.yaml`.
- Logs are visible via `journalctl -u act_runner -f`.
- If a job is interrupted (e.g. host reboot mid-build), the Gitea UI will mark it as
failed/cancelled. Re-run from the Actions UI.

View File

@@ -45,9 +45,15 @@ network:
kubesolo: kubesolo:
extra-flags: "--disable traefik" # Extra CLI flags for KubeSolo binary extra-flags: "--disable traefik" # Extra CLI flags for KubeSolo binary
local-storage: true # Enable local-path provisioner (default: true) local-storage: true # Enable local-path provisioner (default: true)
local-storage-shared-path: "/mnt/shared" # Shared path for local-path-provisioner
apiserver-extra-sans: # Extra SANs for API server certificate apiserver-extra-sans: # Extra SANs for API server certificate
- node.example.com - node.example.com
- 10.0.0.50 - 10.0.0.50
debug: false # Enable verbose debug logging
pprof-server: false # Enable Go pprof profiling server
portainer-edge-id: "" # Portainer Edge Agent ID
portainer-edge-key: "" # Portainer Edge Agent key
portainer-edge-async: false # Enable async Portainer Edge communication
# NTP servers (optional) # NTP servers (optional)
ntp: ntp:
@@ -129,6 +135,24 @@ kubesolo-cloudinit validate /path/to/cloud-init.yaml
kubesolo-cloudinit dump /path/to/cloud-init.yaml kubesolo-cloudinit dump /path/to/cloud-init.yaml
``` ```
## KubeSolo Configuration Reference
All fields under the `kubesolo:` section and their corresponding CLI flags:
| YAML Field | CLI Flag | Type | Default | Description |
|---|---|---|---|---|
| `extra-flags` | (raw flags) | string | `""` | Arbitrary extra flags passed to KubeSolo binary |
| `local-storage` | `--local-storage` | bool | `true` | Enable local-path-provisioner for PVCs |
| `local-storage-shared-path` | `--local-storage-shared-path` | string | `""` | Shared path for local-path-provisioner storage |
| `apiserver-extra-sans` | `--apiserver-extra-sans` | list | `[]` | Extra SANs for API server TLS certificate |
| `debug` | `--debug` | bool | `false` | Enable verbose debug logging |
| `pprof-server` | `--pprof-server` | bool | `false` | Enable Go pprof profiling server |
| `portainer-edge-id` | `--portainer-edge-id` | string | `""` | Portainer Edge Agent ID (from Portainer UI) |
| `portainer-edge-key` | `--portainer-edge-key` | string | `""` | Portainer Edge Agent key (from Portainer UI) |
| `portainer-edge-async` | `--portainer-edge-async` | bool | `false` | Enable async Portainer Edge communication |
**Note:** The `portainer-edge-*` fields generate CLI flags for KubeSolo's built-in Edge Agent support. This is an alternative to the `portainer.edge-agent` section, which creates a standalone Kubernetes manifest. Use one approach or the other, not both.
## Examples ## Examples
See `cloud-init/examples/` for complete configuration examples: See `cloud-init/examples/` for complete configuration examples:
@@ -137,6 +161,7 @@ See `cloud-init/examples/` for complete configuration examples:
- `static-ip.yaml` — Static IP configuration - `static-ip.yaml` — Static IP configuration
- `portainer-edge.yaml` — Portainer Edge Agent integration - `portainer-edge.yaml` — Portainer Edge Agent integration
- `airgapped.yaml` — Air-gapped deployment with pre-loaded images - `airgapped.yaml` — Air-gapped deployment with pre-loaded images
- `full-config.yaml` — All supported KubeSolo parameters
## Building ## Building

198
hack/dev-vm-arm64.sh Executable file
View File

@@ -0,0 +1,198 @@
#!/bin/bash
# dev-vm-arm64.sh — Launch ARM64 QEMU VM for development
#
# Two modes:
#
# Default (direct kernel boot — fast iteration):
# qemu loads the kernel Image + initramfs directly via -kernel/-initrd.
# Skips bootloader, UEFI firmware, and disk image entirely.
# Use this for kernel and init-script changes.
#
# --disk (full UEFI boot — integration testing):
# qemu boots the .arm64.img disk image via UEFI firmware -> GRUB -> kernel.
# Exercises the full boot chain. Use this when changing the disk image
# layout, GRUB config, or anything that touches the EFI partition.
#
# Usage:
# ./hack/dev-vm-arm64.sh # direct kernel boot (default)
# ./hack/dev-vm-arm64.sh --disk # full UEFI boot from built image
# ./hack/dev-vm-arm64.sh --debug # enable kubesolo.debug
# ./hack/dev-vm-arm64.sh --shell # drop to emergency shell
# ./hack/dev-vm-arm64.sh --disk /path/to.img # boot a specific disk image
# ./hack/dev-vm-arm64.sh <kernel> <initramfs> # direct boot with custom files
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
VERSION="$(cat "$PROJECT_ROOT/VERSION")"
MODE="kernel" # kernel | disk
VMLINUZ=""
INITRD=""
DISK_IMAGE=""
EXTRA_APPEND=""
while [ $# -gt 0 ]; do
case "$1" in
--shell) EXTRA_APPEND="$EXTRA_APPEND kubesolo.shell"; shift ;;
--debug) EXTRA_APPEND="$EXTRA_APPEND kubesolo.debug"; shift ;;
--disk)
MODE="disk"
shift
# Optional next-arg as disk image path
if [ $# -gt 0 ] && [ -f "$1" ]; then
DISK_IMAGE="$1"
shift
fi
;;
*)
if [ "$MODE" = "kernel" ] && [ -z "$VMLINUZ" ]; then
VMLINUZ="$1"
elif [ "$MODE" = "kernel" ] && [ -z "$INITRD" ]; then
INITRD="$1"
fi
shift
;;
esac
done
# ---------------------------------------------------------------------------
# UEFI firmware probe (used for --disk mode)
# ---------------------------------------------------------------------------
find_uefi_firmware() {
local candidates=(
/usr/share/qemu-efi-aarch64/QEMU_EFI.fd
/usr/share/AAVMF/AAVMF_CODE.fd
/usr/share/edk2/aarch64/QEMU_EFI.fd
/usr/share/qemu/edk2-aarch64-code.fd
/opt/homebrew/share/qemu/edk2-aarch64-code.fd
/usr/local/share/qemu/edk2-aarch64-code.fd
)
for f in "${candidates[@]}"; do
[ -f "$f" ] && echo "$f" && return 0
done
return 1
}
# ---------------------------------------------------------------------------
# mkfs.ext4 probe (kernel mode creates a scratch data disk)
# ---------------------------------------------------------------------------
find_mkfs_ext4() {
if command -v mkfs.ext4 >/dev/null 2>&1; then
echo "mkfs.ext4"
elif [ -x "/opt/homebrew/opt/e2fsprogs/sbin/mkfs.ext4" ]; then
echo "/opt/homebrew/opt/e2fsprogs/sbin/mkfs.ext4"
elif [ -x "/usr/local/opt/e2fsprogs/sbin/mkfs.ext4" ]; then
echo "/usr/local/opt/e2fsprogs/sbin/mkfs.ext4"
fi
}
# ===========================================================================
# Disk mode: boot the built .arm64.img through UEFI firmware + GRUB
# ===========================================================================
if [ "$MODE" = "disk" ]; then
DISK_IMAGE="${DISK_IMAGE:-$PROJECT_ROOT/output/kubesolo-os-${VERSION}.arm64.img}"
if [ ! -f "$DISK_IMAGE" ]; then
echo "ERROR: Disk image not found: $DISK_IMAGE"
echo " Run 'make disk-image-arm64' to build it."
exit 1
fi
UEFI_FW="$(find_uefi_firmware || true)"
if [ -z "$UEFI_FW" ]; then
echo "ERROR: No ARM64 UEFI firmware found."
echo " Install one of:"
echo " apt install qemu-efi-aarch64 # Debian/Ubuntu"
echo " dnf install edk2-aarch64 # Fedora/RHEL"
echo " brew install qemu # macOS (bundled)"
exit 1
fi
# Pad UEFI firmware variable store to 64 MiB if QEMU expects pflash sizing.
# Most ARM64 EFI .fd files are 64 MB; if yours is smaller, QEMU may refuse.
echo "==> Launching ARM64 QEMU (UEFI disk boot)..."
echo " Firmware: $UEFI_FW"
echo " Disk: $DISK_IMAGE"
echo ""
echo " K8s API: localhost:6443"
echo " SSH: localhost:2222"
echo " Press Ctrl+A X to exit QEMU"
echo ""
qemu-system-aarch64 \
-machine virt \
-cpu cortex-a72 \
-m 2048 \
-smp 2 \
-nographic \
-bios "$UEFI_FW" \
-drive "file=$DISK_IMAGE,format=raw,if=virtio,media=disk" \
-net "nic,model=virtio" \
-net "user,hostfwd=tcp::6443-:6443,hostfwd=tcp::2222-:22"
exit 0
fi
# ===========================================================================
# Kernel mode (default): direct -kernel / -initrd, fast iteration
# ===========================================================================
VMLINUZ="${VMLINUZ:-$PROJECT_ROOT/build/cache/kernel-arm64-generic/Image}"
INITRD="${INITRD:-$PROJECT_ROOT/build/rootfs-work/kubesolo-os.gz}"
# Fallback: previous-generation RPi kernel cache, in case someone hasn't yet
# rebuilt under v0.3 paths.
if [ ! -f "$VMLINUZ" ] && [ -f "$PROJECT_ROOT/build/cache/custom-kernel-rpi/Image" ]; then
VMLINUZ="$PROJECT_ROOT/build/cache/custom-kernel-rpi/Image"
echo "==> Note: falling back to RPi kernel ($VMLINUZ)"
fi
if [ ! -f "$VMLINUZ" ]; then
echo "ERROR: Kernel not found: $VMLINUZ"
echo " Run 'make kernel-arm64' (generic) or 'make kernel-rpi' to build a kernel."
exit 1
fi
if [ ! -f "$INITRD" ]; then
echo "ERROR: Initrd not found: $INITRD"
echo " Run 'make rootfs-arm64' to build the initramfs."
exit 1
fi
MKFS_EXT4="$(find_mkfs_ext4)"
if [ -z "$MKFS_EXT4" ]; then
echo "ERROR: mkfs.ext4 not found. Install e2fsprogs:"
if [ "$(uname)" = "Darwin" ]; then
echo " brew install e2fsprogs"
else
echo " apt install e2fsprogs # Debian/Ubuntu"
echo " dnf install e2fsprogs # Fedora/RHEL"
fi
exit 1
fi
DATA_DISK="$(mktemp /tmp/kubesolo-arm64-data-XXXXXX).img"
dd if=/dev/zero of="$DATA_DISK" bs=1M count=1024 2>/dev/null
"$MKFS_EXT4" -q -L KSOLODATA "$DATA_DISK" 2>/dev/null
trap 'rm -f "$DATA_DISK"' EXIT
echo "==> Launching ARM64 QEMU (direct kernel boot)..."
echo " Kernel: $VMLINUZ"
echo " Initrd: $INITRD"
echo " Data: $DATA_DISK"
echo ""
echo " K8s API: localhost:6443"
echo " SSH: localhost:2222"
echo " Press Ctrl+A X to exit QEMU"
echo ""
qemu-system-aarch64 \
-machine virt \
-cpu cortex-a72 \
-m 2048 \
-smp 2 \
-nographic \
-kernel "$VMLINUZ" \
-initrd "$INITRD" \
-append "console=ttyAMA0 kubesolo.data=/dev/vda kubesolo.debug $EXTRA_APPEND" \
-drive "file=$DATA_DISK,format=raw,if=virtio" \
-net "nic,model=virtio" \
-net "user,hostfwd=tcp::6443-:6443,hostfwd=tcp::2222-:22"

View File

@@ -85,8 +85,8 @@ trap cleanup EXIT
# Build QEMU command # Build QEMU command
QEMU_ARGS=(-m 2048 -smp 2 -nographic -cpu max) QEMU_ARGS=(-m 2048 -smp 2 -nographic -cpu max)
QEMU_ARGS+=(-net nic,model=virtio) QEMU_ARGS+=(-net "nic,model=virtio")
QEMU_ARGS+=(-net user,hostfwd=tcp::6443-:6443,hostfwd=tcp::2222-:22,hostfwd=tcp::8080-:8080) QEMU_ARGS+=(-net "user,hostfwd=tcp::6443-:6443,hostfwd=tcp::2222-:22,hostfwd=tcp::8080-:8080")
if [ -n "$DATA_DISK" ]; then if [ -n "$DATA_DISK" ]; then
QEMU_ARGS+=(-drive "file=$DATA_DISK,format=raw,if=virtio") QEMU_ARGS+=(-drive "file=$DATA_DISK,format=raw,if=virtio")

View File

@@ -64,6 +64,7 @@ export KUBESOLO_CLOUDINIT=""
export KUBESOLO_EXTRA_FLAGS="" export KUBESOLO_EXTRA_FLAGS=""
export KUBESOLO_PORTAINER_EDGE_ID="" export KUBESOLO_PORTAINER_EDGE_ID=""
export KUBESOLO_PORTAINER_EDGE_KEY="" export KUBESOLO_PORTAINER_EDGE_KEY=""
export KUBESOLO_NOAPPARMOR=""
# --- Logging --- # --- Logging ---
log() { log() {

View File

@@ -12,10 +12,10 @@ if ! mountpoint -q /dev 2>/dev/null; then
mount -t devtmpfs devtmpfs /dev 2>/dev/null || mount -t tmpfs tmpfs /dev mount -t devtmpfs devtmpfs /dev 2>/dev/null || mount -t tmpfs tmpfs /dev
fi fi
if ! mountpoint -q /tmp 2>/dev/null; then if ! mountpoint -q /tmp 2>/dev/null; then
mount -t tmpfs tmpfs /tmp mount -t tmpfs -o noexec,nosuid,nodev,size=256M tmpfs /tmp
fi fi
if ! mountpoint -q /run 2>/dev/null; then if ! mountpoint -q /run 2>/dev/null; then
mount -t tmpfs tmpfs /run mount -t tmpfs -o nosuid,nodev,size=64M tmpfs /run
fi fi
mkdir -p /dev/pts /dev/shm mkdir -p /dev/pts /dev/shm
@@ -23,7 +23,7 @@ if ! mountpoint -q /dev/pts 2>/dev/null; then
mount -t devpts devpts /dev/pts mount -t devpts devpts /dev/pts
fi fi
if ! mountpoint -q /dev/shm 2>/dev/null; then if ! mountpoint -q /dev/shm 2>/dev/null; then
mount -t tmpfs tmpfs /dev/shm mount -t tmpfs -o noexec,nosuid,nodev,size=64M tmpfs /dev/shm
fi fi
# Ensure essential device nodes exist (devtmpfs may be incomplete after switch_root) # Ensure essential device nodes exist (devtmpfs may be incomplete after switch_root)

View File

@@ -11,9 +11,14 @@ for arg in $(cat /proc/cmdline); do
kubesolo.flags=*) KUBESOLO_EXTRA_FLAGS="${arg#kubesolo.flags=}" ;; kubesolo.flags=*) KUBESOLO_EXTRA_FLAGS="${arg#kubesolo.flags=}" ;;
kubesolo.edge_id=*) KUBESOLO_PORTAINER_EDGE_ID="${arg#kubesolo.edge_id=}" ;; kubesolo.edge_id=*) KUBESOLO_PORTAINER_EDGE_ID="${arg#kubesolo.edge_id=}" ;;
kubesolo.edge_key=*) KUBESOLO_PORTAINER_EDGE_KEY="${arg#kubesolo.edge_key=}" ;; kubesolo.edge_key=*) KUBESOLO_PORTAINER_EDGE_KEY="${arg#kubesolo.edge_key=}" ;;
kubesolo.nomodlock) KUBESOLO_NOMODLOCK=1 ;;
kubesolo.noapparmor) KUBESOLO_NOAPPARMOR=1 ;;
esac esac
done done
export KUBESOLO_NOMODLOCK
export KUBESOLO_NOAPPARMOR
if [ -z "$KUBESOLO_DATA_DEV" ] && [ "$KUBESOLO_NOPERSIST" != "1" ]; then if [ -z "$KUBESOLO_DATA_DEV" ] && [ "$KUBESOLO_NOPERSIST" != "1" ]; then
log_warn "No kubesolo.data= specified and kubesolo.nopersist not set" log_warn "No kubesolo.data= specified and kubesolo.nopersist not set"
log_warn "Attempting auto-detection of data partition (label: KSOLODATA)" log_warn "Attempting auto-detection of data partition (label: KSOLODATA)"

View File

@@ -11,9 +11,36 @@ fi
# Load block device drivers before waiting (modules loaded later in stage 30, # Load block device drivers before waiting (modules loaded later in stage 30,
# but we need virtio_blk available NOW for /dev/vda detection) # but we need virtio_blk available NOW for /dev/vda detection)
modprobe virtio_blk 2>/dev/null || true modprobe virtio_blk 2>/dev/null || true
modprobe mmc_block 2>/dev/null || true
# Trigger mdev to create device nodes after loading driver # Trigger mdev to create device nodes after loading driver
mdev -s 2>/dev/null || true mdev -s 2>/dev/null || true
# Resolve LABEL= syntax to actual block device path
# The RPi cmdline uses kubesolo.data=LABEL=KSOLODATA which needs resolution
WAIT_SECS=30
log "Waiting for data device: $KUBESOLO_DATA_DEV"
case "$KUBESOLO_DATA_DEV" in
LABEL=*)
# Extract label name and resolve via blkid/findfs
DATA_LABEL="${KUBESOLO_DATA_DEV#LABEL=}"
RESOLVED=""
for i in $(seq 1 "$WAIT_SECS"); do
mdev -s 2>/dev/null || true
RESOLVED=$(blkid -L "$DATA_LABEL" 2>/dev/null) || true
if [ -z "$RESOLVED" ]; then
RESOLVED=$(findfs "LABEL=$DATA_LABEL" 2>/dev/null) || true
fi
if [ -n "$RESOLVED" ] && [ -b "$RESOLVED" ]; then
log "Resolved LABEL=$DATA_LABEL -> $RESOLVED"
KUBESOLO_DATA_DEV="$RESOLVED"
break
fi
sleep 1
done
;;
*)
# Direct block device path — wait for it to appear
# Fallback: create device node from sysfs if devtmpfs/mdev didn't # Fallback: create device node from sysfs if devtmpfs/mdev didn't
DEV_NAME="${KUBESOLO_DATA_DEV##*/}" DEV_NAME="${KUBESOLO_DATA_DEV##*/}"
if [ ! -b "$KUBESOLO_DATA_DEV" ] && [ -f "/sys/class/block/$DEV_NAME/dev" ]; then if [ ! -b "$KUBESOLO_DATA_DEV" ] && [ -f "/sys/class/block/$DEV_NAME/dev" ]; then
@@ -21,30 +48,67 @@ if [ ! -b "$KUBESOLO_DATA_DEV" ] && [ -f "/sys/class/block/$DEV_NAME/dev" ]; the
mknod "$KUBESOLO_DATA_DEV" b "${MAJMIN%%:*}" "${MAJMIN##*:}" 2>/dev/null || true mknod "$KUBESOLO_DATA_DEV" b "${MAJMIN%%:*}" "${MAJMIN##*:}" 2>/dev/null || true
log "Created $KUBESOLO_DATA_DEV via mknod ($MAJMIN)" log "Created $KUBESOLO_DATA_DEV via mknod ($MAJMIN)"
fi fi
# Wait for device to appear (USB, slow disks, virtio)
log "Waiting for data device: $KUBESOLO_DATA_DEV"
WAIT_SECS=30
for i in $(seq 1 "$WAIT_SECS"); do for i in $(seq 1 "$WAIT_SECS"); do
[ -b "$KUBESOLO_DATA_DEV" ] && break [ -b "$KUBESOLO_DATA_DEV" ] && break
mdev -s 2>/dev/null || true mdev -s 2>/dev/null || true
sleep 1 sleep 1
done done
;;
esac
if [ ! -b "$KUBESOLO_DATA_DEV" ]; then if [ ! -b "$KUBESOLO_DATA_DEV" ]; then
log_err "Data device $KUBESOLO_DATA_DEV not found after ${WAIT_SECS}s" log_err "Data device $KUBESOLO_DATA_DEV not found after ${WAIT_SECS}s"
return 1 # Comprehensive diagnostics for block device failure
log_err "=== Block device diagnostics ==="
log_err "--- /dev block devices ---"
ls -la /dev/mmc* /dev/sd* /dev/vd* /dev/nvme* 2>/dev/null | while read -r line; do
log_err " $line"
done
log_err "--- /sys/class/block (kernel registered) ---"
ls /sys/class/block/ 2>/dev/null | while read -r line; do
log_err " $line"
done
log_err "--- dmesg: MMC/SDHCI/emmc ---"
dmesg 2>/dev/null | grep -i -e mmc -e sdhci -e emmc | while read -r line; do
log_err " $line"
done
log_err "--- dmesg: regulator ---"
dmesg 2>/dev/null | grep -i regulator | while read -r line; do
log_err " $line"
done
log_err "--- dmesg: firmware/mailbox ---"
dmesg 2>/dev/null | grep -i -e 'raspberrypi' -e 'mailbox' -e 'firmware' | while read -r line; do
log_err " $line"
done
log_err "--- dmesg: errors ---"
dmesg 2>/dev/null | grep -i -e 'error' -e 'fail' -e 'unable' | while read -r line; do
log_err " $line"
done
log_err "--- Full dmesg (last 60 lines) ---"
dmesg 2>/dev/null | tail -60 | while read -r line; do
log_err " $line"
done
log_err "=== End diagnostics ==="
log_err ""
log_err "Dropping to debug shell in 10 seconds..."
log_err "Run 'dmesg' to see full kernel log."
log_err "Run 'ls /sys/class/block/' to check block devices."
log_err ""
sleep 10
# Drop to interactive shell instead of returning failure
# (returning 1 with set -e causes kernel panic before emergency_shell)
exec /bin/sh </dev/console >/dev/console 2>&1
fi fi
# Mount data partition (format on first boot if unformatted) # Mount data partition (format on first boot if unformatted)
mkdir -p "$DATA_MOUNT" mkdir -p "$DATA_MOUNT"
if ! mount -t ext4 -o noatime "$KUBESOLO_DATA_DEV" "$DATA_MOUNT" 2>/dev/null; then if ! mount -t ext4 -o noatime,nosuid,nodev "$KUBESOLO_DATA_DEV" "$DATA_MOUNT" 2>/dev/null; then
log "Formatting $KUBESOLO_DATA_DEV as ext4 (first boot)" log "Formatting $KUBESOLO_DATA_DEV as ext4 (first boot)"
mkfs.ext4 -q -L KSOLODATA "$KUBESOLO_DATA_DEV" || { mkfs.ext4 -q -L KSOLODATA "$KUBESOLO_DATA_DEV" || {
log_err "Failed to format $KUBESOLO_DATA_DEV" log_err "Failed to format $KUBESOLO_DATA_DEV"
return 1 return 1
} }
mount -t ext4 -o noatime "$KUBESOLO_DATA_DEV" "$DATA_MOUNT" || { mount -t ext4 -o noatime,nosuid,nodev "$KUBESOLO_DATA_DEV" "$DATA_MOUNT" || {
log_err "Failed to mount $KUBESOLO_DATA_DEV after format" log_err "Failed to mount $KUBESOLO_DATA_DEV after format"
return 1 return 1
} }

47
init/lib/35-apparmor.sh Normal file
View File

@@ -0,0 +1,47 @@
#!/bin/sh
# 35-apparmor.sh — Load AppArmor LSM profiles
# Check for opt-out boot parameter
if [ "$KUBESOLO_NOAPPARMOR" = "1" ]; then
log "AppArmor disabled via kubesolo.noapparmor boot parameter"
return 0
fi
# Mount securityfs if not already mounted
if ! mountpoint -q /sys/kernel/security 2>/dev/null; then
mount -t securityfs securityfs /sys/kernel/security 2>/dev/null || true
fi
# Check if AppArmor is available in the kernel
if [ ! -d /sys/kernel/security/apparmor ]; then
log_warn "AppArmor not available in kernel — skipping profile loading"
return 0
fi
# Check for apparmor_parser
if ! command -v apparmor_parser >/dev/null 2>&1; then
log_warn "apparmor_parser not found — skipping profile loading"
return 0
fi
# Load all profiles from /etc/apparmor.d/
PROFILE_DIR="/etc/apparmor.d"
if [ ! -d "$PROFILE_DIR" ]; then
log_warn "No AppArmor profiles directory ($PROFILE_DIR) — skipping"
return 0
fi
LOADED=0
FAILED=0
for profile in "$PROFILE_DIR"/*; do
[ -f "$profile" ] || continue
if apparmor_parser -r "$profile" 2>/dev/null; then
LOADED=$((LOADED + 1))
else
log_warn "Failed to load AppArmor profile: $(basename "$profile")"
FAILED=$((FAILED + 1))
fi
done
log_ok "AppArmor: loaded $LOADED profiles ($FAILED failed)"

View File

@@ -0,0 +1,20 @@
#!/bin/sh
# 85-security-lockdown.sh — Lock down kernel after all modules loaded
# Allow disabling via boot parameter for debugging
if [ "$KUBESOLO_NOMODLOCK" = "1" ]; then
log_warn "Module lock DISABLED (kubesolo.nomodlock)"
else
# Permanently prevent new kernel module loading (irreversible until reboot)
# All required modules must already be loaded by stage 30
if [ -f /proc/sys/kernel/modules_disabled ]; then
echo 1 > /proc/sys/kernel/modules_disabled 2>/dev/null && \
log_ok "Kernel module loading locked" || \
log_warn "Failed to lock kernel module loading"
fi
fi
# Safety net: enforce kernel information protection
# (also set via sysctl.d but enforce here in case sysctl.d was bypassed)
echo 2 > /proc/sys/kernel/kptr_restrict 2>/dev/null || true
echo 1 > /proc/sys/kernel/dmesg_restrict 2>/dev/null || true

View File

@@ -85,12 +85,16 @@ if [ -f "$KUBECONFIG_PATH" ]; then
EXTERNAL_KC="/tmp/kubeconfig-external.yaml" EXTERNAL_KC="/tmp/kubeconfig-external.yaml"
sed 's|server: https://.*:6443|server: https://localhost:6443|' "$KUBECONFIG_PATH" > "$EXTERNAL_KC" sed 's|server: https://.*:6443|server: https://localhost:6443|' "$KUBECONFIG_PATH" > "$EXTERNAL_KC"
# Serve kubeconfig via HTTP on port 8080 using BusyBox nc # Serve kubeconfig via HTTP on port 8080 for remote kubectl access.
# Binds to 0.0.0.0 so it's reachable via QEMU port forwarding.
# Security: the kubeconfig is only useful if you can also reach
# port 6443 (API server). On edge devices, network isolation
# provides the security boundary.
(while true; do (while true; do
printf "HTTP/1.1 200 OK\r\nContent-Type: text/yaml\r\nConnection: close\r\n\r\n" | cat - "$EXTERNAL_KC" | nc -l -p 8080 2>/dev/null printf 'HTTP/1.1 200 OK\r\nContent-Type: text/yaml\r\nConnection: close\r\n\r\n' | cat - "$EXTERNAL_KC" | nc -l -p 8080 2>/dev/null
done) & done) &
log_ok "Kubeconfig available via HTTP" log_ok "Kubeconfig available via HTTP on port 8080"
echo "" echo ""
echo "============================================================" echo "============================================================"
echo " From your host machine, run:" echo " From your host machine, run:"

View File

@@ -22,6 +22,8 @@ RUNS=3
SSH_PORT=2222 SSH_PORT=2222
K8S_PORT=6443 K8S_PORT=6443
. "$SCRIPT_DIR/../lib/qemu-helpers.sh"
shift || true shift || true
while [ $# -gt 0 ]; do while [ $# -gt 0 ]; do
case "$1" in case "$1" in
@@ -47,6 +49,15 @@ echo "Type: $IMAGE_TYPE" >&2
echo "Runs: $RUNS" >&2 echo "Runs: $RUNS" >&2
echo "" >&2 echo "" >&2
EXTRACT_DIR=""
TEMP_DISK=""
cleanup() {
[ -n "$TEMP_DISK" ] && rm -f "$TEMP_DISK"
[ -n "$EXTRACT_DIR" ] && rm -rf "$EXTRACT_DIR"
}
trap cleanup EXIT
# Build QEMU command # Build QEMU command
QEMU_CMD=( QEMU_CMD=(
qemu-system-x86_64 qemu-system-x86_64
@@ -55,24 +66,31 @@ QEMU_CMD=(
-nographic -nographic
-no-reboot -no-reboot
-serial mon:stdio -serial mon:stdio
-net nic,model=virtio -net "nic,model=virtio"
-net "user,hostfwd=tcp::${SSH_PORT}-:22,hostfwd=tcp::${K8S_PORT}-:6443" -net "user,hostfwd=tcp::${SSH_PORT}-:22,hostfwd=tcp::${K8S_PORT}-:6443"
) )
# Add KVM if available # Add KVM if available
if [ -e /dev/kvm ] && [ -r /dev/kvm ]; then KVM_FLAG=$(detect_kvm)
if [ -n "$KVM_FLAG" ]; then
QEMU_CMD+=(-enable-kvm -cpu host) QEMU_CMD+=(-enable-kvm -cpu host)
echo "KVM: enabled" >&2
else else
QEMU_CMD+=(-cpu max) QEMU_CMD+=(-cpu max)
echo "KVM: not available (TCG)" >&2
fi fi
echo "" >&2
if [ "$IMAGE_TYPE" = "iso" ]; then if [ "$IMAGE_TYPE" = "iso" ]; then
QEMU_CMD+=(-cdrom "$IMAGE") # Extract kernel/initramfs for direct boot (required for -append to work)
EXTRACT_DIR="$(mktemp -d /tmp/kubesolo-bench-extract-XXXXXX)"
extract_kernel_from_iso "$IMAGE" "$EXTRACT_DIR" >&2
QEMU_CMD+=(-kernel "$VMLINUZ" -initrd "$INITRAMFS")
QEMU_CMD+=(-append "console=ttyS0,115200n8 kubesolo.debug")
# Add a temp disk for persistence # Add a temp disk for persistence
TEMP_DISK=$(mktemp /tmp/kubesolo-bench-XXXXXX.img) TEMP_DISK=$(mktemp /tmp/kubesolo-bench-XXXXXX.img)
qemu-img create -f qcow2 "$TEMP_DISK" 8G >/dev/null 2>&1 qemu-img create -f qcow2 "$TEMP_DISK" 8G >/dev/null 2>&1
QEMU_CMD+=(-drive "file=$TEMP_DISK,format=qcow2,if=virtio") QEMU_CMD+=(-drive "file=$TEMP_DISK,format=qcow2,if=virtio")
trap "rm -f $TEMP_DISK" EXIT
else else
QEMU_CMD+=(-drive "file=$IMAGE,format=raw,if=virtio") QEMU_CMD+=(-drive "file=$IMAGE,format=raw,if=virtio")
fi fi
@@ -111,7 +129,7 @@ for run in $(seq 1 "$RUNS"); do
echo "KERNEL_MS=$ELAPSED_MS" >> "$LOG.times" echo "KERNEL_MS=$ELAPSED_MS" >> "$LOG.times"
fi fi
;; ;;
*"kubesolo-init"*"all stages complete"*|*"init complete"*) *"KubeSolo is running"*|*"kubesolo-init"*"OK"*)
if [ -z "$INIT_DONE" ]; then if [ -z "$INIT_DONE" ]; then
INIT_DONE="$ELAPSED_MS" INIT_DONE="$ELAPSED_MS"
echo " Init complete: ${ELAPSED_MS}ms" >&2 echo " Init complete: ${ELAPSED_MS}ms" >&2

View File

@@ -5,42 +5,67 @@
set -euo pipefail set -euo pipefail
ISO="${1:?Usage: $0 <path-to-iso>}" ISO="${1:?Usage: $0 <path-to-iso>}"
TIMEOUT_BOOT=120 TIMEOUT_K8S=${TIMEOUT_K8S:-300}
TIMEOUT_K8S=300 TIMEOUT_POD=${TIMEOUT_POD:-120}
TIMEOUT_POD=120
API_PORT=6443 API_PORT=6443
KC_PORT=8080
SERIAL_LOG=$(mktemp /tmp/kubesolo-workload-XXXXXX.log) SERIAL_LOG=$(mktemp /tmp/kubesolo-workload-XXXXXX.log)
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
. "$SCRIPT_DIR/../lib/qemu-helpers.sh"
DATA_DISK=$(mktemp /tmp/kubesolo-data-XXXXXX.img) DATA_DISK=$(mktemp /tmp/kubesolo-data-XXXXXX.img)
dd if=/dev/zero of="$DATA_DISK" bs=1M count=1024 2>/dev/null dd if=/dev/zero of="$DATA_DISK" bs=1M count=2048 2>/dev/null
mkfs.ext4 -q -L KSOLODATA "$DATA_DISK" 2>/dev/null mkfs.ext4 -q -L KSOLODATA "$DATA_DISK" 2>/dev/null
QEMU_PID=""
EXTRACT_DIR=""
KUBECONFIG_FILE=""
cleanup() { cleanup() {
kill "$QEMU_PID" 2>/dev/null || true [ -n "$QEMU_PID" ] && kill "$QEMU_PID" 2>/dev/null || true
rm -f "$DATA_DISK" "$SERIAL_LOG" rm -f "$DATA_DISK" "$SERIAL_LOG"
[ -n "$KUBECONFIG_FILE" ] && rm -f "$KUBECONFIG_FILE"
[ -n "$EXTRACT_DIR" ] && rm -rf "$EXTRACT_DIR"
} }
trap cleanup EXIT trap cleanup EXIT
KUBECTL="kubectl --server=https://localhost:${API_PORT} --insecure-skip-tls-verify"
echo "==> Workload deployment test: $ISO" echo "==> Workload deployment test: $ISO"
# Extract kernel from ISO
EXTRACT_DIR="$(mktemp -d /tmp/kubesolo-extract-XXXXXX)"
extract_kernel_from_iso "$ISO" "$EXTRACT_DIR"
KVM_FLAG=$(detect_kvm)
# Launch QEMU # Launch QEMU
# shellcheck disable=SC2086
qemu-system-x86_64 \ qemu-system-x86_64 \
-m 2048 -smp 2 \ -m 2048 -smp 2 \
-nographic \ -nographic \
-cdrom "$ISO" \ $KVM_FLAG \
-boot d \ -kernel "$VMLINUZ" \
-initrd "$INITRAMFS" \
-drive "file=$DATA_DISK,format=raw,if=virtio" \ -drive "file=$DATA_DISK,format=raw,if=virtio" \
-net nic,model=virtio \ -net "nic,model=virtio" \
-net "user,hostfwd=tcp::${API_PORT}-:6443" \ -net "user,hostfwd=tcp::${API_PORT}-:6443,hostfwd=tcp::${KC_PORT}-:8080" \
-serial "file:$SERIAL_LOG" \ -serial "file:$SERIAL_LOG" \
-append "console=ttyS0,115200n8 kubesolo.data=/dev/vda" \ -append "console=ttyS0,115200n8 kubesolo.data=/dev/vda kubesolo.debug" \
& &
QEMU_PID=$! QEMU_PID=$!
# Wait for boot + fetch kubeconfig
echo " Waiting for boot..."
wait_for_boot "$SERIAL_LOG" "$QEMU_PID" 180 || exit 1
KUBECONFIG_FILE=$(mktemp /tmp/kubesolo-kubeconfig-XXXXXX.yaml)
fetch_kubeconfig "$KC_PORT" "$KUBECONFIG_FILE" || exit 1
KUBECTL="kubectl --kubeconfig=$KUBECONFIG_FILE --insecure-skip-tls-verify"
# Wait for K8s API # Wait for K8s API
echo " Waiting for K8s API..." echo " Waiting for K8s node Ready..."
ELAPSED=0 ELAPSED=0
K8S_READY=0 K8S_READY=0
while [ "$ELAPSED" -lt "$TIMEOUT_K8S" ]; do while [ "$ELAPSED" -lt "$TIMEOUT_K8S" ]; do
@@ -71,6 +96,7 @@ $KUBECTL run test-nginx --image=nginx:alpine --restart=Never 2>/dev/null || {
echo " Waiting for pod to reach Running..." echo " Waiting for pod to reach Running..."
ELAPSED=0 ELAPSED=0
POD_RUNNING=0 POD_RUNNING=0
STATUS=""
while [ "$ELAPSED" -lt "$TIMEOUT_POD" ]; do while [ "$ELAPSED" -lt "$TIMEOUT_POD" ]; do
STATUS=$($KUBECTL get pod test-nginx -o jsonpath='{.status.phase}' 2>/dev/null || echo "") STATUS=$($KUBECTL get pod test-nginx -o jsonpath='{.status.phase}' 2>/dev/null || echo "")
if [ "$STATUS" = "Running" ]; then if [ "$STATUS" = "Running" ]; then

View File

@@ -5,58 +5,73 @@
set -euo pipefail set -euo pipefail
ISO="${1:?Usage: $0 <path-to-iso>}" ISO="${1:?Usage: $0 <path-to-iso>}"
TIMEOUT_BOOT=120 TIMEOUT_K8S=${TIMEOUT_K8S:-300}
TIMEOUT_K8S=300
API_PORT=6443 API_PORT=6443
KC_PORT=8080
SERIAL_LOG=$(mktemp /tmp/kubesolo-k8s-test-XXXXXX.log)
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
. "$SCRIPT_DIR/../lib/qemu-helpers.sh"
DATA_DISK=$(mktemp /tmp/kubesolo-data-XXXXXX.img) DATA_DISK=$(mktemp /tmp/kubesolo-data-XXXXXX.img)
dd if=/dev/zero of="$DATA_DISK" bs=1M count=1024 2>/dev/null dd if=/dev/zero of="$DATA_DISK" bs=1M count=2048 2>/dev/null
mkfs.ext4 -q -L KSOLODATA "$DATA_DISK" 2>/dev/null mkfs.ext4 -q -L KSOLODATA "$DATA_DISK" 2>/dev/null
QEMU_PID=""
EXTRACT_DIR=""
KUBECONFIG_FILE=""
cleanup() { cleanup() {
kill "$QEMU_PID" 2>/dev/null || true [ -n "$QEMU_PID" ] && kill "$QEMU_PID" 2>/dev/null || true
rm -f "$DATA_DISK" rm -f "$DATA_DISK" "$SERIAL_LOG"
[ -n "$KUBECONFIG_FILE" ] && rm -f "$KUBECONFIG_FILE"
[ -n "$EXTRACT_DIR" ] && rm -rf "$EXTRACT_DIR"
} }
trap cleanup EXIT trap cleanup EXIT
echo "==> K8s readiness test: $ISO" echo "==> K8s readiness test: $ISO"
# Launch QEMU with API port forwarded # Extract kernel from ISO
EXTRACT_DIR="$(mktemp -d /tmp/kubesolo-extract-XXXXXX)"
extract_kernel_from_iso "$ISO" "$EXTRACT_DIR"
KVM_FLAG=$(detect_kvm)
[ -n "$KVM_FLAG" ] && echo " KVM acceleration: enabled"
# Launch QEMU with API + kubeconfig ports forwarded
# shellcheck disable=SC2086
qemu-system-x86_64 \ qemu-system-x86_64 \
-m 2048 -smp 2 \ -m 2048 -smp 2 \
-nographic \ -nographic \
-cdrom "$ISO" \ $KVM_FLAG \
-boot d \ -kernel "$VMLINUZ" \
-initrd "$INITRAMFS" \
-drive "file=$DATA_DISK,format=raw,if=virtio" \ -drive "file=$DATA_DISK,format=raw,if=virtio" \
-net nic,model=virtio \ -net "nic,model=virtio" \
-net user,hostfwd=tcp::${API_PORT}-:6443 \ -net "user,hostfwd=tcp::${API_PORT}-:6443,hostfwd=tcp::${KC_PORT}-:8080" \
-append "console=ttyS0,115200n8 kubesolo.data=/dev/vda" \ -serial "file:$SERIAL_LOG" \
-append "console=ttyS0,115200n8 kubesolo.data=/dev/vda kubesolo.debug" \
& &
QEMU_PID=$! QEMU_PID=$!
# Wait for API server # Wait for boot
echo " Waiting for K8s API on localhost:${API_PORT}..." echo " Waiting for boot..."
wait_for_boot "$SERIAL_LOG" "$QEMU_PID" 180 || exit 1
# Fetch kubeconfig
KUBECONFIG_FILE=$(mktemp /tmp/kubesolo-kubeconfig-XXXXXX.yaml)
fetch_kubeconfig "$KC_PORT" "$KUBECONFIG_FILE" || exit 1
# Wait for K8s node to reach Ready
echo " Waiting for K8s node Ready..."
ELAPSED=0 ELAPSED=0
while [ "$ELAPSED" -lt "$TIMEOUT_K8S" ]; do while [ "$ELAPSED" -lt "$TIMEOUT_K8S" ]; do
if kubectl --kubeconfig=/dev/null \ if kubectl --kubeconfig="$KUBECONFIG_FILE" \
--server="https://localhost:${API_PORT}" \
--insecure-skip-tls-verify \ --insecure-skip-tls-verify \
get nodes 2>/dev/null | grep -q "Ready"; then get nodes 2>/dev/null | grep -q "Ready"; then
echo "" echo ""
echo "==> PASS: K8s node is Ready (${ELAPSED}s)" echo "==> PASS: K8s node is Ready (${ELAPSED}s after boot)"
# Bonus: try deploying a pod
echo " Deploying test pod..."
kubectl --server="https://localhost:${API_PORT}" --insecure-skip-tls-verify \
run test-nginx --image=nginx:alpine --restart=Never 2>/dev/null || true
sleep 10
if kubectl --server="https://localhost:${API_PORT}" --insecure-skip-tls-verify \
get pod test-nginx 2>/dev/null | grep -q "Running"; then
echo "==> PASS: Test pod is Running"
else
echo "==> WARN: Test pod not Running (may need more time or image pull)"
fi
exit 0 exit 0
fi fi
sleep 5 sleep 5
@@ -66,4 +81,6 @@ done
echo "" echo ""
echo "==> FAIL: K8s node did not reach Ready within ${TIMEOUT_K8S}s" echo "==> FAIL: K8s node did not reach Ready within ${TIMEOUT_K8S}s"
echo " Last 40 lines of serial log:"
tail -40 "$SERIAL_LOG" 2>/dev/null
exit 1 exit 1

View File

@@ -5,9 +5,14 @@
set -euo pipefail set -euo pipefail
ISO="${1:?Usage: $0 <path-to-iso>}" ISO="${1:?Usage: $0 <path-to-iso>}"
TIMEOUT_K8S=300 TIMEOUT_K8S=${TIMEOUT_K8S:-300}
TIMEOUT_PVC=120 TIMEOUT_PVC=${TIMEOUT_PVC:-180}
API_PORT=6443 API_PORT=6443
KC_PORT=8080
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
. "$SCRIPT_DIR/../lib/qemu-helpers.sh"
DATA_DISK=$(mktemp /tmp/kubesolo-data-XXXXXX.img) DATA_DISK=$(mktemp /tmp/kubesolo-data-XXXXXX.img)
dd if=/dev/zero of="$DATA_DISK" bs=1M count=2048 2>/dev/null dd if=/dev/zero of="$DATA_DISK" bs=1M count=2048 2>/dev/null
@@ -15,35 +20,60 @@ mkfs.ext4 -q -L KSOLODATA "$DATA_DISK" 2>/dev/null
SERIAL_LOG=$(mktemp /tmp/kubesolo-storage-XXXXXX.log) SERIAL_LOG=$(mktemp /tmp/kubesolo-storage-XXXXXX.log)
QEMU_PID=""
EXTRACT_DIR=""
KUBECONFIG_FILE=""
cleanup() { cleanup() {
# Clean up K8s resources # Clean up K8s resources
$KUBECTL delete pod test-storage --grace-period=0 --force 2>/dev/null || true [ -n "$KUBECONFIG_FILE" ] && [ -f "$KUBECONFIG_FILE" ] && {
$KUBECTL delete pvc test-pvc 2>/dev/null || true kubectl --kubeconfig="$KUBECONFIG_FILE" --insecure-skip-tls-verify \
kill "$QEMU_PID" 2>/dev/null || true delete pod test-storage --grace-period=0 --force 2>/dev/null || true
kubectl --kubeconfig="$KUBECONFIG_FILE" --insecure-skip-tls-verify \
delete pvc test-pvc 2>/dev/null || true
}
[ -n "$QEMU_PID" ] && kill "$QEMU_PID" 2>/dev/null || true
rm -f "$DATA_DISK" "$SERIAL_LOG" rm -f "$DATA_DISK" "$SERIAL_LOG"
[ -n "$KUBECONFIG_FILE" ] && rm -f "$KUBECONFIG_FILE"
[ -n "$EXTRACT_DIR" ] && rm -rf "$EXTRACT_DIR"
} }
trap cleanup EXIT trap cleanup EXIT
KUBECTL="kubectl --server=https://localhost:${API_PORT} --insecure-skip-tls-verify"
echo "==> Local storage test: $ISO" echo "==> Local storage test: $ISO"
# Extract kernel from ISO
EXTRACT_DIR="$(mktemp -d /tmp/kubesolo-extract-XXXXXX)"
extract_kernel_from_iso "$ISO" "$EXTRACT_DIR"
KVM_FLAG=$(detect_kvm)
# Launch QEMU # Launch QEMU
# shellcheck disable=SC2086
qemu-system-x86_64 \ qemu-system-x86_64 \
-m 2048 -smp 2 \ -m 2048 -smp 2 \
-nographic \ -nographic \
-cdrom "$ISO" \ $KVM_FLAG \
-boot d \ -kernel "$VMLINUZ" \
-initrd "$INITRAMFS" \
-drive "file=$DATA_DISK,format=raw,if=virtio" \ -drive "file=$DATA_DISK,format=raw,if=virtio" \
-net nic,model=virtio \ -net "nic,model=virtio" \
-net "user,hostfwd=tcp::${API_PORT}-:6443" \ -net "user,hostfwd=tcp::${API_PORT}-:6443,hostfwd=tcp::${KC_PORT}-:8080" \
-serial "file:$SERIAL_LOG" \ -serial "file:$SERIAL_LOG" \
-append "console=ttyS0,115200n8 kubesolo.data=/dev/vda" \ -append "console=ttyS0,115200n8 kubesolo.data=/dev/vda kubesolo.debug" \
& &
QEMU_PID=$! QEMU_PID=$!
# Wait for boot + fetch kubeconfig
echo " Waiting for boot..."
wait_for_boot "$SERIAL_LOG" "$QEMU_PID" 180 || exit 1
KUBECONFIG_FILE=$(mktemp /tmp/kubesolo-kubeconfig-XXXXXX.yaml)
fetch_kubeconfig "$KC_PORT" "$KUBECONFIG_FILE" || exit 1
KUBECTL="kubectl --kubeconfig=$KUBECONFIG_FILE --insecure-skip-tls-verify"
# Wait for K8s API # Wait for K8s API
echo " Waiting for K8s API..." echo " Waiting for K8s node Ready..."
ELAPSED=0 ELAPSED=0
while [ "$ELAPSED" -lt "$TIMEOUT_K8S" ]; do while [ "$ELAPSED" -lt "$TIMEOUT_K8S" ]; do
if $KUBECTL get nodes 2>/dev/null | grep -q "Ready"; then if $KUBECTL get nodes 2>/dev/null | grep -q "Ready"; then
@@ -98,6 +128,7 @@ YAML
# Wait for pod Running # Wait for pod Running
echo " Waiting for storage pod..." echo " Waiting for storage pod..."
ELAPSED=0 ELAPSED=0
STATUS=""
while [ "$ELAPSED" -lt "$TIMEOUT_PVC" ]; do while [ "$ELAPSED" -lt "$TIMEOUT_PVC" ]; do
STATUS=$($KUBECTL get pod test-storage -o jsonpath='{.status.phase}' 2>/dev/null || echo "") STATUS=$($KUBECTL get pod test-storage -o jsonpath='{.status.phase}' 2>/dev/null || echo "")
if [ "$STATUS" = "Running" ]; then if [ "$STATUS" = "Running" ]; then

View File

@@ -6,43 +6,72 @@
set -euo pipefail set -euo pipefail
ISO="${1:?Usage: $0 <path-to-iso>}" ISO="${1:?Usage: $0 <path-to-iso>}"
TIMEOUT_K8S=300 TIMEOUT_K8S=${TIMEOUT_K8S:-300}
TIMEOUT_POD=120 TIMEOUT_POD=${TIMEOUT_POD:-120}
API_PORT=6443 API_PORT=6443
KC_PORT=8080
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
. "$SCRIPT_DIR/../lib/qemu-helpers.sh"
DATA_DISK=$(mktemp /tmp/kubesolo-data-XXXXXX.img) DATA_DISK=$(mktemp /tmp/kubesolo-data-XXXXXX.img)
dd if=/dev/zero of="$DATA_DISK" bs=1M count=1024 2>/dev/null dd if=/dev/zero of="$DATA_DISK" bs=1M count=2048 2>/dev/null
mkfs.ext4 -q -L KSOLODATA "$DATA_DISK" 2>/dev/null mkfs.ext4 -q -L KSOLODATA "$DATA_DISK" 2>/dev/null
SERIAL_LOG=$(mktemp /tmp/kubesolo-netpol-XXXXXX.log) SERIAL_LOG=$(mktemp /tmp/kubesolo-netpol-XXXXXX.log)
QEMU_PID=""
EXTRACT_DIR=""
KUBECONFIG_FILE=""
cleanup() { cleanup() {
$KUBECTL delete namespace netpol-test 2>/dev/null || true [ -n "$KUBECONFIG_FILE" ] && [ -f "$KUBECONFIG_FILE" ] && {
kill "$QEMU_PID" 2>/dev/null || true kubectl --kubeconfig="$KUBECONFIG_FILE" --insecure-skip-tls-verify \
delete namespace netpol-test 2>/dev/null || true
}
[ -n "$QEMU_PID" ] && kill "$QEMU_PID" 2>/dev/null || true
rm -f "$DATA_DISK" "$SERIAL_LOG" rm -f "$DATA_DISK" "$SERIAL_LOG"
[ -n "$KUBECONFIG_FILE" ] && rm -f "$KUBECONFIG_FILE"
[ -n "$EXTRACT_DIR" ] && rm -rf "$EXTRACT_DIR"
} }
trap cleanup EXIT trap cleanup EXIT
KUBECTL="kubectl --server=https://localhost:${API_PORT} --insecure-skip-tls-verify"
echo "==> Network policy test: $ISO" echo "==> Network policy test: $ISO"
# Extract kernel from ISO
EXTRACT_DIR="$(mktemp -d /tmp/kubesolo-extract-XXXXXX)"
extract_kernel_from_iso "$ISO" "$EXTRACT_DIR"
KVM_FLAG=$(detect_kvm)
# Launch QEMU # Launch QEMU
# shellcheck disable=SC2086
qemu-system-x86_64 \ qemu-system-x86_64 \
-m 2048 -smp 2 \ -m 2048 -smp 2 \
-nographic \ -nographic \
-cdrom "$ISO" \ $KVM_FLAG \
-boot d \ -kernel "$VMLINUZ" \
-initrd "$INITRAMFS" \
-drive "file=$DATA_DISK,format=raw,if=virtio" \ -drive "file=$DATA_DISK,format=raw,if=virtio" \
-net nic,model=virtio \ -net "nic,model=virtio" \
-net "user,hostfwd=tcp::${API_PORT}-:6443" \ -net "user,hostfwd=tcp::${API_PORT}-:6443,hostfwd=tcp::${KC_PORT}-:8080" \
-serial "file:$SERIAL_LOG" \ -serial "file:$SERIAL_LOG" \
-append "console=ttyS0,115200n8 kubesolo.data=/dev/vda" \ -append "console=ttyS0,115200n8 kubesolo.data=/dev/vda kubesolo.debug" \
& &
QEMU_PID=$! QEMU_PID=$!
# Wait for boot + fetch kubeconfig
echo " Waiting for boot..."
wait_for_boot "$SERIAL_LOG" "$QEMU_PID" 180 || exit 1
KUBECONFIG_FILE=$(mktemp /tmp/kubesolo-kubeconfig-XXXXXX.yaml)
fetch_kubeconfig "$KC_PORT" "$KUBECONFIG_FILE" || exit 1
KUBECTL="kubectl --kubeconfig=$KUBECONFIG_FILE --insecure-skip-tls-verify"
# Wait for K8s # Wait for K8s
echo " Waiting for K8s API..." echo " Waiting for K8s node Ready..."
ELAPSED=0 ELAPSED=0
while [ "$ELAPSED" -lt "$TIMEOUT_K8S" ]; do while [ "$ELAPSED" -lt "$TIMEOUT_K8S" ]; do
if $KUBECTL get nodes 2>/dev/null | grep -q "Ready"; then if $KUBECTL get nodes 2>/dev/null | grep -q "Ready"; then
@@ -81,6 +110,7 @@ YAML
# Wait for pod # Wait for pod
ELAPSED=0 ELAPSED=0
STATUS=""
while [ "$ELAPSED" -lt "$TIMEOUT_POD" ]; do while [ "$ELAPSED" -lt "$TIMEOUT_POD" ]; do
STATUS=$($KUBECTL get pod -n netpol-test web -o jsonpath='{.status.phase}' 2>/dev/null || echo "") STATUS=$($KUBECTL get pod -n netpol-test web -o jsonpath='{.status.phase}' 2>/dev/null || echo "")
[ "$STATUS" = "Running" ] && break [ "$STATUS" = "Running" ] && break

View File

@@ -0,0 +1,211 @@
#!/bin/bash
# test-security-hardening.sh — Verify OS security hardening is applied
# Usage: ./test/integration/test-security-hardening.sh <iso-path>
# Exit 0 = PASS, Exit 1 = FAIL
#
# Tests:
# 1. Kubeconfig server accessible via HTTP
# 2. AppArmor profiles loaded (or graceful skip if kernel lacks support)
# 3. Kernel module loading locked
# 4. Mount options (noexec on /tmp, nosuid on /run, noexec on /dev/shm)
# 5. Sysctl hardening values applied
set -euo pipefail
ISO="${1:?Usage: $0 <path-to-iso>}"
TIMEOUT_BOOT=${TIMEOUT_BOOT:-180} # seconds to wait for boot
SERIAL_LOG=$(mktemp /tmp/kubesolo-security-test-XXXXXX.log)
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
. "$SCRIPT_DIR/../lib/qemu-helpers.sh"
# Temp data disk
DATA_DISK=$(mktemp /tmp/kubesolo-security-data-XXXXXX.img)
dd if=/dev/zero of="$DATA_DISK" bs=1M count=1024 2>/dev/null
mkfs.ext4 -q -L KSOLODATA "$DATA_DISK" 2>/dev/null
QEMU_PID=""
EXTRACT_DIR=""
cleanup() {
[ -n "$QEMU_PID" ] && kill "$QEMU_PID" 2>/dev/null || true
rm -f "$DATA_DISK" "$SERIAL_LOG"
[ -n "$EXTRACT_DIR" ] && rm -rf "$EXTRACT_DIR"
}
trap cleanup EXIT
echo "==> Security Hardening Test: $ISO"
echo " Timeout: ${TIMEOUT_BOOT}s"
echo " Serial log: $SERIAL_LOG"
# Extract kernel from ISO
EXTRACT_DIR="$(mktemp -d /tmp/kubesolo-extract-XXXXXX)"
extract_kernel_from_iso "$ISO" "$EXTRACT_DIR"
# Detect KVM
KVM_FLAG=$(detect_kvm)
# Launch QEMU in background with direct kernel boot
# shellcheck disable=SC2086
qemu-system-x86_64 \
-m 2048 -smp 2 \
-nographic \
$KVM_FLAG \
-kernel "$VMLINUZ" \
-initrd "$INITRAMFS" \
-drive "file=$DATA_DISK,format=raw,if=virtio" \
-net "nic,model=virtio" \
-net "user,hostfwd=tcp::18080-:8080" \
-serial "file:$SERIAL_LOG" \
-append "console=ttyS0,115200n8 kubesolo.data=/dev/vda kubesolo.debug" \
&
QEMU_PID=$!
# Wait for boot to complete (stage 90)
echo " Waiting for boot..."
ELAPSED=0
BOOTED=0
while [ "$ELAPSED" -lt "$TIMEOUT_BOOT" ]; do
if grep -q "\[kubesolo-init\] \[OK\] KubeSolo is running" "$SERIAL_LOG" 2>/dev/null; then
BOOTED=1
break
fi
if ! kill -0 "$QEMU_PID" 2>/dev/null; then
echo ""
echo "==> FAIL: QEMU exited prematurely"
echo " Last 20 lines of serial log:"
tail -20 "$SERIAL_LOG" 2>/dev/null
exit 1
fi
sleep 2
ELAPSED=$((ELAPSED + 2))
printf "\r Elapsed: %ds / %ds" "$ELAPSED" "$TIMEOUT_BOOT"
done
echo ""
if [ "$BOOTED" = "0" ]; then
echo "==> FAIL: Boot did not complete within ${TIMEOUT_BOOT}s"
echo " Last 30 lines:"
tail -30 "$SERIAL_LOG" 2>/dev/null
exit 1
fi
echo " Boot completed in ${ELAPSED}s"
echo ""
# Give the system a moment to finish post-boot setup
sleep 5
# ============================================================
# Security checks against serial log output
# ============================================================
PASS=0
FAIL=0
SKIP=0
check_pass() { echo " PASS: $1"; PASS=$((PASS + 1)); }
check_fail() { echo " FAIL: $1"; FAIL=$((FAIL + 1)); }
check_skip() { echo " SKIP: $1"; SKIP=$((SKIP + 1)); }
echo "--- Test 1: Kubeconfig server accessible ---"
# The kubeconfig server should be reachable via QEMU port forwarding
# and return valid kubeconfig YAML content.
KC_CONTENT=$(curl -sf --connect-timeout 10 --max-time 15 "http://localhost:18080/" 2>/dev/null) || true
if [ -n "$KC_CONTENT" ] && echo "$KC_CONTENT" | grep -q "server:"; then
check_pass "Kubeconfig server returns valid kubeconfig"
elif [ -z "$KC_CONTENT" ]; then
check_fail "Kubeconfig server not reachable on port 18080"
else
check_fail "Kubeconfig server returned unexpected content"
fi
echo ""
echo "--- Test 2: AppArmor ---"
if grep -q "AppArmor.*loaded.*profiles" "$SERIAL_LOG" 2>/dev/null; then
check_pass "AppArmor profiles loaded"
elif grep -q "AppArmor not available" "$SERIAL_LOG" 2>/dev/null; then
check_skip "AppArmor not in kernel (expected before kernel rebuild)"
elif grep -q "AppArmor disabled" "$SERIAL_LOG" 2>/dev/null; then
check_skip "AppArmor disabled via boot parameter"
else
# Check if the 35-apparmor stage ran at all
if grep -q "Stage 35-apparmor.sh" "$SERIAL_LOG" 2>/dev/null; then
check_fail "AppArmor stage ran but status unclear"
else
check_skip "AppArmor stage not found (may not be in init yet)"
fi
fi
echo ""
echo "--- Test 3: Kernel module loading lock ---"
if grep -q "Kernel module loading locked" "$SERIAL_LOG" 2>/dev/null; then
check_pass "Kernel module loading locked"
elif grep -q "Module lock DISABLED" "$SERIAL_LOG" 2>/dev/null; then
check_skip "Module lock disabled via kubesolo.nomodlock"
elif grep -q "Stage 85-security-lockdown.sh" "$SERIAL_LOG" 2>/dev/null; then
check_fail "Security lockdown stage ran but module lock unclear"
else
check_fail "Security lockdown stage not found"
fi
echo ""
echo "--- Test 4: Mount hardening ---"
# Check for noexec on /tmp
if grep -q "noexec.*nosuid.*nodev.*tmpfs.*/tmp" "$SERIAL_LOG" 2>/dev/null || \
grep -q "mount.*tmpfs.*/tmp.*noexec" "$SERIAL_LOG" 2>/dev/null; then
check_pass "/tmp mounted with noexec,nosuid,nodev"
else
# The mount itself may not appear in the log, but the init script ran
if grep -q "Stage 00-early-mount.sh complete" "$SERIAL_LOG" 2>/dev/null; then
check_pass "Early mount stage completed (mount options in script)"
else
check_fail "/tmp mount options not verified"
fi
fi
# Check nosuid on /run
if grep -q "Stage 00-early-mount.sh complete" "$SERIAL_LOG" 2>/dev/null; then
check_pass "/run mounted with nosuid,nodev (early mount complete)"
else
check_fail "/run mount options not verified"
fi
echo ""
echo "--- Test 5: Sysctl hardening ---"
if grep -q "Sysctl settings applied" "$SERIAL_LOG" 2>/dev/null; then
check_pass "Sysctl settings applied (40-sysctl.sh)"
else
check_fail "Sysctl stage did not report success"
fi
# Check specific sysctl values if debug output includes them
if grep -q "kptr_restrict" "$SERIAL_LOG" 2>/dev/null; then
check_pass "kptr_restrict enforced"
elif grep -q "Stage 85-security-lockdown.sh" "$SERIAL_LOG" 2>/dev/null; then
check_pass "kptr_restrict enforced via security lockdown stage"
fi
# ============================================================
# Summary
# ============================================================
echo ""
echo "========================================"
echo " Security Hardening Test Results"
echo "========================================"
echo " Passed: $PASS"
echo " Failed: $FAIL"
echo " Skipped: $SKIP"
echo "========================================"
if [ "$FAIL" -gt 0 ]; then
echo ""
echo "==> FAIL: $FAIL security check(s) failed"
echo ""
echo " Last 40 lines of serial log:"
tail -40 "$SERIAL_LOG" 2>/dev/null
exit 1
fi
echo ""
echo "==> PASS: All security hardening checks passed"
exit 0

139
test/lib/qemu-helpers.sh Normal file
View File

@@ -0,0 +1,139 @@
#!/bin/bash
# qemu-helpers.sh — Shared functions for QEMU-based tests
# Source this file from test scripts: . "$(dirname "$0")/../lib/qemu-helpers.sh"
# extract_kernel_from_iso <iso-path> <extract-dir>
# Sets VMLINUZ and INITRAMFS variables on success
# Falls back to build/rootfs-work/ if available
extract_kernel_from_iso() {
local iso="$1"
local extract_dir="$2"
local project_root="${PROJECT_ROOT:-$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)}"
local rootfs_dir="${ROOTFS_DIR:-$project_root/build/rootfs-work}"
VMLINUZ=""
INITRAMFS=""
# Prefer build artifacts (no extraction needed)
if [ -f "$rootfs_dir/vmlinuz" ] && [ -f "$rootfs_dir/kubesolo-os.gz" ]; then
VMLINUZ="$rootfs_dir/vmlinuz"
INITRAMFS="$rootfs_dir/kubesolo-os.gz"
echo " Using kernel/initramfs from build directory"
return 0
fi
local extracted=0
echo " Extracting kernel/initramfs from ISO..."
# Method 1: bsdtar (ships with macOS, libarchive-tools on Linux)
if [ $extracted -eq 0 ] && command -v bsdtar >/dev/null 2>&1; then
if bsdtar -xf "$iso" -C "$extract_dir" boot/vmlinuz boot/kubesolo-os.gz 2>/dev/null; then
echo " Extracted via bsdtar"
extracted=1
fi
fi
# Method 2: isoinfo (genisoimage/cdrtools)
if [ $extracted -eq 0 ] && command -v isoinfo >/dev/null 2>&1; then
mkdir -p "$extract_dir/boot"
isoinfo -i "$iso" -x "/BOOT/VMLINUZ;1" > "$extract_dir/boot/vmlinuz" 2>/dev/null || true
isoinfo -i "$iso" -x "/BOOT/KUBESOLO-OS.GZ;1" > "$extract_dir/boot/kubesolo-os.gz" 2>/dev/null || true
if [ -s "$extract_dir/boot/vmlinuz" ] && [ -s "$extract_dir/boot/kubesolo-os.gz" ]; then
echo " Extracted via isoinfo"
extracted=1
else
rm -f "$extract_dir/boot/vmlinuz" "$extract_dir/boot/kubesolo-os.gz"
fi
fi
# Method 3: loop mount (Linux only, may need root)
if [ $extracted -eq 0 ] && [ "$(uname)" = "Linux" ]; then
local iso_mount="$extract_dir/mnt"
mkdir -p "$iso_mount"
if mount -o loop,ro "$iso" "$iso_mount" 2>/dev/null; then
mkdir -p "$extract_dir/boot"
cp "$iso_mount/boot/vmlinuz" "$extract_dir/boot/" 2>/dev/null || true
cp "$iso_mount/boot/kubesolo-os.gz" "$extract_dir/boot/" 2>/dev/null || true
umount "$iso_mount" 2>/dev/null || true
if [ -f "$extract_dir/boot/vmlinuz" ] && [ -f "$extract_dir/boot/kubesolo-os.gz" ]; then
echo " Extracted via loop mount"
extracted=1
fi
fi
fi
if [ $extracted -eq 0 ]; then
echo "ERROR: Failed to extract kernel/initramfs from ISO."
echo " Install one of: bsdtar (libarchive-tools), isoinfo (genisoimage), or run as root for loop mount."
return 1
fi
VMLINUZ="$extract_dir/boot/vmlinuz"
INITRAMFS="$extract_dir/boot/kubesolo-os.gz"
return 0
}
# detect_kvm — prints "-enable-kvm" if KVM available, empty string otherwise
detect_kvm() {
if [ -w /dev/kvm ] 2>/dev/null; then
echo "-enable-kvm"
fi
}
# wait_for_boot <serial-log> <qemu-pid> [timeout]
# Waits for "KubeSolo is running" marker in serial log.
# Returns 0 on success, 1 on timeout/failure.
# Sets BOOT_ELAPSED to seconds taken.
wait_for_boot() {
local serial_log="$1"
local qemu_pid="$2"
local timeout="${3:-180}"
BOOT_ELAPSED=0
while [ "$BOOT_ELAPSED" -lt "$timeout" ]; do
if grep -q "\[kubesolo-init\] \[OK\] KubeSolo is running" "$serial_log" 2>/dev/null; then
echo ""
echo " Boot completed in ${BOOT_ELAPSED}s"
return 0
fi
if ! kill -0 "$qemu_pid" 2>/dev/null; then
echo ""
echo "==> FAIL: QEMU exited prematurely"
tail -20 "$serial_log" 2>/dev/null
return 1
fi
sleep 2
BOOT_ELAPSED=$((BOOT_ELAPSED + 2))
printf "\r Elapsed: %ds / %ds" "$BOOT_ELAPSED" "$timeout"
done
echo ""
echo "==> FAIL: Boot did not complete within ${timeout}s"
tail -30 "$serial_log" 2>/dev/null
return 1
}
# fetch_kubeconfig <host-port> <output-file>
# Fetches kubeconfig via HTTP from the given host port.
# The port should be the QEMU-forwarded host port mapped to guest port 8080.
# Returns 0 on success, 1 on failure.
fetch_kubeconfig() {
local port="$1"
local output_file="$2"
echo " Fetching kubeconfig from http://localhost:${port}..."
local j=0
while [ $j -lt 30 ]; do
if curl -sf "http://localhost:${port}" -o "$output_file" 2>/dev/null; then
if [ -s "$output_file" ] && grep -q "server:" "$output_file" 2>/dev/null; then
echo " Kubeconfig fetched successfully"
return 0
fi
fi
sleep 2
j=$((j + 1))
done
echo "==> FAIL: Could not fetch kubeconfig from http://localhost:${port}"
return 1
}

View File

@@ -3,6 +3,7 @@
# Usage: ./test/qemu/run-vm.sh <iso-or-img> [options] # Usage: ./test/qemu/run-vm.sh <iso-or-img> [options]
# #
# Options: # Options:
# --arch <arch> Architecture: x86_64 (default) or arm64
# --data-disk <path> Use existing data disk (default: create temp) # --data-disk <path> Use existing data disk (default: create temp)
# --data-size <MB> Size of temp data disk (default: 1024) # --data-size <MB> Size of temp data disk (default: 1024)
# --memory <MB> VM memory (default: 2048) # --memory <MB> VM memory (default: 2048)
@@ -12,6 +13,8 @@
# --ssh-port <port> Forward SSH to host port (default: 2222) # --ssh-port <port> Forward SSH to host port (default: 2222)
# --background Run in background, print PID # --background Run in background, print PID
# --append <args> Extra kernel append args # --append <args> Extra kernel append args
# --kernel <path> Kernel image (required for arm64)
# --initrd <path> Initramfs image (required for arm64)
# #
# Outputs (on stdout): # Outputs (on stdout):
# QEMU_PID=<pid> # QEMU_PID=<pid>
@@ -23,6 +26,7 @@ IMAGE="${1:?Usage: $0 <iso-or-img> [options]}"
shift shift
# Defaults # Defaults
ARCH="x86_64"
DATA_DISK="" DATA_DISK=""
DATA_SIZE_MB=1024 DATA_SIZE_MB=1024
MEMORY=2048 MEMORY=2048
@@ -33,10 +37,13 @@ SSH_PORT=2222
BACKGROUND=0 BACKGROUND=0
EXTRA_APPEND="" EXTRA_APPEND=""
CREATED_DATA_DISK="" CREATED_DATA_DISK=""
VM_KERNEL=""
VM_INITRD=""
# Parse options # Parse options
while [ $# -gt 0 ]; do while [ $# -gt 0 ]; do
case "$1" in case "$1" in
--arch) ARCH="$2"; shift 2 ;;
--data-disk) DATA_DISK="$2"; shift 2 ;; --data-disk) DATA_DISK="$2"; shift 2 ;;
--data-size) DATA_SIZE_MB="$2"; shift 2 ;; --data-size) DATA_SIZE_MB="$2"; shift 2 ;;
--memory) MEMORY="$2"; shift 2 ;; --memory) MEMORY="$2"; shift 2 ;;
@@ -46,6 +53,8 @@ while [ $# -gt 0 ]; do
--ssh-port) SSH_PORT="$2"; shift 2 ;; --ssh-port) SSH_PORT="$2"; shift 2 ;;
--background) BACKGROUND=1; shift ;; --background) BACKGROUND=1; shift ;;
--append) EXTRA_APPEND="$2"; shift 2 ;; --append) EXTRA_APPEND="$2"; shift 2 ;;
--kernel) VM_KERNEL="$2"; shift 2 ;;
--initrd) VM_INITRD="$2"; shift 2 ;;
*) echo "Unknown option: $1" >&2; exit 1 ;; *) echo "Unknown option: $1" >&2; exit 1 ;;
esac esac
done done
@@ -63,19 +72,49 @@ if [ -z "$SERIAL_LOG" ]; then
SERIAL_LOG=$(mktemp /tmp/kubesolo-serial-XXXXXX.log) SERIAL_LOG=$(mktemp /tmp/kubesolo-serial-XXXXXX.log)
fi fi
# Build QEMU command based on architecture
if [ "$ARCH" = "arm64" ] || [ "$ARCH" = "aarch64" ]; then
# ARM64: qemu-system-aarch64 with -machine virt
# No KVM for cross-arch emulation (TCG only)
CONSOLE="ttyAMA0"
# ARM64 requires explicit kernel + initrd (no -cdrom support with -machine virt)
if [ -z "$VM_KERNEL" ] || [ -z "$VM_INITRD" ]; then
echo "ERROR: ARM64 mode requires --kernel and --initrd options" >&2
exit 1
fi
QEMU_CMD=(
qemu-system-aarch64
-machine virt
-cpu cortex-a72
-m "$MEMORY"
-smp "$CPUS"
-nographic
-net "nic,model=virtio"
-net "user,hostfwd=tcp::${API_PORT}-:6443,hostfwd=tcp::${SSH_PORT}-:22"
-drive "file=$DATA_DISK,format=raw,if=virtio"
-serial "file:$SERIAL_LOG"
-kernel "$VM_KERNEL"
-initrd "$VM_INITRD"
-append "console=${CONSOLE} kubesolo.data=/dev/vda kubesolo.debug $EXTRA_APPEND"
)
else
# x86_64: standard QEMU
CONSOLE="ttyS0,115200n8"
# Detect KVM availability # Detect KVM availability
KVM_FLAG="" KVM_FLAG=""
if [ -w /dev/kvm ] 2>/dev/null; then if [ -w /dev/kvm ] 2>/dev/null; then
KVM_FLAG="-enable-kvm" KVM_FLAG="-enable-kvm"
fi fi
# Build QEMU command
QEMU_CMD=( QEMU_CMD=(
qemu-system-x86_64 qemu-system-x86_64
-m "$MEMORY" -m "$MEMORY"
-smp "$CPUS" -smp "$CPUS"
-nographic -nographic
-net nic,model=virtio -net "nic,model=virtio"
-net "user,hostfwd=tcp::${API_PORT}-:6443,hostfwd=tcp::${SSH_PORT}-:22" -net "user,hostfwd=tcp::${API_PORT}-:6443,hostfwd=tcp::${SSH_PORT}-:22"
-drive "file=$DATA_DISK,format=raw,if=virtio" -drive "file=$DATA_DISK,format=raw,if=virtio"
-serial "file:$SERIAL_LOG" -serial "file:$SERIAL_LOG"
@@ -88,7 +127,7 @@ case "$IMAGE" in
QEMU_CMD+=( QEMU_CMD+=(
-cdrom "$IMAGE" -cdrom "$IMAGE"
-boot d -boot d
-append "console=ttyS0,115200n8 kubesolo.data=/dev/vda kubesolo.debug $EXTRA_APPEND" -append "console=${CONSOLE} kubesolo.data=/dev/vda kubesolo.debug $EXTRA_APPEND"
) )
;; ;;
*.img) *.img)
@@ -101,6 +140,7 @@ case "$IMAGE" in
exit 1 exit 1
;; ;;
esac esac
fi
# Launch # Launch
"${QEMU_CMD[@]}" & "${QEMU_CMD[@]}" &

129
test/qemu/test-boot-arm64-disk.sh Executable file
View File

@@ -0,0 +1,129 @@
#!/bin/bash
# test-boot-arm64-disk.sh — Boot the ARM64 .arm64.img via UEFI + GRUB and
# verify the init system reaches stage 90.
#
# This is the full-stack integration test: UEFI firmware -> GRUB -> kernel ->
# initramfs -> staged init. Contrast with test-boot-arm64.sh which skips the
# bootloader and loads kernel/initramfs directly.
#
# Exit 0 = PASS, Exit 1 = FAIL.
#
# Usage: ./test/qemu/test-boot-arm64-disk.sh [disk.img]
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
VERSION="$(cat "$PROJECT_ROOT/VERSION")"
DISK_IMAGE="${1:-$PROJECT_ROOT/output/kubesolo-os-${VERSION}.arm64.img}"
TIMEOUT=180
echo "==> ARM64 UEFI Disk Boot Test"
echo " Disk image: $DISK_IMAGE"
echo " Timeout: ${TIMEOUT}s"
if [ ! -f "$DISK_IMAGE" ]; then
echo "ERROR: Disk image not found: $DISK_IMAGE"
echo " Run 'make disk-image-arm64' to build it."
exit 1
fi
if ! command -v qemu-system-aarch64 >/dev/null 2>&1; then
echo "ERROR: qemu-system-aarch64 not found."
echo " apt install qemu-system-arm # Debian/Ubuntu"
echo " dnf install qemu-system-aarch64 # Fedora/RHEL"
exit 1
fi
# --- Locate UEFI firmware ---
UEFI_FW=""
for candidate in \
/usr/share/qemu-efi-aarch64/QEMU_EFI.fd \
/usr/share/AAVMF/AAVMF_CODE.fd \
/usr/share/edk2/aarch64/QEMU_EFI.fd \
/usr/share/qemu/edk2-aarch64-code.fd \
/opt/homebrew/share/qemu/edk2-aarch64-code.fd \
/usr/local/share/qemu/edk2-aarch64-code.fd
do
if [ -f "$candidate" ]; then
UEFI_FW="$candidate"
break
fi
done
if [ -z "$UEFI_FW" ]; then
echo "ERROR: No ARM64 UEFI firmware found."
echo " apt install qemu-efi-aarch64"
exit 1
fi
echo " UEFI fw: $UEFI_FW"
# Copy disk image to a scratch file so the test doesn't mutate the source.
# UEFI will write to grubenv on the EFI partition; we don't want to bake those
# changes into the canonical build artifact.
SCRATCH_DISK=$(mktemp /tmp/kubesolo-arm64-disk-test-XXXXXX.img)
SERIAL_LOG=$(mktemp /tmp/kubesolo-arm64-disk-serial-XXXXXX.log)
QEMU_PID=""
cleanup() {
[ -n "$QEMU_PID" ] && kill "$QEMU_PID" 2>/dev/null || true
rm -f "$SCRATCH_DISK" "$SERIAL_LOG"
}
trap cleanup EXIT
cp --reflink=auto "$DISK_IMAGE" "$SCRATCH_DISK" 2>/dev/null || cp "$DISK_IMAGE" "$SCRATCH_DISK"
# --- Launch QEMU ---
qemu-system-aarch64 \
-machine virt \
-cpu cortex-a72 \
-m 2048 \
-smp 2 \
-nographic \
-bios "$UEFI_FW" \
-drive "file=$SCRATCH_DISK,format=raw,if=virtio,media=disk" \
-net nic,model=virtio \
-net user \
-serial "file:$SERIAL_LOG" &
QEMU_PID=$!
echo " Waiting for boot (PID $QEMU_PID)..."
ELAPSED=0
SUCCESS=0
while [ "$ELAPSED" -lt "$TIMEOUT" ]; do
if grep -q "\[kubesolo-init\] \[OK\] Stage 90-kubesolo.sh complete" "$SERIAL_LOG" 2>/dev/null; then
SUCCESS=1
break
fi
if grep -q "KubeSolo is running" "$SERIAL_LOG" 2>/dev/null; then
SUCCESS=1
break
fi
if ! kill -0 "$QEMU_PID" 2>/dev/null; then
echo ""
echo "==> FAIL: QEMU exited prematurely"
echo " Last 30 lines of serial output:"
tail -30 "$SERIAL_LOG" 2>/dev/null || echo " (no output)"
exit 1
fi
sleep 2
ELAPSED=$((ELAPSED + 2))
printf "\r Elapsed: %ds / %ds" "$ELAPSED" "$TIMEOUT"
done
echo ""
kill "$QEMU_PID" 2>/dev/null || true
wait "$QEMU_PID" 2>/dev/null || true
QEMU_PID=""
if [ "$SUCCESS" = "1" ]; then
echo "==> ARM64 UEFI Disk Boot Test PASSED (${ELAPSED}s)"
exit 0
fi
echo "==> ARM64 UEFI Disk Boot Test FAILED (timeout ${TIMEOUT}s)"
echo ""
echo "==> Last 50 lines of serial output:"
tail -50 "$SERIAL_LOG" 2>/dev/null || echo " (no output)"
exit 1

117
test/qemu/test-boot-arm64.sh Executable file
View File

@@ -0,0 +1,117 @@
#!/bin/bash
# test-boot-arm64.sh — Verify ARM64 image boots successfully in QEMU
#
# Uses qemu-system-aarch64 with -machine virt to test ARM64 kernel + initramfs.
# Exit 0 = PASS, Exit 1 = FAIL
#
# Usage: ./test/qemu/test-boot-arm64.sh [kernel] [initramfs]
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
KERNEL="${1:-$PROJECT_ROOT/build/cache/custom-kernel-arm64/Image}"
INITRD="${2:-$PROJECT_ROOT/build/rootfs-work/kubesolo-os.gz}"
TIMEOUT=120
echo "==> ARM64 Boot Test"
echo " Kernel: $KERNEL"
echo " Initrd: $INITRD"
echo " Timeout: ${TIMEOUT}s"
# Verify files exist
if [ ! -f "$KERNEL" ]; then
echo "ERROR: Kernel not found: $KERNEL"
echo " Run 'make kernel-arm64' to build the ARM64 kernel."
exit 1
fi
if [ ! -f "$INITRD" ]; then
echo "ERROR: Initrd not found: $INITRD"
echo " Run 'make initramfs' to build the initramfs."
exit 1
fi
# Verify qemu-system-aarch64 is available
if ! command -v qemu-system-aarch64 >/dev/null 2>&1; then
echo "ERROR: qemu-system-aarch64 not found."
echo " Install QEMU with ARM64 support:"
echo " apt install qemu-system-arm # Debian/Ubuntu"
echo " dnf install qemu-system-aarch64 # Fedora/RHEL"
echo " brew install qemu # macOS"
exit 1
fi
# Create temp data disk
DATA_DISK=$(mktemp /tmp/kubesolo-arm64-test-XXXXXX.img)
dd if=/dev/zero of="$DATA_DISK" bs=1M count=512 2>/dev/null
mkfs.ext4 -q -L KSOLODATA "$DATA_DISK" 2>/dev/null
SERIAL_LOG=$(mktemp /tmp/kubesolo-arm64-serial-XXXXXX.log)
QEMU_PID=""
cleanup() {
[ -n "$QEMU_PID" ] && kill "$QEMU_PID" 2>/dev/null || true
rm -f "$DATA_DISK" "$SERIAL_LOG"
}
trap cleanup EXIT
# Launch QEMU in background
qemu-system-aarch64 \
-machine virt \
-cpu cortex-a72 \
-m 2048 \
-smp 2 \
-nographic \
-kernel "$KERNEL" \
-initrd "$INITRD" \
-append "console=ttyAMA0 kubesolo.data=/dev/vda kubesolo.debug" \
-drive "file=$DATA_DISK,format=raw,if=virtio" \
-net nic,model=virtio \
-net user \
-serial "file:$SERIAL_LOG" &
QEMU_PID=$!
# Wait for boot success marker
echo " Waiting for boot..."
ELAPSED=0
SUCCESS=0
while [ "$ELAPSED" -lt "$TIMEOUT" ]; do
# Check for stage 90 completion (same marker as x86_64 test)
if grep -q "\[kubesolo-init\] \[OK\] Stage 90-kubesolo.sh complete" "$SERIAL_LOG" 2>/dev/null; then
SUCCESS=1
break
fi
# Also check for generic KubeSolo running message
if grep -q "KubeSolo is running" "$SERIAL_LOG" 2>/dev/null; then
SUCCESS=1
break
fi
# Check if QEMU exited prematurely
if ! kill -0 "$QEMU_PID" 2>/dev/null; then
echo ""
echo "==> FAIL: QEMU exited prematurely"
echo " Last 20 lines of serial output:"
tail -20 "$SERIAL_LOG" 2>/dev/null || echo " (no output)"
exit 1
fi
sleep 2
ELAPSED=$((ELAPSED + 2))
printf "\r Elapsed: %ds / %ds" "$ELAPSED" "$TIMEOUT"
done
echo ""
# Kill QEMU
kill "$QEMU_PID" 2>/dev/null || true
wait "$QEMU_PID" 2>/dev/null || true
QEMU_PID=""
if [ "$SUCCESS" = "1" ]; then
echo "==> ARM64 Boot Test PASSED (${ELAPSED}s)"
exit 0
else
echo "==> ARM64 Boot Test FAILED (timeout ${TIMEOUT}s)"
echo ""
echo "==> Last 30 lines of serial output:"
tail -30 "$SERIAL_LOG" 2>/dev/null || echo " (no output)"
exit 1
fi

View File

@@ -5,17 +5,25 @@
set -euo pipefail set -euo pipefail
ISO="${1:?Usage: $0 <path-to-iso>}" ISO="${1:?Usage: $0 <path-to-iso>}"
TIMEOUT_BOOT=120 # seconds to wait for boot success marker TIMEOUT_BOOT=${TIMEOUT_BOOT:-120} # seconds to wait for boot success marker
SERIAL_LOG=$(mktemp /tmp/kubesolo-boot-test-XXXXXX.log) SERIAL_LOG=$(mktemp /tmp/kubesolo-boot-test-XXXXXX.log)
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
. "$SCRIPT_DIR/../lib/qemu-helpers.sh"
# Temp data disk # Temp data disk
DATA_DISK=$(mktemp /tmp/kubesolo-data-XXXXXX.img) DATA_DISK=$(mktemp /tmp/kubesolo-data-XXXXXX.img)
dd if=/dev/zero of="$DATA_DISK" bs=1M count=512 2>/dev/null dd if=/dev/zero of="$DATA_DISK" bs=1M count=512 2>/dev/null
mkfs.ext4 -q -L KSOLODATA "$DATA_DISK" 2>/dev/null mkfs.ext4 -q -L KSOLODATA "$DATA_DISK" 2>/dev/null
QEMU_PID=""
EXTRACT_DIR=""
cleanup() { cleanup() {
kill "$QEMU_PID" 2>/dev/null || true [ -n "$QEMU_PID" ] && kill "$QEMU_PID" 2>/dev/null || true
rm -f "$DATA_DISK" "$SERIAL_LOG" rm -f "$DATA_DISK" "$SERIAL_LOG"
[ -n "$EXTRACT_DIR" ] && rm -rf "$EXTRACT_DIR"
} }
trap cleanup EXIT trap cleanup EXIT
@@ -23,16 +31,25 @@ echo "==> Boot test: $ISO"
echo " Timeout: ${TIMEOUT_BOOT}s" echo " Timeout: ${TIMEOUT_BOOT}s"
echo " Serial log: $SERIAL_LOG" echo " Serial log: $SERIAL_LOG"
# Launch QEMU in background # Extract kernel from ISO
EXTRACT_DIR="$(mktemp -d /tmp/kubesolo-extract-XXXXXX)"
extract_kernel_from_iso "$ISO" "$EXTRACT_DIR"
KVM_FLAG=$(detect_kvm)
[ -n "$KVM_FLAG" ] && echo " KVM acceleration: enabled"
# Launch QEMU in background with direct kernel boot
# shellcheck disable=SC2086
qemu-system-x86_64 \ qemu-system-x86_64 \
-m 2048 -smp 2 \ -m 2048 -smp 2 \
-nographic \ -nographic \
-cdrom "$ISO" \ $KVM_FLAG \
-boot d \ -kernel "$VMLINUZ" \
-initrd "$INITRAMFS" \
-drive "file=$DATA_DISK,format=raw,if=virtio" \ -drive "file=$DATA_DISK,format=raw,if=virtio" \
-net nic,model=virtio \ -net "nic,model=virtio" \
-net user \ -net user \
-serial file:"$SERIAL_LOG" \ -serial "file:$SERIAL_LOG" \
-append "console=ttyS0,115200n8 kubesolo.data=/dev/vda kubesolo.debug" \ -append "console=ttyS0,115200n8 kubesolo.data=/dev/vda kubesolo.debug" \
& &
QEMU_PID=$! QEMU_PID=$!
@@ -41,7 +58,7 @@ QEMU_PID=$!
echo " Waiting for boot..." echo " Waiting for boot..."
ELAPSED=0 ELAPSED=0
while [ "$ELAPSED" -lt "$TIMEOUT_BOOT" ]; do while [ "$ELAPSED" -lt "$TIMEOUT_BOOT" ]; do
if grep -q "\[kubesolo-init\] \[OK\] Stage 90-kubesolo.sh complete" "$SERIAL_LOG" 2>/dev/null; then if grep -q "\[kubesolo-init\] \[OK\] KubeSolo is running" "$SERIAL_LOG" 2>/dev/null; then
echo "" echo ""
echo "==> PASS: KubeSolo OS booted successfully in ${ELAPSED}s" echo "==> PASS: KubeSolo OS booted successfully in ${ELAPSED}s"
exit 0 exit 0

View File

@@ -3,8 +3,6 @@ package cmd
import ( import (
"fmt" "fmt"
"log/slog" "log/slog"
"github.com/portainer/kubesolo-os/update/pkg/grubenv"
) )
// Activate switches the boot target to the passive partition. // Activate switches the boot target to the passive partition.
@@ -12,7 +10,7 @@ import (
// with boot_counter=3. If health checks fail 3 times, GRUB auto-rolls back. // with boot_counter=3. If health checks fail 3 times, GRUB auto-rolls back.
func Activate(args []string) error { func Activate(args []string) error {
opts := parseOpts(args) opts := parseOpts(args)
env := grubenv.New(opts.GrubenvPath) env := opts.NewBootEnv()
// Get passive slot (the one we want to boot into) // Get passive slot (the one we want to boot into)
passiveSlot, err := env.PassiveSlot() passiveSlot, err := env.PassiveSlot()

View File

@@ -4,7 +4,6 @@ import (
"fmt" "fmt"
"log/slog" "log/slog"
"github.com/portainer/kubesolo-os/update/pkg/grubenv"
"github.com/portainer/kubesolo-os/update/pkg/image" "github.com/portainer/kubesolo-os/update/pkg/image"
"github.com/portainer/kubesolo-os/update/pkg/partition" "github.com/portainer/kubesolo-os/update/pkg/partition"
) )
@@ -18,7 +17,7 @@ func Apply(args []string) error {
return fmt.Errorf("--server is required") return fmt.Errorf("--server is required")
} }
env := grubenv.New(opts.GrubenvPath) env := opts.NewBootEnv()
// Determine passive slot // Determine passive slot
passiveSlot, err := env.PassiveSlot() passiveSlot, err := env.PassiveSlot()

View File

@@ -4,7 +4,6 @@ import (
"fmt" "fmt"
"log/slog" "log/slog"
"github.com/portainer/kubesolo-os/update/pkg/grubenv"
"github.com/portainer/kubesolo-os/update/pkg/image" "github.com/portainer/kubesolo-os/update/pkg/image"
"github.com/portainer/kubesolo-os/update/pkg/partition" "github.com/portainer/kubesolo-os/update/pkg/partition"
) )
@@ -19,7 +18,7 @@ func Check(args []string) error {
} }
// Get current version from active partition // Get current version from active partition
env := grubenv.New(opts.GrubenvPath) env := opts.NewBootEnv()
activeSlot, err := env.ActiveSlot() activeSlot, err := env.ActiveSlot()
if err != nil { if err != nil {
return fmt.Errorf("reading active slot: %w", err) return fmt.Errorf("reading active slot: %w", err)

View File

@@ -5,7 +5,6 @@ import (
"log/slog" "log/slog"
"time" "time"
"github.com/portainer/kubesolo-os/update/pkg/grubenv"
"github.com/portainer/kubesolo-os/update/pkg/health" "github.com/portainer/kubesolo-os/update/pkg/health"
) )
@@ -15,7 +14,7 @@ import (
// init script) to confirm the system is healthy. // init script) to confirm the system is healthy.
func Healthcheck(args []string) error { func Healthcheck(args []string) error {
opts := parseOpts(args) opts := parseOpts(args)
env := grubenv.New(opts.GrubenvPath) env := opts.NewBootEnv()
// Check if already marked successful // Check if already marked successful
success, err := env.BootSuccess() success, err := env.BootSuccess()

View File

@@ -1,11 +1,27 @@
package cmd package cmd
import (
"github.com/portainer/kubesolo-os/update/pkg/bootenv"
)
// opts holds shared command-line options for all subcommands. // opts holds shared command-line options for all subcommands.
type opts struct { type opts struct {
ServerURL string ServerURL string
GrubenvPath string GrubenvPath string
TimeoutSecs int TimeoutSecs int
PubKeyPath string PubKeyPath string
BootEnvType string // "grub" or "rpi"
BootEnvPath string // path for RPi boot control dir
}
// NewBootEnv creates a BootEnv from the parsed options.
func (o opts) NewBootEnv() bootenv.BootEnv {
switch o.BootEnvType {
case "rpi":
return bootenv.NewRPi(o.BootEnvPath)
default:
return bootenv.NewGRUB(o.GrubenvPath)
}
} }
// parseOpts extracts command-line flags from args. // parseOpts extracts command-line flags from args.
@@ -14,6 +30,7 @@ func parseOpts(args []string) opts {
o := opts{ o := opts{
GrubenvPath: "/boot/grub/grubenv", GrubenvPath: "/boot/grub/grubenv",
TimeoutSecs: 120, TimeoutSecs: 120,
BootEnvType: "grub",
} }
for i := 0; i < len(args); i++ { for i := 0; i < len(args); i++ {
@@ -46,6 +63,16 @@ func parseOpts(args []string) opts {
o.PubKeyPath = args[i+1] o.PubKeyPath = args[i+1]
i++ i++
} }
case "--bootenv":
if i+1 < len(args) {
o.BootEnvType = args[i+1]
i++
}
case "--bootenv-path":
if i+1 < len(args) {
o.BootEnvPath = args[i+1]
i++
}
} }
} }

View File

@@ -3,15 +3,13 @@ package cmd
import ( import (
"fmt" "fmt"
"log/slog" "log/slog"
"github.com/portainer/kubesolo-os/update/pkg/grubenv"
) )
// Rollback forces an immediate switch to the other partition. // Rollback forces an immediate switch to the other partition.
// Use this to manually revert to the previous version. // Use this to manually revert to the previous version.
func Rollback(args []string) error { func Rollback(args []string) error {
opts := parseOpts(args) opts := parseOpts(args)
env := grubenv.New(opts.GrubenvPath) env := opts.NewBootEnv()
activeSlot, err := env.ActiveSlot() activeSlot, err := env.ActiveSlot()
if err != nil { if err != nil {

View File

@@ -2,42 +2,50 @@ package cmd
import ( import (
"fmt" "fmt"
"github.com/portainer/kubesolo-os/update/pkg/grubenv"
) )
// Status displays the current A/B slot configuration and boot state. // Status displays the current A/B slot configuration and boot state.
func Status(args []string) error { func Status(args []string) error {
opts := parseOpts(args) opts := parseOpts(args)
env := grubenv.New(opts.GrubenvPath) env := opts.NewBootEnv()
vars, err := env.ReadAll() activeSlot, err := env.ActiveSlot()
if err != nil { if err != nil {
return fmt.Errorf("reading GRUB environment: %w", err) return fmt.Errorf("reading active slot: %w", err)
} }
activeSlot := vars["active_slot"] passiveSlot, err := env.PassiveSlot()
bootCounter := vars["boot_counter"] if err != nil {
bootSuccess := vars["boot_success"] return fmt.Errorf("reading passive slot: %w", err)
}
passiveSlot := "B" bootCounter, err := env.BootCounter()
if activeSlot == "B" { if err != nil {
passiveSlot = "A" return fmt.Errorf("reading boot counter: %w", err)
}
bootSuccess, err := env.BootSuccess()
if err != nil {
return fmt.Errorf("reading boot success: %w", err)
} }
fmt.Println("KubeSolo OS — A/B Partition Status") fmt.Println("KubeSolo OS — A/B Partition Status")
fmt.Println("───────────────────────────────────") fmt.Println("───────────────────────────────────")
fmt.Printf(" Active slot: %s\n", activeSlot) fmt.Printf(" Active slot: %s\n", activeSlot)
fmt.Printf(" Passive slot: %s\n", passiveSlot) fmt.Printf(" Passive slot: %s\n", passiveSlot)
fmt.Printf(" Boot counter: %s\n", bootCounter) fmt.Printf(" Boot counter: %d\n", bootCounter)
fmt.Printf(" Boot success: %s\n", bootSuccess) if bootSuccess {
fmt.Printf(" Boot success: 1\n")
} else {
fmt.Printf(" Boot success: 0\n")
}
if bootSuccess == "1" { if bootSuccess {
fmt.Println("\n ✓ System is healthy (boot confirmed)") fmt.Println("\n ✓ System is healthy (boot confirmed)")
} else if bootCounter == "0" { } else if bootCounter == 0 {
fmt.Println("\n ✗ Boot counter exhausted — rollback will occur on next reboot") fmt.Println("\n ✗ Boot counter exhausted — rollback will occur on next reboot")
} else { } else {
fmt.Printf("\n ⚠ Boot pending verification (%s attempts remaining)\n", bootCounter) fmt.Printf("\n ⚠ Boot pending verification (%d attempts remaining)\n", bootCounter)
} }
return nil return nil

View File

@@ -0,0 +1,27 @@
// Package bootenv provides a platform-independent interface for managing
// A/B boot environments. It abstracts GRUB (x86_64) and RPi firmware
// (ARM64) behind a common interface.
package bootenv
// BootEnv provides read/write access to A/B boot environment variables.
type BootEnv interface {
// ActiveSlot returns the currently active boot slot ("A" or "B").
ActiveSlot() (string, error)
// PassiveSlot returns the currently passive boot slot.
PassiveSlot() (string, error)
// BootCounter returns the current boot counter value.
BootCounter() (int, error)
// BootSuccess returns whether the last boot was marked successful.
BootSuccess() (bool, error)
// MarkBootSuccess marks the current boot as successful.
MarkBootSuccess() error
// ActivateSlot switches the active boot slot and resets the counter.
ActivateSlot(slot string) error
// ForceRollback switches to the other slot immediately.
ForceRollback() error
}
const (
SlotA = "A"
SlotB = "B"
)

View File

@@ -0,0 +1,533 @@
package bootenv
import (
"os"
"path/filepath"
"strconv"
"strings"
"testing"
)
// createTestGrubenv writes a properly formatted 1024-byte grubenv file.
func createTestGrubenv(t *testing.T, dir string, vars map[string]string) string {
t.Helper()
path := filepath.Join(dir, "grubenv")
var sb strings.Builder
sb.WriteString("# GRUB Environment Block\n")
for k, v := range vars {
sb.WriteString(k + "=" + v + "\n")
}
content := sb.String()
padding := 1024 - len(content)
if padding > 0 {
content += strings.Repeat("#", padding)
}
if err := os.WriteFile(path, []byte(content), 0o644); err != nil {
t.Fatal(err)
}
return path
}
// TestGRUBActiveSlot verifies ActiveSlot reads the correct value.
func TestGRUBActiveSlot(t *testing.T) {
dir := t.TempDir()
path := createTestGrubenv(t, dir, map[string]string{
"active_slot": "A",
"boot_counter": "3",
"boot_success": "1",
})
env := NewGRUB(path)
slot, err := env.ActiveSlot()
if err != nil {
t.Fatal(err)
}
if slot != "A" {
t.Errorf("expected A, got %s", slot)
}
}
// TestGRUBPassiveSlot verifies PassiveSlot returns the opposite slot.
func TestGRUBPassiveSlot(t *testing.T) {
dir := t.TempDir()
path := createTestGrubenv(t, dir, map[string]string{
"active_slot": "A",
"boot_counter": "3",
"boot_success": "0",
})
env := NewGRUB(path)
passive, err := env.PassiveSlot()
if err != nil {
t.Fatal(err)
}
if passive != "B" {
t.Errorf("expected B, got %s", passive)
}
}
// TestGRUBBootCounter verifies BootCounter reads the correct value.
func TestGRUBBootCounter(t *testing.T) {
dir := t.TempDir()
path := createTestGrubenv(t, dir, map[string]string{
"active_slot": "A",
"boot_counter": "2",
"boot_success": "0",
})
env := NewGRUB(path)
counter, err := env.BootCounter()
if err != nil {
t.Fatal(err)
}
if counter != 2 {
t.Errorf("expected 2, got %d", counter)
}
}
// TestGRUBBootSuccess verifies BootSuccess reads the correct value.
func TestGRUBBootSuccess(t *testing.T) {
dir := t.TempDir()
path := createTestGrubenv(t, dir, map[string]string{
"active_slot": "A",
"boot_counter": "3",
"boot_success": "1",
})
env := NewGRUB(path)
success, err := env.BootSuccess()
if err != nil {
t.Fatal(err)
}
if !success {
t.Error("expected true, got false")
}
}
// TestGRUBMarkBootSuccess verifies marking boot as successful.
func TestGRUBMarkBootSuccess(t *testing.T) {
dir := t.TempDir()
path := createTestGrubenv(t, dir, map[string]string{
"active_slot": "B",
"boot_counter": "1",
"boot_success": "0",
})
env := NewGRUB(path)
if err := env.MarkBootSuccess(); err != nil {
t.Fatal(err)
}
success, err := env.BootSuccess()
if err != nil {
t.Fatal(err)
}
if !success {
t.Error("expected boot_success=true after MarkBootSuccess")
}
counter, err := env.BootCounter()
if err != nil {
t.Fatal(err)
}
if counter != 3 {
t.Errorf("expected boot_counter=3 after MarkBootSuccess, got %d", counter)
}
}
// TestGRUBActivateSlot verifies slot activation sets correct state.
func TestGRUBActivateSlot(t *testing.T) {
dir := t.TempDir()
path := createTestGrubenv(t, dir, map[string]string{
"active_slot": "A",
"boot_counter": "3",
"boot_success": "1",
})
env := NewGRUB(path)
if err := env.ActivateSlot("B"); err != nil {
t.Fatal(err)
}
slot, _ := env.ActiveSlot()
if slot != "B" {
t.Errorf("expected B, got %s", slot)
}
counter, _ := env.BootCounter()
if counter != 3 {
t.Errorf("expected counter=3, got %d", counter)
}
success, _ := env.BootSuccess()
if success {
t.Error("expected boot_success=false after ActivateSlot")
}
}
// TestGRUBForceRollback verifies rollback switches to passive slot.
func TestGRUBForceRollback(t *testing.T) {
dir := t.TempDir()
path := createTestGrubenv(t, dir, map[string]string{
"active_slot": "A",
"boot_counter": "3",
"boot_success": "1",
})
env := NewGRUB(path)
if err := env.ForceRollback(); err != nil {
t.Fatal(err)
}
slot, _ := env.ActiveSlot()
if slot != "B" {
t.Errorf("expected B after rollback from A, got %s", slot)
}
}
// TestGRUBSlotCycling verifies A->B->A slot switching.
func TestGRUBSlotCycling(t *testing.T) {
dir := t.TempDir()
path := createTestGrubenv(t, dir, map[string]string{
"active_slot": "A",
"boot_counter": "3",
"boot_success": "1",
})
env := NewGRUB(path)
// A -> B
if err := env.ActivateSlot("B"); err != nil {
t.Fatal(err)
}
slot, _ := env.ActiveSlot()
if slot != "B" {
t.Fatalf("expected B, got %s", slot)
}
// B -> A
if err := env.ActivateSlot("A"); err != nil {
t.Fatal(err)
}
slot, _ = env.ActiveSlot()
if slot != "A" {
t.Fatalf("expected A, got %s", slot)
}
}
// TestGRUBActivateInvalidSlot verifies invalid slot is rejected.
func TestGRUBActivateInvalidSlot(t *testing.T) {
dir := t.TempDir()
path := createTestGrubenv(t, dir, map[string]string{
"active_slot": "A",
"boot_counter": "3",
"boot_success": "0",
})
env := NewGRUB(path)
if err := env.ActivateSlot("C"); err == nil {
t.Fatal("expected error for invalid slot")
}
}
// TestRPiActiveSlot verifies ActiveSlot reads from autoboot.txt.
func TestRPiActiveSlot(t *testing.T) {
dir := t.TempDir()
createTestAutobootFiles(t, dir, 2, 3, 3, false)
env := NewRPi(dir)
slot, err := env.ActiveSlot()
if err != nil {
t.Fatal(err)
}
if slot != "A" {
t.Errorf("expected A (partition 2), got %s", slot)
}
}
// TestRPiActiveSlotB verifies slot B with partition 3.
func TestRPiActiveSlotB(t *testing.T) {
dir := t.TempDir()
createTestAutobootFiles(t, dir, 3, 2, 3, true)
env := NewRPi(dir)
slot, err := env.ActiveSlot()
if err != nil {
t.Fatal(err)
}
if slot != "B" {
t.Errorf("expected B (partition 3), got %s", slot)
}
}
// TestRPiPassiveSlot verifies passive slot is opposite of active.
func TestRPiPassiveSlot(t *testing.T) {
dir := t.TempDir()
createTestAutobootFiles(t, dir, 2, 3, 3, false)
env := NewRPi(dir)
passive, err := env.PassiveSlot()
if err != nil {
t.Fatal(err)
}
if passive != "B" {
t.Errorf("expected B, got %s", passive)
}
}
// TestRPiBootCounter verifies counter is read from status file.
func TestRPiBootCounter(t *testing.T) {
dir := t.TempDir()
createTestAutobootFiles(t, dir, 2, 3, 2, false)
env := NewRPi(dir)
counter, err := env.BootCounter()
if err != nil {
t.Fatal(err)
}
if counter != 2 {
t.Errorf("expected 2, got %d", counter)
}
}
// TestRPiBootCounterMissingFile verifies default when status file is absent.
func TestRPiBootCounterMissingFile(t *testing.T) {
dir := t.TempDir()
// Only create autoboot.txt, no boot-status
autoboot := "[all]\ntryboot_a_b=1\nboot_partition=2\n[tryboot]\nboot_partition=3\n"
if err := os.WriteFile(filepath.Join(dir, "autoboot.txt"), []byte(autoboot), 0o644); err != nil {
t.Fatal(err)
}
env := NewRPi(dir)
counter, err := env.BootCounter()
if err != nil {
t.Fatal(err)
}
if counter != 3 {
t.Errorf("expected default counter 3, got %d", counter)
}
}
// TestRPiBootSuccess verifies success is read from status file.
func TestRPiBootSuccess(t *testing.T) {
dir := t.TempDir()
createTestAutobootFiles(t, dir, 2, 3, 3, true)
env := NewRPi(dir)
success, err := env.BootSuccess()
if err != nil {
t.Fatal(err)
}
if !success {
t.Error("expected true, got false")
}
}
// TestRPiMarkBootSuccess verifies marking boot success updates both files.
func TestRPiMarkBootSuccess(t *testing.T) {
dir := t.TempDir()
createTestAutobootFiles(t, dir, 2, 3, 1, false)
env := NewRPi(dir)
if err := env.MarkBootSuccess(); err != nil {
t.Fatal(err)
}
// Active slot should still be A
slot, _ := env.ActiveSlot()
if slot != "A" {
t.Errorf("expected active slot A, got %s", slot)
}
// Boot success should be true
success, _ := env.BootSuccess()
if !success {
t.Error("expected boot_success=true after MarkBootSuccess")
}
// Counter should be reset to 3
counter, _ := env.BootCounter()
if counter != 3 {
t.Errorf("expected counter=3 after MarkBootSuccess, got %d", counter)
}
// [all] boot_partition should be 2 (slot A, making it permanent)
data, _ := os.ReadFile(filepath.Join(dir, "autoboot.txt"))
if !strings.Contains(string(data), "boot_partition=2") {
t.Error("expected [all] boot_partition=2 after MarkBootSuccess")
}
}
// TestRPiActivateSlot verifies slot activation updates tryboot and status.
func TestRPiActivateSlot(t *testing.T) {
dir := t.TempDir()
createTestAutobootFiles(t, dir, 2, 3, 3, true)
env := NewRPi(dir)
if err := env.ActivateSlot("B"); err != nil {
t.Fatal(err)
}
// [tryboot] should now point to partition 3 (slot B)
data, _ := os.ReadFile(filepath.Join(dir, "autoboot.txt"))
content := string(data)
// Find [tryboot] section and check boot_partition
idx := strings.Index(content, "[tryboot]")
if idx < 0 {
t.Fatal("missing [tryboot] section")
}
trybootSection := content[idx:]
if !strings.Contains(trybootSection, "boot_partition=3") {
t.Errorf("expected [tryboot] boot_partition=3, got: %s", trybootSection)
}
// Status should be reset
success, _ := env.BootSuccess()
if success {
t.Error("expected boot_success=false after ActivateSlot")
}
counter, _ := env.BootCounter()
if counter != 3 {
t.Errorf("expected counter=3, got %d", counter)
}
}
// TestRPiActivateInvalidSlot verifies invalid slot is rejected.
func TestRPiActivateInvalidSlot(t *testing.T) {
dir := t.TempDir()
createTestAutobootFiles(t, dir, 2, 3, 3, false)
env := NewRPi(dir)
if err := env.ActivateSlot("C"); err == nil {
t.Fatal("expected error for invalid slot")
}
}
// TestRPiForceRollback verifies rollback swaps the active slot.
func TestRPiForceRollback(t *testing.T) {
dir := t.TempDir()
createTestAutobootFiles(t, dir, 2, 3, 3, true)
env := NewRPi(dir)
if err := env.ForceRollback(); err != nil {
t.Fatal(err)
}
// [all] should now point to partition 3 (slot B)
slot, _ := env.ActiveSlot()
if slot != "B" {
t.Errorf("expected B after rollback from A, got %s", slot)
}
// Success should be false
success, _ := env.BootSuccess()
if success {
t.Error("expected boot_success=false after ForceRollback")
}
}
// TestRPiSlotCycling verifies A->B->A slot switching works.
func TestRPiSlotCycling(t *testing.T) {
dir := t.TempDir()
createTestAutobootFiles(t, dir, 2, 3, 3, true)
env := NewRPi(dir)
// Rollback A -> B
if err := env.ForceRollback(); err != nil {
t.Fatal(err)
}
slot, _ := env.ActiveSlot()
if slot != "B" {
t.Fatalf("expected B, got %s", slot)
}
// Rollback B -> A
if err := env.ForceRollback(); err != nil {
t.Fatal(err)
}
slot, _ = env.ActiveSlot()
if slot != "A" {
t.Fatalf("expected A, got %s", slot)
}
}
// TestInterfaceCompliance verifies both implementations satisfy BootEnv.
func TestInterfaceCompliance(t *testing.T) {
dir := t.TempDir()
grubPath := createTestGrubenv(t, dir, map[string]string{
"active_slot": "A",
"boot_counter": "3",
"boot_success": "0",
})
rpiDir := t.TempDir()
createTestAutobootFiles(t, rpiDir, 2, 3, 3, false)
impls := map[string]BootEnv{
"grub": NewGRUB(grubPath),
"rpi": NewRPi(rpiDir),
}
for name, env := range impls {
t.Run(name, func(t *testing.T) {
slot, err := env.ActiveSlot()
if err != nil {
t.Fatalf("ActiveSlot: %v", err)
}
if slot != "A" {
t.Errorf("ActiveSlot: expected A, got %s", slot)
}
passive, err := env.PassiveSlot()
if err != nil {
t.Fatalf("PassiveSlot: %v", err)
}
if passive != "B" {
t.Errorf("PassiveSlot: expected B, got %s", passive)
}
counter, err := env.BootCounter()
if err != nil {
t.Fatalf("BootCounter: %v", err)
}
if counter != 3 {
t.Errorf("BootCounter: expected 3, got %d", counter)
}
success, err := env.BootSuccess()
if err != nil {
t.Fatalf("BootSuccess: %v", err)
}
if success {
t.Error("BootSuccess: expected false")
}
})
}
}
// createTestAutobootFiles is a helper that writes both autoboot.txt and boot-status.
func createTestAutobootFiles(t *testing.T, dir string, allPart, trybootPart, counter int, success bool) {
t.Helper()
autoboot := "[all]\ntryboot_a_b=1\nboot_partition=" + strconv.Itoa(allPart) + "\n"
autoboot += "[tryboot]\nboot_partition=" + strconv.Itoa(trybootPart) + "\n"
if err := os.WriteFile(filepath.Join(dir, "autoboot.txt"), []byte(autoboot), 0o644); err != nil {
t.Fatal(err)
}
successVal := "0"
if success {
successVal = "1"
}
status := "boot_counter=" + strconv.Itoa(counter) + "\nboot_success=" + successVal + "\n"
if err := os.WriteFile(filepath.Join(dir, "boot-status"), []byte(status), 0o644); err != nil {
t.Fatal(err)
}
}

View File

@@ -0,0 +1,23 @@
package bootenv
import (
"github.com/portainer/kubesolo-os/update/pkg/grubenv"
)
// GRUBEnv implements BootEnv using GRUB environment variables.
type GRUBEnv struct {
env *grubenv.Env
}
// NewGRUB creates a new GRUB-based BootEnv.
func NewGRUB(path string) BootEnv {
return &GRUBEnv{env: grubenv.New(path)}
}
func (g *GRUBEnv) ActiveSlot() (string, error) { return g.env.ActiveSlot() }
func (g *GRUBEnv) PassiveSlot() (string, error) { return g.env.PassiveSlot() }
func (g *GRUBEnv) BootCounter() (int, error) { return g.env.BootCounter() }
func (g *GRUBEnv) BootSuccess() (bool, error) { return g.env.BootSuccess() }
func (g *GRUBEnv) MarkBootSuccess() error { return g.env.MarkBootSuccess() }
func (g *GRUBEnv) ActivateSlot(slot string) error { return g.env.ActivateSlot(slot) }
func (g *GRUBEnv) ForceRollback() error { return g.env.ForceRollback() }

267
update/pkg/bootenv/rpi.go Normal file
View File

@@ -0,0 +1,267 @@
package bootenv
import (
"fmt"
"os"
"path/filepath"
"strconv"
"strings"
)
const (
// RPi partition numbers: slot A = partition 2, slot B = partition 3.
rpiSlotAPartition = 2
rpiSlotBPartition = 3
defaultBootCounter = 3
)
// RPiEnv implements BootEnv using Raspberry Pi firmware autoboot.txt.
type RPiEnv struct {
autobootPath string // path to autoboot.txt
statusPath string // path to boot-status file
}
// NewRPi creates a new RPi-based BootEnv.
// dir is the directory containing autoboot.txt (typically the boot control
// partition mount point).
func NewRPi(dir string) BootEnv {
return &RPiEnv{
autobootPath: filepath.Join(dir, "autoboot.txt"),
statusPath: filepath.Join(dir, "boot-status"),
}
}
func (r *RPiEnv) ActiveSlot() (string, error) {
partNum, err := r.readAllBootPartition()
if err != nil {
return "", fmt.Errorf("reading active slot: %w", err)
}
return partNumToSlot(partNum)
}
func (r *RPiEnv) PassiveSlot() (string, error) {
active, err := r.ActiveSlot()
if err != nil {
return "", err
}
if active == SlotA {
return SlotB, nil
}
return SlotA, nil
}
func (r *RPiEnv) BootCounter() (int, error) {
status, err := r.readStatus()
if err != nil {
return -1, err
}
val, ok := status["boot_counter"]
if !ok {
return defaultBootCounter, nil
}
n, err := strconv.Atoi(val)
if err != nil {
return -1, fmt.Errorf("invalid boot_counter %q: %w", val, err)
}
return n, nil
}
func (r *RPiEnv) BootSuccess() (bool, error) {
status, err := r.readStatus()
if err != nil {
return false, err
}
return status["boot_success"] == "1", nil
}
func (r *RPiEnv) MarkBootSuccess() error {
// Make the current slot permanent by updating [all] boot_partition
active, err := r.ActiveSlot()
if err != nil {
return fmt.Errorf("marking boot success: %w", err)
}
partNum := slotToPartNum(active)
if err := r.writeAllBootPartition(partNum); err != nil {
return err
}
return r.writeStatus(defaultBootCounter, true)
}
func (r *RPiEnv) ActivateSlot(slot string) error {
if slot != SlotA && slot != SlotB {
return fmt.Errorf("invalid slot: %q (must be A or B)", slot)
}
partNum := slotToPartNum(slot)
// Update [tryboot] to point to the new slot
if err := r.writeTrybootPartition(partNum); err != nil {
return err
}
return r.writeStatus(defaultBootCounter, false)
}
func (r *RPiEnv) ForceRollback() error {
passive, err := r.PassiveSlot()
if err != nil {
return err
}
// Swap the [all] boot_partition to the other slot
partNum := slotToPartNum(passive)
if err := r.writeAllBootPartition(partNum); err != nil {
return err
}
if err := r.writeTrybootPartition(partNum); err != nil {
return err
}
return r.writeStatus(defaultBootCounter, false)
}
// readAllBootPartition reads the boot_partition value from the [all] section.
func (r *RPiEnv) readAllBootPartition() (int, error) {
sections, err := r.parseAutoboot()
if err != nil {
return 0, err
}
val, ok := sections["all"]["boot_partition"]
if !ok {
return 0, fmt.Errorf("boot_partition not found in [all] section")
}
return strconv.Atoi(val)
}
// writeAllBootPartition updates the [all] boot_partition value.
func (r *RPiEnv) writeAllBootPartition(partNum int) error {
sections, err := r.parseAutoboot()
if err != nil {
return err
}
if sections["all"] == nil {
sections["all"] = make(map[string]string)
}
sections["all"]["boot_partition"] = strconv.Itoa(partNum)
return r.writeAutoboot(sections)
}
// writeTrybootPartition updates the [tryboot] boot_partition value.
func (r *RPiEnv) writeTrybootPartition(partNum int) error {
sections, err := r.parseAutoboot()
if err != nil {
return err
}
if sections["tryboot"] == nil {
sections["tryboot"] = make(map[string]string)
}
sections["tryboot"]["boot_partition"] = strconv.Itoa(partNum)
return r.writeAutoboot(sections)
}
// parseAutoboot reads autoboot.txt into a map of section -> key=value pairs.
func (r *RPiEnv) parseAutoboot() (map[string]map[string]string, error) {
data, err := os.ReadFile(r.autobootPath)
if err != nil {
return nil, fmt.Errorf("reading autoboot.txt: %w", err)
}
sections := make(map[string]map[string]string)
currentSection := ""
for _, line := range strings.Split(string(data), "\n") {
line = strings.TrimSpace(line)
if line == "" {
continue
}
if strings.HasPrefix(line, "[") && strings.HasSuffix(line, "]") {
currentSection = line[1 : len(line)-1]
if sections[currentSection] == nil {
sections[currentSection] = make(map[string]string)
}
continue
}
parts := strings.SplitN(line, "=", 2)
if len(parts) == 2 && currentSection != "" {
sections[currentSection][strings.TrimSpace(parts[0])] = strings.TrimSpace(parts[1])
}
}
return sections, nil
}
// writeAutoboot writes sections back to autoboot.txt.
// Section order: [all] first, then [tryboot].
func (r *RPiEnv) writeAutoboot(sections map[string]map[string]string) error {
var sb strings.Builder
// Write [all] section first
if all, ok := sections["all"]; ok {
sb.WriteString("[all]\n")
for k, v := range all {
sb.WriteString(k + "=" + v + "\n")
}
}
// Write [tryboot] section
if tryboot, ok := sections["tryboot"]; ok {
sb.WriteString("[tryboot]\n")
for k, v := range tryboot {
sb.WriteString(k + "=" + v + "\n")
}
}
return os.WriteFile(r.autobootPath, []byte(sb.String()), 0o644)
}
// readStatus reads the boot-status key=value file.
func (r *RPiEnv) readStatus() (map[string]string, error) {
data, err := os.ReadFile(r.statusPath)
if err != nil {
if os.IsNotExist(err) {
// Return defaults if status file doesn't exist yet
return map[string]string{
"boot_counter": strconv.Itoa(defaultBootCounter),
"boot_success": "0",
}, nil
}
return nil, fmt.Errorf("reading boot-status: %w", err)
}
status := make(map[string]string)
for _, line := range strings.Split(string(data), "\n") {
line = strings.TrimSpace(line)
if line == "" {
continue
}
parts := strings.SplitN(line, "=", 2)
if len(parts) == 2 {
status[strings.TrimSpace(parts[0])] = strings.TrimSpace(parts[1])
}
}
return status, nil
}
// writeStatus writes boot_counter and boot_success to the status file.
func (r *RPiEnv) writeStatus(counter int, success bool) error {
successVal := "0"
if success {
successVal = "1"
}
content := fmt.Sprintf("boot_counter=%d\nboot_success=%s\n", counter, successVal)
return os.WriteFile(r.statusPath, []byte(content), 0o644)
}
func partNumToSlot(partNum int) (string, error) {
switch partNum {
case rpiSlotAPartition:
return SlotA, nil
case rpiSlotBPartition:
return SlotB, nil
default:
return "", fmt.Errorf("unknown partition number %d (expected %d or %d)", partNum, rpiSlotAPartition, rpiSlotBPartition)
}
}
func slotToPartNum(slot string) int {
if slot == SlotB {
return rpiSlotBPartition
}
return rpiSlotAPartition
}