From e372df578bc08b9af39217c1a2f77b4ec48927b8 Mon Sep 17 00:00:00 2001 From: Adolfo Delorenzo Date: Wed, 11 Feb 2026 10:18:42 -0600 Subject: [PATCH] feat: initial Phase 1 PoC scaffolding for KubeSolo OS MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Complete Phase 1 implementation of KubeSolo OS — an immutable, bootable Linux distribution built on Tiny Core Linux for running KubeSolo single-node Kubernetes. Build system: - Makefile with fetch, rootfs, initramfs, iso, disk-image targets - Dockerfile.builder for reproducible builds - Scripts to download Tiny Core, extract rootfs, inject KubeSolo, pack initramfs, and create bootable ISO/disk images Init system (10 POSIX sh stages): - Early mount (proc/sys/dev/cgroup2), cmdline parsing, persistent mount with bind-mounts, kernel module loading, sysctl, DHCP networking, hostname, clock sync, containerd prep, KubeSolo exec Shared libraries: - functions.sh (device wait, IP lookup, config helpers) - network.sh (static IP, config persistence, interface detection) - health.sh (containerd, API server, node readiness checks) - Emergency shell for boot failure debugging Testing: - QEMU boot test with serial log marker detection - K8s readiness test with kubectl verification - Persistence test (reboot + verify state survives) - Workload deployment test (nginx pod) - Local storage test (PVC + local-path provisioner) - Network policy test - Reusable run-vm.sh launcher Developer tools: - dev-vm.sh (interactive QEMU with port forwarding) - rebuild-initramfs.sh (fast iteration) - inject-ssh.sh (dropbear SSH for debugging) - extract-kernel-config.sh + kernel-audit.sh Documentation: - Full design document with architecture research - Boot flow documentation covering all 10 init stages - Cloud-init examples (DHCP, static IP, Portainer Edge, air-gapped) Co-Authored-By: Claude Opus 4.6 --- .gitignore | 25 + CLAUDE.md | 428 +++++++++ Makefile | 176 ++++ README.md | 86 ++ VERSION | 1 + build/Dockerfile.builder | 34 + build/config/kernel-audit.sh | 169 ++++ build/config/modules.list | 31 + build/config/versions.env | 19 + build/rootfs/etc/kubesolo/defaults.yaml | 22 + build/rootfs/etc/sysctl.d/k8s.conf | 17 + build/rootfs/usr/lib/kubesolo-os/health.sh | 63 ++ build/rootfs/usr/lib/kubesolo-os/network.sh | 80 ++ build/scripts/create-disk-image.sh | 110 +++ build/scripts/create-iso.sh | 140 +++ build/scripts/extract-core.sh | 83 ++ build/scripts/fetch-components.sh | 72 ++ build/scripts/inject-kubesolo.sh | 124 +++ build/scripts/pack-initramfs.sh | 23 + cloud-init/examples/airgapped.yaml | 29 + cloud-init/examples/dhcp.yaml | 18 + cloud-init/examples/portainer-edge.yaml | 26 + cloud-init/examples/static-ip.yaml | 17 + docs/boot-flow.md | 181 ++++ docs/design/kubesolo-os-design.md | 945 ++++++++++++++++++++ hack/dev-vm.sh | 84 ++ hack/extract-kernel-config.sh | 48 + hack/inject-ssh.sh | 82 ++ hack/rebuild-initramfs.sh | 18 + init/emergency-shell.sh | 48 + init/init.sh | 87 ++ init/lib/00-early-mount.sh | 23 + init/lib/10-parse-cmdline.sh | 27 + init/lib/20-persistent-mount.sh | 47 + init/lib/30-kernel-modules.sh | 28 + init/lib/40-sysctl.sh | 20 + init/lib/50-network.sh | 64 ++ init/lib/60-hostname.sh | 24 + init/lib/70-clock.sh | 19 + init/lib/80-containerd.sh | 22 + init/lib/90-kubesolo.sh | 38 + init/lib/functions.sh | 75 ++ test/integration/test-deploy-workload.sh | 97 ++ test/integration/test-k8s-ready.sh | 69 ++ test/integration/test-local-storage.sh | 126 +++ test/integration/test-network-policy.sh | 119 +++ test/kernel/check-config.sh | 23 + test/qemu/run-vm.sh | 120 +++ test/qemu/test-boot.sh | 65 ++ test/qemu/test-persistence.sh | 100 +++ 50 files changed, 4392 insertions(+) create mode 100644 .gitignore create mode 100644 CLAUDE.md create mode 100644 Makefile create mode 100644 README.md create mode 100644 VERSION create mode 100644 build/Dockerfile.builder create mode 100755 build/config/kernel-audit.sh create mode 100644 build/config/modules.list create mode 100644 build/config/versions.env create mode 100644 build/rootfs/etc/kubesolo/defaults.yaml create mode 100644 build/rootfs/etc/sysctl.d/k8s.conf create mode 100755 build/rootfs/usr/lib/kubesolo-os/health.sh create mode 100755 build/rootfs/usr/lib/kubesolo-os/network.sh create mode 100755 build/scripts/create-disk-image.sh create mode 100755 build/scripts/create-iso.sh create mode 100755 build/scripts/extract-core.sh create mode 100755 build/scripts/fetch-components.sh create mode 100755 build/scripts/inject-kubesolo.sh create mode 100755 build/scripts/pack-initramfs.sh create mode 100644 cloud-init/examples/airgapped.yaml create mode 100644 cloud-init/examples/dhcp.yaml create mode 100644 cloud-init/examples/portainer-edge.yaml create mode 100644 cloud-init/examples/static-ip.yaml create mode 100644 docs/boot-flow.md create mode 100644 docs/design/kubesolo-os-design.md create mode 100755 hack/dev-vm.sh create mode 100755 hack/extract-kernel-config.sh create mode 100755 hack/inject-ssh.sh create mode 100755 hack/rebuild-initramfs.sh create mode 100755 init/emergency-shell.sh create mode 100755 init/init.sh create mode 100755 init/lib/00-early-mount.sh create mode 100755 init/lib/10-parse-cmdline.sh create mode 100755 init/lib/20-persistent-mount.sh create mode 100755 init/lib/30-kernel-modules.sh create mode 100755 init/lib/40-sysctl.sh create mode 100755 init/lib/50-network.sh create mode 100755 init/lib/60-hostname.sh create mode 100755 init/lib/70-clock.sh create mode 100755 init/lib/80-containerd.sh create mode 100755 init/lib/90-kubesolo.sh create mode 100755 init/lib/functions.sh create mode 100755 test/integration/test-deploy-workload.sh create mode 100755 test/integration/test-k8s-ready.sh create mode 100755 test/integration/test-local-storage.sh create mode 100755 test/integration/test-network-policy.sh create mode 100755 test/kernel/check-config.sh create mode 100755 test/qemu/run-vm.sh create mode 100755 test/qemu/test-boot.sh create mode 100755 test/qemu/test-persistence.sh diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..ed6f832 --- /dev/null +++ b/.gitignore @@ -0,0 +1,25 @@ +# Build artifacts +output/ +build/cache/ +build/rootfs-work/ + +# Generated files +*.iso +*.img +*.gz +*.squashfs + +# Editor +.vscode/ +.idea/ +*.swp +*.swo +*~ + +# OS +.DS_Store +Thumbs.db + +# Go +update/update-agent +cloud-init/cloud-init-parser diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..5699d94 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,428 @@ +# CLAUDE.md — KubeSolo OS + +## Project Overview + +**KubeSolo OS** is an immutable, bootable Linux distribution purpose-built to run [KubeSolo](https://github.com/portainer/kubesolo) — Portainer's ultra-lightweight single-node Kubernetes distribution. It combines Tiny Core Linux's minimal footprint (~11 MB) with KubeSolo's single-binary K8s packaging to create an appliance-like Kubernetes node with atomic A/B updates. + +**Design document:** See `docs/design/kubesolo-os-design.md` for full architecture research, competitive analysis, and technical specifications. + +**Target:** Edge/IoT devices, single-node K8s appliances, air-gapped deployments, resource-constrained hardware. + +--- + +## Repository Structure + +``` +kubesolo-os/ +├── CLAUDE.md # This file +├── README.md # Project README +├── Makefile # Top-level build orchestration +├── VERSION # Semver version (e.g., 0.1.0) +│ +├── docs/ +│ ├── design/ +│ │ └── kubesolo-os-design.md # Full architecture document +│ ├── boot-flow.md # Boot sequence documentation +│ ├── update-flow.md # Atomic update documentation +│ └── cloud-init.md # Configuration reference +│ +├── build/ # Build system +│ ├── Dockerfile.builder # Containerized build environment +│ ├── build.sh # Main build script (orchestrator) +│ ├── config/ +│ │ ├── kernel-audit.sh # Verify kernel config requirements +│ │ ├── kernel-config.fragment # Custom kernel config overrides (if needed) +│ │ └── modules.list # Required kernel modules list +│ ├── rootfs/ # Files injected into initramfs +│ │ ├── sbin/ +│ │ │ └── init # Custom init script +│ │ ├── etc/ +│ │ │ ├── os-release # OS identification +│ │ │ ├── sysctl.d/ +│ │ │ │ └── k8s.conf # Kernel parameters for K8s +│ │ │ └── kubesolo/ +│ │ │ └── defaults.yaml # Default KubeSolo config +│ │ └── usr/ +│ │ └── lib/ +│ │ └── kubesolo-os/ +│ │ ├── functions.sh # Shared shell functions +│ │ ├── network.sh # Network configuration helpers +│ │ └── health.sh # Health check functions +│ ├── grub/ +│ │ ├── grub.cfg # A/B boot GRUB config +│ │ └── grub-env-defaults # Default GRUB environment vars +│ └── scripts/ +│ ├── fetch-components.sh # Download Tiny Core, KubeSolo, deps +│ ├── extract-core.sh # Extract and prepare Tiny Core rootfs +│ ├── inject-kubesolo.sh # Add KubeSolo + deps to rootfs +│ ├── pack-initramfs.sh # Repack initramfs (core.gz → kubesolo-os.gz) +│ ├── create-iso.sh # Build bootable ISO +│ ├── create-disk-image.sh # Build raw disk image with A/B partitions +│ └── create-oci-image.sh # Build OCI container image (future) +│ +├── init/ # Init system source +│ ├── init.sh # Main init script (becomes /sbin/init) +│ ├── lib/ +│ │ ├── 00-early-mount.sh # Mount proc, sys, dev, tmpfs +│ │ ├── 10-parse-cmdline.sh # Parse kernel boot parameters +│ │ ├── 20-persistent-mount.sh # Mount + bind persistent data partition +│ │ ├── 30-kernel-modules.sh # Load required kernel modules +│ │ ├── 40-sysctl.sh # Apply sysctl settings +│ │ ├── 50-network.sh # Network configuration (cloud-init/DHCP) +│ │ ├── 60-hostname.sh # Set hostname +│ │ ├── 70-clock.sh # NTP / system clock +│ │ ├── 80-containerd.sh # Start containerd +│ │ └── 90-kubesolo.sh # Start KubeSolo (final stage) +│ └── emergency-shell.sh # Drop to shell on boot failure +│ +├── update/ # Atomic update agent +│ ├── go.mod +│ ├── go.sum +│ ├── main.go # Update agent entrypoint +│ ├── cmd/ +│ │ ├── check.go # Check for available updates +│ │ ├── apply.go # Download + write to passive partition +│ │ ├── activate.go # Update GRUB, set boot counter +│ │ ├── rollback.go # Force rollback to previous partition +│ │ └── healthcheck.go # Post-boot health verification +│ ├── pkg/ +│ │ ├── grubenv/ # GRUB environment manipulation +│ │ │ └── grubenv.go +│ │ ├── partition/ # Partition detection and management +│ │ │ └── partition.go +│ │ ├── image/ # Image download, verify, write +│ │ │ └── image.go +│ │ └── health/ # K8s + containerd health checks +│ │ └── health.go +│ └── deploy/ +│ └── update-cronjob.yaml # K8s CronJob manifest for auto-updates +│ +├── cloud-init/ # Cloud-init implementation +│ ├── cloud-init.go # Lightweight cloud-init parser +│ ├── network.go # Network config from cloud-init +│ ├── kubesolo.go # KubeSolo config from cloud-init +│ └── examples/ +│ ├── dhcp.yaml # DHCP example +│ ├── static-ip.yaml # Static IP example +│ ├── portainer-edge.yaml # Portainer Edge integration +│ └── airgapped.yaml # Air-gapped deployment +│ +├── test/ # Testing +│ ├── Makefile # Test orchestration +│ ├── qemu/ +│ │ ├── run-vm.sh # Launch QEMU VM with built image +│ │ ├── test-boot.sh # Automated boot test +│ │ ├── test-persistence.sh # Reboot + verify state survives +│ │ ├── test-update.sh # A/B update cycle test +│ │ └── test-rollback.sh # Forced rollback test +│ ├── integration/ +│ │ ├── test-k8s-ready.sh # Verify K8s node reaches Ready +│ │ ├── test-deploy-workload.sh # Deploy nginx, verify pod running +│ │ ├── test-local-storage.sh # PVC with local-path provisioner +│ │ └── test-network-policy.sh # Basic network policy enforcement +│ └── kernel/ +│ └── check-config.sh # Validate kernel config requirements +│ +└── hack/ # Developer utilities + ├── dev-vm.sh # Quick-launch dev VM (QEMU) + ├── rebuild-initramfs.sh # Fast rebuild for dev iteration + ├── inject-ssh.sh # Add SSH extension for debugging + └── extract-kernel-config.sh # Pull /proc/config.gz from running TC +``` + +--- + +## Architecture Summary + +### Core Concept + +KubeSolo OS is a **remastered Tiny Core Linux** where the initramfs (`core.gz`) is rebuilt to include KubeSolo and all its dependencies. The result is a single bootable image that: + +1. Boots kernel + initramfs into RAM (read-only SquashFS root) +2. Mounts a persistent ext4 partition for K8s state +3. Bind-mounts writable paths (`/var/lib/kubesolo`, `/var/lib/containerd`, etc.) +4. Loads kernel modules (br_netfilter, overlay, veth, etc.) +5. Configures networking (cloud-init → persistent config → DHCP fallback) +6. Starts containerd, then KubeSolo +7. Kubernetes API becomes available; node reaches Ready + +### Partition Layout + +``` +Disk (minimum 8 GB): + Part 1: EFI/Boot (256 MB, FAT32) — GRUB + A/B boot logic + Part 2: System A (512 MB, ext4) — vmlinuz + kubesolo-os.gz (active) + Part 3: System B (512 MB, ext4) — vmlinuz + kubesolo-os.gz (passive) + Part 4: Data (remaining, ext4) — persistent K8s state +``` + +### Persistent Paths (survive updates) + +| Mount Point | Content | On Data Partition | +|---|---|---| +| `/var/lib/kubesolo` | K8s state, certs, SQLite DB | `/mnt/data/kubesolo` | +| `/var/lib/containerd` | Container images + layers | `/mnt/data/containerd` | +| `/etc/kubesolo` | Node configuration | `/mnt/data/etc-kubesolo` | +| `/var/log` | System + K8s logs | `/mnt/data/log` | +| `/usr/local` | User data, extra binaries | `/mnt/data/usr-local` | + +### Atomic Updates + +A/B partition scheme with GRUB fallback counter: +- Update writes new image to passive partition +- GRUB boots new partition with `boot_counter=3` +- Health check sets `boot_success=1` on success +- On 3 consecutive failures (counter reaches 0), GRUB auto-rolls back + +--- + +## Technology Stack + +| Component | Technology | Rationale | +|---|---|---| +| Base OS | Tiny Core Linux 17.0 (Micro Core, x86_64) | 11 MB, RAM-resident, SquashFS root | +| Kernel | Tiny Core stock (6.x) or custom build | Must have cgroup v2, namespaces, netfilter | +| Kubernetes | KubeSolo (single binary) | Single-node K8s, SQLite backend, bundled runtime | +| Container runtime | containerd + runc (bundled in KubeSolo) | Industry standard, KubeSolo dependency | +| Init | Custom shell script (POSIX sh) | Minimal, no systemd dependency | +| Bootloader | GRUB 2 (EFI + BIOS) | A/B partition support, env variables | +| Update agent | Go binary | Single static binary, K8s client-go | +| Cloud-init parser | Go binary or shell script | First-boot configuration | +| Build system | Bash + Make + Docker (builder container) | Reproducible builds | +| Testing | QEMU/KVM + shell scripts | Automated boot + integration tests | + +--- + +## Development Guidelines + +### Shell Scripts (init system, build scripts) + +- **POSIX sh** — no bashisms in init scripts (BusyBox ash compatibility) +- **Shellcheck** all scripts: `shellcheck -s sh