From 36311ed4f4f87a77960095950c68da6a42b5b282 Mon Sep 17 00:00:00 2001 From: Adolfo Delorenzo Date: Wed, 11 Feb 2026 23:40:06 -0600 Subject: [PATCH] docs: update README for all phases complete, add CHANGELOG README.md rewritten to reflect all 5 design-doc phases complete with sections for custom kernel, cloud-init, atomic updates, monitoring, full make targets table, and documentation links. CHANGELOG.md created with detailed v0.1.0 release notes covering all features across all phases. Co-Authored-By: Claude Opus 4.6 --- CHANGELOG.md | 80 ++++++++++++++++++++++ README.md | 184 +++++++++++++++++++++++++++++++++++++++++++-------- 2 files changed, 237 insertions(+), 27 deletions(-) create mode 100644 CHANGELOG.md diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..77a753e --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,80 @@ +# Changelog + +All notable changes to KubeSolo OS are documented in this file. + +Format based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), +versioning follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +## [0.1.0] - 2026-02-12 + +First release with all 5 design-doc phases complete. ISO boots and runs K8s pods. + +### Added + +#### Custom Kernel +- Custom kernel build (6.18.2-tinycore64) with container-critical configs +- Added CONFIG_CGROUP_BPF, CONFIG_DEVTMPFS, CONFIG_DEVTMPFS_MOUNT, CONFIG_MEMCG, CONFIG_CFS_BANDWIDTH +- Stripped unnecessary subsystems (sound, GPU, wireless, Bluetooth, etc.) +- Selective kernel module install — only modules.list + transitive deps in initramfs + +#### Init System (Phase 1) +- POSIX sh init system with staged boot (00-early-mount through 90-kubesolo) +- switch_root from initramfs to SquashFS root +- Persistent data partition mount with bind-mounts for K8s state +- Kernel module loading, sysctl tuning, network, hostname, NTP +- Emergency shell fallback on boot failure +- Device node creation via mknod fallback from sysfs + +#### Cloud-Init (Phase 2) +- Go-based cloud-init parser (~2.7 MB static binary) +- Network configuration: DHCP and static IP modes +- Hostname and machine-id generation +- KubeSolo configuration (node-name, extra flags) +- Portainer Edge Agent integration via K8s manifest injection +- Persistent config saved to /mnt/data/ for next-boot fast path +- 22 Go tests + +#### A/B Atomic Updates (Phase 3) +- 4-partition GPT disk image: EFI + System A + System B + Data +- GRUB 2 bootloader with A/B slot selection and boot counter rollback +- Go update agent (~6.0 MB static binary) with check, apply, activate, rollback commands +- Health check: containerd + K8s API + node Ready verification +- Update server protocol: HTTP serving latest.json + image files +- K8s CronJob for automated update checks (every 6 hours) +- Zero external Go dependencies — uses kubectl/ctr exec commands + +#### Production Hardening (Phase 4) +- Ed25519 image signing with pure Go stdlib (zero external deps) +- Key generation, signing, and verification CLI commands +- Portainer Edge Agent deployment via cloud-init +- SSH extension injection for debugging (hack/inject-ssh.sh) +- Boot time and resource usage benchmarks +- Deployment guide documentation + +#### Distribution & Fleet Management (Phase 5) +- Gitea Actions CI/CD (test + build + shellcheck on push, release on tags) +- OCI container image packaging (scratch-based) +- Prometheus metrics endpoint (zero-dependency text exposition format) +- USB provisioning script with cloud-init injection +- ARM64 cross-compilation support + +#### Build System +- Makefile with full build orchestration +- Dockerized reproducible builds (build/Dockerfile.builder) +- Component fetching with version pinning +- ISO and raw disk image creation +- Fast rebuild path (`make quick`) + +#### Documentation +- Architecture design document +- Boot flow reference +- A/B update flow reference +- Cloud-init configuration reference +- Deployment and operations guide + +### Fixed +- Replaced `grep -oP` with POSIX-safe `sed` in functions.sh (BusyBox compatibility) +- Replaced `grep -qiE` with `grep -qi -e` pattern (POSIX compliance) +- Fixed KVM flag handling in dev-vm.sh (bash array context) +- Added iptables table pre-initialization before kube-proxy start (nf_tables issue) +- Added /dev/kmsg and /etc/machine-id creation for kubelet diff --git a/README.md b/README.md index 493292f..9e962ae 100644 --- a/README.md +++ b/README.md @@ -2,18 +2,23 @@ An immutable, bootable Linux distribution purpose-built for [KubeSolo](https://github.com/portainer/kubesolo) — Portainer's ultra-lightweight single-node Kubernetes. -> **Status:** Phase 1 — Proof of Concept +> **Status:** All 5 phases complete. Boots and runs K8s workloads. ## What is this? KubeSolo OS combines **Tiny Core Linux** (~11 MB) with **KubeSolo** (single-binary Kubernetes) to create an appliance-like K8s node that: -- Boots to a functional Kubernetes cluster in ~30 seconds +- Boots to a functional Kubernetes cluster in ~35 seconds - Runs entirely from RAM with a read-only SquashFS root - Persists K8s state across reboots via a dedicated data partition -- Targets < 100 MB total image size (OS + K8s) +- Uses a custom kernel (6.18.2-tinycore64) optimized for containers +- Supports first-boot configuration via cloud-init YAML +- Performs atomic A/B updates with automatic GRUB-based rollback +- Signs update images with Ed25519 for integrity verification +- Exposes Prometheus metrics for monitoring +- Integrates with Portainer Edge for fleet management +- Ships as ISO, raw disk image, or OCI container - Requires no SSH, no package manager, no writable system files -- Supports atomic A/B updates with automatic rollback (Phase 3) **Target use cases:** IoT/IIoT edge, air-gapped deployments, single-node K8s appliances, kiosk/POS systems, resource-constrained hardware. @@ -23,63 +28,188 @@ KubeSolo OS combines **Tiny Core Linux** (~11 MB) with **KubeSolo** (single-bina # Fetch Tiny Core ISO + KubeSolo binary make fetch +# Build custom kernel (first time only, ~25 min, cached) +make kernel + +# Build Go binaries +make build-cloudinit build-update-agent + # Build bootable ISO -make iso +make rootfs initramfs iso # Test in QEMU make dev-vm ``` +Or build everything at once inside Docker: + +```bash +make docker-build +``` + ## Requirements **Build host:** - Linux x86_64 with root/sudo (for loop mounts) +- Go 1.22+ (for cloud-init and update agent) - Tools: `cpio`, `gzip`, `wget`, `curl`, `syslinux` (or use `make docker-build`) **Runtime:** -- x86_64 hardware or VM +- x86_64 hardware or VM (ARM64 cross-compilation available) - 512 MB RAM minimum (1 GB+ recommended) - 8 GB disk (for persistent data partition) ## Architecture ``` -Boot Media → Kernel + Initramfs (kubesolo-os.gz) - │ - ├── SquashFS root (read-only, in RAM) - ├── Persistent data partition (ext4, bind-mounted) - │ ├── /var/lib/kubesolo (K8s state, certs, SQLite) - │ ├── /var/lib/containerd (container images) - │ └── /etc/kubesolo (node configuration) - ├── Custom init (POSIX sh, staged boot) - └── KubeSolo (exec replaces init as PID 1) +Boot Media (ISO or Disk Image) + │ + ├── GRUB 2 bootloader (A/B slot selection, rollback counter) + │ + └── Kernel + Initramfs (kubesolo-os.gz) + │ + ├── switch_root → SquashFS root (read-only, in RAM) + ├── Persistent data partition (ext4, bind-mounted) + │ ├── /var/lib/kubesolo (K8s state, certs, SQLite) + │ ├── /var/lib/containerd (container images) + │ └── /etc/kubesolo (node configuration) + ├── Custom init (POSIX sh, staged boot 00→90) + │ └── Stage 45: cloud-init (Go binary) + ├── containerd (bundled with KubeSolo) + └── KubeSolo (single-binary K8s) +``` + +### Partition Layout (Disk Image) + +``` +GPT Disk (minimum 8 GB): + Part 1: EFI/Boot (256 MB, FAT32) — GRUB + A/B boot logic + Part 2: System A (512 MB, ext4) — vmlinuz + kubesolo-os.gz (active) + Part 3: System B (512 MB, ext4) — vmlinuz + kubesolo-os.gz (passive) + Part 4: Data (remaining, ext4) — persistent K8s state ``` See [docs/design/kubesolo-os-design.md](docs/design/kubesolo-os-design.md) for the full architecture document. +## Custom Kernel + +The stock Tiny Core 17.0 kernel lacks several configs required for containers. KubeSolo OS builds a custom kernel (6.18.2-tinycore64) that adds: + +- `CONFIG_CGROUP_BPF` — cgroup v2 device control via BPF +- `CONFIG_DEVTMPFS` / `CONFIG_DEVTMPFS_MOUNT` — automatic /dev node creation +- `CONFIG_MEMCG` — memory cgroup controller +- `CONFIG_CFS_BANDWIDTH` — CPU bandwidth throttling + +Unnecessary subsystems (sound, GPU, wireless, Bluetooth, etc.) are stripped to keep the kernel minimal. Build is cached in `build/cache/custom-kernel/`. + +## Cloud-Init + +First-boot configuration via a simple YAML schema: + +```yaml +hostname: edge-node-01 +network: + mode: static + address: 192.168.1.100/24 + gateway: 192.168.1.1 + dns: + - 8.8.8.8 +kubesolo: + node-name: edge-node-01 +portainer: + edge_id: "your-edge-id" + edge_key: "your-edge-key" +``` + +See [docs/cloud-init.md](docs/cloud-init.md) and the [examples](cloud-init/examples/). + +## Atomic Updates + +A/B partition scheme with GRUB boot counter for automatic rollback: + +1. Update agent downloads new image to passive partition +2. GRUB boots new partition with `boot_counter=3` +3. Health check verifies containerd + K8s API + node Ready → sets `boot_success=1` +4. On 3 consecutive boot failures, GRUB auto-rolls back to previous slot + +Updates can be signed with Ed25519 for integrity verification. A K8s CronJob checks for updates every 6 hours. + +See [docs/update-flow.md](docs/update-flow.md). + +## Monitoring + +The update agent exposes Prometheus metrics on port 9100: + +```bash +kubesolo-update metrics --listen :9100 +``` + +Metrics include: `kubesolo_os_info`, `boot_success`, `boot_counter`, `uptime_seconds`, `update_available`, `memory_total_bytes`, `memory_available_bytes`. + ## Project Structure ``` -├── CLAUDE.md # AI-assisted development instructions ├── Makefile # Build orchestration -├── build/ # Build scripts, configs, rootfs overlays +├── build/ # Build scripts, kernel config, rootfs overlays +│ └── scripts/ +│ ├── build-kernel.sh # Custom kernel compilation +│ ├── fetch-components.sh # Download components +│ ├── create-iso.sh # Bootable ISO +│ ├── create-disk-image.sh # A/B partition disk image +│ └── create-oci-image.sh # OCI container image ├── init/ # Custom init system (POSIX sh) -├── update/ # Atomic update agent (Go, Phase 3) -├── cloud-init/ # First-boot configuration (Phase 2) -├── test/ # QEMU-based automated tests -├── hack/ # Developer utilities -└── docs/ # Design documents +│ ├── init.sh # Main init + switch_root +│ └── lib/ # Staged boot scripts (00-90) +├── cloud-init/ # Go cloud-init parser +├── update/ # Go atomic update agent +├── test/ # QEMU-based automated tests + benchmarks +├── hack/ # Developer utilities (dev-vm, SSH, USB) +├── docs/ # Documentation +│ ├── design/ # Architecture design document +│ ├── boot-flow.md # Boot sequence reference +│ ├── update-flow.md # A/B update reference +│ ├── cloud-init.md # Cloud-init configuration reference +│ └── deployment-guide.md # Deployment and operations guide +└── .gitea/workflows/ # CI/CD (Gitea Actions) ``` +## Make Targets + +| Target | Description | +|--------|-------------| +| `make fetch` | Download Tiny Core ISO + KubeSolo binary | +| `make kernel` | Build custom kernel (cached) | +| `make build-cloudinit` | Compile cloud-init Go binary | +| `make build-update-agent` | Compile update agent Go binary | +| `make rootfs` | Extract Tiny Core + inject KubeSolo | +| `make initramfs` | Pack initramfs (kubesolo-os.gz) | +| `make iso` | Create bootable ISO | +| `make disk-image` | Create A/B partition disk image | +| `make oci-image` | Package as OCI container | +| `make build-cross` | Cross-compile for amd64 + arm64 | +| `make docker-build` | Build everything in Docker | +| `make quick` | Fast rebuild (re-inject + repack + ISO) | +| `make dev-vm` | Launch QEMU dev VM | +| `make test-all` | Run all tests | + +## Documentation + +- [Architecture Design](docs/design/kubesolo-os-design.md) — full research and technical specification +- [Boot Flow](docs/boot-flow.md) — boot sequence from GRUB to K8s Ready +- [Update Flow](docs/update-flow.md) — A/B atomic update mechanism +- [Cloud-Init](docs/cloud-init.md) — first-boot configuration reference +- [Deployment Guide](docs/deployment-guide.md) — installation, operations, troubleshooting + ## Roadmap | Phase | Scope | Status | |-------|-------|--------| -| 1 | PoC: boot Tiny Core + KubeSolo, verify K8s | 🚧 In Progress | -| 2 | Persistent storage, cloud-init, networking | Planned | -| 3 | A/B atomic updates, GRUB, rollback | Planned | -| 4 | Production hardening, signing, Portainer Edge | Planned | -| 5 | OCI distribution, ARM64, fleet management | Planned | +| 1 | PoC: boot Tiny Core + KubeSolo, verify K8s | Complete | +| 2 | Cloud-init Go parser, network, hostname | Complete | +| 3 | A/B atomic updates, GRUB, rollback agent | Complete | +| 4 | Ed25519 signing, Portainer Edge, SSH extension | Complete | +| 5 | CI/CD, OCI distribution, Prometheus metrics, ARM64 | Complete | +| - | Custom kernel build for container runtime fixes | Complete | ## License