docs: update README for all phases complete, add CHANGELOG

README.md rewritten to reflect all 5 design-doc phases complete with sections for custom kernel, cloud-init, atomic updates, monitoring, full make targets table, and documentation links. CHANGELOG.md created with detailed v0.1.0 release notes covering all features across all phases. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-11 23:40:06 -06:00
parent 39732488ef
commit 36311ed4f4
2 changed files with 237 additions and 27 deletions
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -0,0 +1,80 @@
+# Changelog
+
+All notable changes to KubeSolo OS are documented in this file.
+
+Format based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+versioning follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [0.1.0] - 2026-02-12
+
+First release with all 5 design-doc phases complete. ISO boots and runs K8s pods.
+
+### Added
+
+#### Custom Kernel
+- Custom kernel build (6.18.2-tinycore64) with container-critical configs
+- Added CONFIG_CGROUP_BPF, CONFIG_DEVTMPFS, CONFIG_DEVTMPFS_MOUNT, CONFIG_MEMCG, CONFIG_CFS_BANDWIDTH
+- Stripped unnecessary subsystems (sound, GPU, wireless, Bluetooth, etc.)
+- Selective kernel module install — only modules.list + transitive deps in initramfs
+
+#### Init System (Phase 1)
+- POSIX sh init system with staged boot (00-early-mount through 90-kubesolo)
+- switch_root from initramfs to SquashFS root
+- Persistent data partition mount with bind-mounts for K8s state
+- Kernel module loading, sysctl tuning, network, hostname, NTP
+- Emergency shell fallback on boot failure
+- Device node creation via mknod fallback from sysfs
+
+#### Cloud-Init (Phase 2)
+- Go-based cloud-init parser (~2.7 MB static binary)
+- Network configuration: DHCP and static IP modes
+- Hostname and machine-id generation
+- KubeSolo configuration (node-name, extra flags)
+- Portainer Edge Agent integration via K8s manifest injection
+- Persistent config saved to /mnt/data/ for next-boot fast path
+- 22 Go tests
+
+#### A/B Atomic Updates (Phase 3)
+- 4-partition GPT disk image: EFI + System A + System B + Data
+- GRUB 2 bootloader with A/B slot selection and boot counter rollback
+- Go update agent (~6.0 MB static binary) with check, apply, activate, rollback commands
+- Health check: containerd + K8s API + node Ready verification
+- Update server protocol: HTTP serving latest.json + image files
+- K8s CronJob for automated update checks (every 6 hours)
+- Zero external Go dependencies — uses kubectl/ctr exec commands
+
+#### Production Hardening (Phase 4)
+- Ed25519 image signing with pure Go stdlib (zero external deps)
+- Key generation, signing, and verification CLI commands
+- Portainer Edge Agent deployment via cloud-init
+- SSH extension injection for debugging (hack/inject-ssh.sh)
+- Boot time and resource usage benchmarks
+- Deployment guide documentation
+
+#### Distribution & Fleet Management (Phase 5)
+- Gitea Actions CI/CD (test + build + shellcheck on push, release on tags)
+- OCI container image packaging (scratch-based)
+- Prometheus metrics endpoint (zero-dependency text exposition format)
+- USB provisioning script with cloud-init injection
+- ARM64 cross-compilation support
+
+#### Build System
+- Makefile with full build orchestration
+- Dockerized reproducible builds (build/Dockerfile.builder)
+- Component fetching with version pinning
+- ISO and raw disk image creation
+- Fast rebuild path (`make quick`)
+
+#### Documentation
+- Architecture design document
+- Boot flow reference
+- A/B update flow reference
+- Cloud-init configuration reference
+- Deployment and operations guide
+
+### Fixed
+- Replaced `grep -oP` with POSIX-safe `sed` in functions.sh (BusyBox compatibility)
+- Replaced `grep -qiE` with `grep -qi -e` pattern (POSIX compliance)
+- Fixed KVM flag handling in dev-vm.sh (bash array context)
+- Added iptables table pre-initialization before kube-proxy start (nf_tables issue)
+- Added /dev/kmsg and /etc/machine-id creation for kubelet
--- a/README.md
+++ b/README.md
@@ -2,18 +2,23 @@

 An immutable, bootable Linux distribution purpose-built for [KubeSolo](https://github.com/portainer/kubesolo) — Portainer's ultra-lightweight single-node Kubernetes.

-> **Status:** Phase 1 — Proof of Concept
+> **Status:** All 5 phases complete. Boots and runs K8s workloads.

 ## What is this?

 KubeSolo OS combines **Tiny Core Linux** (~11 MB) with **KubeSolo** (single-binary Kubernetes) to create an appliance-like K8s node that:

- Boots to a functional Kubernetes cluster in ~30 seconds
+- Boots to a functional Kubernetes cluster in ~35 seconds
 - Runs entirely from RAM with a read-only SquashFS root
 - Persists K8s state across reboots via a dedicated data partition
- Targets < 100 MB total image size (OS + K8s)
+- Uses a custom kernel (6.18.2-tinycore64) optimized for containers
+- Supports first-boot configuration via cloud-init YAML
+- Performs atomic A/B updates with automatic GRUB-based rollback
+- Signs update images with Ed25519 for integrity verification
+- Exposes Prometheus metrics for monitoring
+- Integrates with Portainer Edge for fleet management
+- Ships as ISO, raw disk image, or OCI container
 - Requires no SSH, no package manager, no writable system files
- Supports atomic A/B updates with automatic rollback (Phase 3)

 **Target use cases:** IoT/IIoT edge, air-gapped deployments, single-node K8s appliances, kiosk/POS systems, resource-constrained hardware.

@@ -23,63 +28,188 @@ KubeSolo OS combines **Tiny Core Linux** (~11 MB) with **KubeSolo** (single-bina
 # Fetch Tiny Core ISO + KubeSolo binary
 make fetch

+# Build custom kernel (first time only, ~25 min, cached)
+make kernel
+
+# Build Go binaries
+make build-cloudinit build-update-agent
+
 # Build bootable ISO
-make iso
+make rootfs initramfs iso

 # Test in QEMU
 make dev-vm
 ```

+Or build everything at once inside Docker:
+
+```bash
+make docker-build
+```
+
 ## Requirements

 **Build host:**
 - Linux x86_64 with root/sudo (for loop mounts)
+- Go 1.22+ (for cloud-init and update agent)
 - Tools: `cpio`, `gzip`, `wget`, `curl`, `syslinux` (or use `make docker-build`)

 **Runtime:**
- x86_64 hardware or VM
+- x86_64 hardware or VM (ARM64 cross-compilation available)
 - 512 MB RAM minimum (1 GB+ recommended)
 - 8 GB disk (for persistent data partition)

 ## Architecture

 ```
-Boot Media → Kernel + Initramfs (kubesolo-os.gz)
+Boot Media (ISO or Disk Image)
  │
-                  ├── SquashFS root (read-only, in RAM)
+  ├── GRUB 2 bootloader (A/B slot selection, rollback counter)
+  │
+  └── Kernel + Initramfs (kubesolo-os.gz)
+        │
+        ├── switch_root → SquashFS root (read-only, in RAM)
        ├── Persistent data partition (ext4, bind-mounted)
        │     ├── /var/lib/kubesolo   (K8s state, certs, SQLite)
        │     ├── /var/lib/containerd  (container images)
        │     └── /etc/kubesolo        (node configuration)
-                  ├── Custom init (POSIX sh, staged boot)
-                  └── KubeSolo (exec replaces init as PID 1)
+        ├── Custom init (POSIX sh, staged boot 00→90)
+        │     └── Stage 45: cloud-init (Go binary)
+        ├── containerd (bundled with KubeSolo)
+        └── KubeSolo (single-binary K8s)
+```
+
+### Partition Layout (Disk Image)
+
+```
+GPT Disk (minimum 8 GB):
+  Part 1: EFI/Boot   (256 MB, FAT32)  — GRUB + A/B boot logic
+  Part 2: System A   (512 MB, ext4)   — vmlinuz + kubesolo-os.gz (active)
+  Part 3: System B   (512 MB, ext4)   — vmlinuz + kubesolo-os.gz (passive)
+  Part 4: Data       (remaining, ext4) — persistent K8s state
 ```

 See [docs/design/kubesolo-os-design.md](docs/design/kubesolo-os-design.md) for the full architecture document.

+## Custom Kernel
+
+The stock Tiny Core 17.0 kernel lacks several configs required for containers. KubeSolo OS builds a custom kernel (6.18.2-tinycore64) that adds:
+
+- `CONFIG_CGROUP_BPF` — cgroup v2 device control via BPF
+- `CONFIG_DEVTMPFS` / `CONFIG_DEVTMPFS_MOUNT` — automatic /dev node creation
+- `CONFIG_MEMCG` — memory cgroup controller
+- `CONFIG_CFS_BANDWIDTH` — CPU bandwidth throttling
+
+Unnecessary subsystems (sound, GPU, wireless, Bluetooth, etc.) are stripped to keep the kernel minimal. Build is cached in `build/cache/custom-kernel/`.
+
+## Cloud-Init
+
+First-boot configuration via a simple YAML schema:
+
+```yaml
+hostname: edge-node-01
+network:
+  mode: static
+  address: 192.168.1.100/24
+  gateway: 192.168.1.1
+  dns:
+    - 8.8.8.8
+kubesolo:
+  node-name: edge-node-01
+portainer:
+  edge_id: "your-edge-id"
+  edge_key: "your-edge-key"
+```
+
+See [docs/cloud-init.md](docs/cloud-init.md) and the [examples](cloud-init/examples/).
+
+## Atomic Updates
+
+A/B partition scheme with GRUB boot counter for automatic rollback:
+
+1. Update agent downloads new image to passive partition
+2. GRUB boots new partition with `boot_counter=3`
+3. Health check verifies containerd + K8s API + node Ready → sets `boot_success=1`
+4. On 3 consecutive boot failures, GRUB auto-rolls back to previous slot
+
+Updates can be signed with Ed25519 for integrity verification. A K8s CronJob checks for updates every 6 hours.
+
+See [docs/update-flow.md](docs/update-flow.md).
+
+## Monitoring
+
+The update agent exposes Prometheus metrics on port 9100:
+
+```bash
+kubesolo-update metrics --listen :9100
+```
+
+Metrics include: `kubesolo_os_info`, `boot_success`, `boot_counter`, `uptime_seconds`, `update_available`, `memory_total_bytes`, `memory_available_bytes`.
+
 ## Project Structure

 ```
-├── CLAUDE.md              # AI-assisted development instructions
 ├── Makefile               # Build orchestration
-├── build/                 # Build scripts, configs, rootfs overlays
+├── build/                 # Build scripts, kernel config, rootfs overlays
+│   └── scripts/
+│       ├── build-kernel.sh        # Custom kernel compilation
+│       ├── fetch-components.sh    # Download components
+│       ├── create-iso.sh          # Bootable ISO
+│       ├── create-disk-image.sh   # A/B partition disk image
+│       └── create-oci-image.sh    # OCI container image
 ├── init/                  # Custom init system (POSIX sh)
-├── update/                # Atomic update agent (Go, Phase 3)
-├── cloud-init/            # First-boot configuration (Phase 2)
-├── test/                  # QEMU-based automated tests
-├── hack/                  # Developer utilities
-└── docs/                  # Design documents
+│   ├── init.sh            # Main init + switch_root
+│   └── lib/               # Staged boot scripts (00-90)
+├── cloud-init/            # Go cloud-init parser
+├── update/                # Go atomic update agent
+├── test/                  # QEMU-based automated tests + benchmarks
+├── hack/                  # Developer utilities (dev-vm, SSH, USB)
+├── docs/                  # Documentation
+│   ├── design/            # Architecture design document
+│   ├── boot-flow.md       # Boot sequence reference
+│   ├── update-flow.md     # A/B update reference
+│   ├── cloud-init.md      # Cloud-init configuration reference
+│   └── deployment-guide.md # Deployment and operations guide
+└── .gitea/workflows/      # CI/CD (Gitea Actions)
 ```

+## Make Targets
+
+| Target | Description |
+|--------|-------------|
+| `make fetch` | Download Tiny Core ISO + KubeSolo binary |
+| `make kernel` | Build custom kernel (cached) |
+| `make build-cloudinit` | Compile cloud-init Go binary |
+| `make build-update-agent` | Compile update agent Go binary |
+| `make rootfs` | Extract Tiny Core + inject KubeSolo |
+| `make initramfs` | Pack initramfs (kubesolo-os.gz) |
+| `make iso` | Create bootable ISO |
+| `make disk-image` | Create A/B partition disk image |
+| `make oci-image` | Package as OCI container |
+| `make build-cross` | Cross-compile for amd64 + arm64 |
+| `make docker-build` | Build everything in Docker |
+| `make quick` | Fast rebuild (re-inject + repack + ISO) |
+| `make dev-vm` | Launch QEMU dev VM |
+| `make test-all` | Run all tests |
+
+## Documentation
+
+- [Architecture Design](docs/design/kubesolo-os-design.md) — full research and technical specification
+- [Boot Flow](docs/boot-flow.md) — boot sequence from GRUB to K8s Ready
+- [Update Flow](docs/update-flow.md) — A/B atomic update mechanism
+- [Cloud-Init](docs/cloud-init.md) — first-boot configuration reference
+- [Deployment Guide](docs/deployment-guide.md) — installation, operations, troubleshooting
+
 ## Roadmap

 | Phase | Scope | Status |
 |-------|-------|--------|
-| 1 | PoC: boot Tiny Core + KubeSolo, verify K8s | 🚧 In Progress |
-| 2 | Persistent storage, cloud-init, networking | Planned |
-| 3 | A/B atomic updates, GRUB, rollback | Planned |
-| 4 | Production hardening, signing, Portainer Edge | Planned |
-| 5 | OCI distribution, ARM64, fleet management | Planned |
+| 1 | PoC: boot Tiny Core + KubeSolo, verify K8s | Complete |
+| 2 | Cloud-init Go parser, network, hostname | Complete |
+| 3 | A/B atomic updates, GRUB, rollback agent | Complete |
+| 4 | Ed25519 signing, Portainer Edge, SSH extension | Complete |
+| 5 | CI/CD, OCI distribution, Prometheus metrics, ARM64 | Complete |
+| - | Custom kernel build for container runtime fixes | Complete |

 ## License