adelorenzo/kubesolo-os
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 2 Wiki Activity

feat: initial Phase 1 PoC scaffolding for KubeSolo OS

Browse Source 
Complete Phase 1 implementation of KubeSolo OS — an immutable, bootable
Linux distribution built on Tiny Core Linux for running KubeSolo
single-node Kubernetes.

Build system:
- Makefile with fetch, rootfs, initramfs, iso, disk-image targets
- Dockerfile.builder for reproducible builds
- Scripts to download Tiny Core, extract rootfs, inject KubeSolo,
  pack initramfs, and create bootable ISO/disk images

Init system (10 POSIX sh stages):
- Early mount (proc/sys/dev/cgroup2), cmdline parsing, persistent
  mount with bind-mounts, kernel module loading, sysctl, DHCP
  networking, hostname, clock sync, containerd prep, KubeSolo exec

Shared libraries:
- functions.sh (device wait, IP lookup, config helpers)
- network.sh (static IP, config persistence, interface detection)
- health.sh (containerd, API server, node readiness checks)
- Emergency shell for boot failure debugging

Testing:
- QEMU boot test with serial log marker detection
- K8s readiness test with kubectl verification
- Persistence test (reboot + verify state survives)
- Workload deployment test (nginx pod)
- Local storage test (PVC + local-path provisioner)
- Network policy test
- Reusable run-vm.sh launcher

Developer tools:
- dev-vm.sh (interactive QEMU with port forwarding)
- rebuild-initramfs.sh (fast iteration)
- inject-ssh.sh (dropbear SSH for debugging)
- extract-kernel-config.sh + kernel-audit.sh

Documentation:
- Full design document with architecture research
- Boot flow documentation covering all 10 init stages
- Cloud-init examples (DHCP, static IP, Portainer Edge, air-gapped)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Adolfo Delorenzo
2026-02-11 10:18:42 -06:00
commit e372df578b
50 changed files with 4392 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

428
CLAUDE.md Normal file
View File

@@ -0,0 +1,428 @@
# CLAUDE.md — KubeSolo OS
## Project Overview
**KubeSolo OS** is an immutable, bootable Linux distribution purpose-built to run [KubeSolo](https://github.com/portainer/kubesolo) — Portainer's ultra-lightweight single-node Kubernetes distribution. It combines Tiny Core Linux's minimal footprint (~11 MB) with KubeSolo's single-binary K8s packaging to create an appliance-like Kubernetes node with atomic A/B updates.
**Design document:** See `docs/design/kubesolo-os-design.md` for full architecture research, competitive analysis, and technical specifications.
**Target:** Edge/IoT devices, single-node K8s appliances, air-gapped deployments, resource-constrained hardware.
---
## Repository Structure
```
kubesolo-os/
├── CLAUDE.md # This file
├── README.md # Project README
├── Makefile # Top-level build orchestration
├── VERSION # Semver version (e.g., 0.1.0)
│
├── docs/
│ ├── design/
│ │ └── kubesolo-os-design.md # Full architecture document
│ ├── boot-flow.md # Boot sequence documentation
│ ├── update-flow.md # Atomic update documentation
│ └── cloud-init.md # Configuration reference
│
├── build/ # Build system
│ ├── Dockerfile.builder # Containerized build environment
│ ├── build.sh # Main build script (orchestrator)
│ ├── config/
│ │ ├── kernel-audit.sh # Verify kernel config requirements
│ │ ├── kernel-config.fragment # Custom kernel config overrides (if needed)
│ │ └── modules.list # Required kernel modules list
│ ├── rootfs/ # Files injected into initramfs
│ │ ├── sbin/
│ │ │ └── init # Custom init script
│ │ ├── etc/
│ │ │ ├── os-release # OS identification
│ │ │ ├── sysctl.d/
│ │ │ │ └── k8s.conf # Kernel parameters for K8s
│ │ │ └── kubesolo/
│ │ │ └── defaults.yaml # Default KubeSolo config
│ │ └── usr/
│ │ └── lib/
│ │ └── kubesolo-os/
│ │ ├── functions.sh # Shared shell functions
│ │ ├── network.sh # Network configuration helpers
│ │ └── health.sh # Health check functions
│ ├── grub/
│ │ ├── grub.cfg # A/B boot GRUB config
│ │ └── grub-env-defaults # Default GRUB environment vars
│ └── scripts/
│ ├── fetch-components.sh # Download Tiny Core, KubeSolo, deps
│ ├── extract-core.sh # Extract and prepare Tiny Core rootfs
│ ├── inject-kubesolo.sh # Add KubeSolo + deps to rootfs
│ ├── pack-initramfs.sh # Repack initramfs (core.gz → kubesolo-os.gz)
│ ├── create-iso.sh # Build bootable ISO
│ ├── create-disk-image.sh # Build raw disk image with A/B partitions
│ └── create-oci-image.sh # Build OCI container image (future)
│
├── init/ # Init system source
│ ├── init.sh # Main init script (becomes /sbin/init)
│ ├── lib/
│ │ ├── 00-early-mount.sh # Mount proc, sys, dev, tmpfs
│ │ ├── 10-parse-cmdline.sh # Parse kernel boot parameters
│ │ ├── 20-persistent-mount.sh # Mount + bind persistent data partition
│ │ ├── 30-kernel-modules.sh # Load required kernel modules
│ │ ├── 40-sysctl.sh # Apply sysctl settings
│ │ ├── 50-network.sh # Network configuration (cloud-init/DHCP)
│ │ ├── 60-hostname.sh # Set hostname
│ │ ├── 70-clock.sh # NTP / system clock
│ │ ├── 80-containerd.sh # Start containerd
│ │ └── 90-kubesolo.sh # Start KubeSolo (final stage)
│ └── emergency-shell.sh # Drop to shell on boot failure
│
├── update/ # Atomic update agent
│ ├── go.mod
│ ├── go.sum
│ ├── main.go # Update agent entrypoint
│ ├── cmd/
│ │ ├── check.go # Check for available updates
│ │ ├── apply.go # Download + write to passive partition
│ │ ├── activate.go # Update GRUB, set boot counter
│ │ ├── rollback.go # Force rollback to previous partition
│ │ └── healthcheck.go # Post-boot health verification
│ ├── pkg/
│ │ ├── grubenv/ # GRUB environment manipulation
│ │ │ └── grubenv.go
│ │ ├── partition/ # Partition detection and management
│ │ │ └── partition.go
│ │ ├── image/ # Image download, verify, write
│ │ │ └── image.go
│ │ └── health/ # K8s + containerd health checks
│ │ └── health.go
│ └── deploy/
│ └── update-cronjob.yaml # K8s CronJob manifest for auto-updates
│
├── cloud-init/ # Cloud-init implementation
│ ├── cloud-init.go # Lightweight cloud-init parser
│ ├── network.go # Network config from cloud-init
│ ├── kubesolo.go # KubeSolo config from cloud-init
│ └── examples/
│ ├── dhcp.yaml # DHCP example
│ ├── static-ip.yaml # Static IP example
│ ├── portainer-edge.yaml # Portainer Edge integration
│ └── airgapped.yaml # Air-gapped deployment
│
├── test/ # Testing
│ ├── Makefile # Test orchestration
│ ├── qemu/
│ │ ├── run-vm.sh # Launch QEMU VM with built image
│ │ ├── test-boot.sh # Automated boot test
│ │ ├── test-persistence.sh # Reboot + verify state survives
│ │ ├── test-update.sh # A/B update cycle test
│ │ └── test-rollback.sh # Forced rollback test
│ ├── integration/
│ │ ├── test-k8s-ready.sh # Verify K8s node reaches Ready
│ │ ├── test-deploy-workload.sh # Deploy nginx, verify pod running
│ │ ├── test-local-storage.sh # PVC with local-path provisioner
│ │ └── test-network-policy.sh # Basic network policy enforcement
│ └── kernel/
│ └── check-config.sh # Validate kernel config requirements
│
└── hack/ # Developer utilities
├── dev-vm.sh # Quick-launch dev VM (QEMU)
├── rebuild-initramfs.sh # Fast rebuild for dev iteration
├── inject-ssh.sh # Add SSH extension for debugging
└── extract-kernel-config.sh # Pull /proc/config.gz from running TC
```
---
## Architecture Summary
### Core Concept
KubeSolo OS is a **remastered Tiny Core Linux** where the initramfs (`core.gz`) is rebuilt to include KubeSolo and all its dependencies. The result is a single bootable image that:
1. Boots kernel + initramfs into RAM (read-only SquashFS root)
2. Mounts a persistent ext4 partition for K8s state
3. Bind-mounts writable paths (`/var/lib/kubesolo`, `/var/lib/containerd`, etc.)
4. Loads kernel modules (br_netfilter, overlay, veth, etc.)
5. Configures networking (cloud-init → persistent config → DHCP fallback)
6. Starts containerd, then KubeSolo
7. Kubernetes API becomes available; node reaches Ready
### Partition Layout
```
Disk (minimum 8 GB):
Part 1: EFI/Boot (256 MB, FAT32) — GRUB + A/B boot logic
Part 2: System A (512 MB, ext4) — vmlinuz + kubesolo-os.gz (active)
Part 3: System B (512 MB, ext4) — vmlinuz + kubesolo-os.gz (passive)
Part 4: Data (remaining, ext4) — persistent K8s state
```
### Persistent Paths (survive updates)
| Mount Point | Content | On Data Partition |
|---|---|---|
| `/var/lib/kubesolo` | K8s state, certs, SQLite DB | `/mnt/data/kubesolo` |
| `/var/lib/containerd` | Container images + layers | `/mnt/data/containerd` |
| `/etc/kubesolo` | Node configuration | `/mnt/data/etc-kubesolo` |
| `/var/log` | System + K8s logs | `/mnt/data/log` |
| `/usr/local` | User data, extra binaries | `/mnt/data/usr-local` |
### Atomic Updates
A/B partition scheme with GRUB fallback counter:
- Update writes new image to passive partition
- GRUB boots new partition with `boot_counter=3`
- Health check sets `boot_success=1` on success
- On 3 consecutive failures (counter reaches 0), GRUB auto-rolls back
---
## Technology Stack
| Component | Technology | Rationale |
|---|---|---|
| Base OS | Tiny Core Linux 17.0 (Micro Core, x86_64) | 11 MB, RAM-resident, SquashFS root |
| Kernel | Tiny Core stock (6.x) or custom build | Must have cgroup v2, namespaces, netfilter |
| Kubernetes | KubeSolo (single binary) | Single-node K8s, SQLite backend, bundled runtime |
| Container runtime | containerd + runc (bundled in KubeSolo) | Industry standard, KubeSolo dependency |
| Init | Custom shell script (POSIX sh) | Minimal, no systemd dependency |
| Bootloader | GRUB 2 (EFI + BIOS) | A/B partition support, env variables |
| Update agent | Go binary | Single static binary, K8s client-go |
| Cloud-init parser | Go binary or shell script | First-boot configuration |
| Build system | Bash + Make + Docker (builder container) | Reproducible builds |
| Testing | QEMU/KVM + shell scripts | Automated boot + integration tests |
---
## Development Guidelines
### Shell Scripts (init system, build scripts)
- **POSIX sh** — no bashisms in init scripts (BusyBox ash compatibility)
- **Shellcheck** all scripts: `shellcheck -s sh <script>`
- Use `set -euo pipefail` in build scripts (bash)
- Use `set -e` in init scripts (POSIX sh, no pipefail)
- Quote all variable expansions: `"$var"` not `$var`
- Use `$(command)` not backticks
- Functions for reusable logic; source shared libraries from `/usr/lib/kubesolo-os/`
- Log to stderr with prefix: `echo "[kubesolo-init] message" >&2`
- Init stages are numbered (`00-`, `10-`, ...) and sourced in order
### Go Code (update agent, cloud-init parser)
- **Go 1.22+**
- Build static binaries: `CGO_ENABLED=0 go build -ldflags='-s -w' -o binary`
- Use `client-go` for Kubernetes health checks
- Minimal dependencies — this runs on a tiny system
- Error handling: wrap errors with context (`fmt.Errorf("failed to X: %w", err)`)
- Use structured logging (`log/slog`)
- Unit tests required for `pkg/` packages
- No network calls in tests (mock interfaces)
### Build System
- **Makefile** targets:
- `make fetch` — download Tiny Core ISO, KubeSolo binary, dependencies
- `make rootfs` — extract core.gz, inject KubeSolo, prepare rootfs
- `make initramfs` — repack rootfs into kubesolo-os.gz
- `make iso` — create bootable ISO
- `make disk-image` — create raw disk image with A/B partitions
- `make test-boot` — launch QEMU, verify boot + K8s ready
- `make test-update` — full A/B update cycle test
- `make test-all` — run all tests
- `make clean` — remove build artifacts
- `make docker-build` — run entire build inside Docker (reproducible)
- **Reproducible builds** — pin all component versions in `build/config/versions.env`:
```bash
TINYCORE_VERSION=17.0
TINYCORE_ARCH=x86_64
KUBESOLO_VERSION=latest # pin to specific release when available
CONTAINERD_VERSION=1.7.x # if fetching separately
GRUB_VERSION=2.12
```
- **Builder container** — `build/Dockerfile.builder` with all build tools (cpio, gzip, grub-mkimage, squashfs-tools, qemu for testing)
- All downloads go to `build/cache/` (gitignored, reused across builds)
- Build output goes to `output/` (gitignored)
### Testing
- **Every change must pass `make test-boot`** — the image boots and K8s reaches Ready
- QEMU tests use `-nographic` and serial console for CI compatibility
- Test timeout: 120 seconds for boot, 300 seconds for K8s ready
- Integration tests use `kubectl` against the VM's forwarded API port
- Kernel config audit runs as a build-time check, not runtime
### Git Workflow
- Branch naming: `feat/`, `fix/`, `docs/`, `test/`, `build/`
- Commit messages: conventional commits (`feat:`, `fix:`, `build:`, `test:`, `docs:`)
- Tag releases: `v0.1.0`, `v0.2.0`, etc.
- `.gitignore`: `build/cache/`, `output/`, `*.iso`, `*.img`, `*.gz` (build artifacts)
---
## Key Kernel Requirements
The Tiny Core kernel **MUST** have these configs. Run `build/config/kernel-audit.sh` against the kernel to verify:
```
# Mandatory — cgroup v2
CONFIG_CGROUPS=y
CONFIG_CGROUP_CPUACCT=y
CONFIG_CGROUP_DEVICE=y
CONFIG_CGROUP_FREEZER=y
CONFIG_CGROUP_SCHED=y
CONFIG_CGROUP_PIDS=y
CONFIG_MEMCG=y
# Mandatory — namespaces
CONFIG_NAMESPACES=y
CONFIG_NET_NS=y
CONFIG_PID_NS=y
CONFIG_USER_NS=y
CONFIG_UTS_NS=y
CONFIG_IPC_NS=y
# Mandatory — filesystem
CONFIG_OVERLAY_FS=y|m
CONFIG_SQUASHFS=y
# Mandatory — networking
CONFIG_BRIDGE=y|m
CONFIG_NETFILTER=y
CONFIG_NF_NAT=y|m
CONFIG_IP_NF_IPTABLES=y|m
CONFIG_IP_NF_NAT=y|m
CONFIG_IP_NF_FILTER=y|m
CONFIG_VETH=y|m
CONFIG_VXLAN=y|m
# Recommended
CONFIG_BPF_SYSCALL=y
CONFIG_SECCOMP=y
CONFIG_CRYPTO_SHA256=y|m
```
If the stock Tiny Core kernel is missing any mandatory config, the project must either:
1. Load the feature as a kernel module (if `=m`)
2. Custom-compile the kernel with the missing options enabled
---
## Boot Parameters
KubeSolo OS uses kernel command line parameters for runtime configuration:
| Parameter | Default | Description |
|---|---|---|
| `kubesolo.data=<device>` | (required) | Block device for persistent data partition |
| `kubesolo.debug` | (off) | Enable verbose init logging |
| `kubesolo.shell` | (off) | Drop to emergency shell instead of booting |
| `kubesolo.nopersist` | (off) | Run fully in RAM (no persistent mount) |
| `kubesolo.cloudinit=<path>` | `/mnt/data/etc-kubesolo/cloud-init.yaml` | Cloud-init config file |
| `kubesolo.flags=<flags>` | (none) | Extra flags passed to KubeSolo binary |
| `quiet` | (off) | Suppress kernel boot messages |
---
## Phase 1 Scope (Current)
The immediate goal is a **Proof of Concept** that boots to a functional K8s node:
### Deliverables
1. `build/scripts/fetch-components.sh` — downloads Tiny Core ISO + KubeSolo
2. `build/scripts/extract-core.sh` — extracts Tiny Core rootfs from ISO
3. `build/config/kernel-audit.sh` — checks kernel config against requirements
4. `init/init.sh` + `init/lib/*.sh` — modular init system
5. `build/scripts/inject-kubesolo.sh` — adds KubeSolo + deps to rootfs
6. `build/scripts/pack-initramfs.sh` — repacks into kubesolo-os.gz
7. `build/scripts/create-iso.sh` — creates bootable ISO (syslinux, simpler than GRUB for PoC)
8. `test/qemu/run-vm.sh` — launches QEMU with the ISO
9. `test/qemu/test-boot.sh` — automated boot + K8s readiness check
10. `Makefile` — ties it all together
### NOT in Phase 1
- A/B partitions (Phase 3)
- GRUB bootloader (Phase 3 — use syslinux/isolinux for PoC ISO)
- Update agent (Phase 3)
- Cloud-init parser (Phase 2)
- OCI image distribution (Phase 5)
- ARM64 support (Phase 5)
### Success Criteria
- `make iso` produces a bootable ISO < 100 MB
- ISO boots in QEMU in < 30 seconds to login/shell
- KubeSolo starts and `kubectl get nodes` shows node Ready within 120 seconds
- A test pod (`nginx`) can be deployed and reaches Running state
- System root is read-only (writes to `/usr`, `/bin`, `/sbin` fail)
- Reboot preserves K8s state (pods, services survive restart)
---
## Common Tasks
### First-time setup
```bash
# Clone and enter repo
git clone <repo-url> kubesolo-os && cd kubesolo-os
# Fetch all components (downloads to build/cache/)
make fetch
# Full build → ISO
make iso
# Boot in QEMU for testing
make test-boot
```
### Rebuild after init script changes
```bash
# Fast path: just repack initramfs and rebuild ISO
make initramfs iso
```
### Run all tests
```bash
make test-all
```
### Debug a failing boot
```bash
# Boot with serial console attached
./hack/dev-vm.sh
# Or boot to emergency shell
./hack/dev-vm.sh --shell
```
### Add SSH for debugging (dev only)
```bash
./hack/inject-ssh.sh output/kubesolo-os.gz
# Rebuilds initramfs with dropbear SSH + your ~/.ssh/id_rsa.pub
```
---
## Important Constraints
1. **No systemd** — Tiny Core uses BusyBox init; our custom init is pure POSIX sh
2. **No package manager at runtime** — everything needed must be in the initramfs
3. **BusyBox userland** — commands may have limited flags vs GNU coreutils (test with BusyBox)
4. **Static binaries preferred** — Go binaries must be `CGO_ENABLED=0`; avoid glibc runtime deps
5. **KubeSolo bundles containerd** — do NOT install a separate containerd; use what KubeSolo ships
6. **Memory budget** — target 512 MB minimum RAM; OS overhead should be < 100 MB
7. **Disk image must be self-contained** — no network access required during boot (air-gap safe)
8. **Kernel modules** — only modules present in the initramfs are available; no runtime module install
---
## External References
- [KubeSolo GitHub](https://github.com/portainer/kubesolo)
- [Tiny Core Linux](http://www.tinycorelinux.net)
- [Tiny Core Remastering Wiki](http://wiki.tinycorelinux.net/doku.php?id=wiki:remastering)
- [Tiny Core Into the Core](http://wiki.tinycorelinux.net/doku.php?id=wiki:into_the_core)
- [Talos Linux](https://www.talos.dev) — reference for immutable K8s OS patterns
- [Kairos](https://kairos.io) — reference for OCI-based immutable OS distribution
- [Kubernetes Node Requirements](https://kubernetes.io/docs/setup/production-environment/container-runtimes/)
- [cgroup v2 Documentation](https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html)