# CLAUDE.md — KubeSolo OS

## Project Overview

**KubeSolo OS** is an immutable, bootable Linux distribution purpose-built to run [KubeSolo](https://github.com/portainer/kubesolo) — Portainer's ultra-lightweight single-node Kubernetes distribution. It combines Tiny Core Linux's minimal footprint (~11 MB) with KubeSolo's single-binary K8s packaging to create an appliance-like Kubernetes node with atomic A/B updates.

**Design document:** See `docs/design/kubesolo-os-design.md` for full architecture research, competitive analysis, and technical specifications.

**Target:** Edge/IoT devices, single-node K8s appliances, air-gapped deployments, resource-constrained hardware.

---

## Repository Structure

```
kubesolo-os/
├── CLAUDE.md                          # This file
├── README.md                          # Project README
├── Makefile                           # Top-level build orchestration
├── VERSION                            # Semver version (e.g., 0.1.0)
│
├── docs/
│   ├── design/
│   │   └── kubesolo-os-design.md      # Full architecture document
│   ├── boot-flow.md                   # Boot sequence documentation
│   ├── update-flow.md                 # Atomic update documentation
│   └── cloud-init.md                  # Configuration reference
│
├── build/                             # Build system
│   ├── Dockerfile.builder             # Containerized build environment
│   ├── build.sh                       # Main build script (orchestrator)
│   ├── config/
│   │   ├── kernel-audit.sh            # Verify kernel config requirements
│   │   ├── kernel-config.fragment     # Custom kernel config overrides (if needed)
│   │   └── modules.list              # Required kernel modules list
│   ├── rootfs/                        # Files injected into initramfs
│   │   ├── sbin/
│   │   │   └── init                   # Custom init script
│   │   ├── etc/
│   │   │   ├── os-release             # OS identification
│   │   │   ├── sysctl.d/
│   │   │   │   └── k8s.conf          # Kernel parameters for K8s
│   │   │   └── kubesolo/
│   │   │       └── defaults.yaml      # Default KubeSolo config
│   │   └── usr/
│   │       └── lib/
│   │           └── kubesolo-os/
│   │               ├── functions.sh   # Shared shell functions
│   │               ├── network.sh     # Network configuration helpers
│   │               └── health.sh      # Health check functions
│   ├── grub/
│   │   ├── grub.cfg                   # A/B boot GRUB config
│   │   └── grub-env-defaults         # Default GRUB environment vars
│   └── scripts/
│       ├── fetch-components.sh        # Download Tiny Core, KubeSolo, deps
│       ├── extract-core.sh            # Extract and prepare Tiny Core rootfs
│       ├── inject-kubesolo.sh         # Add KubeSolo + deps to rootfs
│       ├── pack-initramfs.sh          # Repack initramfs (core.gz → kubesolo-os.gz)
│       ├── create-iso.sh             # Build bootable ISO
│       ├── create-disk-image.sh       # Build raw disk image with A/B partitions
│       └── create-oci-image.sh        # Build OCI container image (future)
│
├── init/                              # Init system source
│   ├── init.sh                        # Main init script (becomes /sbin/init)
│   ├── lib/
│   │   ├── 00-early-mount.sh         # Mount proc, sys, dev, tmpfs
│   │   ├── 10-parse-cmdline.sh       # Parse kernel boot parameters
│   │   ├── 20-persistent-mount.sh    # Mount + bind persistent data partition
│   │   ├── 30-kernel-modules.sh      # Load required kernel modules
│   │   ├── 40-sysctl.sh             # Apply sysctl settings
│   │   ├── 50-network.sh            # Network configuration (cloud-init/DHCP)
│   │   ├── 60-hostname.sh           # Set hostname
│   │   ├── 70-clock.sh              # NTP / system clock
│   │   ├── 80-containerd.sh         # Start containerd
│   │   └── 90-kubesolo.sh           # Start KubeSolo (final stage)
│   └── emergency-shell.sh            # Drop to shell on boot failure
│
├── update/                            # Atomic update agent
│   ├── go.mod
│   ├── go.sum
│   ├── main.go                        # Update agent entrypoint
│   ├── cmd/
│   │   ├── check.go                   # Check for available updates
│   │   ├── apply.go                   # Download + write to passive partition
│   │   ├── activate.go                # Update GRUB, set boot counter
│   │   ├── rollback.go                # Force rollback to previous partition
│   │   └── healthcheck.go            # Post-boot health verification
│   ├── pkg/
│   │   ├── grubenv/                   # GRUB environment manipulation
│   │   │   └── grubenv.go
│   │   ├── partition/                 # Partition detection and management
│   │   │   └── partition.go
│   │   ├── image/                     # Image download, verify, write
│   │   │   └── image.go
│   │   └── health/                    # K8s + containerd health checks
│   │       └── health.go
│   └── deploy/
│       └── update-cronjob.yaml        # K8s CronJob manifest for auto-updates
│
├── cloud-init/                        # Cloud-init implementation
│   ├── cloud-init.go                  # Lightweight cloud-init parser
│   ├── network.go                     # Network config from cloud-init
│   ├── kubesolo.go                    # KubeSolo config from cloud-init
│   └── examples/
│       ├── dhcp.yaml                  # DHCP example
│       ├── static-ip.yaml            # Static IP example
│       ├── portainer-edge.yaml        # Portainer Edge integration
│       └── airgapped.yaml            # Air-gapped deployment
│
├── test/                              # Testing
│   ├── Makefile                       # Test orchestration
│   ├── qemu/
│   │   ├── run-vm.sh                  # Launch QEMU VM with built image
│   │   ├── test-boot.sh              # Automated boot test
│   │   ├── test-persistence.sh       # Reboot + verify state survives
│   │   ├── test-update.sh            # A/B update cycle test
│   │   └── test-rollback.sh          # Forced rollback test
│   ├── integration/
│   │   ├── test-k8s-ready.sh         # Verify K8s node reaches Ready
│   │   ├── test-deploy-workload.sh   # Deploy nginx, verify pod running
│   │   ├── test-local-storage.sh     # PVC with local-path provisioner
│   │   └── test-network-policy.sh    # Basic network policy enforcement
│   └── kernel/
│       └── check-config.sh           # Validate kernel config requirements
│
└── hack/                              # Developer utilities
    ├── dev-vm.sh                      # Quick-launch dev VM (QEMU)
    ├── rebuild-initramfs.sh          # Fast rebuild for dev iteration
    ├── inject-ssh.sh                  # Add SSH extension for debugging
    └── extract-kernel-config.sh      # Pull /proc/config.gz from running TC
```

---

## Architecture Summary

### Core Concept

KubeSolo OS is a **remastered Tiny Core Linux** where the initramfs (`core.gz`) is rebuilt to include KubeSolo and all its dependencies. The result is a single bootable image that:

1. Boots kernel + initramfs into RAM (read-only SquashFS root)
2. Mounts a persistent ext4 partition for K8s state
3. Bind-mounts writable paths (`/var/lib/kubesolo`, `/var/lib/containerd`, etc.)
4. Loads kernel modules (br_netfilter, overlay, veth, etc.)
5. Configures networking (cloud-init → persistent config → DHCP fallback)
6. Starts containerd, then KubeSolo
7. Kubernetes API becomes available; node reaches Ready

### Partition Layout

```
Disk (minimum 8 GB):
  Part 1: EFI/Boot    (256 MB, FAT32)  — GRUB + A/B boot logic
  Part 2: System A    (512 MB, ext4)   — vmlinuz + kubesolo-os.gz (active)
  Part 3: System B    (512 MB, ext4)   — vmlinuz + kubesolo-os.gz (passive)
  Part 4: Data        (remaining, ext4) — persistent K8s state
```

### Persistent Paths (survive updates)

| Mount Point | Content | On Data Partition |
|---|---|---|
| `/var/lib/kubesolo` | K8s state, certs, SQLite DB | `/mnt/data/kubesolo` |
| `/var/lib/containerd` | Container images + layers | `/mnt/data/containerd` |
| `/etc/kubesolo` | Node configuration | `/mnt/data/etc-kubesolo` |
| `/var/log` | System + K8s logs | `/mnt/data/log` |
| `/usr/local` | User data, extra binaries | `/mnt/data/usr-local` |

### Atomic Updates

A/B partition scheme with GRUB fallback counter:
- Update writes new image to passive partition
- GRUB boots new partition with `boot_counter=3`
- Health check sets `boot_success=1` on success
- On 3 consecutive failures (counter reaches 0), GRUB auto-rolls back

---

## Technology Stack

| Component | Technology | Rationale |
|---|---|---|
| Base OS | Tiny Core Linux 17.0 (Micro Core, x86_64) | 11 MB, RAM-resident, SquashFS root |
| Kernel | Tiny Core stock (6.x) or custom build | Must have cgroup v2, namespaces, netfilter |
| Kubernetes | KubeSolo (single binary) | Single-node K8s, SQLite backend, bundled runtime |
| Container runtime | containerd + runc (bundled in KubeSolo) | Industry standard, KubeSolo dependency |
| Init | Custom shell script (POSIX sh) | Minimal, no systemd dependency |
| Bootloader | GRUB 2 (EFI + BIOS) | A/B partition support, env variables |
| Update agent | Go binary | Single static binary, K8s client-go |
| Cloud-init parser | Go binary or shell script | First-boot configuration |
| Build system | Bash + Make + Docker (builder container) | Reproducible builds |
| Testing | QEMU/KVM + shell scripts | Automated boot + integration tests |

---

## Development Guidelines

### Shell Scripts (init system, build scripts)

- **POSIX sh** — no bashisms in init scripts (BusyBox ash compatibility)
- **Shellcheck** all scripts: `shellcheck -s sh <script>`
- Use `set -euo pipefail` in build scripts (bash)
- Use `set -e` in init scripts (POSIX sh, no pipefail)
- Quote all variable expansions: `"$var"` not `$var`
- Use `$(command)` not backticks
- Functions for reusable logic; source shared libraries from `/usr/lib/kubesolo-os/`
- Log to stderr with prefix: `echo "[kubesolo-init] message" >&2`
- Init stages are numbered (`00-`, `10-`, ...) and sourced in order

### Go Code (update agent, cloud-init parser)

- **Go 1.22+**
- Build static binaries: `CGO_ENABLED=0 go build -ldflags='-s -w' -o binary`
- Use `client-go` for Kubernetes health checks
- Minimal dependencies — this runs on a tiny system
- Error handling: wrap errors with context (`fmt.Errorf("failed to X: %w", err)`)
- Use structured logging (`log/slog`)
- Unit tests required for `pkg/` packages
- No network calls in tests (mock interfaces)

### Build System

- **Makefile** targets:
  - `make fetch` — download Tiny Core ISO, KubeSolo binary, dependencies
  - `make rootfs` — extract core.gz, inject KubeSolo, prepare rootfs
  - `make initramfs` — repack rootfs into kubesolo-os.gz
  - `make iso` — create bootable ISO
  - `make disk-image` — create raw disk image with A/B partitions
  - `make test-boot` — launch QEMU, verify boot + K8s ready
  - `make test-update` — full A/B update cycle test
  - `make test-all` — run all tests
  - `make clean` — remove build artifacts
  - `make docker-build` — run entire build inside Docker (reproducible)
- **Reproducible builds** — pin all component versions in `build/config/versions.env`:
  ```bash
  TINYCORE_VERSION=17.0
  TINYCORE_ARCH=x86_64
  KUBESOLO_VERSION=latest  # pin to specific release when available
  CONTAINERD_VERSION=1.7.x # if fetching separately
  GRUB_VERSION=2.12
  ```
- **Builder container** — `build/Dockerfile.builder` with all build tools (cpio, gzip, grub-mkimage, squashfs-tools, qemu for testing)
- All downloads go to `build/cache/` (gitignored, reused across builds)
- Build output goes to `output/` (gitignored)

### Testing

- **Every change must pass `make test-boot`** — the image boots and K8s reaches Ready
- QEMU tests use `-nographic` and serial console for CI compatibility
- Test timeout: 120 seconds for boot, 300 seconds for K8s ready
- Integration tests use `kubectl` against the VM's forwarded API port
- Kernel config audit runs as a build-time check, not runtime

### Git Workflow

- Branch naming: `feat/`, `fix/`, `docs/`, `test/`, `build/`
- Commit messages: conventional commits (`feat:`, `fix:`, `build:`, `test:`, `docs:`)
- Tag releases: `v0.1.0`, `v0.2.0`, etc.
- `.gitignore`: `build/cache/`, `output/`, `*.iso`, `*.img`, `*.gz` (build artifacts)

---

## Key Kernel Requirements

The Tiny Core kernel **MUST** have these configs. Run `build/config/kernel-audit.sh` against the kernel to verify:

```
# Mandatory — cgroup v2
CONFIG_CGROUPS=y
CONFIG_CGROUP_CPUACCT=y
CONFIG_CGROUP_DEVICE=y
CONFIG_CGROUP_FREEZER=y
CONFIG_CGROUP_SCHED=y
CONFIG_CGROUP_PIDS=y
CONFIG_MEMCG=y

# Mandatory — namespaces
CONFIG_NAMESPACES=y
CONFIG_NET_NS=y
CONFIG_PID_NS=y
CONFIG_USER_NS=y
CONFIG_UTS_NS=y
CONFIG_IPC_NS=y

# Mandatory — filesystem
CONFIG_OVERLAY_FS=y|m
CONFIG_SQUASHFS=y

# Mandatory — networking
CONFIG_BRIDGE=y|m
CONFIG_NETFILTER=y
CONFIG_NF_NAT=y|m
CONFIG_IP_NF_IPTABLES=y|m
CONFIG_IP_NF_NAT=y|m
CONFIG_IP_NF_FILTER=y|m
CONFIG_VETH=y|m
CONFIG_VXLAN=y|m

# Recommended
CONFIG_BPF_SYSCALL=y
CONFIG_SECCOMP=y
CONFIG_CRYPTO_SHA256=y|m
```

If the stock Tiny Core kernel is missing any mandatory config, the project must either:
1. Load the feature as a kernel module (if `=m`)
2. Custom-compile the kernel with the missing options enabled

---

## Boot Parameters

KubeSolo OS uses kernel command line parameters for runtime configuration:

| Parameter | Default | Description |
|---|---|---|
| `kubesolo.data=<device>` | (required) | Block device for persistent data partition |
| `kubesolo.debug` | (off) | Enable verbose init logging |
| `kubesolo.shell` | (off) | Drop to emergency shell instead of booting |
| `kubesolo.nopersist` | (off) | Run fully in RAM (no persistent mount) |
| `kubesolo.cloudinit=<path>` | `/mnt/data/etc-kubesolo/cloud-init.yaml` | Cloud-init config file |
| `kubesolo.flags=<flags>` | (none) | Extra flags passed to KubeSolo binary |
| `quiet` | (off) | Suppress kernel boot messages |

---

## Phase 1 Scope (Current)

The immediate goal is a **Proof of Concept** that boots to a functional K8s node:

### Deliverables
1. `build/scripts/fetch-components.sh` — downloads Tiny Core ISO + KubeSolo
2. `build/scripts/extract-core.sh` — extracts Tiny Core rootfs from ISO
3. `build/config/kernel-audit.sh` — checks kernel config against requirements
4. `init/init.sh` + `init/lib/*.sh` — modular init system
5. `build/scripts/inject-kubesolo.sh` — adds KubeSolo + deps to rootfs
6. `build/scripts/pack-initramfs.sh` — repacks into kubesolo-os.gz
7. `build/scripts/create-iso.sh` — creates bootable ISO (syslinux, simpler than GRUB for PoC)
8. `test/qemu/run-vm.sh` — launches QEMU with the ISO
9. `test/qemu/test-boot.sh` — automated boot + K8s readiness check
10. `Makefile` — ties it all together

### NOT in Phase 1
- A/B partitions (Phase 3)
- GRUB bootloader (Phase 3 — use syslinux/isolinux for PoC ISO)
- Update agent (Phase 3)
- Cloud-init parser (Phase 2)
- OCI image distribution (Phase 5)
- ARM64 support (Phase 5)

### Success Criteria
- `make iso` produces a bootable ISO < 100 MB
- ISO boots in QEMU in < 30 seconds to login/shell
- KubeSolo starts and `kubectl get nodes` shows node Ready within 120 seconds
- A test pod (`nginx`) can be deployed and reaches Running state
- System root is read-only (writes to `/usr`, `/bin`, `/sbin` fail)
- Reboot preserves K8s state (pods, services survive restart)

---

## Common Tasks

### First-time setup
```bash
# Clone and enter repo
git clone <repo-url> kubesolo-os && cd kubesolo-os

# Fetch all components (downloads to build/cache/)
make fetch

# Full build → ISO
make iso

# Boot in QEMU for testing
make test-boot
```

### Rebuild after init script changes
```bash
# Fast path: just repack initramfs and rebuild ISO
make initramfs iso
```

### Run all tests
```bash
make test-all
```

### Debug a failing boot
```bash
# Boot with serial console attached
./hack/dev-vm.sh

# Or boot to emergency shell
./hack/dev-vm.sh --shell
```

### Add SSH for debugging (dev only)
```bash
./hack/inject-ssh.sh output/kubesolo-os.gz
# Rebuilds initramfs with dropbear SSH + your ~/.ssh/id_rsa.pub
```

---

## Important Constraints

1. **No systemd** — Tiny Core uses BusyBox init; our custom init is pure POSIX sh
2. **No package manager at runtime** — everything needed must be in the initramfs
3. **BusyBox userland** — commands may have limited flags vs GNU coreutils (test with BusyBox)
4. **Static binaries preferred** — Go binaries must be `CGO_ENABLED=0`; avoid glibc runtime deps
5. **KubeSolo bundles containerd** — do NOT install a separate containerd; use what KubeSolo ships
6. **Memory budget** — target 512 MB minimum RAM; OS overhead should be < 100 MB
7. **Disk image must be self-contained** — no network access required during boot (air-gap safe)
8. **Kernel modules** — only modules present in the initramfs are available; no runtime module install

---

## External References

- [KubeSolo GitHub](https://github.com/portainer/kubesolo)
- [Tiny Core Linux](http://www.tinycorelinux.net)
- [Tiny Core Remastering Wiki](http://wiki.tinycorelinux.net/doku.php?id=wiki:remastering)
- [Tiny Core Into the Core](http://wiki.tinycorelinux.net/doku.php?id=wiki:into_the_core)
- [Talos Linux](https://www.talos.dev) — reference for immutable K8s OS patterns
- [Kairos](https://kairos.io) — reference for OCI-based immutable OS distribution
- [Kubernetes Node Requirements](https://kubernetes.io/docs/setup/production-environment/container-runtimes/)
- [cgroup v2 Documentation](https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html)