kubesolo-os/CLAUDE.md

CLAUDE.md — KubeSolo OS

Project Overview

KubeSolo OS is an immutable, bootable Linux distribution purpose-built to run KubeSolo — Portainer's ultra-lightweight single-node Kubernetes distribution. It combines Tiny Core Linux's minimal footprint (~11 MB) with KubeSolo's single-binary K8s packaging to create an appliance-like Kubernetes node with atomic A/B updates.

Design document: See docs/design/kubesolo-os-design.md for full architecture research, competitive analysis, and technical specifications.

Target: Edge/IoT devices, single-node K8s appliances, air-gapped deployments, resource-constrained hardware.

---

Repository Structure

kubesolo-os/
├── CLAUDE.md                          # This file
├── README.md                          # Project README
├── Makefile                           # Top-level build orchestration
├── VERSION                            # Semver version (e.g., 0.1.0)
│
├── docs/
│   ├── design/
│   │   └── kubesolo-os-design.md      # Full architecture document
│   ├── boot-flow.md                   # Boot sequence documentation
│   ├── update-flow.md                 # Atomic update documentation
│   └── cloud-init.md                  # Configuration reference
│
├── build/                             # Build system
│   ├── Dockerfile.builder             # Containerized build environment
│   ├── build.sh                       # Main build script (orchestrator)
│   ├── config/
│   │   ├── kernel-audit.sh            # Verify kernel config requirements
│   │   ├── kernel-config.fragment     # Custom kernel config overrides (if needed)
│   │   └── modules.list              # Required kernel modules list
│   ├── rootfs/                        # Files injected into initramfs
│   │   ├── sbin/
│   │   │   └── init                   # Custom init script
│   │   ├── etc/
│   │   │   ├── os-release             # OS identification
│   │   │   ├── sysctl.d/
│   │   │   │   └── k8s.conf          # Kernel parameters for K8s
│   │   │   └── kubesolo/
│   │   │       └── defaults.yaml      # Default KubeSolo config
│   │   └── usr/
│   │       └── lib/
│   │           └── kubesolo-os/
│   │               ├── functions.sh   # Shared shell functions
│   │               ├── network.sh     # Network configuration helpers
│   │               └── health.sh      # Health check functions
│   ├── grub/
│   │   ├── grub.cfg                   # A/B boot GRUB config
│   │   └── grub-env-defaults         # Default GRUB environment vars
│   └── scripts/
│       ├── fetch-components.sh        # Download Tiny Core, KubeSolo, deps
│       ├── extract-core.sh            # Extract and prepare Tiny Core rootfs
│       ├── inject-kubesolo.sh         # Add KubeSolo + deps to rootfs
│       ├── pack-initramfs.sh          # Repack initramfs (core.gz → kubesolo-os.gz)
│       ├── create-iso.sh             # Build bootable ISO
│       ├── create-disk-image.sh       # Build raw disk image with A/B partitions
│       └── create-oci-image.sh        # Build OCI container image (future)
│
├── init/                              # Init system source
│   ├── init.sh                        # Main init script (becomes /sbin/init)
│   ├── lib/
│   │   ├── 00-early-mount.sh         # Mount proc, sys, dev, tmpfs
│   │   ├── 10-parse-cmdline.sh       # Parse kernel boot parameters
│   │   ├── 20-persistent-mount.sh    # Mount + bind persistent data partition
│   │   ├── 30-kernel-modules.sh      # Load required kernel modules
│   │   ├── 40-sysctl.sh             # Apply sysctl settings
│   │   ├── 50-network.sh            # Network configuration (cloud-init/DHCP)
│   │   ├── 60-hostname.sh           # Set hostname
│   │   ├── 70-clock.sh              # NTP / system clock
│   │   ├── 80-containerd.sh         # Start containerd
│   │   └── 90-kubesolo.sh           # Start KubeSolo (final stage)
│   └── emergency-shell.sh            # Drop to shell on boot failure
│
├── update/                            # Atomic update agent
│   ├── go.mod
│   ├── go.sum
│   ├── main.go                        # Update agent entrypoint
│   ├── cmd/
│   │   ├── check.go                   # Check for available updates
│   │   ├── apply.go                   # Download + write to passive partition
│   │   ├── activate.go                # Update GRUB, set boot counter
│   │   ├── rollback.go                # Force rollback to previous partition
│   │   └── healthcheck.go            # Post-boot health verification
│   ├── pkg/
│   │   ├── grubenv/                   # GRUB environment manipulation
│   │   │   └── grubenv.go
│   │   ├── partition/                 # Partition detection and management
│   │   │   └── partition.go
│   │   ├── image/                     # Image download, verify, write
│   │   │   └── image.go
│   │   └── health/                    # K8s + containerd health checks
│   │       └── health.go
│   └── deploy/
│       └── update-cronjob.yaml        # K8s CronJob manifest for auto-updates
│
├── cloud-init/                        # Cloud-init implementation
│   ├── cloud-init.go                  # Lightweight cloud-init parser
│   ├── network.go                     # Network config from cloud-init
│   ├── kubesolo.go                    # KubeSolo config from cloud-init
│   └── examples/
│       ├── dhcp.yaml                  # DHCP example
│       ├── static-ip.yaml            # Static IP example
│       ├── portainer-edge.yaml        # Portainer Edge integration
│       └── airgapped.yaml            # Air-gapped deployment
│
├── test/                              # Testing
│   ├── Makefile                       # Test orchestration
│   ├── qemu/
│   │   ├── run-vm.sh                  # Launch QEMU VM with built image
│   │   ├── test-boot.sh              # Automated boot test
│   │   ├── test-persistence.sh       # Reboot + verify state survives
│   │   ├── test-update.sh            # A/B update cycle test
│   │   └── test-rollback.sh          # Forced rollback test
│   ├── integration/
│   │   ├── test-k8s-ready.sh         # Verify K8s node reaches Ready
│   │   ├── test-deploy-workload.sh   # Deploy nginx, verify pod running
│   │   ├── test-local-storage.sh     # PVC with local-path provisioner
│   │   └── test-network-policy.sh    # Basic network policy enforcement
│   └── kernel/
│       └── check-config.sh           # Validate kernel config requirements
│
└── hack/                              # Developer utilities
    ├── dev-vm.sh                      # Quick-launch dev VM (QEMU)
    ├── rebuild-initramfs.sh          # Fast rebuild for dev iteration
    ├── inject-ssh.sh                  # Add SSH extension for debugging
    └── extract-kernel-config.sh      # Pull /proc/config.gz from running TC

---

Architecture Summary

Core Concept

KubeSolo OS is a remastered Tiny Core Linux where the initramfs (core.gz) is rebuilt to include KubeSolo and all its dependencies. The result is a single bootable image that:

1. Boots kernel + initramfs into RAM (read-only SquashFS root)
2. Mounts a persistent ext4 partition for K8s state
3. Bind-mounts writable paths (/var/lib/kubesolo, /var/lib/containerd, etc.)
4. Loads kernel modules (br_netfilter, overlay, veth, etc.)
5. Configures networking (cloud-init → persistent config → DHCP fallback)
6. Starts containerd, then KubeSolo
7. Kubernetes API becomes available; node reaches Ready

Partition Layout

Disk (minimum 8 GB):
  Part 1: EFI/Boot    (256 MB, FAT32)  — GRUB + A/B boot logic
  Part 2: System A    (512 MB, ext4)   — vmlinuz + kubesolo-os.gz (active)
  Part 3: System B    (512 MB, ext4)   — vmlinuz + kubesolo-os.gz (passive)
  Part 4: Data        (remaining, ext4) — persistent K8s state

Persistent Paths (survive updates)

Mount Point	Content	On Data Partition
/var/lib/kubesolo	K8s state, certs, SQLite DB	/mnt/data/kubesolo
/var/lib/containerd	Container images + layers	/mnt/data/containerd
/etc/kubesolo	Node configuration	/mnt/data/etc-kubesolo
/var/log	System + K8s logs	/mnt/data/log
/usr/local	User data, extra binaries	/mnt/data/usr-local

Atomic Updates

A/B partition scheme with GRUB fallback counter:

• Update writes new image to passive partition
• GRUB boots new partition with boot_counter=3
• Health check sets boot_success=1 on success
• On 3 consecutive failures (counter reaches 0), GRUB auto-rolls back

---

Technology Stack

Component	Technology	Rationale
Base OS	Tiny Core Linux 17.0 (Micro Core, x86_64)	11 MB, RAM-resident, SquashFS root
Kernel	Tiny Core stock (6.x) or custom build	Must have cgroup v2, namespaces, netfilter
Kubernetes	KubeSolo (single binary)	Single-node K8s, SQLite backend, bundled runtime
Container runtime	containerd + runc (bundled in KubeSolo)	Industry standard, KubeSolo dependency
Init	Custom shell script (POSIX sh)	Minimal, no systemd dependency
Bootloader	GRUB 2 (EFI + BIOS)	A/B partition support, env variables
Update agent	Go binary	Single static binary, K8s client-go
Cloud-init parser	Go binary or shell script	First-boot configuration
Build system	Bash + Make + Docker (builder container)	Reproducible builds
Testing	QEMU/KVM + shell scripts	Automated boot + integration tests

---

Development Guidelines

Shell Scripts (init system, build scripts)

• POSIX sh — no bashisms in init scripts (BusyBox ash compatibility)
• Shellcheck all scripts: shellcheck -s sh <script>
• Use set -euo pipefail in build scripts (bash)
• Use set -e in init scripts (POSIX sh, no pipefail)
• Quote all variable expansions: "$var" not $var
• Use $(command) not backticks
• Functions for reusable logic; source shared libraries from /usr/lib/kubesolo-os/
• Log to stderr with prefix: echo "[kubesolo-init] message" >&2
• Init stages are numbered (00-, 10-, ...) and sourced in order

Go Code (update agent, cloud-init parser)

• Go 1.22+
• Build static binaries: CGO_ENABLED=0 go build -ldflags='-s -w' -o binary
• Use client-go for Kubernetes health checks
• Minimal dependencies — this runs on a tiny system
• Error handling: wrap errors with context (fmt.Errorf("failed to X: %w", err))
• Use structured logging (log/slog)
• Unit tests required for pkg/ packages
• No network calls in tests (mock interfaces)

Build System

• Makefile targets:
  • make fetch — download Tiny Core ISO, KubeSolo binary, dependencies
  • make rootfs — extract core.gz, inject KubeSolo, prepare rootfs
  • make initramfs — repack rootfs into kubesolo-os.gz
  • make iso — create bootable ISO
  • make disk-image — create raw disk image with A/B partitions
  • make test-boot — launch QEMU, verify boot + K8s ready
  • make test-update — full A/B update cycle test
  • make test-all — run all tests
  • make clean — remove build artifacts
  • make docker-build — run entire build inside Docker (reproducible)
• Reproducible builds — pin all component versions in build/config/versions.env:

  TINYCORE_VERSION=17.0
  TINYCORE_ARCH=x86_64
  KUBESOLO_VERSION=latest  # pin to specific release when available
  CONTAINERD_VERSION=1.7.x # if fetching separately
  GRUB_VERSION=2.12
  

• Builder container — build/Dockerfile.builder with all build tools (cpio, gzip, grub-mkimage, squashfs-tools, qemu for testing)
• All downloads go to build/cache/ (gitignored, reused across builds)
• Build output goes to output/ (gitignored)

Testing

• Every change must pass make test-boot — the image boots and K8s reaches Ready
• QEMU tests use -nographic and serial console for CI compatibility
• Test timeout: 120 seconds for boot, 300 seconds for K8s ready
• Integration tests use kubectl against the VM's forwarded API port
• Kernel config audit runs as a build-time check, not runtime

Git Workflow

• Branch naming: feat/, fix/, docs/, test/, build/
• Commit messages: conventional commits (feat:, fix:, build:, test:, docs:)
• Tag releases: v0.1.0, v0.2.0, etc.
• .gitignore: build/cache/, output/, *.iso, *.img, *.gz (build artifacts)

---

Key Kernel Requirements

The Tiny Core kernel MUST have these configs. Run build/config/kernel-audit.sh against the kernel to verify:

# Mandatory — cgroup v2
CONFIG_CGROUPS=y
CONFIG_CGROUP_CPUACCT=y
CONFIG_CGROUP_DEVICE=y
CONFIG_CGROUP_FREEZER=y
CONFIG_CGROUP_SCHED=y
CONFIG_CGROUP_PIDS=y
CONFIG_MEMCG=y

# Mandatory — namespaces
CONFIG_NAMESPACES=y
CONFIG_NET_NS=y
CONFIG_PID_NS=y
CONFIG_USER_NS=y
CONFIG_UTS_NS=y
CONFIG_IPC_NS=y

# Mandatory — filesystem
CONFIG_OVERLAY_FS=y|m
CONFIG_SQUASHFS=y

# Mandatory — networking
CONFIG_BRIDGE=y|m
CONFIG_NETFILTER=y
CONFIG_NF_NAT=y|m
CONFIG_IP_NF_IPTABLES=y|m
CONFIG_IP_NF_NAT=y|m
CONFIG_IP_NF_FILTER=y|m
CONFIG_VETH=y|m
CONFIG_VXLAN=y|m

# Recommended
CONFIG_BPF_SYSCALL=y
CONFIG_SECCOMP=y
CONFIG_CRYPTO_SHA256=y|m

If the stock Tiny Core kernel is missing any mandatory config, the project must either:

1. Load the feature as a kernel module (if =m)
2. Custom-compile the kernel with the missing options enabled

---

Boot Parameters

KubeSolo OS uses kernel command line parameters for runtime configuration:

Parameter	Default	Description
kubesolo.data=<device>	(required)	Block device for persistent data partition
kubesolo.debug	(off)	Enable verbose init logging
kubesolo.shell	(off)	Drop to emergency shell instead of booting
kubesolo.nopersist	(off)	Run fully in RAM (no persistent mount)
kubesolo.cloudinit=<path>	/mnt/data/etc-kubesolo/cloud-init.yaml	Cloud-init config file
kubesolo.flags=<flags>	(none)	Extra flags passed to KubeSolo binary
quiet	(off)	Suppress kernel boot messages

---

Phase 1 Scope (Current)

The immediate goal is a Proof of Concept that boots to a functional K8s node:

Deliverables

1. build/scripts/fetch-components.sh — downloads Tiny Core ISO + KubeSolo
2. build/scripts/extract-core.sh — extracts Tiny Core rootfs from ISO
3. build/config/kernel-audit.sh — checks kernel config against requirements
4. init/init.sh + init/lib/*.sh — modular init system
5. build/scripts/inject-kubesolo.sh — adds KubeSolo + deps to rootfs
6. build/scripts/pack-initramfs.sh — repacks into kubesolo-os.gz
7. build/scripts/create-iso.sh — creates bootable ISO (syslinux, simpler than GRUB for PoC)
8. test/qemu/run-vm.sh — launches QEMU with the ISO
9. test/qemu/test-boot.sh — automated boot + K8s readiness check
10. Makefile — ties it all together

NOT in Phase 1

• A/B partitions (Phase 3)
• GRUB bootloader (Phase 3 — use syslinux/isolinux for PoC ISO)
• Update agent (Phase 3)
• Cloud-init parser (Phase 2)
• OCI image distribution (Phase 5)
• ARM64 support (Phase 5)

Success Criteria

• make iso produces a bootable ISO < 100 MB
• ISO boots in QEMU in < 30 seconds to login/shell
• KubeSolo starts and kubectl get nodes shows node Ready within 120 seconds
• A test pod (nginx) can be deployed and reaches Running state
• System root is read-only (writes to /usr, /bin, /sbin fail)
• Reboot preserves K8s state (pods, services survive restart)

---

Common Tasks

First-time setup

# Clone and enter repo
git clone <repo-url> kubesolo-os && cd kubesolo-os

# Fetch all components (downloads to build/cache/)
make fetch

# Full build → ISO
make iso

# Boot in QEMU for testing
make test-boot

Rebuild after init script changes

# Fast path: just repack initramfs and rebuild ISO
make initramfs iso

Run all tests

make test-all

Debug a failing boot

# Boot with serial console attached
./hack/dev-vm.sh

# Or boot to emergency shell
./hack/dev-vm.sh --shell

Add SSH for debugging (dev only)

./hack/inject-ssh.sh output/kubesolo-os.gz
# Rebuilds initramfs with dropbear SSH + your ~/.ssh/id_rsa.pub

---

Important Constraints

1. No systemd — Tiny Core uses BusyBox init; our custom init is pure POSIX sh
2. No package manager at runtime — everything needed must be in the initramfs
3. BusyBox userland — commands may have limited flags vs GNU coreutils (test with BusyBox)
4. Static binaries preferred — Go binaries must be CGO_ENABLED=0; avoid glibc runtime deps
5. KubeSolo bundles containerd — do NOT install a separate containerd; use what KubeSolo ships
6. Memory budget — target 512 MB minimum RAM; OS overhead should be < 100 MB
7. Disk image must be self-contained — no network access required during boot (air-gap safe)
8. Kernel modules — only modules present in the initramfs are available; no runtime module install

---

External References

• KubeSolo GitHub
• Tiny Core Linux
• Tiny Core Remastering Wiki
• Tiny Core Into the Core
• Talos Linux — reference for immutable K8s OS patterns
• Kairos — reference for OCI-based immutable OS distribution
• Kubernetes Node Requirements
• cgroup v2 Documentation