feat: initial Phase 1 PoC scaffolding for KubeSolo OS
Complete Phase 1 implementation of KubeSolo OS — an immutable, bootable Linux distribution built on Tiny Core Linux for running KubeSolo single-node Kubernetes. Build system: - Makefile with fetch, rootfs, initramfs, iso, disk-image targets - Dockerfile.builder for reproducible builds - Scripts to download Tiny Core, extract rootfs, inject KubeSolo, pack initramfs, and create bootable ISO/disk images Init system (10 POSIX sh stages): - Early mount (proc/sys/dev/cgroup2), cmdline parsing, persistent mount with bind-mounts, kernel module loading, sysctl, DHCP networking, hostname, clock sync, containerd prep, KubeSolo exec Shared libraries: - functions.sh (device wait, IP lookup, config helpers) - network.sh (static IP, config persistence, interface detection) - health.sh (containerd, API server, node readiness checks) - Emergency shell for boot failure debugging Testing: - QEMU boot test with serial log marker detection - K8s readiness test with kubectl verification - Persistence test (reboot + verify state survives) - Workload deployment test (nginx pod) - Local storage test (PVC + local-path provisioner) - Network policy test - Reusable run-vm.sh launcher Developer tools: - dev-vm.sh (interactive QEMU with port forwarding) - rebuild-initramfs.sh (fast iteration) - inject-ssh.sh (dropbear SSH for debugging) - extract-kernel-config.sh + kernel-audit.sh Documentation: - Full design document with architecture research - Boot flow documentation covering all 10 init stages - Cloud-init examples (DHCP, static IP, Portainer Edge, air-gapped) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
25
.gitignore
vendored
Normal file
25
.gitignore
vendored
Normal file
@@ -0,0 +1,25 @@
|
|||||||
|
# Build artifacts
|
||||||
|
output/
|
||||||
|
build/cache/
|
||||||
|
build/rootfs-work/
|
||||||
|
|
||||||
|
# Generated files
|
||||||
|
*.iso
|
||||||
|
*.img
|
||||||
|
*.gz
|
||||||
|
*.squashfs
|
||||||
|
|
||||||
|
# Editor
|
||||||
|
.vscode/
|
||||||
|
.idea/
|
||||||
|
*.swp
|
||||||
|
*.swo
|
||||||
|
*~
|
||||||
|
|
||||||
|
# OS
|
||||||
|
.DS_Store
|
||||||
|
Thumbs.db
|
||||||
|
|
||||||
|
# Go
|
||||||
|
update/update-agent
|
||||||
|
cloud-init/cloud-init-parser
|
||||||
428
CLAUDE.md
Normal file
428
CLAUDE.md
Normal file
@@ -0,0 +1,428 @@
|
|||||||
|
# CLAUDE.md — KubeSolo OS
|
||||||
|
|
||||||
|
## Project Overview
|
||||||
|
|
||||||
|
**KubeSolo OS** is an immutable, bootable Linux distribution purpose-built to run [KubeSolo](https://github.com/portainer/kubesolo) — Portainer's ultra-lightweight single-node Kubernetes distribution. It combines Tiny Core Linux's minimal footprint (~11 MB) with KubeSolo's single-binary K8s packaging to create an appliance-like Kubernetes node with atomic A/B updates.
|
||||||
|
|
||||||
|
**Design document:** See `docs/design/kubesolo-os-design.md` for full architecture research, competitive analysis, and technical specifications.
|
||||||
|
|
||||||
|
**Target:** Edge/IoT devices, single-node K8s appliances, air-gapped deployments, resource-constrained hardware.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Repository Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
kubesolo-os/
|
||||||
|
├── CLAUDE.md # This file
|
||||||
|
├── README.md # Project README
|
||||||
|
├── Makefile # Top-level build orchestration
|
||||||
|
├── VERSION # Semver version (e.g., 0.1.0)
|
||||||
|
│
|
||||||
|
├── docs/
|
||||||
|
│ ├── design/
|
||||||
|
│ │ └── kubesolo-os-design.md # Full architecture document
|
||||||
|
│ ├── boot-flow.md # Boot sequence documentation
|
||||||
|
│ ├── update-flow.md # Atomic update documentation
|
||||||
|
│ └── cloud-init.md # Configuration reference
|
||||||
|
│
|
||||||
|
├── build/ # Build system
|
||||||
|
│ ├── Dockerfile.builder # Containerized build environment
|
||||||
|
│ ├── build.sh # Main build script (orchestrator)
|
||||||
|
│ ├── config/
|
||||||
|
│ │ ├── kernel-audit.sh # Verify kernel config requirements
|
||||||
|
│ │ ├── kernel-config.fragment # Custom kernel config overrides (if needed)
|
||||||
|
│ │ └── modules.list # Required kernel modules list
|
||||||
|
│ ├── rootfs/ # Files injected into initramfs
|
||||||
|
│ │ ├── sbin/
|
||||||
|
│ │ │ └── init # Custom init script
|
||||||
|
│ │ ├── etc/
|
||||||
|
│ │ │ ├── os-release # OS identification
|
||||||
|
│ │ │ ├── sysctl.d/
|
||||||
|
│ │ │ │ └── k8s.conf # Kernel parameters for K8s
|
||||||
|
│ │ │ └── kubesolo/
|
||||||
|
│ │ │ └── defaults.yaml # Default KubeSolo config
|
||||||
|
│ │ └── usr/
|
||||||
|
│ │ └── lib/
|
||||||
|
│ │ └── kubesolo-os/
|
||||||
|
│ │ ├── functions.sh # Shared shell functions
|
||||||
|
│ │ ├── network.sh # Network configuration helpers
|
||||||
|
│ │ └── health.sh # Health check functions
|
||||||
|
│ ├── grub/
|
||||||
|
│ │ ├── grub.cfg # A/B boot GRUB config
|
||||||
|
│ │ └── grub-env-defaults # Default GRUB environment vars
|
||||||
|
│ └── scripts/
|
||||||
|
│ ├── fetch-components.sh # Download Tiny Core, KubeSolo, deps
|
||||||
|
│ ├── extract-core.sh # Extract and prepare Tiny Core rootfs
|
||||||
|
│ ├── inject-kubesolo.sh # Add KubeSolo + deps to rootfs
|
||||||
|
│ ├── pack-initramfs.sh # Repack initramfs (core.gz → kubesolo-os.gz)
|
||||||
|
│ ├── create-iso.sh # Build bootable ISO
|
||||||
|
│ ├── create-disk-image.sh # Build raw disk image with A/B partitions
|
||||||
|
│ └── create-oci-image.sh # Build OCI container image (future)
|
||||||
|
│
|
||||||
|
├── init/ # Init system source
|
||||||
|
│ ├── init.sh # Main init script (becomes /sbin/init)
|
||||||
|
│ ├── lib/
|
||||||
|
│ │ ├── 00-early-mount.sh # Mount proc, sys, dev, tmpfs
|
||||||
|
│ │ ├── 10-parse-cmdline.sh # Parse kernel boot parameters
|
||||||
|
│ │ ├── 20-persistent-mount.sh # Mount + bind persistent data partition
|
||||||
|
│ │ ├── 30-kernel-modules.sh # Load required kernel modules
|
||||||
|
│ │ ├── 40-sysctl.sh # Apply sysctl settings
|
||||||
|
│ │ ├── 50-network.sh # Network configuration (cloud-init/DHCP)
|
||||||
|
│ │ ├── 60-hostname.sh # Set hostname
|
||||||
|
│ │ ├── 70-clock.sh # NTP / system clock
|
||||||
|
│ │ ├── 80-containerd.sh # Start containerd
|
||||||
|
│ │ └── 90-kubesolo.sh # Start KubeSolo (final stage)
|
||||||
|
│ └── emergency-shell.sh # Drop to shell on boot failure
|
||||||
|
│
|
||||||
|
├── update/ # Atomic update agent
|
||||||
|
│ ├── go.mod
|
||||||
|
│ ├── go.sum
|
||||||
|
│ ├── main.go # Update agent entrypoint
|
||||||
|
│ ├── cmd/
|
||||||
|
│ │ ├── check.go # Check for available updates
|
||||||
|
│ │ ├── apply.go # Download + write to passive partition
|
||||||
|
│ │ ├── activate.go # Update GRUB, set boot counter
|
||||||
|
│ │ ├── rollback.go # Force rollback to previous partition
|
||||||
|
│ │ └── healthcheck.go # Post-boot health verification
|
||||||
|
│ ├── pkg/
|
||||||
|
│ │ ├── grubenv/ # GRUB environment manipulation
|
||||||
|
│ │ │ └── grubenv.go
|
||||||
|
│ │ ├── partition/ # Partition detection and management
|
||||||
|
│ │ │ └── partition.go
|
||||||
|
│ │ ├── image/ # Image download, verify, write
|
||||||
|
│ │ │ └── image.go
|
||||||
|
│ │ └── health/ # K8s + containerd health checks
|
||||||
|
│ │ └── health.go
|
||||||
|
│ └── deploy/
|
||||||
|
│ └── update-cronjob.yaml # K8s CronJob manifest for auto-updates
|
||||||
|
│
|
||||||
|
├── cloud-init/ # Cloud-init implementation
|
||||||
|
│ ├── cloud-init.go # Lightweight cloud-init parser
|
||||||
|
│ ├── network.go # Network config from cloud-init
|
||||||
|
│ ├── kubesolo.go # KubeSolo config from cloud-init
|
||||||
|
│ └── examples/
|
||||||
|
│ ├── dhcp.yaml # DHCP example
|
||||||
|
│ ├── static-ip.yaml # Static IP example
|
||||||
|
│ ├── portainer-edge.yaml # Portainer Edge integration
|
||||||
|
│ └── airgapped.yaml # Air-gapped deployment
|
||||||
|
│
|
||||||
|
├── test/ # Testing
|
||||||
|
│ ├── Makefile # Test orchestration
|
||||||
|
│ ├── qemu/
|
||||||
|
│ │ ├── run-vm.sh # Launch QEMU VM with built image
|
||||||
|
│ │ ├── test-boot.sh # Automated boot test
|
||||||
|
│ │ ├── test-persistence.sh # Reboot + verify state survives
|
||||||
|
│ │ ├── test-update.sh # A/B update cycle test
|
||||||
|
│ │ └── test-rollback.sh # Forced rollback test
|
||||||
|
│ ├── integration/
|
||||||
|
│ │ ├── test-k8s-ready.sh # Verify K8s node reaches Ready
|
||||||
|
│ │ ├── test-deploy-workload.sh # Deploy nginx, verify pod running
|
||||||
|
│ │ ├── test-local-storage.sh # PVC with local-path provisioner
|
||||||
|
│ │ └── test-network-policy.sh # Basic network policy enforcement
|
||||||
|
│ └── kernel/
|
||||||
|
│ └── check-config.sh # Validate kernel config requirements
|
||||||
|
│
|
||||||
|
└── hack/ # Developer utilities
|
||||||
|
├── dev-vm.sh # Quick-launch dev VM (QEMU)
|
||||||
|
├── rebuild-initramfs.sh # Fast rebuild for dev iteration
|
||||||
|
├── inject-ssh.sh # Add SSH extension for debugging
|
||||||
|
└── extract-kernel-config.sh # Pull /proc/config.gz from running TC
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Architecture Summary
|
||||||
|
|
||||||
|
### Core Concept
|
||||||
|
|
||||||
|
KubeSolo OS is a **remastered Tiny Core Linux** where the initramfs (`core.gz`) is rebuilt to include KubeSolo and all its dependencies. The result is a single bootable image that:
|
||||||
|
|
||||||
|
1. Boots kernel + initramfs into RAM (read-only SquashFS root)
|
||||||
|
2. Mounts a persistent ext4 partition for K8s state
|
||||||
|
3. Bind-mounts writable paths (`/var/lib/kubesolo`, `/var/lib/containerd`, etc.)
|
||||||
|
4. Loads kernel modules (br_netfilter, overlay, veth, etc.)
|
||||||
|
5. Configures networking (cloud-init → persistent config → DHCP fallback)
|
||||||
|
6. Starts containerd, then KubeSolo
|
||||||
|
7. Kubernetes API becomes available; node reaches Ready
|
||||||
|
|
||||||
|
### Partition Layout
|
||||||
|
|
||||||
|
```
|
||||||
|
Disk (minimum 8 GB):
|
||||||
|
Part 1: EFI/Boot (256 MB, FAT32) — GRUB + A/B boot logic
|
||||||
|
Part 2: System A (512 MB, ext4) — vmlinuz + kubesolo-os.gz (active)
|
||||||
|
Part 3: System B (512 MB, ext4) — vmlinuz + kubesolo-os.gz (passive)
|
||||||
|
Part 4: Data (remaining, ext4) — persistent K8s state
|
||||||
|
```
|
||||||
|
|
||||||
|
### Persistent Paths (survive updates)
|
||||||
|
|
||||||
|
| Mount Point | Content | On Data Partition |
|
||||||
|
|---|---|---|
|
||||||
|
| `/var/lib/kubesolo` | K8s state, certs, SQLite DB | `/mnt/data/kubesolo` |
|
||||||
|
| `/var/lib/containerd` | Container images + layers | `/mnt/data/containerd` |
|
||||||
|
| `/etc/kubesolo` | Node configuration | `/mnt/data/etc-kubesolo` |
|
||||||
|
| `/var/log` | System + K8s logs | `/mnt/data/log` |
|
||||||
|
| `/usr/local` | User data, extra binaries | `/mnt/data/usr-local` |
|
||||||
|
|
||||||
|
### Atomic Updates
|
||||||
|
|
||||||
|
A/B partition scheme with GRUB fallback counter:
|
||||||
|
- Update writes new image to passive partition
|
||||||
|
- GRUB boots new partition with `boot_counter=3`
|
||||||
|
- Health check sets `boot_success=1` on success
|
||||||
|
- On 3 consecutive failures (counter reaches 0), GRUB auto-rolls back
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Technology Stack
|
||||||
|
|
||||||
|
| Component | Technology | Rationale |
|
||||||
|
|---|---|---|
|
||||||
|
| Base OS | Tiny Core Linux 17.0 (Micro Core, x86_64) | 11 MB, RAM-resident, SquashFS root |
|
||||||
|
| Kernel | Tiny Core stock (6.x) or custom build | Must have cgroup v2, namespaces, netfilter |
|
||||||
|
| Kubernetes | KubeSolo (single binary) | Single-node K8s, SQLite backend, bundled runtime |
|
||||||
|
| Container runtime | containerd + runc (bundled in KubeSolo) | Industry standard, KubeSolo dependency |
|
||||||
|
| Init | Custom shell script (POSIX sh) | Minimal, no systemd dependency |
|
||||||
|
| Bootloader | GRUB 2 (EFI + BIOS) | A/B partition support, env variables |
|
||||||
|
| Update agent | Go binary | Single static binary, K8s client-go |
|
||||||
|
| Cloud-init parser | Go binary or shell script | First-boot configuration |
|
||||||
|
| Build system | Bash + Make + Docker (builder container) | Reproducible builds |
|
||||||
|
| Testing | QEMU/KVM + shell scripts | Automated boot + integration tests |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Development Guidelines
|
||||||
|
|
||||||
|
### Shell Scripts (init system, build scripts)
|
||||||
|
|
||||||
|
- **POSIX sh** — no bashisms in init scripts (BusyBox ash compatibility)
|
||||||
|
- **Shellcheck** all scripts: `shellcheck -s sh <script>`
|
||||||
|
- Use `set -euo pipefail` in build scripts (bash)
|
||||||
|
- Use `set -e` in init scripts (POSIX sh, no pipefail)
|
||||||
|
- Quote all variable expansions: `"$var"` not `$var`
|
||||||
|
- Use `$(command)` not backticks
|
||||||
|
- Functions for reusable logic; source shared libraries from `/usr/lib/kubesolo-os/`
|
||||||
|
- Log to stderr with prefix: `echo "[kubesolo-init] message" >&2`
|
||||||
|
- Init stages are numbered (`00-`, `10-`, ...) and sourced in order
|
||||||
|
|
||||||
|
### Go Code (update agent, cloud-init parser)
|
||||||
|
|
||||||
|
- **Go 1.22+**
|
||||||
|
- Build static binaries: `CGO_ENABLED=0 go build -ldflags='-s -w' -o binary`
|
||||||
|
- Use `client-go` for Kubernetes health checks
|
||||||
|
- Minimal dependencies — this runs on a tiny system
|
||||||
|
- Error handling: wrap errors with context (`fmt.Errorf("failed to X: %w", err)`)
|
||||||
|
- Use structured logging (`log/slog`)
|
||||||
|
- Unit tests required for `pkg/` packages
|
||||||
|
- No network calls in tests (mock interfaces)
|
||||||
|
|
||||||
|
### Build System
|
||||||
|
|
||||||
|
- **Makefile** targets:
|
||||||
|
- `make fetch` — download Tiny Core ISO, KubeSolo binary, dependencies
|
||||||
|
- `make rootfs` — extract core.gz, inject KubeSolo, prepare rootfs
|
||||||
|
- `make initramfs` — repack rootfs into kubesolo-os.gz
|
||||||
|
- `make iso` — create bootable ISO
|
||||||
|
- `make disk-image` — create raw disk image with A/B partitions
|
||||||
|
- `make test-boot` — launch QEMU, verify boot + K8s ready
|
||||||
|
- `make test-update` — full A/B update cycle test
|
||||||
|
- `make test-all` — run all tests
|
||||||
|
- `make clean` — remove build artifacts
|
||||||
|
- `make docker-build` — run entire build inside Docker (reproducible)
|
||||||
|
- **Reproducible builds** — pin all component versions in `build/config/versions.env`:
|
||||||
|
```bash
|
||||||
|
TINYCORE_VERSION=17.0
|
||||||
|
TINYCORE_ARCH=x86_64
|
||||||
|
KUBESOLO_VERSION=latest # pin to specific release when available
|
||||||
|
CONTAINERD_VERSION=1.7.x # if fetching separately
|
||||||
|
GRUB_VERSION=2.12
|
||||||
|
```
|
||||||
|
- **Builder container** — `build/Dockerfile.builder` with all build tools (cpio, gzip, grub-mkimage, squashfs-tools, qemu for testing)
|
||||||
|
- All downloads go to `build/cache/` (gitignored, reused across builds)
|
||||||
|
- Build output goes to `output/` (gitignored)
|
||||||
|
|
||||||
|
### Testing
|
||||||
|
|
||||||
|
- **Every change must pass `make test-boot`** — the image boots and K8s reaches Ready
|
||||||
|
- QEMU tests use `-nographic` and serial console for CI compatibility
|
||||||
|
- Test timeout: 120 seconds for boot, 300 seconds for K8s ready
|
||||||
|
- Integration tests use `kubectl` against the VM's forwarded API port
|
||||||
|
- Kernel config audit runs as a build-time check, not runtime
|
||||||
|
|
||||||
|
### Git Workflow
|
||||||
|
|
||||||
|
- Branch naming: `feat/`, `fix/`, `docs/`, `test/`, `build/`
|
||||||
|
- Commit messages: conventional commits (`feat:`, `fix:`, `build:`, `test:`, `docs:`)
|
||||||
|
- Tag releases: `v0.1.0`, `v0.2.0`, etc.
|
||||||
|
- `.gitignore`: `build/cache/`, `output/`, `*.iso`, `*.img`, `*.gz` (build artifacts)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Key Kernel Requirements
|
||||||
|
|
||||||
|
The Tiny Core kernel **MUST** have these configs. Run `build/config/kernel-audit.sh` against the kernel to verify:
|
||||||
|
|
||||||
|
```
|
||||||
|
# Mandatory — cgroup v2
|
||||||
|
CONFIG_CGROUPS=y
|
||||||
|
CONFIG_CGROUP_CPUACCT=y
|
||||||
|
CONFIG_CGROUP_DEVICE=y
|
||||||
|
CONFIG_CGROUP_FREEZER=y
|
||||||
|
CONFIG_CGROUP_SCHED=y
|
||||||
|
CONFIG_CGROUP_PIDS=y
|
||||||
|
CONFIG_MEMCG=y
|
||||||
|
|
||||||
|
# Mandatory — namespaces
|
||||||
|
CONFIG_NAMESPACES=y
|
||||||
|
CONFIG_NET_NS=y
|
||||||
|
CONFIG_PID_NS=y
|
||||||
|
CONFIG_USER_NS=y
|
||||||
|
CONFIG_UTS_NS=y
|
||||||
|
CONFIG_IPC_NS=y
|
||||||
|
|
||||||
|
# Mandatory — filesystem
|
||||||
|
CONFIG_OVERLAY_FS=y|m
|
||||||
|
CONFIG_SQUASHFS=y
|
||||||
|
|
||||||
|
# Mandatory — networking
|
||||||
|
CONFIG_BRIDGE=y|m
|
||||||
|
CONFIG_NETFILTER=y
|
||||||
|
CONFIG_NF_NAT=y|m
|
||||||
|
CONFIG_IP_NF_IPTABLES=y|m
|
||||||
|
CONFIG_IP_NF_NAT=y|m
|
||||||
|
CONFIG_IP_NF_FILTER=y|m
|
||||||
|
CONFIG_VETH=y|m
|
||||||
|
CONFIG_VXLAN=y|m
|
||||||
|
|
||||||
|
# Recommended
|
||||||
|
CONFIG_BPF_SYSCALL=y
|
||||||
|
CONFIG_SECCOMP=y
|
||||||
|
CONFIG_CRYPTO_SHA256=y|m
|
||||||
|
```
|
||||||
|
|
||||||
|
If the stock Tiny Core kernel is missing any mandatory config, the project must either:
|
||||||
|
1. Load the feature as a kernel module (if `=m`)
|
||||||
|
2. Custom-compile the kernel with the missing options enabled
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Boot Parameters
|
||||||
|
|
||||||
|
KubeSolo OS uses kernel command line parameters for runtime configuration:
|
||||||
|
|
||||||
|
| Parameter | Default | Description |
|
||||||
|
|---|---|---|
|
||||||
|
| `kubesolo.data=<device>` | (required) | Block device for persistent data partition |
|
||||||
|
| `kubesolo.debug` | (off) | Enable verbose init logging |
|
||||||
|
| `kubesolo.shell` | (off) | Drop to emergency shell instead of booting |
|
||||||
|
| `kubesolo.nopersist` | (off) | Run fully in RAM (no persistent mount) |
|
||||||
|
| `kubesolo.cloudinit=<path>` | `/mnt/data/etc-kubesolo/cloud-init.yaml` | Cloud-init config file |
|
||||||
|
| `kubesolo.flags=<flags>` | (none) | Extra flags passed to KubeSolo binary |
|
||||||
|
| `quiet` | (off) | Suppress kernel boot messages |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 1 Scope (Current)
|
||||||
|
|
||||||
|
The immediate goal is a **Proof of Concept** that boots to a functional K8s node:
|
||||||
|
|
||||||
|
### Deliverables
|
||||||
|
1. `build/scripts/fetch-components.sh` — downloads Tiny Core ISO + KubeSolo
|
||||||
|
2. `build/scripts/extract-core.sh` — extracts Tiny Core rootfs from ISO
|
||||||
|
3. `build/config/kernel-audit.sh` — checks kernel config against requirements
|
||||||
|
4. `init/init.sh` + `init/lib/*.sh` — modular init system
|
||||||
|
5. `build/scripts/inject-kubesolo.sh` — adds KubeSolo + deps to rootfs
|
||||||
|
6. `build/scripts/pack-initramfs.sh` — repacks into kubesolo-os.gz
|
||||||
|
7. `build/scripts/create-iso.sh` — creates bootable ISO (syslinux, simpler than GRUB for PoC)
|
||||||
|
8. `test/qemu/run-vm.sh` — launches QEMU with the ISO
|
||||||
|
9. `test/qemu/test-boot.sh` — automated boot + K8s readiness check
|
||||||
|
10. `Makefile` — ties it all together
|
||||||
|
|
||||||
|
### NOT in Phase 1
|
||||||
|
- A/B partitions (Phase 3)
|
||||||
|
- GRUB bootloader (Phase 3 — use syslinux/isolinux for PoC ISO)
|
||||||
|
- Update agent (Phase 3)
|
||||||
|
- Cloud-init parser (Phase 2)
|
||||||
|
- OCI image distribution (Phase 5)
|
||||||
|
- ARM64 support (Phase 5)
|
||||||
|
|
||||||
|
### Success Criteria
|
||||||
|
- `make iso` produces a bootable ISO < 100 MB
|
||||||
|
- ISO boots in QEMU in < 30 seconds to login/shell
|
||||||
|
- KubeSolo starts and `kubectl get nodes` shows node Ready within 120 seconds
|
||||||
|
- A test pod (`nginx`) can be deployed and reaches Running state
|
||||||
|
- System root is read-only (writes to `/usr`, `/bin`, `/sbin` fail)
|
||||||
|
- Reboot preserves K8s state (pods, services survive restart)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Common Tasks
|
||||||
|
|
||||||
|
### First-time setup
|
||||||
|
```bash
|
||||||
|
# Clone and enter repo
|
||||||
|
git clone <repo-url> kubesolo-os && cd kubesolo-os
|
||||||
|
|
||||||
|
# Fetch all components (downloads to build/cache/)
|
||||||
|
make fetch
|
||||||
|
|
||||||
|
# Full build → ISO
|
||||||
|
make iso
|
||||||
|
|
||||||
|
# Boot in QEMU for testing
|
||||||
|
make test-boot
|
||||||
|
```
|
||||||
|
|
||||||
|
### Rebuild after init script changes
|
||||||
|
```bash
|
||||||
|
# Fast path: just repack initramfs and rebuild ISO
|
||||||
|
make initramfs iso
|
||||||
|
```
|
||||||
|
|
||||||
|
### Run all tests
|
||||||
|
```bash
|
||||||
|
make test-all
|
||||||
|
```
|
||||||
|
|
||||||
|
### Debug a failing boot
|
||||||
|
```bash
|
||||||
|
# Boot with serial console attached
|
||||||
|
./hack/dev-vm.sh
|
||||||
|
|
||||||
|
# Or boot to emergency shell
|
||||||
|
./hack/dev-vm.sh --shell
|
||||||
|
```
|
||||||
|
|
||||||
|
### Add SSH for debugging (dev only)
|
||||||
|
```bash
|
||||||
|
./hack/inject-ssh.sh output/kubesolo-os.gz
|
||||||
|
# Rebuilds initramfs with dropbear SSH + your ~/.ssh/id_rsa.pub
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Important Constraints
|
||||||
|
|
||||||
|
1. **No systemd** — Tiny Core uses BusyBox init; our custom init is pure POSIX sh
|
||||||
|
2. **No package manager at runtime** — everything needed must be in the initramfs
|
||||||
|
3. **BusyBox userland** — commands may have limited flags vs GNU coreutils (test with BusyBox)
|
||||||
|
4. **Static binaries preferred** — Go binaries must be `CGO_ENABLED=0`; avoid glibc runtime deps
|
||||||
|
5. **KubeSolo bundles containerd** — do NOT install a separate containerd; use what KubeSolo ships
|
||||||
|
6. **Memory budget** — target 512 MB minimum RAM; OS overhead should be < 100 MB
|
||||||
|
7. **Disk image must be self-contained** — no network access required during boot (air-gap safe)
|
||||||
|
8. **Kernel modules** — only modules present in the initramfs are available; no runtime module install
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## External References
|
||||||
|
|
||||||
|
- [KubeSolo GitHub](https://github.com/portainer/kubesolo)
|
||||||
|
- [Tiny Core Linux](http://www.tinycorelinux.net)
|
||||||
|
- [Tiny Core Remastering Wiki](http://wiki.tinycorelinux.net/doku.php?id=wiki:remastering)
|
||||||
|
- [Tiny Core Into the Core](http://wiki.tinycorelinux.net/doku.php?id=wiki:into_the_core)
|
||||||
|
- [Talos Linux](https://www.talos.dev) — reference for immutable K8s OS patterns
|
||||||
|
- [Kairos](https://kairos.io) — reference for OCI-based immutable OS distribution
|
||||||
|
- [Kubernetes Node Requirements](https://kubernetes.io/docs/setup/production-environment/container-runtimes/)
|
||||||
|
- [cgroup v2 Documentation](https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html)
|
||||||
176
Makefile
Normal file
176
Makefile
Normal file
@@ -0,0 +1,176 @@
|
|||||||
|
.PHONY: all fetch rootfs initramfs iso disk-image \
|
||||||
|
test-boot test-k8s test-persistence test-deploy test-storage test-all \
|
||||||
|
dev-vm dev-vm-shell quick docker-build shellcheck \
|
||||||
|
kernel-audit clean distclean help
|
||||||
|
|
||||||
|
SHELL := /bin/bash
|
||||||
|
VERSION := $(shell cat VERSION)
|
||||||
|
BUILD_DIR := build
|
||||||
|
CACHE_DIR := $(BUILD_DIR)/cache
|
||||||
|
OUTPUT_DIR := output
|
||||||
|
ROOTFS_DIR := $(BUILD_DIR)/rootfs-work
|
||||||
|
|
||||||
|
# Load component versions
|
||||||
|
include $(BUILD_DIR)/config/versions.env
|
||||||
|
|
||||||
|
# Default target
|
||||||
|
all: iso
|
||||||
|
|
||||||
|
# =============================================================================
|
||||||
|
# Download external components
|
||||||
|
# =============================================================================
|
||||||
|
fetch:
|
||||||
|
@echo "==> Fetching components..."
|
||||||
|
@mkdir -p $(CACHE_DIR)
|
||||||
|
$(BUILD_DIR)/scripts/fetch-components.sh
|
||||||
|
|
||||||
|
# =============================================================================
|
||||||
|
# Build stages
|
||||||
|
# =============================================================================
|
||||||
|
rootfs: fetch
|
||||||
|
@echo "==> Preparing rootfs..."
|
||||||
|
$(BUILD_DIR)/scripts/extract-core.sh
|
||||||
|
$(BUILD_DIR)/scripts/inject-kubesolo.sh
|
||||||
|
|
||||||
|
initramfs: rootfs
|
||||||
|
@echo "==> Packing initramfs..."
|
||||||
|
$(BUILD_DIR)/scripts/pack-initramfs.sh
|
||||||
|
|
||||||
|
iso: initramfs
|
||||||
|
@echo "==> Creating bootable ISO..."
|
||||||
|
$(BUILD_DIR)/scripts/create-iso.sh
|
||||||
|
@echo "==> Built: $(OUTPUT_DIR)/$(OS_NAME)-$(VERSION).iso"
|
||||||
|
|
||||||
|
disk-image: initramfs
|
||||||
|
@echo "==> Creating disk image..."
|
||||||
|
$(BUILD_DIR)/scripts/create-disk-image.sh
|
||||||
|
@echo "==> Built: $(OUTPUT_DIR)/$(OS_NAME)-$(VERSION).img"
|
||||||
|
|
||||||
|
# =============================================================================
|
||||||
|
# Kernel validation
|
||||||
|
# =============================================================================
|
||||||
|
kernel-audit:
|
||||||
|
@echo "==> Auditing kernel configuration..."
|
||||||
|
$(BUILD_DIR)/config/kernel-audit.sh
|
||||||
|
|
||||||
|
# =============================================================================
|
||||||
|
# Testing
|
||||||
|
# =============================================================================
|
||||||
|
test-boot: iso
|
||||||
|
@echo "==> Testing boot in QEMU..."
|
||||||
|
test/qemu/test-boot.sh $(OUTPUT_DIR)/$(OS_NAME)-$(VERSION).iso
|
||||||
|
|
||||||
|
test-k8s: iso
|
||||||
|
@echo "==> Testing K8s readiness..."
|
||||||
|
test/integration/test-k8s-ready.sh $(OUTPUT_DIR)/$(OS_NAME)-$(VERSION).iso
|
||||||
|
|
||||||
|
test-persistence: disk-image
|
||||||
|
@echo "==> Testing persistence across reboot..."
|
||||||
|
test/qemu/test-persistence.sh $(OUTPUT_DIR)/$(OS_NAME)-$(VERSION).img
|
||||||
|
|
||||||
|
test-deploy: iso
|
||||||
|
@echo "==> Testing workload deployment..."
|
||||||
|
test/integration/test-deploy-workload.sh $(OUTPUT_DIR)/$(OS_NAME)-$(VERSION).iso
|
||||||
|
|
||||||
|
test-storage: iso
|
||||||
|
@echo "==> Testing local storage provisioning..."
|
||||||
|
test/integration/test-local-storage.sh $(OUTPUT_DIR)/$(OS_NAME)-$(VERSION).iso
|
||||||
|
|
||||||
|
test-all: test-boot test-k8s test-persistence
|
||||||
|
|
||||||
|
# Full integration test suite (requires more time)
|
||||||
|
test-integration: test-k8s test-deploy test-storage
|
||||||
|
|
||||||
|
# =============================================================================
|
||||||
|
# Code quality
|
||||||
|
# =============================================================================
|
||||||
|
shellcheck:
|
||||||
|
@echo "==> Running shellcheck on init scripts..."
|
||||||
|
shellcheck -s sh init/init.sh init/lib/*.sh init/emergency-shell.sh
|
||||||
|
@echo "==> Running shellcheck on build scripts..."
|
||||||
|
shellcheck -s bash build/scripts/*.sh build/config/kernel-audit.sh
|
||||||
|
@echo "==> Running shellcheck on test scripts..."
|
||||||
|
shellcheck -s bash test/qemu/*.sh test/integration/*.sh test/kernel/*.sh
|
||||||
|
@echo "==> Running shellcheck on hack scripts..."
|
||||||
|
shellcheck -s bash hack/*.sh
|
||||||
|
@echo "==> All shellcheck checks passed"
|
||||||
|
|
||||||
|
# =============================================================================
|
||||||
|
# Development helpers
|
||||||
|
# =============================================================================
|
||||||
|
dev-vm: iso
|
||||||
|
@echo "==> Launching dev VM..."
|
||||||
|
hack/dev-vm.sh $(OUTPUT_DIR)/$(OS_NAME)-$(VERSION).iso
|
||||||
|
|
||||||
|
dev-vm-shell: iso
|
||||||
|
@echo "==> Launching dev VM (emergency shell)..."
|
||||||
|
hack/dev-vm.sh $(OUTPUT_DIR)/$(OS_NAME)-$(VERSION).iso --shell
|
||||||
|
|
||||||
|
dev-vm-debug: iso
|
||||||
|
@echo "==> Launching dev VM (debug mode)..."
|
||||||
|
hack/dev-vm.sh $(OUTPUT_DIR)/$(OS_NAME)-$(VERSION).iso --debug
|
||||||
|
|
||||||
|
# Fast rebuild: only repack initramfs + ISO (skip fetch/extract)
|
||||||
|
quick:
|
||||||
|
@echo "==> Quick rebuild (repack + ISO only)..."
|
||||||
|
$(BUILD_DIR)/scripts/inject-kubesolo.sh
|
||||||
|
$(BUILD_DIR)/scripts/pack-initramfs.sh
|
||||||
|
$(BUILD_DIR)/scripts/create-iso.sh
|
||||||
|
@echo "==> Quick rebuild complete: $(OUTPUT_DIR)/$(OS_NAME)-$(VERSION).iso"
|
||||||
|
|
||||||
|
# =============================================================================
|
||||||
|
# Docker-based reproducible build
|
||||||
|
# =============================================================================
|
||||||
|
docker-build:
|
||||||
|
@echo "==> Building in Docker..."
|
||||||
|
docker build -t kubesolo-os-builder -f $(BUILD_DIR)/Dockerfile.builder .
|
||||||
|
docker run --rm --privileged \
|
||||||
|
-v $(PWD)/$(OUTPUT_DIR):/output \
|
||||||
|
-v $(PWD)/$(CACHE_DIR):/cache \
|
||||||
|
kubesolo-os-builder make iso OUTPUT_DIR=/output CACHE_DIR=/cache
|
||||||
|
|
||||||
|
# =============================================================================
|
||||||
|
# Cleanup
|
||||||
|
# =============================================================================
|
||||||
|
clean:
|
||||||
|
@echo "==> Cleaning build artifacts..."
|
||||||
|
rm -rf $(ROOTFS_DIR) $(OUTPUT_DIR)
|
||||||
|
@echo "==> Clean. (Cache preserved in $(CACHE_DIR); use 'make distclean' to remove)"
|
||||||
|
|
||||||
|
distclean: clean
|
||||||
|
rm -rf $(CACHE_DIR)
|
||||||
|
|
||||||
|
# =============================================================================
|
||||||
|
# Help
|
||||||
|
# =============================================================================
|
||||||
|
help:
|
||||||
|
@echo "KubeSolo OS Build System (v$(VERSION))"
|
||||||
|
@echo ""
|
||||||
|
@echo "Build targets:"
|
||||||
|
@echo " make fetch Download Tiny Core ISO, KubeSolo, dependencies"
|
||||||
|
@echo " make rootfs Extract + prepare rootfs with KubeSolo"
|
||||||
|
@echo " make initramfs Repack rootfs into kubesolo-os.gz"
|
||||||
|
@echo " make iso Create bootable ISO (default target)"
|
||||||
|
@echo " make disk-image Create raw disk image with boot + data partitions"
|
||||||
|
@echo " make quick Fast rebuild (re-inject + repack + ISO only)"
|
||||||
|
@echo " make docker-build Reproducible build inside Docker"
|
||||||
|
@echo ""
|
||||||
|
@echo "Test targets:"
|
||||||
|
@echo " make test-boot Boot ISO in QEMU, verify boot success"
|
||||||
|
@echo " make test-k8s Boot + verify K8s node reaches Ready"
|
||||||
|
@echo " make test-persist Reboot disk image, verify state persists"
|
||||||
|
@echo " make test-deploy Deploy nginx pod, verify Running"
|
||||||
|
@echo " make test-storage Test PVC with local-path provisioner"
|
||||||
|
@echo " make test-all Run core tests (boot + k8s + persistence)"
|
||||||
|
@echo " make test-integ Run full integration suite"
|
||||||
|
@echo ""
|
||||||
|
@echo "Dev targets:"
|
||||||
|
@echo " make dev-vm Launch interactive QEMU VM"
|
||||||
|
@echo " make dev-vm-shell Launch QEMU VM -> emergency shell"
|
||||||
|
@echo " make dev-vm-debug Launch QEMU VM with debug logging"
|
||||||
|
@echo " make kernel-audit Check kernel config against requirements"
|
||||||
|
@echo " make shellcheck Lint all shell scripts"
|
||||||
|
@echo ""
|
||||||
|
@echo "Cleanup:"
|
||||||
|
@echo " make clean Remove build artifacts (preserve cache)"
|
||||||
|
@echo " make distclean Remove everything including cache"
|
||||||
86
README.md
Normal file
86
README.md
Normal file
@@ -0,0 +1,86 @@
|
|||||||
|
# KubeSolo OS
|
||||||
|
|
||||||
|
An immutable, bootable Linux distribution purpose-built for [KubeSolo](https://github.com/portainer/kubesolo) — Portainer's ultra-lightweight single-node Kubernetes.
|
||||||
|
|
||||||
|
> **Status:** Phase 1 — Proof of Concept
|
||||||
|
|
||||||
|
## What is this?
|
||||||
|
|
||||||
|
KubeSolo OS combines **Tiny Core Linux** (~11 MB) with **KubeSolo** (single-binary Kubernetes) to create an appliance-like K8s node that:
|
||||||
|
|
||||||
|
- Boots to a functional Kubernetes cluster in ~30 seconds
|
||||||
|
- Runs entirely from RAM with a read-only SquashFS root
|
||||||
|
- Persists K8s state across reboots via a dedicated data partition
|
||||||
|
- Targets < 100 MB total image size (OS + K8s)
|
||||||
|
- Requires no SSH, no package manager, no writable system files
|
||||||
|
- Supports atomic A/B updates with automatic rollback (Phase 3)
|
||||||
|
|
||||||
|
**Target use cases:** IoT/IIoT edge, air-gapped deployments, single-node K8s appliances, kiosk/POS systems, resource-constrained hardware.
|
||||||
|
|
||||||
|
## Quick Start
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Fetch Tiny Core ISO + KubeSolo binary
|
||||||
|
make fetch
|
||||||
|
|
||||||
|
# Build bootable ISO
|
||||||
|
make iso
|
||||||
|
|
||||||
|
# Test in QEMU
|
||||||
|
make dev-vm
|
||||||
|
```
|
||||||
|
|
||||||
|
## Requirements
|
||||||
|
|
||||||
|
**Build host:**
|
||||||
|
- Linux x86_64 with root/sudo (for loop mounts)
|
||||||
|
- Tools: `cpio`, `gzip`, `wget`, `curl`, `syslinux` (or use `make docker-build`)
|
||||||
|
|
||||||
|
**Runtime:**
|
||||||
|
- x86_64 hardware or VM
|
||||||
|
- 512 MB RAM minimum (1 GB+ recommended)
|
||||||
|
- 8 GB disk (for persistent data partition)
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
Boot Media → Kernel + Initramfs (kubesolo-os.gz)
|
||||||
|
│
|
||||||
|
├── SquashFS root (read-only, in RAM)
|
||||||
|
├── Persistent data partition (ext4, bind-mounted)
|
||||||
|
│ ├── /var/lib/kubesolo (K8s state, certs, SQLite)
|
||||||
|
│ ├── /var/lib/containerd (container images)
|
||||||
|
│ └── /etc/kubesolo (node configuration)
|
||||||
|
├── Custom init (POSIX sh, staged boot)
|
||||||
|
└── KubeSolo (exec replaces init as PID 1)
|
||||||
|
```
|
||||||
|
|
||||||
|
See [docs/design/kubesolo-os-design.md](docs/design/kubesolo-os-design.md) for the full architecture document.
|
||||||
|
|
||||||
|
## Project Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
├── CLAUDE.md # AI-assisted development instructions
|
||||||
|
├── Makefile # Build orchestration
|
||||||
|
├── build/ # Build scripts, configs, rootfs overlays
|
||||||
|
├── init/ # Custom init system (POSIX sh)
|
||||||
|
├── update/ # Atomic update agent (Go, Phase 3)
|
||||||
|
├── cloud-init/ # First-boot configuration (Phase 2)
|
||||||
|
├── test/ # QEMU-based automated tests
|
||||||
|
├── hack/ # Developer utilities
|
||||||
|
└── docs/ # Design documents
|
||||||
|
```
|
||||||
|
|
||||||
|
## Roadmap
|
||||||
|
|
||||||
|
| Phase | Scope | Status |
|
||||||
|
|-------|-------|--------|
|
||||||
|
| 1 | PoC: boot Tiny Core + KubeSolo, verify K8s | 🚧 In Progress |
|
||||||
|
| 2 | Persistent storage, cloud-init, networking | Planned |
|
||||||
|
| 3 | A/B atomic updates, GRUB, rollback | Planned |
|
||||||
|
| 4 | Production hardening, signing, Portainer Edge | Planned |
|
||||||
|
| 5 | OCI distribution, ARM64, fleet management | Planned |
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
TBD
|
||||||
34
build/Dockerfile.builder
Normal file
34
build/Dockerfile.builder
Normal file
@@ -0,0 +1,34 @@
|
|||||||
|
FROM ubuntu:24.04
|
||||||
|
|
||||||
|
ENV DEBIAN_FRONTEND=noninteractive
|
||||||
|
|
||||||
|
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||||
|
bash \
|
||||||
|
bsdtar \
|
||||||
|
cpio \
|
||||||
|
curl \
|
||||||
|
dosfstools \
|
||||||
|
e2fsprogs \
|
||||||
|
fdisk \
|
||||||
|
genisoimage \
|
||||||
|
gzip \
|
||||||
|
isolinux \
|
||||||
|
losetup \
|
||||||
|
make \
|
||||||
|
parted \
|
||||||
|
squashfs-tools \
|
||||||
|
syslinux \
|
||||||
|
syslinux-common \
|
||||||
|
syslinux-utils \
|
||||||
|
wget \
|
||||||
|
xorriso \
|
||||||
|
xz-utils \
|
||||||
|
&& rm -rf /var/lib/apt/lists/*
|
||||||
|
|
||||||
|
WORKDIR /build
|
||||||
|
COPY . /build
|
||||||
|
|
||||||
|
RUN chmod +x build/scripts/*.sh build/config/*.sh
|
||||||
|
|
||||||
|
ENTRYPOINT ["/usr/bin/make"]
|
||||||
|
CMD ["iso"]
|
||||||
169
build/config/kernel-audit.sh
Executable file
169
build/config/kernel-audit.sh
Executable file
@@ -0,0 +1,169 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# kernel-audit.sh — Verify kernel config has all required features for KubeSolo
|
||||||
|
# Usage: ./kernel-audit.sh [/path/to/kernel/.config]
|
||||||
|
# If no path given, attempts to read from /proc/config.gz or boot config
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
RED='\033[0;31m'
|
||||||
|
GREEN='\033[0;32m'
|
||||||
|
YELLOW='\033[1;33m'
|
||||||
|
NC='\033[0m'
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
|
||||||
|
# --- Locate kernel config ---
|
||||||
|
find_kernel_config() {
|
||||||
|
if [[ -n "${1:-}" ]] && [[ -f "$1" ]]; then
|
||||||
|
echo "$1"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
# Try /proc/config.gz (if CONFIG_IKCONFIG_PROC=y)
|
||||||
|
if [[ -f /proc/config.gz ]]; then
|
||||||
|
local tmp
|
||||||
|
tmp=$(mktemp)
|
||||||
|
zcat /proc/config.gz > "$tmp"
|
||||||
|
echo "$tmp"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
# Try /boot/config-$(uname -r)
|
||||||
|
local boot_config="/boot/config-$(uname -r)"
|
||||||
|
if [[ -f "$boot_config" ]]; then
|
||||||
|
echo "$boot_config"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
echo ""
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
CONFIG_FILE=$(find_kernel_config "${1:-}") || {
|
||||||
|
echo -e "${RED}ERROR: Cannot find kernel config.${NC}"
|
||||||
|
echo "Provide path as argument, or ensure /proc/config.gz or /boot/config-\$(uname -r) exists."
|
||||||
|
exit 1
|
||||||
|
}
|
||||||
|
|
||||||
|
echo "==> Auditing kernel config: $CONFIG_FILE"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
PASS=0
|
||||||
|
FAIL=0
|
||||||
|
WARN=0
|
||||||
|
|
||||||
|
check_config() {
|
||||||
|
local option="$1"
|
||||||
|
local required="$2" # "mandatory" or "recommended"
|
||||||
|
local description="$3"
|
||||||
|
|
||||||
|
local value
|
||||||
|
value=$(grep -E "^${option}=" "$CONFIG_FILE" 2>/dev/null || true)
|
||||||
|
|
||||||
|
if [[ -n "$value" ]]; then
|
||||||
|
local setting="${value#*=}"
|
||||||
|
echo -e " ${GREEN}✓${NC} ${option}=${setting} — ${description}"
|
||||||
|
((PASS++))
|
||||||
|
elif grep -qE "^# ${option} is not set" "$CONFIG_FILE" 2>/dev/null; then
|
||||||
|
if [[ "$required" == "mandatory" ]]; then
|
||||||
|
echo -e " ${RED}✗${NC} ${option} is NOT SET — ${description} [REQUIRED]"
|
||||||
|
((FAIL++))
|
||||||
|
else
|
||||||
|
echo -e " ${YELLOW}△${NC} ${option} is NOT SET — ${description} [recommended]"
|
||||||
|
((WARN++))
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
if [[ "$required" == "mandatory" ]]; then
|
||||||
|
echo -e " ${RED}?${NC} ${option} not found in config — ${description} [REQUIRED]"
|
||||||
|
((FAIL++))
|
||||||
|
else
|
||||||
|
echo -e " ${YELLOW}?${NC} ${option} not found in config — ${description} [recommended]"
|
||||||
|
((WARN++))
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# --- cgroup v2 ---
|
||||||
|
echo "cgroup v2:"
|
||||||
|
check_config CONFIG_CGROUPS mandatory "Control groups support"
|
||||||
|
check_config CONFIG_CGROUP_CPUACCT mandatory "CPU accounting"
|
||||||
|
check_config CONFIG_CGROUP_DEVICE mandatory "Device controller"
|
||||||
|
check_config CONFIG_CGROUP_FREEZER mandatory "Freezer controller"
|
||||||
|
check_config CONFIG_CGROUP_SCHED mandatory "CPU scheduler controller"
|
||||||
|
check_config CONFIG_CGROUP_PIDS mandatory "PIDs controller"
|
||||||
|
check_config CONFIG_MEMCG mandatory "Memory controller"
|
||||||
|
check_config CONFIG_CGROUP_BPF recommended "BPF controller"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# --- Namespaces ---
|
||||||
|
echo "Namespaces:"
|
||||||
|
check_config CONFIG_NAMESPACES mandatory "Namespace support"
|
||||||
|
check_config CONFIG_NET_NS mandatory "Network namespaces"
|
||||||
|
check_config CONFIG_PID_NS mandatory "PID namespaces"
|
||||||
|
check_config CONFIG_USER_NS mandatory "User namespaces"
|
||||||
|
check_config CONFIG_UTS_NS mandatory "UTS namespaces"
|
||||||
|
check_config CONFIG_IPC_NS mandatory "IPC namespaces"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# --- Filesystem ---
|
||||||
|
echo "Filesystem:"
|
||||||
|
check_config CONFIG_OVERLAY_FS mandatory "OverlayFS (containerd)"
|
||||||
|
check_config CONFIG_SQUASHFS mandatory "SquashFS (Tiny Core root)"
|
||||||
|
check_config CONFIG_BLK_DEV_LOOP mandatory "Loop device (SquashFS mount)"
|
||||||
|
check_config CONFIG_EXT4_FS mandatory "ext4 (persistent partition)"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# --- Networking ---
|
||||||
|
echo "Networking:"
|
||||||
|
check_config CONFIG_BRIDGE mandatory "Bridge (K8s pod networking)"
|
||||||
|
check_config CONFIG_NETFILTER mandatory "Netfilter framework"
|
||||||
|
check_config CONFIG_NF_NAT mandatory "NAT support"
|
||||||
|
check_config CONFIG_NF_CONNTRACK mandatory "Connection tracking"
|
||||||
|
check_config CONFIG_IP_NF_IPTABLES mandatory "iptables"
|
||||||
|
check_config CONFIG_IP_NF_NAT mandatory "iptables NAT"
|
||||||
|
check_config CONFIG_IP_NF_FILTER mandatory "iptables filter"
|
||||||
|
check_config CONFIG_VETH mandatory "Virtual ethernet pairs"
|
||||||
|
check_config CONFIG_VXLAN mandatory "VXLAN (overlay networking)"
|
||||||
|
check_config CONFIG_NET_SCH_HTB recommended "HTB qdisc (bandwidth limiting)"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# --- Security ---
|
||||||
|
echo "Security:"
|
||||||
|
check_config CONFIG_SECCOMP recommended "Seccomp (container security)"
|
||||||
|
check_config CONFIG_SECCOMP_FILTER recommended "Seccomp BPF filter"
|
||||||
|
check_config CONFIG_BPF_SYSCALL recommended "BPF syscall"
|
||||||
|
check_config CONFIG_AUDIT recommended "Audit framework"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# --- Crypto ---
|
||||||
|
echo "Crypto:"
|
||||||
|
check_config CONFIG_CRYPTO_SHA256 recommended "SHA-256 (image verification)"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# --- IPVS (optional, for kube-proxy IPVS mode) ---
|
||||||
|
echo "IPVS (optional, kube-proxy IPVS mode):"
|
||||||
|
check_config CONFIG_IP_VS recommended "IPVS core"
|
||||||
|
check_config CONFIG_IP_VS_RR recommended "IPVS round-robin"
|
||||||
|
check_config CONFIG_IP_VS_WRR recommended "IPVS weighted round-robin"
|
||||||
|
check_config CONFIG_IP_VS_SH recommended "IPVS source hashing"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# --- Summary ---
|
||||||
|
echo "========================================"
|
||||||
|
echo -e " ${GREEN}Passed:${NC} $PASS"
|
||||||
|
echo -e " ${RED}Failed:${NC} $FAIL"
|
||||||
|
echo -e " ${YELLOW}Warnings:${NC} $WARN"
|
||||||
|
echo "========================================"
|
||||||
|
|
||||||
|
if [[ $FAIL -gt 0 ]]; then
|
||||||
|
echo ""
|
||||||
|
echo -e "${RED}FAIL: $FAIL mandatory kernel config(s) missing.${NC}"
|
||||||
|
echo "Options:"
|
||||||
|
echo " 1. Check if missing features are available as loadable modules (=m)"
|
||||||
|
echo " 2. Recompile the kernel with missing options enabled"
|
||||||
|
echo " 3. Use a different kernel (e.g., Alpine Linux kernel)"
|
||||||
|
exit 1
|
||||||
|
else
|
||||||
|
echo ""
|
||||||
|
echo -e "${GREEN}PASS: All mandatory kernel configs present.${NC}"
|
||||||
|
if [[ $WARN -gt 0 ]]; then
|
||||||
|
echo -e "${YELLOW}Note: $WARN recommended configs missing (non-blocking).${NC}"
|
||||||
|
fi
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
31
build/config/modules.list
Normal file
31
build/config/modules.list
Normal file
@@ -0,0 +1,31 @@
|
|||||||
|
# Kernel modules loaded at boot by init
|
||||||
|
# One module per line. Lines starting with # are ignored.
|
||||||
|
# Modules are loaded in order listed.
|
||||||
|
|
||||||
|
# Networking — bridge and netfilter (required for K8s pod networking)
|
||||||
|
br_netfilter
|
||||||
|
bridge
|
||||||
|
veth
|
||||||
|
vxlan
|
||||||
|
|
||||||
|
# Netfilter / iptables (required for kube-proxy and service routing)
|
||||||
|
ip_tables
|
||||||
|
iptable_nat
|
||||||
|
iptable_filter
|
||||||
|
iptable_mangle
|
||||||
|
nf_nat
|
||||||
|
nf_conntrack
|
||||||
|
nf_conntrack_netlink
|
||||||
|
|
||||||
|
# Filesystem — overlay (required for containerd)
|
||||||
|
overlay
|
||||||
|
|
||||||
|
# Conntrack (required for K8s services)
|
||||||
|
nf_conntrack
|
||||||
|
|
||||||
|
# Optional — useful for CNI plugins and diagnostics
|
||||||
|
tun
|
||||||
|
ip_vs
|
||||||
|
ip_vs_rr
|
||||||
|
ip_vs_wrr
|
||||||
|
ip_vs_sh
|
||||||
19
build/config/versions.env
Normal file
19
build/config/versions.env
Normal file
@@ -0,0 +1,19 @@
|
|||||||
|
# KubeSolo OS Component Versions
|
||||||
|
# All external dependencies pinned here for reproducible builds
|
||||||
|
|
||||||
|
# Tiny Core Linux
|
||||||
|
TINYCORE_VERSION=17.0
|
||||||
|
TINYCORE_ARCH=x86_64
|
||||||
|
TINYCORE_MIRROR=http://www.tinycorelinux.net
|
||||||
|
TINYCORE_ISO=CorePure64-${TINYCORE_VERSION}.iso
|
||||||
|
TINYCORE_ISO_URL=${TINYCORE_MIRROR}/${TINYCORE_VERSION%%.*}.x/${TINYCORE_ARCH}/release/${TINYCORE_ISO}
|
||||||
|
|
||||||
|
# KubeSolo
|
||||||
|
KUBESOLO_INSTALL_URL=https://get.kubesolo.io
|
||||||
|
|
||||||
|
# Build tools (used inside builder container)
|
||||||
|
GRUB_VERSION=2.12
|
||||||
|
SYSLINUX_VERSION=6.03
|
||||||
|
|
||||||
|
# Output naming
|
||||||
|
OS_NAME=kubesolo-os
|
||||||
22
build/rootfs/etc/kubesolo/defaults.yaml
Normal file
22
build/rootfs/etc/kubesolo/defaults.yaml
Normal file
@@ -0,0 +1,22 @@
|
|||||||
|
# KubeSolo OS — Default KubeSolo Configuration
|
||||||
|
# These defaults are used when no cloud-init or persistent config is found.
|
||||||
|
# Overridden by: /etc/kubesolo/config.yaml (persistent) or cloud-init
|
||||||
|
|
||||||
|
# Data directory for K8s state (certs, etcd/sqlite, manifests)
|
||||||
|
data-dir: /var/lib/kubesolo
|
||||||
|
|
||||||
|
# Enable local-path provisioner for PersistentVolumeClaims
|
||||||
|
local-storage: true
|
||||||
|
|
||||||
|
# API server will listen on all interfaces
|
||||||
|
bind-address: 0.0.0.0
|
||||||
|
|
||||||
|
# Cluster CIDR ranges
|
||||||
|
cluster-cidr: 10.42.0.0/16
|
||||||
|
service-cidr: 10.43.0.0/16
|
||||||
|
|
||||||
|
# Disable components not needed for single-node
|
||||||
|
# (KubeSolo may handle this internally)
|
||||||
|
# disable:
|
||||||
|
# - traefik
|
||||||
|
# - servicelb
|
||||||
17
build/rootfs/etc/sysctl.d/k8s.conf
Normal file
17
build/rootfs/etc/sysctl.d/k8s.conf
Normal file
@@ -0,0 +1,17 @@
|
|||||||
|
# Kubernetes networking requirements
|
||||||
|
net.bridge.bridge-nf-call-iptables = 1
|
||||||
|
net.bridge.bridge-nf-call-ip6tables = 1
|
||||||
|
net.ipv4.ip_forward = 1
|
||||||
|
|
||||||
|
# inotify limits (containerd + kubelet watch requirements)
|
||||||
|
fs.inotify.max_user_instances = 1024
|
||||||
|
fs.inotify.max_user_watches = 524288
|
||||||
|
|
||||||
|
# Connection tracking (kube-proxy)
|
||||||
|
net.netfilter.nf_conntrack_max = 131072
|
||||||
|
|
||||||
|
# File descriptor limits
|
||||||
|
fs.file-max = 1048576
|
||||||
|
|
||||||
|
# Disable swap (K8s requirement — though we have no swap anyway)
|
||||||
|
vm.swappiness = 0
|
||||||
63
build/rootfs/usr/lib/kubesolo-os/health.sh
Executable file
63
build/rootfs/usr/lib/kubesolo-os/health.sh
Executable file
@@ -0,0 +1,63 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
# health.sh — Health check functions for KubeSolo OS
|
||||||
|
# Used by init health monitoring and update agent rollback logic
|
||||||
|
# POSIX sh only.
|
||||||
|
|
||||||
|
KUBECONFIG_PATH="/var/lib/kubesolo/pki/admin/admin.kubeconfig"
|
||||||
|
|
||||||
|
# Check if containerd socket is responding
|
||||||
|
check_containerd() {
|
||||||
|
[ -S /run/containerd/containerd.sock ] || return 1
|
||||||
|
# If ctr is available, try listing containers
|
||||||
|
if command -v ctr >/dev/null 2>&1; then
|
||||||
|
ctr --connect-timeout 5s version >/dev/null 2>&1
|
||||||
|
else
|
||||||
|
return 0 # socket exists, assume ok
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# Check if the K8s API server is responding
|
||||||
|
check_apiserver() {
|
||||||
|
kubeconfig="${1:-$KUBECONFIG_PATH}"
|
||||||
|
if [ ! -f "$kubeconfig" ]; then
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
if command -v kubectl >/dev/null 2>&1; then
|
||||||
|
kubectl --kubeconfig="$kubeconfig" get --raw /healthz >/dev/null 2>&1
|
||||||
|
elif command -v curl >/dev/null 2>&1; then
|
||||||
|
# Fallback: direct API call
|
||||||
|
server=$(sed -n 's/.*server: *//p' "$kubeconfig" 2>/dev/null | head -1)
|
||||||
|
[ -n "$server" ] && curl -sk "${server}/healthz" >/dev/null 2>&1
|
||||||
|
else
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# Check if the node has reached Ready status
|
||||||
|
check_node_ready() {
|
||||||
|
kubeconfig="${1:-$KUBECONFIG_PATH}"
|
||||||
|
[ -f "$kubeconfig" ] || return 1
|
||||||
|
command -v kubectl >/dev/null 2>&1 || return 1
|
||||||
|
kubectl --kubeconfig="$kubeconfig" get nodes 2>/dev/null | grep -q "Ready"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Combined health check — returns 0 only if all components are healthy
|
||||||
|
check_health() {
|
||||||
|
check_containerd || return 1
|
||||||
|
check_apiserver || return 1
|
||||||
|
check_node_ready || return 1
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
|
||||||
|
# Wait for system to become healthy with timeout
|
||||||
|
wait_for_healthy() {
|
||||||
|
timeout="${1:-300}"
|
||||||
|
interval="${2:-5}"
|
||||||
|
elapsed=0
|
||||||
|
while [ "$elapsed" -lt "$timeout" ]; do
|
||||||
|
check_health && return 0
|
||||||
|
sleep "$interval"
|
||||||
|
elapsed=$((elapsed + interval))
|
||||||
|
done
|
||||||
|
return 1
|
||||||
|
}
|
||||||
80
build/rootfs/usr/lib/kubesolo-os/network.sh
Executable file
80
build/rootfs/usr/lib/kubesolo-os/network.sh
Executable file
@@ -0,0 +1,80 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
# network.sh — Network configuration helpers for KubeSolo OS init
|
||||||
|
# Sourced by init stages. POSIX sh only.
|
||||||
|
|
||||||
|
# Configure a static IP address on an interface
|
||||||
|
# Usage: static_ip <iface> <ip/prefix> <gateway> [dns1] [dns2]
|
||||||
|
static_ip() {
|
||||||
|
iface="$1" addr="$2" gw="$3" dns1="${4:-}" dns2="${5:-}"
|
||||||
|
|
||||||
|
ip link set "$iface" up
|
||||||
|
ip addr add "$addr" dev "$iface"
|
||||||
|
ip route add default via "$gw" dev "$iface"
|
||||||
|
|
||||||
|
# Write resolv.conf
|
||||||
|
: > /etc/resolv.conf
|
||||||
|
[ -n "$dns1" ] && echo "nameserver $dns1" >> /etc/resolv.conf
|
||||||
|
[ -n "$dns2" ] && echo "nameserver $dns2" >> /etc/resolv.conf
|
||||||
|
}
|
||||||
|
|
||||||
|
# Save current network configuration for persistence across reboots
|
||||||
|
# Writes a shell script that can be sourced to restore networking
|
||||||
|
save_network_config() {
|
||||||
|
dest="${1:-/mnt/data/network/interfaces.sh}"
|
||||||
|
mkdir -p "$(dirname "$dest")"
|
||||||
|
|
||||||
|
iface=""
|
||||||
|
for d in /sys/class/net/*; do
|
||||||
|
name="$(basename "$d")"
|
||||||
|
case "$name" in lo|docker*|veth*|br*|cni*) continue ;; esac
|
||||||
|
iface="$name"
|
||||||
|
break
|
||||||
|
done
|
||||||
|
[ -z "$iface" ] && return 1
|
||||||
|
|
||||||
|
addr=$(ip -4 addr show "$iface" | sed -n 's/.*inet \([0-9./]*\).*/\1/p' | head -1)
|
||||||
|
gw=$(ip route show default 2>/dev/null | sed -n 's/default via \([0-9.]*\).*/\1/p' | head -1)
|
||||||
|
|
||||||
|
cat > "$dest" << SCRIPT
|
||||||
|
#!/bin/sh
|
||||||
|
# Auto-saved network config — generated by KubeSolo OS
|
||||||
|
ip link set $iface up
|
||||||
|
ip addr add $addr dev $iface
|
||||||
|
ip route add default via $gw dev $iface
|
||||||
|
SCRIPT
|
||||||
|
|
||||||
|
# Append DNS if resolv.conf has entries
|
||||||
|
if [ -f /etc/resolv.conf ]; then
|
||||||
|
echo ": > /etc/resolv.conf" >> "$dest"
|
||||||
|
sed -n 's/^nameserver \(.*\)/echo "nameserver \1" >> \/etc\/resolv.conf/p' \
|
||||||
|
/etc/resolv.conf >> "$dest"
|
||||||
|
fi
|
||||||
|
|
||||||
|
chmod +x "$dest"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Get the primary network interface name
|
||||||
|
get_primary_iface() {
|
||||||
|
for d in /sys/class/net/*; do
|
||||||
|
name="$(basename "$d")"
|
||||||
|
case "$name" in lo|docker*|veth*|br*|cni*) continue ;; esac
|
||||||
|
echo "$name"
|
||||||
|
return 0
|
||||||
|
done
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
# Wait for link on an interface
|
||||||
|
wait_for_link() {
|
||||||
|
iface="$1"
|
||||||
|
timeout="${2:-15}"
|
||||||
|
i=0
|
||||||
|
while [ "$i" -lt "$timeout" ]; do
|
||||||
|
if ip link show "$iface" 2>/dev/null | grep -q 'state UP'; then
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
sleep 1
|
||||||
|
i=$((i + 1))
|
||||||
|
done
|
||||||
|
return 1
|
||||||
|
}
|
||||||
110
build/scripts/create-disk-image.sh
Executable file
110
build/scripts/create-disk-image.sh
Executable file
@@ -0,0 +1,110 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# create-disk-image.sh — Create a raw disk image with boot + data partitions
|
||||||
|
# Phase 1: simple layout (boot + data). Phase 3 adds A/B system partitions.
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
|
||||||
|
ROOTFS_DIR="${ROOTFS_DIR:-$PROJECT_ROOT/build/rootfs-work}"
|
||||||
|
OUTPUT_DIR="${OUTPUT_DIR:-$PROJECT_ROOT/output}"
|
||||||
|
VERSION="$(cat "$PROJECT_ROOT/VERSION")"
|
||||||
|
OS_NAME="kubesolo-os"
|
||||||
|
|
||||||
|
IMG_OUTPUT="$OUTPUT_DIR/${OS_NAME}-${VERSION}.img"
|
||||||
|
IMG_SIZE_MB="${IMG_SIZE_MB:-2048}" # 2 GB default
|
||||||
|
|
||||||
|
VMLINUZ="$ROOTFS_DIR/vmlinuz"
|
||||||
|
INITRAMFS="$ROOTFS_DIR/kubesolo-os.gz"
|
||||||
|
|
||||||
|
for f in "$VMLINUZ" "$INITRAMFS"; do
|
||||||
|
[ -f "$f" ] || { echo "ERROR: Missing $f — run 'make initramfs'"; exit 1; }
|
||||||
|
done
|
||||||
|
|
||||||
|
echo "==> Creating ${IMG_SIZE_MB}MB disk image..."
|
||||||
|
mkdir -p "$OUTPUT_DIR"
|
||||||
|
|
||||||
|
# Create sparse image
|
||||||
|
dd if=/dev/zero of="$IMG_OUTPUT" bs=1M count=0 seek="$IMG_SIZE_MB" 2>/dev/null
|
||||||
|
|
||||||
|
# Partition: 256MB boot (ext4) + rest data (ext4)
|
||||||
|
# Using sfdisk for scriptability
|
||||||
|
sfdisk "$IMG_OUTPUT" << EOF
|
||||||
|
label: dos
|
||||||
|
unit: sectors
|
||||||
|
|
||||||
|
# Boot partition: 256 MB, bootable
|
||||||
|
start=2048, size=524288, type=83, bootable
|
||||||
|
# Data partition: remaining space
|
||||||
|
start=526336, type=83
|
||||||
|
EOF
|
||||||
|
|
||||||
|
# Set up loop device
|
||||||
|
LOOP=$(losetup --show -fP "$IMG_OUTPUT")
|
||||||
|
echo "==> Loop device: $LOOP"
|
||||||
|
|
||||||
|
cleanup() {
|
||||||
|
umount "${LOOP}p1" 2>/dev/null || true
|
||||||
|
umount "${LOOP}p2" 2>/dev/null || true
|
||||||
|
losetup -d "$LOOP" 2>/dev/null || true
|
||||||
|
rm -rf "$MNT_BOOT" "$MNT_DATA" 2>/dev/null || true
|
||||||
|
}
|
||||||
|
trap cleanup EXIT
|
||||||
|
|
||||||
|
# Format partitions
|
||||||
|
mkfs.ext4 -q -L KSOLOBOOT "${LOOP}p1"
|
||||||
|
mkfs.ext4 -q -L KSOLODATA "${LOOP}p2"
|
||||||
|
|
||||||
|
# Mount and populate boot partition
|
||||||
|
MNT_BOOT=$(mktemp -d)
|
||||||
|
MNT_DATA=$(mktemp -d)
|
||||||
|
|
||||||
|
mount "${LOOP}p1" "$MNT_BOOT"
|
||||||
|
mount "${LOOP}p2" "$MNT_DATA"
|
||||||
|
|
||||||
|
# Install syslinux + kernel + initramfs to boot partition
|
||||||
|
mkdir -p "$MNT_BOOT/boot/syslinux"
|
||||||
|
cp "$VMLINUZ" "$MNT_BOOT/boot/vmlinuz"
|
||||||
|
cp "$INITRAMFS" "$MNT_BOOT/boot/kubesolo-os.gz"
|
||||||
|
|
||||||
|
# Syslinux config for disk boot (extlinux)
|
||||||
|
cat > "$MNT_BOOT/boot/syslinux/syslinux.cfg" << 'EOF'
|
||||||
|
DEFAULT kubesolo
|
||||||
|
TIMEOUT 30
|
||||||
|
PROMPT 0
|
||||||
|
|
||||||
|
LABEL kubesolo
|
||||||
|
KERNEL /boot/vmlinuz
|
||||||
|
INITRD /boot/kubesolo-os.gz
|
||||||
|
APPEND quiet kubesolo.data=LABEL=KSOLODATA
|
||||||
|
|
||||||
|
LABEL kubesolo-debug
|
||||||
|
KERNEL /boot/vmlinuz
|
||||||
|
INITRD /boot/kubesolo-os.gz
|
||||||
|
APPEND kubesolo.data=LABEL=KSOLODATA kubesolo.debug console=ttyS0,115200n8
|
||||||
|
|
||||||
|
LABEL kubesolo-shell
|
||||||
|
KERNEL /boot/vmlinuz
|
||||||
|
INITRD /boot/kubesolo-os.gz
|
||||||
|
APPEND kubesolo.shell console=ttyS0,115200n8
|
||||||
|
EOF
|
||||||
|
|
||||||
|
# Install extlinux bootloader
|
||||||
|
if command -v extlinux >/dev/null 2>&1; then
|
||||||
|
extlinux --install "$MNT_BOOT/boot/syslinux" 2>/dev/null || {
|
||||||
|
echo "WARN: extlinux install failed — image may not be directly bootable"
|
||||||
|
echo " Use with QEMU -kernel/-initrd flags instead"
|
||||||
|
}
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Prepare data partition structure
|
||||||
|
for dir in kubesolo containerd etc-kubesolo log usr-local network; do
|
||||||
|
mkdir -p "$MNT_DATA/$dir"
|
||||||
|
done
|
||||||
|
|
||||||
|
sync
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "==> Disk image created: $IMG_OUTPUT"
|
||||||
|
echo " Size: $(du -h "$IMG_OUTPUT" | cut -f1)"
|
||||||
|
echo " Boot partition (KSOLOBOOT): kernel + initramfs"
|
||||||
|
echo " Data partition (KSOLODATA): persistent K8s state"
|
||||||
140
build/scripts/create-iso.sh
Executable file
140
build/scripts/create-iso.sh
Executable file
@@ -0,0 +1,140 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# create-iso.sh — Create a bootable ISO from kernel + initramfs
|
||||||
|
# Uses isolinux (syslinux) for Phase 1 simplicity (GRUB in Phase 3)
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
|
||||||
|
ROOTFS_DIR="${ROOTFS_DIR:-$PROJECT_ROOT/build/rootfs-work}"
|
||||||
|
OUTPUT_DIR="${OUTPUT_DIR:-$PROJECT_ROOT/output}"
|
||||||
|
VERSION="$(cat "$PROJECT_ROOT/VERSION")"
|
||||||
|
OS_NAME="kubesolo-os"
|
||||||
|
|
||||||
|
ISO_STAGING="$ROOTFS_DIR/iso-staging"
|
||||||
|
ISO_OUTPUT="$OUTPUT_DIR/${OS_NAME}-${VERSION}.iso"
|
||||||
|
|
||||||
|
VMLINUZ="$ROOTFS_DIR/vmlinuz"
|
||||||
|
INITRAMFS="$ROOTFS_DIR/kubesolo-os.gz"
|
||||||
|
|
||||||
|
# Validate inputs
|
||||||
|
for f in "$VMLINUZ" "$INITRAMFS"; do
|
||||||
|
if [ ! -f "$f" ]; then
|
||||||
|
echo "ERROR: Missing required file: $f"
|
||||||
|
echo "Run 'make initramfs' first."
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
# Check for required tools
|
||||||
|
for cmd in mkisofs xorriso genisoimage; do
|
||||||
|
if command -v "$cmd" >/dev/null 2>&1; then
|
||||||
|
MKISO_CMD="$cmd"
|
||||||
|
break
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
if [ -z "${MKISO_CMD:-}" ]; then
|
||||||
|
echo "ERROR: Need mkisofs, genisoimage, or xorriso to create ISO"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# --- Stage ISO contents ---
|
||||||
|
rm -rf "$ISO_STAGING"
|
||||||
|
mkdir -p "$ISO_STAGING/boot/isolinux"
|
||||||
|
|
||||||
|
cp "$VMLINUZ" "$ISO_STAGING/boot/vmlinuz"
|
||||||
|
cp "$INITRAMFS" "$ISO_STAGING/boot/kubesolo-os.gz"
|
||||||
|
|
||||||
|
# Find isolinux.bin
|
||||||
|
ISOLINUX_BIN=""
|
||||||
|
for path in /usr/lib/ISOLINUX/isolinux.bin /usr/lib/syslinux/isolinux.bin \
|
||||||
|
/usr/share/syslinux/isolinux.bin /usr/lib/syslinux/bios/isolinux.bin; do
|
||||||
|
[ -f "$path" ] && ISOLINUX_BIN="$path" && break
|
||||||
|
done
|
||||||
|
|
||||||
|
if [ -z "$ISOLINUX_BIN" ]; then
|
||||||
|
echo "ERROR: Cannot find isolinux.bin. Install syslinux/isolinux package."
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
cp "$ISOLINUX_BIN" "$ISO_STAGING/boot/isolinux/"
|
||||||
|
|
||||||
|
# Copy ldlinux.c32 if it exists (needed by syslinux 6+)
|
||||||
|
LDLINUX_DIR="$(dirname "$ISOLINUX_BIN")"
|
||||||
|
for mod in ldlinux.c32 libcom32.c32 libutil.c32 mboot.c32; do
|
||||||
|
[ -f "$LDLINUX_DIR/$mod" ] && cp "$LDLINUX_DIR/$mod" "$ISO_STAGING/boot/isolinux/"
|
||||||
|
done
|
||||||
|
|
||||||
|
# Isolinux config
|
||||||
|
cat > "$ISO_STAGING/boot/isolinux/isolinux.cfg" << 'EOF'
|
||||||
|
DEFAULT kubesolo
|
||||||
|
TIMEOUT 30
|
||||||
|
PROMPT 0
|
||||||
|
|
||||||
|
LABEL kubesolo
|
||||||
|
MENU LABEL KubeSolo OS
|
||||||
|
KERNEL /boot/vmlinuz
|
||||||
|
INITRD /boot/kubesolo-os.gz
|
||||||
|
APPEND quiet kubesolo.data=LABEL=KSOLODATA
|
||||||
|
|
||||||
|
LABEL kubesolo-debug
|
||||||
|
MENU LABEL KubeSolo OS (debug)
|
||||||
|
KERNEL /boot/vmlinuz
|
||||||
|
INITRD /boot/kubesolo-os.gz
|
||||||
|
APPEND kubesolo.data=LABEL=KSOLODATA kubesolo.debug console=ttyS0,115200n8
|
||||||
|
|
||||||
|
LABEL kubesolo-shell
|
||||||
|
MENU LABEL KubeSolo OS (emergency shell)
|
||||||
|
KERNEL /boot/vmlinuz
|
||||||
|
INITRD /boot/kubesolo-os.gz
|
||||||
|
APPEND kubesolo.shell console=ttyS0,115200n8
|
||||||
|
|
||||||
|
LABEL kubesolo-nopersist
|
||||||
|
MENU LABEL KubeSolo OS (RAM only, no persistence)
|
||||||
|
KERNEL /boot/vmlinuz
|
||||||
|
INITRD /boot/kubesolo-os.gz
|
||||||
|
APPEND kubesolo.nopersist
|
||||||
|
EOF
|
||||||
|
|
||||||
|
# --- Create ISO ---
|
||||||
|
mkdir -p "$OUTPUT_DIR"
|
||||||
|
|
||||||
|
case "$MKISO_CMD" in
|
||||||
|
xorriso)
|
||||||
|
xorriso -as mkisofs \
|
||||||
|
-o "$ISO_OUTPUT" \
|
||||||
|
-isohybrid-mbr /usr/lib/ISOLINUX/isohdpfx.bin 2>/dev/null || true \
|
||||||
|
-c boot/isolinux/boot.cat \
|
||||||
|
-b boot/isolinux/isolinux.bin \
|
||||||
|
-no-emul-boot \
|
||||||
|
-boot-load-size 4 \
|
||||||
|
-boot-info-table \
|
||||||
|
"$ISO_STAGING"
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
"$MKISO_CMD" \
|
||||||
|
-o "$ISO_OUTPUT" \
|
||||||
|
-b boot/isolinux/isolinux.bin \
|
||||||
|
-c boot/isolinux/boot.cat \
|
||||||
|
-no-emul-boot \
|
||||||
|
-boot-load-size 4 \
|
||||||
|
-boot-info-table \
|
||||||
|
-J -R -V "KUBESOLOOS" \
|
||||||
|
"$ISO_STAGING"
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
# Make ISO hybrid-bootable (USB stick)
|
||||||
|
if command -v isohybrid >/dev/null 2>&1; then
|
||||||
|
isohybrid "$ISO_OUTPUT" 2>/dev/null || true
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Clean staging
|
||||||
|
rm -rf "$ISO_STAGING"
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "==> ISO created: $ISO_OUTPUT"
|
||||||
|
echo " Size: $(du -h "$ISO_OUTPUT" | cut -f1)"
|
||||||
|
echo ""
|
||||||
|
echo " Boot in QEMU: make dev-vm"
|
||||||
|
echo " Write to USB: dd if=$ISO_OUTPUT of=/dev/sdX bs=4M status=progress"
|
||||||
83
build/scripts/extract-core.sh
Executable file
83
build/scripts/extract-core.sh
Executable file
@@ -0,0 +1,83 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# extract-core.sh — Extract Tiny Core Linux rootfs from ISO
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
|
||||||
|
CACHE_DIR="${CACHE_DIR:-$PROJECT_ROOT/build/cache}"
|
||||||
|
ROOTFS_DIR="${ROOTFS_DIR:-$PROJECT_ROOT/build/rootfs-work}"
|
||||||
|
|
||||||
|
# shellcheck source=../config/versions.env
|
||||||
|
. "$SCRIPT_DIR/../config/versions.env"
|
||||||
|
|
||||||
|
TC_ISO="$CACHE_DIR/$TINYCORE_ISO"
|
||||||
|
ISO_MNT="$ROOTFS_DIR/iso-mount"
|
||||||
|
|
||||||
|
if [ ! -f "$TC_ISO" ]; then
|
||||||
|
echo "ERROR: Tiny Core ISO not found: $TC_ISO"
|
||||||
|
echo "Run 'make fetch' first."
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Clean previous rootfs
|
||||||
|
rm -rf "$ROOTFS_DIR"
|
||||||
|
mkdir -p "$ROOTFS_DIR" "$ISO_MNT"
|
||||||
|
|
||||||
|
# --- Mount ISO and extract kernel + initramfs ---
|
||||||
|
echo "==> Mounting ISO: $TC_ISO"
|
||||||
|
mount -o loop,ro "$TC_ISO" "$ISO_MNT" 2>/dev/null || {
|
||||||
|
# Fallback for non-root: use 7z or bsdtar
|
||||||
|
echo " mount failed (need root?), trying bsdtar..."
|
||||||
|
mkdir -p "$ISO_MNT"
|
||||||
|
bsdtar xf "$TC_ISO" -C "$ISO_MNT" 2>/dev/null || {
|
||||||
|
echo " bsdtar failed, trying 7z..."
|
||||||
|
7z x -o"$ISO_MNT" "$TC_ISO" >/dev/null 2>&1
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
# Find vmlinuz and core.gz (path varies by Tiny Core version/arch)
|
||||||
|
VMLINUZ=""
|
||||||
|
COREGZ=""
|
||||||
|
for f in "$ISO_MNT"/boot/vmlinuz64 "$ISO_MNT"/boot/vmlinuz; do
|
||||||
|
[ -f "$f" ] && VMLINUZ="$f" && break
|
||||||
|
done
|
||||||
|
for f in "$ISO_MNT"/boot/corepure64.gz "$ISO_MNT"/boot/core.gz; do
|
||||||
|
[ -f "$f" ] && COREGZ="$f" && break
|
||||||
|
done
|
||||||
|
|
||||||
|
if [ -z "$VMLINUZ" ] || [ -z "$COREGZ" ]; then
|
||||||
|
echo "ERROR: Could not find vmlinuz/core.gz in ISO"
|
||||||
|
echo "ISO contents:"
|
||||||
|
find "$ISO_MNT" -type f
|
||||||
|
umount "$ISO_MNT" 2>/dev/null || true
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "==> Found kernel: $VMLINUZ"
|
||||||
|
echo "==> Found initramfs: $COREGZ"
|
||||||
|
|
||||||
|
# Copy kernel
|
||||||
|
cp "$VMLINUZ" "$ROOTFS_DIR/vmlinuz"
|
||||||
|
|
||||||
|
# --- Extract initramfs (core.gz → rootfs) ---
|
||||||
|
echo "==> Extracting initramfs..."
|
||||||
|
mkdir -p "$ROOTFS_DIR/rootfs"
|
||||||
|
cd "$ROOTFS_DIR/rootfs"
|
||||||
|
zcat "$COREGZ" | cpio -idm 2>/dev/null
|
||||||
|
|
||||||
|
# Unmount ISO
|
||||||
|
cd "$PROJECT_ROOT"
|
||||||
|
umount "$ISO_MNT" 2>/dev/null || true
|
||||||
|
rm -rf "$ISO_MNT"
|
||||||
|
|
||||||
|
echo "==> Rootfs extracted: $ROOTFS_DIR/rootfs"
|
||||||
|
echo " Size: $(du -sh "$ROOTFS_DIR/rootfs" | cut -f1)"
|
||||||
|
echo " Kernel: $ROOTFS_DIR/vmlinuz ($(du -h "$ROOTFS_DIR/vmlinuz" | cut -f1))"
|
||||||
|
|
||||||
|
# --- Audit kernel config if available ---
|
||||||
|
if [ -f "$ROOTFS_DIR/rootfs/proc/config.gz" ]; then
|
||||||
|
echo "==> Kernel config found in rootfs, auditing..."
|
||||||
|
"$SCRIPT_DIR/../config/kernel-audit.sh" <(zcat "$ROOTFS_DIR/rootfs/proc/config.gz") || true
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "==> Extract complete."
|
||||||
72
build/scripts/fetch-components.sh
Executable file
72
build/scripts/fetch-components.sh
Executable file
@@ -0,0 +1,72 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# fetch-components.sh — Download Tiny Core ISO and KubeSolo binary
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
|
||||||
|
CACHE_DIR="${CACHE_DIR:-$PROJECT_ROOT/build/cache}"
|
||||||
|
|
||||||
|
# Load versions
|
||||||
|
# shellcheck source=../config/versions.env
|
||||||
|
. "$SCRIPT_DIR/../config/versions.env"
|
||||||
|
|
||||||
|
mkdir -p "$CACHE_DIR"
|
||||||
|
|
||||||
|
# --- Tiny Core Linux ISO ---
|
||||||
|
TC_ISO="$CACHE_DIR/$TINYCORE_ISO"
|
||||||
|
TC_URL="${TINYCORE_MIRROR}/${TINYCORE_VERSION%%.*}.x/${TINYCORE_ARCH}/release/${TINYCORE_ISO}"
|
||||||
|
|
||||||
|
if [ -f "$TC_ISO" ]; then
|
||||||
|
echo "==> Tiny Core ISO already cached: $TC_ISO"
|
||||||
|
else
|
||||||
|
echo "==> Downloading Tiny Core Linux ${TINYCORE_VERSION} (${TINYCORE_ARCH})..."
|
||||||
|
echo " URL: $TC_URL"
|
||||||
|
wget -q --show-progress -O "$TC_ISO" "$TC_URL" || {
|
||||||
|
# Fallback: try alternate mirror structure
|
||||||
|
TC_URL_ALT="${TINYCORE_MIRROR}/${TINYCORE_VERSION%%.*}.x/${TINYCORE_ARCH}/release/CorePure64-current.iso"
|
||||||
|
echo " Primary URL failed, trying: $TC_URL_ALT"
|
||||||
|
wget -q --show-progress -O "$TC_ISO" "$TC_URL_ALT"
|
||||||
|
}
|
||||||
|
echo "==> Downloaded: $TC_ISO ($(du -h "$TC_ISO" | cut -f1))"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# --- KubeSolo ---
|
||||||
|
KUBESOLO_INSTALLER="$CACHE_DIR/install-kubesolo.sh"
|
||||||
|
KUBESOLO_BIN="$CACHE_DIR/kubesolo"
|
||||||
|
|
||||||
|
if [ -f "$KUBESOLO_BIN" ]; then
|
||||||
|
echo "==> KubeSolo binary already cached: $KUBESOLO_BIN"
|
||||||
|
else
|
||||||
|
echo "==> Downloading KubeSolo installer..."
|
||||||
|
curl -sfL "$KUBESOLO_INSTALL_URL" -o "$KUBESOLO_INSTALLER"
|
||||||
|
|
||||||
|
echo "==> Extracting KubeSolo binary..."
|
||||||
|
echo " NOTE: The installer normally runs 'install'. We extract the binary URL instead."
|
||||||
|
echo " For Phase 1 PoC, install KubeSolo on a host and copy the binary."
|
||||||
|
echo ""
|
||||||
|
echo " Manual step required:"
|
||||||
|
echo " 1. On a Linux x86_64 host: curl -sfL https://get.kubesolo.io | sudo sh -"
|
||||||
|
echo " 2. Copy /usr/local/bin/kubesolo to: $KUBESOLO_BIN"
|
||||||
|
echo " 3. Re-run: make rootfs"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Try to extract download URL from installer script
|
||||||
|
BINARY_URL=$(grep -oP 'https://[^ ]+kubesolo[^ ]+' "$KUBESOLO_INSTALLER" 2>/dev/null | head -1 || true)
|
||||||
|
if [ -n "$BINARY_URL" ]; then
|
||||||
|
echo " Attempting direct download from: $BINARY_URL"
|
||||||
|
curl -sfL "$BINARY_URL" -o "$KUBESOLO_BIN" && chmod +x "$KUBESOLO_BIN" || {
|
||||||
|
echo " Direct download failed. Use manual step above."
|
||||||
|
}
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ -f "$KUBESOLO_BIN" ]; then
|
||||||
|
echo "==> KubeSolo binary: $KUBESOLO_BIN ($(du -h "$KUBESOLO_BIN" | cut -f1))"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
# --- Summary ---
|
||||||
|
echo ""
|
||||||
|
echo "==> Component cache:"
|
||||||
|
ls -lh "$CACHE_DIR"/ 2>/dev/null || true
|
||||||
|
echo ""
|
||||||
|
echo "==> Fetch complete."
|
||||||
124
build/scripts/inject-kubesolo.sh
Executable file
124
build/scripts/inject-kubesolo.sh
Executable file
@@ -0,0 +1,124 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# inject-kubesolo.sh — Add KubeSolo binary, init system, and configs to rootfs
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
|
||||||
|
CACHE_DIR="${CACHE_DIR:-$PROJECT_ROOT/build/cache}"
|
||||||
|
ROOTFS_DIR="${ROOTFS_DIR:-$PROJECT_ROOT/build/rootfs-work}"
|
||||||
|
ROOTFS="$ROOTFS_DIR/rootfs"
|
||||||
|
VERSION="$(cat "$PROJECT_ROOT/VERSION")"
|
||||||
|
|
||||||
|
if [ ! -d "$ROOTFS" ]; then
|
||||||
|
echo "ERROR: Rootfs not found: $ROOTFS"
|
||||||
|
echo "Run extract-core.sh first."
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
KUBESOLO_BIN="$CACHE_DIR/kubesolo"
|
||||||
|
if [ ! -f "$KUBESOLO_BIN" ]; then
|
||||||
|
echo "ERROR: KubeSolo binary not found: $KUBESOLO_BIN"
|
||||||
|
echo "See fetch-components.sh output for instructions."
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "==> Injecting KubeSolo into rootfs..."
|
||||||
|
|
||||||
|
# --- 1. KubeSolo binary ---
|
||||||
|
mkdir -p "$ROOTFS/usr/local/bin"
|
||||||
|
cp "$KUBESOLO_BIN" "$ROOTFS/usr/local/bin/kubesolo"
|
||||||
|
chmod +x "$ROOTFS/usr/local/bin/kubesolo"
|
||||||
|
echo " Installed KubeSolo binary ($(du -h "$KUBESOLO_BIN" | cut -f1))"
|
||||||
|
|
||||||
|
# --- 2. Custom init system ---
|
||||||
|
echo " Installing init system..."
|
||||||
|
|
||||||
|
# Main init
|
||||||
|
cp "$PROJECT_ROOT/init/init.sh" "$ROOTFS/sbin/init"
|
||||||
|
chmod +x "$ROOTFS/sbin/init"
|
||||||
|
|
||||||
|
# Init stages
|
||||||
|
mkdir -p "$ROOTFS/usr/lib/kubesolo-os/init.d"
|
||||||
|
for stage in "$PROJECT_ROOT"/init/lib/*.sh; do
|
||||||
|
[ -f "$stage" ] || continue
|
||||||
|
cp "$stage" "$ROOTFS/usr/lib/kubesolo-os/init.d/"
|
||||||
|
chmod +x "$ROOTFS/usr/lib/kubesolo-os/init.d/$(basename "$stage")"
|
||||||
|
done
|
||||||
|
echo " Installed $(ls "$ROOTFS/usr/lib/kubesolo-os/init.d/" | wc -l) init stages"
|
||||||
|
|
||||||
|
# Shared functions
|
||||||
|
if [ -f "$PROJECT_ROOT/init/lib/functions.sh" ]; then
|
||||||
|
cp "$PROJECT_ROOT/init/lib/functions.sh" "$ROOTFS/usr/lib/kubesolo-os/functions.sh"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Emergency shell
|
||||||
|
if [ -f "$PROJECT_ROOT/init/emergency-shell.sh" ]; then
|
||||||
|
cp "$PROJECT_ROOT/init/emergency-shell.sh" "$ROOTFS/usr/lib/kubesolo-os/emergency-shell.sh"
|
||||||
|
chmod +x "$ROOTFS/usr/lib/kubesolo-os/emergency-shell.sh"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Shared library scripts (network, health)
|
||||||
|
for lib in network.sh health.sh; do
|
||||||
|
src="$PROJECT_ROOT/build/rootfs/usr/lib/kubesolo-os/$lib"
|
||||||
|
[ -f "$src" ] && cp "$src" "$ROOTFS/usr/lib/kubesolo-os/$lib"
|
||||||
|
done
|
||||||
|
|
||||||
|
# --- 3. Kernel modules list ---
|
||||||
|
cp "$PROJECT_ROOT/build/config/modules.list" "$ROOTFS/usr/lib/kubesolo-os/modules.list"
|
||||||
|
|
||||||
|
# --- 4. Sysctl config ---
|
||||||
|
mkdir -p "$ROOTFS/etc/sysctl.d"
|
||||||
|
cp "$PROJECT_ROOT/build/rootfs/etc/sysctl.d/k8s.conf" "$ROOTFS/etc/sysctl.d/k8s.conf"
|
||||||
|
|
||||||
|
# --- 5. OS metadata ---
|
||||||
|
echo "$VERSION" > "$ROOTFS/etc/kubesolo-os-version"
|
||||||
|
|
||||||
|
cat > "$ROOTFS/etc/os-release" << EOF
|
||||||
|
NAME="KubeSolo OS"
|
||||||
|
VERSION="$VERSION"
|
||||||
|
ID=kubesolo-os
|
||||||
|
VERSION_ID=$VERSION
|
||||||
|
PRETTY_NAME="KubeSolo OS $VERSION"
|
||||||
|
HOME_URL="https://github.com/portainer/kubesolo"
|
||||||
|
BUG_REPORT_URL="https://github.com/portainer/kubesolo/issues"
|
||||||
|
EOF
|
||||||
|
|
||||||
|
# --- 6. Default KubeSolo config ---
|
||||||
|
mkdir -p "$ROOTFS/etc/kubesolo"
|
||||||
|
if [ -f "$PROJECT_ROOT/build/rootfs/etc/kubesolo/defaults.yaml" ]; then
|
||||||
|
cp "$PROJECT_ROOT/build/rootfs/etc/kubesolo/defaults.yaml" "$ROOTFS/etc/kubesolo/defaults.yaml"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# --- 7. Essential directories ---
|
||||||
|
mkdir -p "$ROOTFS/var/lib/kubesolo"
|
||||||
|
mkdir -p "$ROOTFS/var/lib/containerd"
|
||||||
|
mkdir -p "$ROOTFS/etc/kubesolo"
|
||||||
|
mkdir -p "$ROOTFS/etc/cni/net.d"
|
||||||
|
mkdir -p "$ROOTFS/opt/cni/bin"
|
||||||
|
mkdir -p "$ROOTFS/var/log"
|
||||||
|
mkdir -p "$ROOTFS/usr/local"
|
||||||
|
mkdir -p "$ROOTFS/mnt/data"
|
||||||
|
mkdir -p "$ROOTFS/run/containerd"
|
||||||
|
|
||||||
|
# --- 8. Ensure /etc/hosts and /etc/resolv.conf exist ---
|
||||||
|
if [ ! -f "$ROOTFS/etc/hosts" ]; then
|
||||||
|
cat > "$ROOTFS/etc/hosts" << EOF
|
||||||
|
127.0.0.1 localhost
|
||||||
|
::1 localhost
|
||||||
|
EOF
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ ! -f "$ROOTFS/etc/resolv.conf" ]; then
|
||||||
|
cat > "$ROOTFS/etc/resolv.conf" << EOF
|
||||||
|
nameserver 8.8.8.8
|
||||||
|
nameserver 1.1.1.1
|
||||||
|
EOF
|
||||||
|
fi
|
||||||
|
|
||||||
|
# --- Summary ---
|
||||||
|
echo ""
|
||||||
|
echo "==> Injection complete. Rootfs contents:"
|
||||||
|
echo " Total size: $(du -sh "$ROOTFS" | cut -f1)"
|
||||||
|
echo " KubeSolo: $(du -h "$ROOTFS/usr/local/bin/kubesolo" | cut -f1)"
|
||||||
|
echo " Init stages: $(ls "$ROOTFS/usr/lib/kubesolo-os/init.d/" | wc -l)"
|
||||||
|
echo ""
|
||||||
23
build/scripts/pack-initramfs.sh
Executable file
23
build/scripts/pack-initramfs.sh
Executable file
@@ -0,0 +1,23 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# pack-initramfs.sh — Repack modified rootfs into kubesolo-os.gz
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
|
||||||
|
ROOTFS_DIR="${ROOTFS_DIR:-$PROJECT_ROOT/build/rootfs-work}"
|
||||||
|
ROOTFS="$ROOTFS_DIR/rootfs"
|
||||||
|
OUTPUT="$ROOTFS_DIR/kubesolo-os.gz"
|
||||||
|
|
||||||
|
if [ ! -d "$ROOTFS" ]; then
|
||||||
|
echo "ERROR: Rootfs not found: $ROOTFS"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "==> Packing initramfs..."
|
||||||
|
|
||||||
|
cd "$ROOTFS"
|
||||||
|
find . | cpio -o -H newc 2>/dev/null | gzip -9 > "$OUTPUT"
|
||||||
|
|
||||||
|
echo "==> Built: $OUTPUT"
|
||||||
|
echo " Size: $(du -h "$OUTPUT" | cut -f1)"
|
||||||
|
echo " (Original Tiny Core core.gz is ~11 MB for reference)"
|
||||||
29
cloud-init/examples/airgapped.yaml
Normal file
29
cloud-init/examples/airgapped.yaml
Normal file
@@ -0,0 +1,29 @@
|
|||||||
|
# KubeSolo OS Cloud-Init — Air-Gapped Deployment
|
||||||
|
# For environments with no internet access.
|
||||||
|
# All container images must be pre-loaded into containerd.
|
||||||
|
#
|
||||||
|
# Place at: /mnt/data/etc-kubesolo/cloud-init.yaml
|
||||||
|
|
||||||
|
hostname: airgap-node-01
|
||||||
|
|
||||||
|
network:
|
||||||
|
mode: static
|
||||||
|
interface: eth0
|
||||||
|
address: 10.0.0.50/24
|
||||||
|
gateway: 10.0.0.1
|
||||||
|
dns:
|
||||||
|
- 10.0.0.1
|
||||||
|
|
||||||
|
kubesolo:
|
||||||
|
local-storage: true
|
||||||
|
# Disable components that need internet
|
||||||
|
extra-flags: "--disable traefik --disable servicelb"
|
||||||
|
|
||||||
|
# Pre-loaded images (Phase 2+: auto-import at boot)
|
||||||
|
# Images must be placed as tar files on the data partition at:
|
||||||
|
# /mnt/data/images/*.tar
|
||||||
|
# They will be imported into containerd on first boot.
|
||||||
|
airgap:
|
||||||
|
import-images: true
|
||||||
|
images-dir: /mnt/data/images
|
||||||
|
# registry-mirror: "" # Optional: local registry mirror
|
||||||
18
cloud-init/examples/dhcp.yaml
Normal file
18
cloud-init/examples/dhcp.yaml
Normal file
@@ -0,0 +1,18 @@
|
|||||||
|
# KubeSolo OS Cloud-Init — DHCP Configuration (Default)
|
||||||
|
# Place at: /mnt/data/etc-kubesolo/cloud-init.yaml (on data partition)
|
||||||
|
# Or pass via boot param: kubesolo.cloudinit=/path/to/this.yaml
|
||||||
|
|
||||||
|
hostname: kubesolo-node
|
||||||
|
|
||||||
|
network:
|
||||||
|
mode: dhcp
|
||||||
|
# interface: eth0 # Optional: specify interface (auto-detected if omitted)
|
||||||
|
# dns: # Optional: override DHCP-provided DNS
|
||||||
|
# - 8.8.8.8
|
||||||
|
# - 1.1.1.1
|
||||||
|
|
||||||
|
kubesolo:
|
||||||
|
# extra-flags: "" # Additional flags for KubeSolo binary
|
||||||
|
# local-storage: true
|
||||||
|
# apiserver-extra-sans:
|
||||||
|
# - kubesolo.local
|
||||||
26
cloud-init/examples/portainer-edge.yaml
Normal file
26
cloud-init/examples/portainer-edge.yaml
Normal file
@@ -0,0 +1,26 @@
|
|||||||
|
# KubeSolo OS Cloud-Init — Portainer Edge Agent Integration
|
||||||
|
# This config connects the KubeSolo node to a Portainer Business instance
|
||||||
|
# via the Edge Agent for remote management.
|
||||||
|
#
|
||||||
|
# Place at: /mnt/data/etc-kubesolo/cloud-init.yaml
|
||||||
|
|
||||||
|
hostname: edge-node-01
|
||||||
|
|
||||||
|
network:
|
||||||
|
mode: dhcp
|
||||||
|
|
||||||
|
kubesolo:
|
||||||
|
local-storage: true
|
||||||
|
# extra-flags: ""
|
||||||
|
|
||||||
|
# Portainer Edge Agent configuration
|
||||||
|
# After KubeSolo starts, deploy the Edge Agent as a workload
|
||||||
|
portainer:
|
||||||
|
edge-agent:
|
||||||
|
enabled: true
|
||||||
|
# Get these values from Portainer → Environments → Add Environment → Edge Agent
|
||||||
|
edge-id: "your-edge-id-here"
|
||||||
|
edge-key: "your-edge-key-here"
|
||||||
|
portainer-url: "https://portainer.example.com"
|
||||||
|
# Optional: specify Edge Agent version
|
||||||
|
# image: portainer/agent:latest
|
||||||
17
cloud-init/examples/static-ip.yaml
Normal file
17
cloud-init/examples/static-ip.yaml
Normal file
@@ -0,0 +1,17 @@
|
|||||||
|
# KubeSolo OS Cloud-Init — Static IP Configuration
|
||||||
|
# Place at: /mnt/data/etc-kubesolo/cloud-init.yaml
|
||||||
|
|
||||||
|
hostname: kubesolo-edge-01
|
||||||
|
|
||||||
|
network:
|
||||||
|
mode: static
|
||||||
|
interface: eth0
|
||||||
|
address: 192.168.1.100/24
|
||||||
|
gateway: 192.168.1.1
|
||||||
|
dns:
|
||||||
|
- 8.8.8.8
|
||||||
|
- 8.8.4.4
|
||||||
|
|
||||||
|
kubesolo:
|
||||||
|
extra-flags: "--apiserver-extra-sans kubesolo-edge-01.local"
|
||||||
|
local-storage: true
|
||||||
181
docs/boot-flow.md
Normal file
181
docs/boot-flow.md
Normal file
@@ -0,0 +1,181 @@
|
|||||||
|
# KubeSolo OS — Boot Flow
|
||||||
|
|
||||||
|
This document describes the boot sequence from power-on to a running Kubernetes node.
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
```
|
||||||
|
BIOS/UEFI → Bootloader (isolinux) → Linux Kernel → initramfs → /sbin/init
|
||||||
|
→ Stage 00: Mount virtual filesystems
|
||||||
|
→ Stage 10: Parse boot parameters
|
||||||
|
→ Stage 20: Mount persistent storage
|
||||||
|
→ Stage 30: Load kernel modules
|
||||||
|
→ Stage 40: Apply sysctl settings
|
||||||
|
→ Stage 50: Configure networking
|
||||||
|
→ Stage 60: Set hostname
|
||||||
|
→ Stage 70: Set system clock
|
||||||
|
→ Stage 80: Prepare containerd prerequisites
|
||||||
|
→ Stage 90: exec KubeSolo (becomes PID 1)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Stage Details
|
||||||
|
|
||||||
|
### Bootloader (isolinux/syslinux)
|
||||||
|
|
||||||
|
The ISO uses isolinux with several boot options:
|
||||||
|
|
||||||
|
| Label | Description |
|
||||||
|
|-------|-------------|
|
||||||
|
| `kubesolo` | Normal boot (default, 3s timeout) |
|
||||||
|
| `kubesolo-debug` | Boot with verbose init logging + serial console |
|
||||||
|
| `kubesolo-shell` | Drop to emergency shell immediately |
|
||||||
|
| `kubesolo-nopersist` | Run fully in RAM, no persistent mount |
|
||||||
|
|
||||||
|
Kernel command line always includes `kubesolo.data=LABEL=KSOLODATA` to specify the persistent data partition.
|
||||||
|
|
||||||
|
### Stage 00 — Early Mount (`00-early-mount.sh`)
|
||||||
|
|
||||||
|
Mounts essential virtual filesystems before anything else can work:
|
||||||
|
|
||||||
|
- `/proc` — process information
|
||||||
|
- `/sys` — sysfs (device/driver info)
|
||||||
|
- `/dev` — devtmpfs (block devices)
|
||||||
|
- `/tmp`, `/run` — tmpfs scratch space
|
||||||
|
- `/dev/pts`, `/dev/shm` — pseudo-terminals, shared memory
|
||||||
|
- `/sys/fs/cgroup` — cgroup v2 unified hierarchy (v1 fallback if unavailable)
|
||||||
|
|
||||||
|
### Stage 10 — Parse Cmdline (`10-parse-cmdline.sh`)
|
||||||
|
|
||||||
|
Reads `/proc/cmdline` and sets environment variables:
|
||||||
|
|
||||||
|
| Boot Parameter | Variable | Description |
|
||||||
|
|---------------|----------|-------------|
|
||||||
|
| `kubesolo.data=<dev>` | `KUBESOLO_DATA_DEV` | Block device for persistent data |
|
||||||
|
| `kubesolo.debug` | `KUBESOLO_DEBUG` | Enables `set -x` for trace logging |
|
||||||
|
| `kubesolo.shell` | `KUBESOLO_SHELL` | Drop to shell after this stage |
|
||||||
|
| `kubesolo.nopersist` | `KUBESOLO_NOPERSIST` | Skip persistent mount |
|
||||||
|
| `kubesolo.cloudinit=<path>` | `KUBESOLO_CLOUDINIT` | Cloud-init config file path |
|
||||||
|
| `kubesolo.flags=<flags>` | `KUBESOLO_EXTRA_FLAGS` | Extra flags for KubeSolo binary |
|
||||||
|
|
||||||
|
If `kubesolo.data` is not specified, auto-detects a partition with label `KSOLODATA` via `blkid`. If none found, falls back to RAM-only mode.
|
||||||
|
|
||||||
|
### Stage 20 — Persistent Mount (`20-persistent-mount.sh`)
|
||||||
|
|
||||||
|
If not in RAM-only mode:
|
||||||
|
|
||||||
|
1. Waits up to 30s for the data device to appear (handles slow USB, virtio)
|
||||||
|
2. Mounts the ext4 data partition at `/mnt/data`
|
||||||
|
3. Creates directory structure on first boot
|
||||||
|
4. Bind-mounts persistent directories:
|
||||||
|
|
||||||
|
| Source (data partition) | Mount Point | Content |
|
||||||
|
|------------------------|-------------|---------|
|
||||||
|
| `/mnt/data/kubesolo` | `/var/lib/kubesolo` | K8s state, certs, SQLite DB |
|
||||||
|
| `/mnt/data/containerd` | `/var/lib/containerd` | Container images + layers |
|
||||||
|
| `/mnt/data/etc-kubesolo` | `/etc/kubesolo` | Node configuration |
|
||||||
|
| `/mnt/data/log` | `/var/log` | System + K8s logs |
|
||||||
|
| `/mnt/data/usr-local` | `/usr/local` | User binaries |
|
||||||
|
|
||||||
|
In RAM-only mode, these directories are backed by tmpfs and lost on reboot.
|
||||||
|
|
||||||
|
### Stage 30 — Kernel Modules (`30-kernel-modules.sh`)
|
||||||
|
|
||||||
|
Loads kernel modules listed in `/usr/lib/kubesolo-os/modules.list`:
|
||||||
|
|
||||||
|
- `br_netfilter`, `bridge`, `veth`, `vxlan` — K8s pod networking
|
||||||
|
- `ip_tables`, `iptable_nat`, `nf_nat`, `nf_conntrack` — service routing
|
||||||
|
- `overlay` — containerd storage driver
|
||||||
|
- `ip_vs`, `ip_vs_rr`, `ip_vs_wrr`, `ip_vs_sh` — optional IPVS mode
|
||||||
|
|
||||||
|
Modules that fail to load are logged as warnings (may be built into the kernel).
|
||||||
|
|
||||||
|
### Stage 40 — Sysctl (`40-sysctl.sh`)
|
||||||
|
|
||||||
|
Applies kernel parameters from `/etc/sysctl.d/k8s.conf`:
|
||||||
|
|
||||||
|
- `net.bridge.bridge-nf-call-iptables = 1` — K8s requirement
|
||||||
|
- `net.ipv4.ip_forward = 1` — pod-to-pod routing
|
||||||
|
- `fs.inotify.max_user_watches = 524288` — kubelet/containerd watchers
|
||||||
|
- `net.netfilter.nf_conntrack_max = 131072` — service connection tracking
|
||||||
|
- `vm.swappiness = 0` — no swap (K8s requirement)
|
||||||
|
|
||||||
|
### Stage 50 — Network (`50-network.sh`)
|
||||||
|
|
||||||
|
Priority order:
|
||||||
|
1. **Saved config** — `/mnt/data/network/interfaces.sh` (from previous boot)
|
||||||
|
2. **Cloud-init** — parsed from `cloud-init.yaml` (Phase 2: Go parser)
|
||||||
|
3. **DHCP fallback** — `udhcpc` on first non-virtual interface
|
||||||
|
|
||||||
|
Brings up loopback, finds the first physical interface (skipping lo, docker, veth, br, cni), and runs DHCP.
|
||||||
|
|
||||||
|
### Stage 60 — Hostname (`60-hostname.sh`)
|
||||||
|
|
||||||
|
Priority order:
|
||||||
|
1. Saved hostname from data partition
|
||||||
|
2. Generated from MAC address of primary interface (`kubesolo-XXXXXX`)
|
||||||
|
|
||||||
|
Writes to `/etc/hostname` and appends to `/etc/hosts`.
|
||||||
|
|
||||||
|
### Stage 70 — Clock (`70-clock.sh`)
|
||||||
|
|
||||||
|
Best-effort time synchronization:
|
||||||
|
1. Try `hwclock -s` (hardware clock)
|
||||||
|
2. Try NTP in background (non-blocking) via `ntpd` or `ntpdate`
|
||||||
|
3. Log warning if no time source available
|
||||||
|
|
||||||
|
Non-blocking because NTP failure shouldn't prevent boot.
|
||||||
|
|
||||||
|
### Stage 80 — Containerd (`80-containerd.sh`)
|
||||||
|
|
||||||
|
Ensures containerd prerequisites:
|
||||||
|
- Creates `/run/containerd`, `/var/lib/containerd`
|
||||||
|
- Creates CNI directories (`/etc/cni/net.d`, `/opt/cni/bin`)
|
||||||
|
- Loads custom containerd config if present
|
||||||
|
|
||||||
|
KubeSolo manages the actual containerd lifecycle internally.
|
||||||
|
|
||||||
|
### Stage 90 — KubeSolo (`90-kubesolo.sh`)
|
||||||
|
|
||||||
|
Final stage — **exec replaces the init process**:
|
||||||
|
|
||||||
|
1. Verifies `/usr/local/bin/kubesolo` exists
|
||||||
|
2. Builds command line: `--path /var/lib/kubesolo --local-storage true`
|
||||||
|
3. Adds hostname as extra SAN for API server certificate
|
||||||
|
4. Appends any extra flags from boot params or config file
|
||||||
|
5. `exec kubesolo $ARGS` — KubeSolo becomes PID 1
|
||||||
|
|
||||||
|
After this, KubeSolo starts containerd, kubelet, API server, and all K8s components. The node should reach Ready status within 60-120 seconds.
|
||||||
|
|
||||||
|
## Failure Handling
|
||||||
|
|
||||||
|
If any stage returns non-zero, `/sbin/init` calls `emergency_shell()` which:
|
||||||
|
1. Logs the failure to serial console
|
||||||
|
2. Drops to `/bin/sh` for debugging
|
||||||
|
3. User can type `exit` to retry the boot sequence
|
||||||
|
|
||||||
|
If `kubesolo.shell` is passed as a boot parameter, the system drops to shell immediately after Stage 10 (cmdline parsing).
|
||||||
|
|
||||||
|
## Debugging
|
||||||
|
|
||||||
|
### Serial Console
|
||||||
|
|
||||||
|
All init stages log to stderr with the prefix `[kubesolo-init]`. Boot with
|
||||||
|
`console=ttyS0,115200n8` (default in debug mode) to see output on serial.
|
||||||
|
|
||||||
|
### Boot Markers
|
||||||
|
|
||||||
|
Test scripts look for these markers in the serial log:
|
||||||
|
- `[kubesolo-init] [OK] Stage 90-kubesolo.sh complete` — full boot success
|
||||||
|
- `[kubesolo-init] [ERROR]` — stage failure
|
||||||
|
|
||||||
|
### Emergency Shell
|
||||||
|
|
||||||
|
From the emergency shell:
|
||||||
|
```sh
|
||||||
|
dmesg | tail -50 # Kernel messages
|
||||||
|
cat /proc/cmdline # Boot parameters
|
||||||
|
cat /proc/mounts # Current mounts
|
||||||
|
blkid # Block devices and labels
|
||||||
|
ip addr # Network interfaces
|
||||||
|
ls /usr/lib/kubesolo-os/init.d/ # Available init stages
|
||||||
|
```
|
||||||
945
docs/design/kubesolo-os-design.md
Normal file
945
docs/design/kubesolo-os-design.md
Normal file
@@ -0,0 +1,945 @@
|
|||||||
|
# KubeSolo OS — Bootable Immutable Kubernetes Distribution
|
||||||
|
|
||||||
|
## Design Research: KubeSolo + Tiny Core Linux
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Executive Summary
|
||||||
|
|
||||||
|
This document outlines the architecture for **KubeSolo OS** — an immutable, bootable Linux distribution purpose-built to run KubeSolo (Portainer's single-node Kubernetes distribution) with atomic updates. The design combines the minimal footprint of Tiny Core Linux with KubeSolo's single-binary K8s packaging to create an appliance-like Kubernetes node that boots directly into a production-ready cluster.
|
||||||
|
|
||||||
|
**Target use cases:** IoT/IIoT edge devices, single-node K8s appliances, air-gapped deployments, embedded systems, kiosk/POS systems, and resource-constrained hardware.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Component Analysis
|
||||||
|
|
||||||
|
### 2.1 KubeSolo
|
||||||
|
|
||||||
|
**Source:** https://github.com/portainer/kubesolo
|
||||||
|
|
||||||
|
KubeSolo is Portainer's production-ready, ultra-lightweight single-node Kubernetes distribution designed for edge and IoT scenarios.
|
||||||
|
|
||||||
|
**Architecture highlights:**
|
||||||
|
|
||||||
|
- **Single binary** — all K8s components bundled into one executable
|
||||||
|
- **SQLite backend** — uses Kine to replace etcd, eliminating cluster coordination overhead
|
||||||
|
- **Bundled runtime** — ships containerd, runc, CoreDNS, and CNI plugins
|
||||||
|
- **No scheduler** — replaced by a custom `NodeSetter` admission webhook (single-node, no scheduling decisions needed)
|
||||||
|
- **Dual libc support** — detects and supports both glibc and musl (Alpine) environments
|
||||||
|
- **Offline-ready** — designed for air-gapped deployments; all images can be preloaded
|
||||||
|
- **Portainer Edge integration** — optional remote management via `--portainer-edge-*` flags
|
||||||
|
|
||||||
|
**Runtime requirements:**
|
||||||
|
|
||||||
|
| Requirement | Minimum | Recommended |
|
||||||
|
|---|---|---|
|
||||||
|
| RAM | 512 MB | 1 GB+ |
|
||||||
|
| Kernel | 3.10+ (legacy) | 5.8+ (cgroup v2) |
|
||||||
|
| Storage | ~500 MB (binary) | 2 GB+ (with workloads) |
|
||||||
|
|
||||||
|
**Key kernel dependencies:**
|
||||||
|
|
||||||
|
- cgroup v2 (kernel 5.8+) — `CONFIG_CGROUP`, `CONFIG_CGROUP_CPUACCT`, `CONFIG_CGROUP_DEVICE`, `CONFIG_CGROUP_FREEZER`, `CONFIG_CGROUP_SCHED`, `CONFIG_CGROUP_PIDS`, `CONFIG_CGROUP_NET_CLASSID`
|
||||||
|
- Namespaces — `CONFIG_NAMESPACES`, `CONFIG_NET_NS`, `CONFIG_PID_NS`, `CONFIG_USER_NS`, `CONFIG_UTS_NS`, `CONFIG_IPC_NS`
|
||||||
|
- Networking — `CONFIG_BRIDGE`, `CONFIG_NETFILTER`, `CONFIG_VETH`, `CONFIG_VXLAN`, `CONFIG_IP_NF_IPTABLES`, `CONFIG_IP_NF_NAT`
|
||||||
|
- Filesystem — `CONFIG_OVERLAY_FS`, `CONFIG_SQUASHFS`
|
||||||
|
- Modules required at runtime: `br_netfilter`, `overlay`, `ip_tables`, `iptable_nat`, `iptable_filter`
|
||||||
|
|
||||||
|
**Installation & operation:**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Standard install
|
||||||
|
curl -sfL https://get.kubesolo.io | sudo sh -
|
||||||
|
|
||||||
|
# Kubeconfig location
|
||||||
|
/var/lib/kubesolo/pki/admin/admin.kubeconfig
|
||||||
|
|
||||||
|
# Key flags
|
||||||
|
--path /var/lib/kubesolo # config directory
|
||||||
|
--apiserver-extra-sans # additional TLS SANs
|
||||||
|
--local-storage true # enable local-path provisioner
|
||||||
|
--portainer-edge-id # Portainer Edge agent ID
|
||||||
|
--portainer-edge-key # Portainer Edge agent key
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2.2 Tiny Core Linux
|
||||||
|
|
||||||
|
**Source:** http://www.tinycorelinux.net
|
||||||
|
|
||||||
|
Tiny Core Linux is an ultra-minimal Linux distribution (11–17 MB) that runs entirely in RAM.
|
||||||
|
|
||||||
|
**Architecture highlights:**
|
||||||
|
|
||||||
|
- **Micro Core** — 11 MB: kernel + root filesystem + basic kernel modules (no GUI)
|
||||||
|
- **RAM-resident** — entire OS loaded into memory at boot; disk only needed for persistence
|
||||||
|
- **SquashFS root** — read-only compressed filesystem, inherently immutable
|
||||||
|
- **Extension system** — `.tcz` packages (SquashFS-compressed) mounted or copied at boot
|
||||||
|
- **Three operational modes:**
|
||||||
|
1. **Cloud/Default** — pure RAM, nothing persists across reboots
|
||||||
|
2. **Mount mode** — extensions stored in `/tce` directory, loop-mounted at boot
|
||||||
|
3. **Copy mode** — extensions copied into RAM from persistent storage
|
||||||
|
|
||||||
|
**Key concepts for this design:**
|
||||||
|
|
||||||
|
- `/tce` directory on persistent storage holds extensions and configuration
|
||||||
|
- `onboot.lst` — list of extensions to auto-mount at boot
|
||||||
|
- `filetool.sh` + `/opt/.filetool.lst` — backup/restore mechanism for persistent files
|
||||||
|
- Boot codes control behavior: `tce=`, `base`, `norestore`, `noswap`, etc.
|
||||||
|
- Custom remastering: extract `core.gz` → modify → repack → create bootable image
|
||||||
|
- Frugal install: `vmlinuz` + `core.gz` + bootloader + `/tce` directory
|
||||||
|
|
||||||
|
**Kernel:** Ships modern Linux kernel (6.x series in v17.0), supports x86, x86_64, ARM.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Competitive Landscape — Existing Immutable K8s OSes
|
||||||
|
|
||||||
|
### 3.1 Comparison Matrix
|
||||||
|
|
||||||
|
| Feature | Talos Linux | Bottlerocket | Flatcar Linux | Kairos | **KubeSolo OS** (proposed) |
|
||||||
|
|---|---|---|---|---|---|
|
||||||
|
| **Footprint** | ~80 MB | ~500 MB | ~700 MB | Varies (base distro) | **~50–80 MB** |
|
||||||
|
| **Immutability** | Radical (12 binaries) | Strong (read-only root) | Moderate (read-only /usr) | Strong (overlayFS) | **Strong (SquashFS root)** |
|
||||||
|
| **SSH access** | None (API only) | Disabled (container shell) | Yes | Optional | **Optional (extension)** |
|
||||||
|
| **Update model** | A/B partitions | A/B partitions | A/B partitions (ChromeOS) | A/B partitions (OCI) | **A/B partitions** |
|
||||||
|
| **K8s variants** | Multi-node, HA | Multi-node (EKS) | Multi-node (any) | Multi-node (any) | **Single-node only** |
|
||||||
|
| **Management** | talosctl (mTLS API) | API (localhost) | Ignition + SSH | Cloud-init, K8s CRDs | **API + cloud-init** |
|
||||||
|
| **Base OS** | Custom (Go userland) | Custom (Bottlerocket) | Gentoo-derived | Any Linux (meta-distro) | **Tiny Core Linux** |
|
||||||
|
| **Target** | Cloud + Edge | AWS (primarily) | Cloud + Bare metal | Edge + Bare metal | **Edge + IoT** |
|
||||||
|
| **Configuration** | Machine config YAML | TOML settings | Ignition JSON | Cloud-init YAML | **Cloud-init + boot codes** |
|
||||||
|
|
||||||
|
### 3.2 Key Lessons from Each
|
||||||
|
|
||||||
|
**From Talos Linux:**
|
||||||
|
- API-only management is powerful but aggressive — provide as optional mode
|
||||||
|
- 12-binary minimalism is aspirational; KubeSolo's single binary aligns well
|
||||||
|
- System extensions as SquashFS overlays in initramfs = directly applicable to Tiny Core's `.tcz` model
|
||||||
|
- A/B partition with GRUB fallback counter for automatic rollback
|
||||||
|
|
||||||
|
**From Bottlerocket:**
|
||||||
|
- Bootstrap containers for customization — useful pattern for pre-deploying workloads
|
||||||
|
- Host containers for privileged operations (debugging, admin access)
|
||||||
|
- Tightly coupled OS+K8s versions simplifies compatibility testing
|
||||||
|
|
||||||
|
**From Flatcar Linux:**
|
||||||
|
- Ignition for first-boot declarative config — consider cloud-init equivalent
|
||||||
|
- ChromeOS-style update engine is battle-tested
|
||||||
|
- Dynamic kernel module loading — Tiny Core's extension system provides similar flexibility
|
||||||
|
|
||||||
|
**From Kairos:**
|
||||||
|
- Container-based OS distribution (OCI images) — enables `docker pull` for OS updates
|
||||||
|
- P2P mesh clustering via libp2p — interesting for edge fleet bootstrapping
|
||||||
|
- Meta-distribution approach: don't reinvent, augment
|
||||||
|
- Static kernel+initrd shipped in container image = truly atomic full-stack updates
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Architecture Design
|
||||||
|
|
||||||
|
### 4.1 High-Level Architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
┌──────────────────────────────────────────────────────┐
|
||||||
|
│ BOOT MEDIA │
|
||||||
|
│ ┌──────────┐ ┌──────────┐ ┌────────────────────┐ │
|
||||||
|
│ │ GRUB/ │ │ Partition│ │ Partition B │ │
|
||||||
|
│ │ Syslinux │ │ A │ │ (passive) │ │
|
||||||
|
│ │ (EFI/ │ │ (active) │ │ │ │
|
||||||
|
│ │ BIOS) │ │ │ │ vmlinuz │ │
|
||||||
|
│ │ │ │ vmlinuz │ │ kubesolo-os.gz │ │
|
||||||
|
│ │ Fallback │ │ kubesolo-│ │ extensions.tcz │ │
|
||||||
|
│ │ counter │ │ os.gz │ │ │ │
|
||||||
|
│ │ │ │ ext.tcz │ │ │ │
|
||||||
|
│ └──────────┘ └──────────┘ └────────────────────┘ │
|
||||||
|
│ │
|
||||||
|
│ ┌──────────────────────────────────────────────────┐│
|
||||||
|
│ │ Persistent Data Partition ││
|
||||||
|
│ │ /var/lib/kubesolo/ (K8s state, SQLite DB) ││
|
||||||
|
│ │ /var/lib/containerd/ (container images/layers) ││
|
||||||
|
│ │ /etc/kubesolo/ (node config) ││
|
||||||
|
│ │ /var/log/ (logs, optional) ││
|
||||||
|
│ │ /usr/local/ (user data) ││
|
||||||
|
│ └──────────────────────────────────────────────────┘│
|
||||||
|
└──────────────────────────────────────────────────────┘
|
||||||
|
|
||||||
|
BOOT FLOW
|
||||||
|
│
|
||||||
|
┌────────────▼────────────┐
|
||||||
|
│ GRUB loads vmlinuz + │
|
||||||
|
│ kubesolo-os.gz from │
|
||||||
|
│ active partition │
|
||||||
|
└────────────┬────────────┘
|
||||||
|
│
|
||||||
|
┌────────────▼────────────┐
|
||||||
|
│ Kernel boots, mounts │
|
||||||
|
│ SquashFS root (ro) │
|
||||||
|
│ in RAM │
|
||||||
|
└────────────┬────────────┘
|
||||||
|
│
|
||||||
|
┌────────────▼────────────┐
|
||||||
|
│ init: mount persistent │
|
||||||
|
│ partition, bind-mount │
|
||||||
|
│ writable paths │
|
||||||
|
└────────────┬────────────┘
|
||||||
|
│
|
||||||
|
┌────────────▼────────────┐
|
||||||
|
│ Load kernel modules: │
|
||||||
|
│ br_netfilter, overlay, │
|
||||||
|
│ ip_tables, veth │
|
||||||
|
└────────────┬────────────┘
|
||||||
|
│
|
||||||
|
┌────────────▼────────────┐
|
||||||
|
│ Configure networking │
|
||||||
|
│ (cloud-init or static) │
|
||||||
|
└────────────┬────────────┘
|
||||||
|
│
|
||||||
|
┌────────────▼────────────┐
|
||||||
|
│ Start KubeSolo │
|
||||||
|
│ (single binary) │
|
||||||
|
└────────────┬────────────┘
|
||||||
|
│
|
||||||
|
┌────────────▼────────────┐
|
||||||
|
│ K8s API available │
|
||||||
|
│ Node ready for │
|
||||||
|
│ workloads │
|
||||||
|
└─────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4.2 Partition Layout
|
||||||
|
|
||||||
|
```
|
||||||
|
Disk Layout (minimum 8 GB recommended):
|
||||||
|
|
||||||
|
┌──────────────────────────────────────────────────────────┐
|
||||||
|
│ Partition 1: EFI/Boot (256 MB, FAT32) │
|
||||||
|
│ /EFI/BOOT/bootx64.efi (or /boot/grub for BIOS) │
|
||||||
|
│ grub.cfg with A/B logic + fallback counter │
|
||||||
|
├──────────────────────────────────────────────────────────┤
|
||||||
|
│ Partition 2: System A (512 MB, SquashFS image, read-only)│
|
||||||
|
│ vmlinuz │
|
||||||
|
│ kubesolo-os.gz (initramfs: core.gz + KubeSolo ext) │
|
||||||
|
├──────────────────────────────────────────────────────────┤
|
||||||
|
│ Partition 3: System B (512 MB, SquashFS image, read-only)│
|
||||||
|
│ (passive — receives updates, swaps with A) │
|
||||||
|
├──────────────────────────────────────────────────────────┤
|
||||||
|
│ Partition 4: Persistent Data (remaining space, ext4) │
|
||||||
|
│ /var/lib/kubesolo/ → K8s state, certs, SQLite │
|
||||||
|
│ /var/lib/containerd/ → container images & layers │
|
||||||
|
│ /etc/kubesolo/ → node configuration │
|
||||||
|
│ /etc/network/ → network config │
|
||||||
|
│ /var/log/ → system + K8s logs │
|
||||||
|
│ /usr/local/ → user extensions │
|
||||||
|
└──────────────────────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4.3 Filesystem Mount Strategy
|
||||||
|
|
||||||
|
At boot, the init system constructs the runtime filesystem:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Root: SquashFS from initramfs (read-only, in RAM)
|
||||||
|
/ → tmpfs (RAM) + SquashFS overlay (ro)
|
||||||
|
|
||||||
|
# Persistent bind mounts from data partition
|
||||||
|
/var/lib/kubesolo → /mnt/data/kubesolo (rw)
|
||||||
|
/var/lib/containerd → /mnt/data/containerd (rw)
|
||||||
|
/etc/kubesolo → /mnt/data/etc-kubesolo (rw)
|
||||||
|
/etc/resolv.conf → /mnt/data/resolv.conf (rw)
|
||||||
|
/var/log → /mnt/data/log (rw)
|
||||||
|
/usr/local → /mnt/data/usr-local (rw)
|
||||||
|
|
||||||
|
# Everything else: read-only or tmpfs
|
||||||
|
/tmp → tmpfs
|
||||||
|
/run → tmpfs
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4.4 Custom Initramfs (kubesolo-os.gz)
|
||||||
|
|
||||||
|
The initramfs is the core of the distribution — a remastered Tiny Core `core.gz` with KubeSolo baked in:
|
||||||
|
|
||||||
|
```
|
||||||
|
kubesolo-os.gz (cpio+gzip archive)
|
||||||
|
├── bin/ # BusyBox symlinks
|
||||||
|
├── sbin/
|
||||||
|
│ └── init # Custom init script (see §4.5)
|
||||||
|
├── lib/
|
||||||
|
│ └── modules/ # Kernel modules (br_netfilter, overlay, etc.)
|
||||||
|
├── usr/
|
||||||
|
│ └── local/
|
||||||
|
│ └── bin/
|
||||||
|
│ └── kubesolo # KubeSolo binary
|
||||||
|
├── opt/
|
||||||
|
│ ├── containerd/ # containerd + runc + CNI plugins
|
||||||
|
│ │ ├── bin/
|
||||||
|
│ │ │ ├── containerd
|
||||||
|
│ │ │ ├── containerd-shim-runc-v2
|
||||||
|
│ │ │ └── runc
|
||||||
|
│ │ └── cni/
|
||||||
|
│ │ └── bin/ # CNI plugins (bridge, host-local, loopback, etc.)
|
||||||
|
│ └── kubesolo-os/
|
||||||
|
│ ├── cloud-init.yaml # Default cloud-init config
|
||||||
|
│ └── update-agent # Atomic update agent binary
|
||||||
|
├── etc/
|
||||||
|
│ ├── os-release # KubeSolo OS identification
|
||||||
|
│ ├── kubesolo/
|
||||||
|
│ │ └── config.yaml # Default KubeSolo config
|
||||||
|
│ └── sysctl.d/
|
||||||
|
│ └── k8s.conf # Kernel parameters for K8s
|
||||||
|
└── var/
|
||||||
|
└── lib/
|
||||||
|
└── kubesolo/ # Mount point (bind-mounted to persistent)
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4.5 Init System
|
||||||
|
|
||||||
|
A custom init script replaces Tiny Core's default init to implement the appliance boot flow:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
#!/bin/sh
|
||||||
|
# /sbin/init — KubeSolo OS init
|
||||||
|
set -e
|
||||||
|
|
||||||
|
# 1. Mount essential filesystems
|
||||||
|
mount -t proc proc /proc
|
||||||
|
mount -t sysfs sysfs /sys
|
||||||
|
mount -t devtmpfs devtmpfs /dev
|
||||||
|
mount -t tmpfs tmpfs /tmp
|
||||||
|
mount -t tmpfs tmpfs /run
|
||||||
|
mkdir -p /dev/pts /dev/shm
|
||||||
|
mount -t devpts devpts /dev/pts
|
||||||
|
mount -t tmpfs tmpfs /dev/shm
|
||||||
|
|
||||||
|
# 2. Parse boot parameters
|
||||||
|
PERSISTENT_DEV=""
|
||||||
|
for arg in $(cat /proc/cmdline); do
|
||||||
|
case "$arg" in
|
||||||
|
kubesolo.data=*) PERSISTENT_DEV="${arg#kubesolo.data=}" ;;
|
||||||
|
kubesolo.debug) set -x ;;
|
||||||
|
kubesolo.shell) exec /bin/sh ;; # Emergency shell
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
|
||||||
|
# 3. Mount persistent data partition
|
||||||
|
if [ -n "$PERSISTENT_DEV" ]; then
|
||||||
|
mkdir -p /mnt/data
|
||||||
|
# Wait for device (USB, slow disks)
|
||||||
|
for i in $(seq 1 30); do
|
||||||
|
[ -b "$PERSISTENT_DEV" ] && break
|
||||||
|
sleep 1
|
||||||
|
done
|
||||||
|
mount -t ext4 "$PERSISTENT_DEV" /mnt/data
|
||||||
|
|
||||||
|
# Create directory structure on first boot
|
||||||
|
for dir in kubesolo containerd etc-kubesolo log usr-local network; do
|
||||||
|
mkdir -p /mnt/data/$dir
|
||||||
|
done
|
||||||
|
|
||||||
|
# Bind mount persistent paths
|
||||||
|
mount --bind /mnt/data/kubesolo /var/lib/kubesolo
|
||||||
|
mount --bind /mnt/data/containerd /var/lib/containerd
|
||||||
|
mount --bind /mnt/data/etc-kubesolo /etc/kubesolo
|
||||||
|
mount --bind /mnt/data/log /var/log
|
||||||
|
mount --bind /mnt/data/usr-local /usr/local
|
||||||
|
fi
|
||||||
|
|
||||||
|
# 4. Load required kernel modules
|
||||||
|
modprobe br_netfilter
|
||||||
|
modprobe overlay
|
||||||
|
modprobe ip_tables
|
||||||
|
modprobe iptable_nat
|
||||||
|
modprobe iptable_filter
|
||||||
|
modprobe veth
|
||||||
|
modprobe vxlan
|
||||||
|
|
||||||
|
# 5. Set kernel parameters
|
||||||
|
sysctl -w net.bridge.bridge-nf-call-iptables=1
|
||||||
|
sysctl -w net.bridge.bridge-nf-call-ip6tables=1
|
||||||
|
sysctl -w net.ipv4.ip_forward=1
|
||||||
|
sysctl -w fs.inotify.max_user_instances=1024
|
||||||
|
sysctl -w fs.inotify.max_user_watches=524288
|
||||||
|
|
||||||
|
# 6. Configure networking
|
||||||
|
# Priority: cloud-init > persistent config > DHCP fallback
|
||||||
|
if [ -f /mnt/data/network/interfaces ]; then
|
||||||
|
# Apply saved network config
|
||||||
|
configure_network /mnt/data/network/interfaces
|
||||||
|
elif [ -f /mnt/data/etc-kubesolo/cloud-init.yaml ]; then
|
||||||
|
# First boot: apply cloud-init
|
||||||
|
apply_cloud_init /mnt/data/etc-kubesolo/cloud-init.yaml
|
||||||
|
else
|
||||||
|
# Fallback: DHCP on first interface
|
||||||
|
udhcpc -i eth0 -s /usr/share/udhcpc/default.script
|
||||||
|
fi
|
||||||
|
|
||||||
|
# 7. Set hostname
|
||||||
|
if [ -f /mnt/data/etc-kubesolo/hostname ]; then
|
||||||
|
hostname $(cat /mnt/data/etc-kubesolo/hostname)
|
||||||
|
else
|
||||||
|
hostname kubesolo-$(cat /sys/class/net/eth0/address | tr -d ':' | tail -c 6)
|
||||||
|
fi
|
||||||
|
|
||||||
|
# 8. Start containerd
|
||||||
|
containerd --config /etc/kubesolo/containerd-config.toml &
|
||||||
|
sleep 2 # Wait for socket
|
||||||
|
|
||||||
|
# 9. Start KubeSolo
|
||||||
|
exec /usr/local/bin/kubesolo \
|
||||||
|
--path /var/lib/kubesolo \
|
||||||
|
--local-storage true \
|
||||||
|
$(cat /etc/kubesolo/extra-flags 2>/dev/null || true)
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4.6 Atomic Update System
|
||||||
|
|
||||||
|
#### Update Flow
|
||||||
|
|
||||||
|
```
|
||||||
|
UPDATE PROCESS
|
||||||
|
│
|
||||||
|
┌─────────────▼──────────────┐
|
||||||
|
│ 1. Download new OS image │
|
||||||
|
│ (kubesolo-os-v2.img) │
|
||||||
|
│ Verify checksum + sig │
|
||||||
|
└─────────────┬──────────────┘
|
||||||
|
│
|
||||||
|
┌─────────────▼──────────────┐
|
||||||
|
│ 2. Write image to PASSIVE │
|
||||||
|
│ partition (B if A active) │
|
||||||
|
└─────────────┬──────────────┘
|
||||||
|
│
|
||||||
|
┌─────────────▼──────────────┐
|
||||||
|
│ 3. Update GRUB: │
|
||||||
|
│ - Set next boot → B │
|
||||||
|
│ - Set boot_counter = 3 │
|
||||||
|
└─────────────┬──────────────┘
|
||||||
|
│
|
||||||
|
┌─────────────▼──────────────┐
|
||||||
|
│ 4. Reboot │
|
||||||
|
└─────────────┬──────────────┘
|
||||||
|
│
|
||||||
|
┌─────────────▼──────────────┐
|
||||||
|
│ 5. GRUB boots partition B │
|
||||||
|
│ Decrements boot_counter │
|
||||||
|
└─────────────┬──────────────┘
|
||||||
|
│
|
||||||
|
┌──────────┴──────────┐
|
||||||
|
│ │
|
||||||
|
┌─────▼─────┐ ┌─────▼─────┐
|
||||||
|
│ Boot OK │ │ Boot FAIL │
|
||||||
|
│ │ │ │
|
||||||
|
│ Health │ │ Counter │
|
||||||
|
│ check OK │ │ hits 0 │
|
||||||
|
│ │ │ │
|
||||||
|
│ Mark B as │ │ GRUB auto │
|
||||||
|
│ default │ │ rollback │
|
||||||
|
│ Clear │ │ to A │
|
||||||
|
│ counter │ │ │
|
||||||
|
└───────────┘ └───────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
#### GRUB Configuration for A/B Boot
|
||||||
|
|
||||||
|
```grub
|
||||||
|
# /boot/grub/grub.cfg
|
||||||
|
|
||||||
|
set default=0
|
||||||
|
set timeout=3
|
||||||
|
|
||||||
|
# Saved environment variables:
|
||||||
|
# active_slot = A or B
|
||||||
|
# boot_counter = 3 (decremented each boot, 0 = rollback)
|
||||||
|
# boot_success = 0 (set to 1 by health check)
|
||||||
|
|
||||||
|
load_env
|
||||||
|
|
||||||
|
# If last boot failed and counter expired, swap slots
|
||||||
|
if [ "${boot_success}" != "1" ]; then
|
||||||
|
if [ "${boot_counter}" = "0" ]; then
|
||||||
|
if [ "${active_slot}" = "A" ]; then
|
||||||
|
set active_slot=B
|
||||||
|
else
|
||||||
|
set active_slot=A
|
||||||
|
fi
|
||||||
|
save_env active_slot
|
||||||
|
set boot_counter=3
|
||||||
|
save_env boot_counter
|
||||||
|
else
|
||||||
|
# Decrement counter
|
||||||
|
if [ "${boot_counter}" = "3" ]; then set boot_counter=2; fi
|
||||||
|
if [ "${boot_counter}" = "2" ]; then set boot_counter=1; fi
|
||||||
|
if [ "${boot_counter}" = "1" ]; then set boot_counter=0; fi
|
||||||
|
save_env boot_counter
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
set boot_success=0
|
||||||
|
save_env boot_success
|
||||||
|
|
||||||
|
# Boot from active slot
|
||||||
|
if [ "${active_slot}" = "A" ]; then
|
||||||
|
set root=(hd0,gpt2)
|
||||||
|
else
|
||||||
|
set root=(hd0,gpt3)
|
||||||
|
fi
|
||||||
|
|
||||||
|
menuentry "KubeSolo OS" {
|
||||||
|
linux /vmlinuz kubesolo.data=/dev/sda4 quiet
|
||||||
|
initrd /kubesolo-os.gz
|
||||||
|
}
|
||||||
|
|
||||||
|
menuentry "KubeSolo OS (emergency shell)" {
|
||||||
|
linux /vmlinuz kubesolo.data=/dev/sda4 kubesolo.shell
|
||||||
|
initrd /kubesolo-os.gz
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Update Agent
|
||||||
|
|
||||||
|
A lightweight Go binary that runs as a Kubernetes CronJob or DaemonSet:
|
||||||
|
|
||||||
|
```
|
||||||
|
kubesolo-update-agent responsibilities:
|
||||||
|
1. Poll update server (HTTPS) or watch OCI registry for new tags
|
||||||
|
2. Download + verify new system image (SHA256 + optional GPG signature)
|
||||||
|
3. Write to passive partition (dd or equivalent)
|
||||||
|
4. Update GRUB environment (grub-editenv)
|
||||||
|
5. Trigger reboot (via Kubernetes node drain → reboot)
|
||||||
|
6. Post-boot health check:
|
||||||
|
- KubeSolo API reachable?
|
||||||
|
- containerd healthy?
|
||||||
|
- Node Ready in kubectl?
|
||||||
|
If all pass → set boot_success=1
|
||||||
|
If any fail → leave boot_success=0 (auto-rollback on next reboot)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Update distribution models:**
|
||||||
|
|
||||||
|
1. **HTTP/S server** — host images on a simple file server; agent polls for `latest.json`
|
||||||
|
2. **OCI registry** — tag system images as container images; agent pulls new tags
|
||||||
|
3. **USB drive** — for air-gapped: plug USB with new image, agent detects and applies
|
||||||
|
4. **Portainer Edge** — leverage existing Portainer Edge infrastructure for fleet updates
|
||||||
|
|
||||||
|
### 4.7 Configuration System
|
||||||
|
|
||||||
|
#### First Boot (cloud-init)
|
||||||
|
|
||||||
|
The system uses a simplified cloud-init compatible with Tiny Core's environment:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
# /etc/kubesolo/cloud-init.yaml (placed on data partition before first boot)
|
||||||
|
#cloud-config
|
||||||
|
|
||||||
|
hostname: edge-node-001
|
||||||
|
|
||||||
|
network:
|
||||||
|
version: 2
|
||||||
|
ethernets:
|
||||||
|
eth0:
|
||||||
|
dhcp4: false
|
||||||
|
addresses:
|
||||||
|
- 192.168.1.100/24
|
||||||
|
gateway4: 192.168.1.1
|
||||||
|
nameservers:
|
||||||
|
addresses:
|
||||||
|
- 8.8.8.8
|
||||||
|
- 1.1.1.1
|
||||||
|
|
||||||
|
kubesolo:
|
||||||
|
extra-sans:
|
||||||
|
- edge-node-001.local
|
||||||
|
- 192.168.1.100
|
||||||
|
local-storage: true
|
||||||
|
portainer:
|
||||||
|
edge-id: "your-edge-id"
|
||||||
|
edge-key: "your-edge-key"
|
||||||
|
|
||||||
|
ssh:
|
||||||
|
enabled: false # Set true to enable SSH extension
|
||||||
|
authorized_keys:
|
||||||
|
- "ssh-rsa AAAA..."
|
||||||
|
|
||||||
|
ntp:
|
||||||
|
servers:
|
||||||
|
- pool.ntp.org
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Runtime Configuration
|
||||||
|
|
||||||
|
Post-boot configuration changes via Kubernetes API:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Access from the node (kubeconfig is at known path)
|
||||||
|
export KUBECONFIG=/var/lib/kubesolo/pki/admin/admin.kubeconfig
|
||||||
|
kubectl get nodes
|
||||||
|
kubectl apply -f workload.yaml
|
||||||
|
|
||||||
|
# Remote access via Portainer Edge or direct API
|
||||||
|
# (if apiserver-extra-sans includes remote IP/DNS)
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Build Process
|
||||||
|
|
||||||
|
### 5.1 Build Pipeline
|
||||||
|
|
||||||
|
```
|
||||||
|
BUILD PIPELINE
|
||||||
|
|
||||||
|
┌─────────────────────┐ ┌──────────────────────┐
|
||||||
|
│ 1. Fetch Tiny Core │────▶│ 2. Extract core.gz │
|
||||||
|
│ Micro Core ISO │ │ (cpio -idmv) │
|
||||||
|
└─────────────────────┘ └──────────┬───────────┘
|
||||||
|
│
|
||||||
|
┌──────────▼───────────┐
|
||||||
|
│ 3. Inject KubeSolo │
|
||||||
|
│ binary + deps │
|
||||||
|
│ (containerd, runc,│
|
||||||
|
│ CNI, modules) │
|
||||||
|
└──────────┬───────────┘
|
||||||
|
│
|
||||||
|
┌──────────▼───────────┐
|
||||||
|
│ 4. Replace /sbin/init│
|
||||||
|
│ with custom init │
|
||||||
|
└──────────┬───────────┘
|
||||||
|
│
|
||||||
|
┌──────────▼───────────┐
|
||||||
|
│ 5. Repack initramfs │
|
||||||
|
│ (find . | cpio -o │
|
||||||
|
│ | gzip > ks-os.gz│
|
||||||
|
└──────────┬───────────┘
|
||||||
|
│
|
||||||
|
┌──────────▼───────────┐
|
||||||
|
│ 6. Verify kernel has │
|
||||||
|
│ required configs │
|
||||||
|
│ (cgroup v2, ns, │
|
||||||
|
│ netfilter, etc.) │
|
||||||
|
└──────────┬───────────┘
|
||||||
|
│
|
||||||
|
┌────────────────────────┼────────────────────┐
|
||||||
|
│ │ │
|
||||||
|
┌─────────▼─────────┐ ┌─────────▼─────────┐ ┌──────▼───────┐
|
||||||
|
│ 7a. Create ISO │ │ 7b. Create raw │ │ 7c. Create │
|
||||||
|
│ (bootable │ │ disk image │ │ OCI │
|
||||||
|
│ media) │ │ (dd to disk) │ │ image │
|
||||||
|
└───────────────────┘ └───────────────────┘ └──────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
### 5.2 Build Script (Skeleton)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
#!/bin/bash
|
||||||
|
# build-kubesolo-os.sh
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
VERSION="${1:?Usage: $0 <version>}"
|
||||||
|
WORK_DIR="$(mktemp -d)"
|
||||||
|
OUTPUT_DIR="./output"
|
||||||
|
|
||||||
|
# --- 1. Download components ---
|
||||||
|
echo "==> Downloading Tiny Core Micro Core..."
|
||||||
|
wget -q "http://www.tinycorelinux.net/17.x/x86_64/release/CorePure64-17.0.iso" \
|
||||||
|
-O "$WORK_DIR/core.iso"
|
||||||
|
|
||||||
|
echo "==> Downloading KubeSolo..."
|
||||||
|
curl -sfL https://get.kubesolo.io -o "$WORK_DIR/install-kubesolo.sh"
|
||||||
|
# Or: download specific release binary from GitHub
|
||||||
|
|
||||||
|
# --- 2. Extract Tiny Core ---
|
||||||
|
mkdir -p "$WORK_DIR/iso" "$WORK_DIR/rootfs"
|
||||||
|
mount -o loop "$WORK_DIR/core.iso" "$WORK_DIR/iso"
|
||||||
|
cp "$WORK_DIR/iso/boot/vmlinuz64" "$WORK_DIR/vmlinuz"
|
||||||
|
cd "$WORK_DIR/rootfs"
|
||||||
|
zcat "$WORK_DIR/iso/boot/corepure64.gz" | cpio -idmv 2>/dev/null
|
||||||
|
umount "$WORK_DIR/iso"
|
||||||
|
|
||||||
|
# --- 3. Inject KubeSolo + dependencies ---
|
||||||
|
# KubeSolo binary
|
||||||
|
mkdir -p usr/local/bin
|
||||||
|
cp /path/to/kubesolo usr/local/bin/kubesolo
|
||||||
|
chmod +x usr/local/bin/kubesolo
|
||||||
|
|
||||||
|
# containerd + runc + CNI (extracted from KubeSolo bundle or downloaded separately)
|
||||||
|
mkdir -p opt/cni/bin
|
||||||
|
# ... copy containerd, runc, CNI plugins
|
||||||
|
|
||||||
|
# Required kernel modules (if not already in core.gz)
|
||||||
|
# ... may need to compile or extract from Tiny Core extensions
|
||||||
|
|
||||||
|
# --- 4. Custom init ---
|
||||||
|
cat > sbin/init << 'INIT'
|
||||||
|
#!/bin/sh
|
||||||
|
# ... (init script from §4.5)
|
||||||
|
INIT
|
||||||
|
chmod +x sbin/init
|
||||||
|
|
||||||
|
# --- 5. Sysctl + OS metadata ---
|
||||||
|
mkdir -p etc/sysctl.d
|
||||||
|
cat > etc/sysctl.d/k8s.conf << EOF
|
||||||
|
net.bridge.bridge-nf-call-iptables = 1
|
||||||
|
net.bridge.bridge-nf-call-ip6tables = 1
|
||||||
|
net.ipv4.ip_forward = 1
|
||||||
|
fs.inotify.max_user_instances = 1024
|
||||||
|
fs.inotify.max_user_watches = 524288
|
||||||
|
EOF
|
||||||
|
|
||||||
|
cat > etc/os-release << EOF
|
||||||
|
NAME="KubeSolo OS"
|
||||||
|
VERSION="$VERSION"
|
||||||
|
ID=kubesolo-os
|
||||||
|
VERSION_ID=$VERSION
|
||||||
|
PRETTY_NAME="KubeSolo OS $VERSION"
|
||||||
|
HOME_URL="https://github.com/portainer/kubesolo"
|
||||||
|
EOF
|
||||||
|
|
||||||
|
# --- 6. Repack initramfs ---
|
||||||
|
find . | cpio -o -H newc 2>/dev/null | gzip -9 > "$WORK_DIR/kubesolo-os.gz"
|
||||||
|
|
||||||
|
# --- 7. Create disk image ---
|
||||||
|
mkdir -p "$OUTPUT_DIR"
|
||||||
|
create_disk_image "$WORK_DIR/vmlinuz" "$WORK_DIR/kubesolo-os.gz" \
|
||||||
|
"$OUTPUT_DIR/kubesolo-os-${VERSION}.img"
|
||||||
|
|
||||||
|
echo "==> Built: $OUTPUT_DIR/kubesolo-os-${VERSION}.img"
|
||||||
|
```
|
||||||
|
|
||||||
|
### 5.3 Alternative: Kairos-based Build (Container-first)
|
||||||
|
|
||||||
|
For faster iteration, leverage the Kairos framework to get A/B updates, P2P mesh, and OCI distribution for free:
|
||||||
|
|
||||||
|
```dockerfile
|
||||||
|
# Dockerfile.kubesolo-os
|
||||||
|
FROM quay.io/kairos/core-alpine:latest
|
||||||
|
|
||||||
|
# Install KubeSolo
|
||||||
|
RUN curl -sfL https://get.kubesolo.io | sh -
|
||||||
|
|
||||||
|
# Pre-configure
|
||||||
|
COPY kubesolo-config.yaml /etc/kubesolo/config.yaml
|
||||||
|
COPY cloud-init-defaults.yaml /system/oem/
|
||||||
|
|
||||||
|
# Kernel modules
|
||||||
|
RUN apk add --no-cache \
|
||||||
|
linux-lts \
|
||||||
|
iptables \
|
||||||
|
iproute2 \
|
||||||
|
conntrack-tools
|
||||||
|
|
||||||
|
# Sysctl
|
||||||
|
COPY k8s-sysctl.conf /etc/sysctl.d/
|
||||||
|
|
||||||
|
# Build: docker build -t kubesolo-os:v1 .
|
||||||
|
# Flash: Use AuroraBoot or Kairos tooling to convert OCI → bootable image
|
||||||
|
```
|
||||||
|
|
||||||
|
**Advantages of Kairos approach:**
|
||||||
|
- A/B atomic updates with rollback — built-in
|
||||||
|
- OCI-based distribution — `docker push` your OS
|
||||||
|
- P2P mesh bootstrapping — nodes find each other
|
||||||
|
- Kubernetes-native upgrades — `kubectl apply` to upgrade the OS
|
||||||
|
- Proven in production edge deployments
|
||||||
|
|
||||||
|
**Trade-offs:**
|
||||||
|
- Larger footprint than pure Tiny Core remaster (~200–400 MB vs ~50–80 MB)
|
||||||
|
- Dependency on Kairos project maintenance
|
||||||
|
- Less control over boot process internals
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Kernel Considerations
|
||||||
|
|
||||||
|
### 6.1 Tiny Core Kernel Audit
|
||||||
|
|
||||||
|
Tiny Core 17.0 ships a modern 6.x kernel, which should have cgroup v2 support compiled in. However, the **kernel config must be verified** for these critical options:
|
||||||
|
|
||||||
|
```
|
||||||
|
# MANDATORY for KubeSolo
|
||||||
|
CONFIG_CGROUPS=y
|
||||||
|
CONFIG_CGROUP_CPUACCT=y
|
||||||
|
CONFIG_CGROUP_DEVICE=y
|
||||||
|
CONFIG_CGROUP_FREEZER=y
|
||||||
|
CONFIG_CGROUP_SCHED=y
|
||||||
|
CONFIG_CGROUP_PIDS=y
|
||||||
|
CONFIG_MEMCG=y
|
||||||
|
CONFIG_CGROUP_BPF=y
|
||||||
|
|
||||||
|
CONFIG_NAMESPACES=y
|
||||||
|
CONFIG_NET_NS=y
|
||||||
|
CONFIG_PID_NS=y
|
||||||
|
CONFIG_USER_NS=y
|
||||||
|
CONFIG_UTS_NS=y
|
||||||
|
CONFIG_IPC_NS=y
|
||||||
|
|
||||||
|
CONFIG_OVERLAY_FS=y # or =m (module)
|
||||||
|
CONFIG_BRIDGE=y # or =m
|
||||||
|
CONFIG_NETFILTER=y
|
||||||
|
CONFIG_NF_NAT=y
|
||||||
|
CONFIG_IP_NF_IPTABLES=y
|
||||||
|
CONFIG_IP_NF_NAT=y
|
||||||
|
CONFIG_IP_NF_FILTER=y
|
||||||
|
CONFIG_VETH=y # or =m
|
||||||
|
CONFIG_VXLAN=y # or =m
|
||||||
|
|
||||||
|
CONFIG_SQUASHFS=y # For Tiny Core's own extension system
|
||||||
|
CONFIG_BLK_DEV_LOOP=y # For SquashFS mounting
|
||||||
|
|
||||||
|
# RECOMMENDED
|
||||||
|
CONFIG_BPF_SYSCALL=y # For modern CNI plugins
|
||||||
|
CONFIG_CRYPTO_SHA256=y # For image verification
|
||||||
|
CONFIG_SECCOMP=y # Container security
|
||||||
|
CONFIG_AUDIT=y # Audit logging
|
||||||
|
```
|
||||||
|
|
||||||
|
If the stock Tiny Core kernel lacks any of these, options are:
|
||||||
|
|
||||||
|
1. **Load as modules** — if compiled as `=m`, load via `modprobe` in init
|
||||||
|
2. **Recompile kernel** — use Tiny Core's kernel build process with custom config
|
||||||
|
3. **Use a different kernel** — e.g., pull the kernel from Alpine Linux or build from mainline
|
||||||
|
|
||||||
|
### 6.2 Custom Kernel Build (if needed)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# On a Tiny Core build system
|
||||||
|
tce-load -wi compiletc linux-6.x-source
|
||||||
|
cd /usr/src/linux-6.x
|
||||||
|
cp /path/to/kubesolo-kernel.config .config
|
||||||
|
make oldconfig
|
||||||
|
make -j$(nproc) bzImage modules
|
||||||
|
# Extract vmlinuz and required modules
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. Security Model
|
||||||
|
|
||||||
|
### 7.1 Layered Security
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────────────────────────────────────┐
|
||||||
|
│ APPLICATION LAYER │
|
||||||
|
│ Kubernetes RBAC + Network Policies │
|
||||||
|
│ Pod Security Standards │
|
||||||
|
│ Seccomp / AppArmor profiles │
|
||||||
|
├─────────────────────────────────────────┤
|
||||||
|
│ CONTAINER RUNTIME LAYER │
|
||||||
|
│ containerd with default seccomp │
|
||||||
|
│ Read-only container rootfs │
|
||||||
|
│ User namespace mapping (optional) │
|
||||||
|
├─────────────────────────────────────────┤
|
||||||
|
│ OS LAYER │
|
||||||
|
│ SquashFS root (read-only, in RAM) │
|
||||||
|
│ No package manager │
|
||||||
|
│ No SSH by default │
|
||||||
|
│ Minimal userland (BusyBox only) │
|
||||||
|
│ No compiler, no debugger │
|
||||||
|
├─────────────────────────────────────────┤
|
||||||
|
│ BOOT LAYER │
|
||||||
|
│ Signed images (GPG verification) │
|
||||||
|
│ Secure Boot (optional, UEFI) │
|
||||||
|
│ A/B rollback on tamper/failure │
|
||||||
|
│ TPM-based attestation (optional) │
|
||||||
|
└─────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
### 7.2 Attack Surface Comparison
|
||||||
|
|
||||||
|
| Attack Vector | Traditional Linux | KubeSolo OS |
|
||||||
|
|---|---|---|
|
||||||
|
| Package manager exploit | Possible (apt/yum) | **Eliminated** (no pkg manager) |
|
||||||
|
| SSH brute force | Common | **Eliminated** (no SSH default) |
|
||||||
|
| Writable system files | Yes (/etc, /usr) | **Eliminated** (SquashFS ro) |
|
||||||
|
| Persistent rootkit | Survives reboot | **Eliminated** (RAM-only root) |
|
||||||
|
| Kernel module injection | Possible | **Mitigated** (only preloaded modules) |
|
||||||
|
| Local privilege escalation | Various paths | **Reduced** (minimal binaries) |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. Implementation Roadmap
|
||||||
|
|
||||||
|
### Phase 1 — Proof of Concept (2–3 weeks)
|
||||||
|
|
||||||
|
**Goal:** Boot Tiny Core + KubeSolo, validate K8s functionality.
|
||||||
|
|
||||||
|
1. Download Tiny Core Micro Core 17.0 (x86_64)
|
||||||
|
2. Extract `core.gz`, inject KubeSolo binary
|
||||||
|
3. Create custom init that starts KubeSolo
|
||||||
|
4. Verify kernel has required configs (cgroup v2, namespaces, netfilter)
|
||||||
|
5. Build bootable ISO, test in QEMU/KVM
|
||||||
|
6. Deploy a test workload (nginx pod)
|
||||||
|
7. Validate: `kubectl get nodes` shows Ready
|
||||||
|
|
||||||
|
**Success criteria:** Single ISO boots to functional K8s node in < 30 seconds.
|
||||||
|
|
||||||
|
### Phase 2 — Persistence + Immutability (2–3 weeks)
|
||||||
|
|
||||||
|
**Goal:** Persistent K8s state across reboots, immutable root.
|
||||||
|
|
||||||
|
1. Implement persistent data partition with bind mounts
|
||||||
|
2. Verify K8s state survives reboot (pods, services, PVCs)
|
||||||
|
3. Verify SQLite DB integrity across unclean shutdowns
|
||||||
|
4. Lock down root filesystem (verify read-only enforcement)
|
||||||
|
5. Test: corrupt system files → verify RAM-only root is unaffected
|
||||||
|
|
||||||
|
### Phase 3 — Atomic Updates + Rollback (3–4 weeks)
|
||||||
|
|
||||||
|
**Goal:** A/B partition updates with automatic rollback.
|
||||||
|
|
||||||
|
1. Implement GRUB A/B boot configuration
|
||||||
|
2. Build update agent (Go binary)
|
||||||
|
3. Implement health check + `boot_success` flag
|
||||||
|
4. Test update cycle: A → B → verify → mark good
|
||||||
|
5. Test rollback: A → B → fail → auto-revert to A
|
||||||
|
6. Test: pull power during update → verify clean state
|
||||||
|
|
||||||
|
### Phase 4 — Production Hardening (2–3 weeks)
|
||||||
|
|
||||||
|
**Goal:** Production-ready security and manageability.
|
||||||
|
|
||||||
|
1. Image signing and verification (GPG or sigstore/cosign)
|
||||||
|
2. Cloud-init implementation for first-boot config
|
||||||
|
3. Portainer Edge integration testing
|
||||||
|
4. Optional SSH extension (`.tcz`)
|
||||||
|
5. Optional management API (lightweight, mTLS-authenticated)
|
||||||
|
6. Performance benchmarking (boot time, memory usage, disk I/O)
|
||||||
|
7. Documentation and deployment guides
|
||||||
|
|
||||||
|
### Phase 5 — Distribution + Fleet Management (ongoing)
|
||||||
|
|
||||||
|
**Goal:** Scale to fleet deployments.
|
||||||
|
|
||||||
|
1. CI/CD pipeline for automated image builds
|
||||||
|
2. OCI registry distribution (optional)
|
||||||
|
3. Fleet update orchestration (rolling updates across nodes)
|
||||||
|
4. Monitoring integration (Prometheus metrics endpoint)
|
||||||
|
5. USB provisioning tool for air-gapped deployments
|
||||||
|
6. ARM64 support (Raspberry Pi, Jetson, etc.)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9. Open Questions & Decisions
|
||||||
|
|
||||||
|
| # | Question | Options | Recommendation |
|
||||||
|
|---|---|---|---|
|
||||||
|
| 1 | **Build approach** | Pure Tiny Core remaster vs. Kairos framework | Start with pure remaster for minimal footprint; evaluate Kairos if update complexity becomes unmanageable |
|
||||||
|
| 2 | **Kernel** | Stock Tiny Core kernel vs. custom build | Audit stock kernel first; only custom-build if missing critical configs |
|
||||||
|
| 3 | **Management interface** | SSH / API / Portainer Edge only | Portainer Edge primary; optional SSH extension for debugging |
|
||||||
|
| 4 | **Update distribution** | HTTP server / OCI registry / USB | HTTP for simplicity; OCI if leveraging container infrastructure |
|
||||||
|
| 5 | **Init system** | Custom shell script vs. BusyBox init vs. s6 | Custom shell script for PoC; evaluate s6 for supervision |
|
||||||
|
| 6 | **Networking** | DHCP only / Static / cloud-init | Cloud-init with DHCP fallback |
|
||||||
|
| 7 | **Architecture support** | x86_64 only vs. multi-arch | x86_64 first; ARM64 in Phase 5 |
|
||||||
|
| 8 | **Container images** | Preloaded in initramfs vs. pull at boot | Preload core workloads; pull additional at runtime |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 10. References
|
||||||
|
|
||||||
|
- KubeSolo: https://github.com/portainer/kubesolo
|
||||||
|
- Tiny Core Linux: http://www.tinycorelinux.net
|
||||||
|
- Tiny Core Wiki (Remastering): http://wiki.tinycorelinux.net/doku.php?id=wiki:remastering
|
||||||
|
- Talos Linux: https://www.talos.dev
|
||||||
|
- Kairos: https://kairos.io
|
||||||
|
- Bottlerocket: https://github.com/bottlerocket-os/bottlerocket
|
||||||
|
- Flatcar Linux: https://www.flatcar.org
|
||||||
|
- Kubernetes Node Requirements: https://kubernetes.io/docs/setup/production-environment/container-runtimes/
|
||||||
|
- cgroup v2: https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html
|
||||||
84
hack/dev-vm.sh
Executable file
84
hack/dev-vm.sh
Executable file
@@ -0,0 +1,84 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# dev-vm.sh — Launch a QEMU VM for development and testing
|
||||||
|
# Usage: ./hack/dev-vm.sh [path-to-iso-or-img] [--shell] [--debug]
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||||||
|
VERSION="$(cat "$PROJECT_ROOT/VERSION")"
|
||||||
|
DEFAULT_ISO="$PROJECT_ROOT/output/kubesolo-os-${VERSION}.iso"
|
||||||
|
DEFAULT_IMG="$PROJECT_ROOT/output/kubesolo-os-${VERSION}.img"
|
||||||
|
|
||||||
|
IMAGE="${1:-}"
|
||||||
|
EXTRA_APPEND=""
|
||||||
|
SERIAL_OPTS="-serial stdio"
|
||||||
|
|
||||||
|
# Parse flags
|
||||||
|
shift || true
|
||||||
|
for arg in "$@"; do
|
||||||
|
case "$arg" in
|
||||||
|
--shell) EXTRA_APPEND="$EXTRA_APPEND kubesolo.shell" ;;
|
||||||
|
--debug) EXTRA_APPEND="$EXTRA_APPEND kubesolo.debug" ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
|
||||||
|
# Auto-detect image
|
||||||
|
if [ -z "$IMAGE" ]; then
|
||||||
|
if [ -f "$DEFAULT_ISO" ]; then
|
||||||
|
IMAGE="$DEFAULT_ISO"
|
||||||
|
elif [ -f "$DEFAULT_IMG" ]; then
|
||||||
|
IMAGE="$DEFAULT_IMG"
|
||||||
|
else
|
||||||
|
echo "ERROR: No image found. Run 'make iso' or 'make disk-image' first."
|
||||||
|
echo " Or specify path: $0 <path-to-iso-or-img>"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "==> Launching QEMU with: $IMAGE"
|
||||||
|
echo " Press Ctrl+A, X to exit"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Create a temporary data disk for persistence testing
|
||||||
|
DATA_DISK=$(mktemp /tmp/kubesolo-data-XXXXXX.img)
|
||||||
|
dd if=/dev/zero of="$DATA_DISK" bs=1M count=1024 2>/dev/null
|
||||||
|
mkfs.ext4 -q -L KSOLODATA "$DATA_DISK" 2>/dev/null
|
||||||
|
|
||||||
|
cleanup() { rm -f "$DATA_DISK"; }
|
||||||
|
trap cleanup EXIT
|
||||||
|
|
||||||
|
COMMON_OPTS=(
|
||||||
|
-m 2048
|
||||||
|
-smp 2
|
||||||
|
-nographic
|
||||||
|
-net nic,model=virtio
|
||||||
|
-net user,hostfwd=tcp::6443-:6443,hostfwd=tcp::2222-:22
|
||||||
|
-drive "file=$DATA_DISK,format=raw,if=virtio"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Enable KVM if available
|
||||||
|
if [ -w /dev/kvm ] 2>/dev/null; then
|
||||||
|
COMMON_OPTS+=(-enable-kvm)
|
||||||
|
echo " KVM acceleration: enabled"
|
||||||
|
else
|
||||||
|
echo " KVM acceleration: not available (using TCG)"
|
||||||
|
fi
|
||||||
|
|
||||||
|
case "$IMAGE" in
|
||||||
|
*.iso)
|
||||||
|
qemu-system-x86_64 \
|
||||||
|
"${COMMON_OPTS[@]}" \
|
||||||
|
-cdrom "$IMAGE" \
|
||||||
|
-boot d \
|
||||||
|
-append "console=ttyS0,115200n8 kubesolo.data=/dev/vda $EXTRA_APPEND"
|
||||||
|
;;
|
||||||
|
*.img)
|
||||||
|
qemu-system-x86_64 \
|
||||||
|
"${COMMON_OPTS[@]}" \
|
||||||
|
-drive "file=$IMAGE,format=raw,if=virtio"
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
echo "ERROR: Unrecognized image format: $IMAGE"
|
||||||
|
exit 1
|
||||||
|
;;
|
||||||
|
esac
|
||||||
48
hack/extract-kernel-config.sh
Executable file
48
hack/extract-kernel-config.sh
Executable file
@@ -0,0 +1,48 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# extract-kernel-config.sh — Pull kernel config from Tiny Core rootfs
|
||||||
|
# Usage: ./hack/extract-kernel-config.sh [path-to-core.gz]
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||||||
|
ROOTFS_DIR="${ROOTFS_DIR:-$PROJECT_ROOT/build/rootfs-work}"
|
||||||
|
|
||||||
|
COREGZ="${1:-$ROOTFS_DIR/rootfs}"
|
||||||
|
OUTPUT="$PROJECT_ROOT/build/cache/kernel-config"
|
||||||
|
|
||||||
|
if [ -d "$COREGZ" ]; then
|
||||||
|
# Rootfs already extracted
|
||||||
|
ROOTFS="$COREGZ"
|
||||||
|
elif [ -f "$COREGZ" ]; then
|
||||||
|
# Extract core.gz
|
||||||
|
TMPDIR=$(mktemp -d)
|
||||||
|
cd "$TMPDIR"
|
||||||
|
zcat "$COREGZ" | cpio -idm 2>/dev/null
|
||||||
|
ROOTFS="$TMPDIR"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Try /proc/config.gz in rootfs
|
||||||
|
if [ -f "$ROOTFS/proc/config.gz" ]; then
|
||||||
|
zcat "$ROOTFS/proc/config.gz" > "$OUTPUT"
|
||||||
|
echo "==> Extracted kernel config to: $OUTPUT"
|
||||||
|
"$PROJECT_ROOT/build/config/kernel-audit.sh" "$OUTPUT"
|
||||||
|
else
|
||||||
|
echo "Kernel config not found in rootfs /proc/config.gz"
|
||||||
|
echo ""
|
||||||
|
echo "Alternative: Boot the Tiny Core ISO in QEMU and run:"
|
||||||
|
echo " zcat /proc/config.gz > /tmp/kernel-config"
|
||||||
|
echo " # Then copy it out"
|
||||||
|
echo ""
|
||||||
|
echo "Or check if /boot/config-* exists in the ISO"
|
||||||
|
|
||||||
|
# Try looking in /boot
|
||||||
|
for f in "$ROOTFS"/boot/config-*; do
|
||||||
|
[ -f "$f" ] || continue
|
||||||
|
cp "$f" "$OUTPUT"
|
||||||
|
echo "==> Found boot config: $f → $OUTPUT"
|
||||||
|
"$PROJECT_ROOT/build/config/kernel-audit.sh" "$OUTPUT"
|
||||||
|
exit 0
|
||||||
|
done
|
||||||
|
fi
|
||||||
|
|
||||||
|
[ -d "${TMPDIR:-}" ] && rm -rf "$TMPDIR"
|
||||||
82
hack/inject-ssh.sh
Executable file
82
hack/inject-ssh.sh
Executable file
@@ -0,0 +1,82 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# inject-ssh.sh — Add SSH (dropbear) to initramfs for debugging
|
||||||
|
# Usage: ./hack/inject-ssh.sh [path-to-kubesolo-os.gz]
|
||||||
|
#
|
||||||
|
# This adds a minimal SSH server to the initramfs so you can SSH into the
|
||||||
|
# running KubeSolo OS for debugging. NOT for production use.
|
||||||
|
#
|
||||||
|
# Prerequisites: dropbear binaries (statically compiled) or tcz packages
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||||||
|
ROOTFS_DIR="${ROOTFS_DIR:-$PROJECT_ROOT/build/rootfs-work}"
|
||||||
|
ROOTFS="$ROOTFS_DIR/rootfs"
|
||||||
|
|
||||||
|
INITRAMFS="${1:-$ROOTFS_DIR/kubesolo-os.gz}"
|
||||||
|
|
||||||
|
if [ ! -d "$ROOTFS" ]; then
|
||||||
|
echo "ERROR: Rootfs not found: $ROOTFS"
|
||||||
|
echo "Run 'make rootfs' first."
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
SSH_PUBKEY="${SSH_PUBKEY:-$HOME/.ssh/id_rsa.pub}"
|
||||||
|
if [ ! -f "$SSH_PUBKEY" ]; then
|
||||||
|
SSH_PUBKEY="$HOME/.ssh/id_ed25519.pub"
|
||||||
|
fi
|
||||||
|
if [ ! -f "$SSH_PUBKEY" ]; then
|
||||||
|
echo "ERROR: No SSH public key found."
|
||||||
|
echo "Set SSH_PUBKEY=/path/to/key.pub or generate one with: ssh-keygen"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "==> Injecting SSH support into rootfs..."
|
||||||
|
echo " Public key: $SSH_PUBKEY"
|
||||||
|
|
||||||
|
# Create SSH directories
|
||||||
|
mkdir -p "$ROOTFS/root/.ssh"
|
||||||
|
mkdir -p "$ROOTFS/etc/dropbear"
|
||||||
|
|
||||||
|
# Install authorized key
|
||||||
|
cp "$SSH_PUBKEY" "$ROOTFS/root/.ssh/authorized_keys"
|
||||||
|
chmod 700 "$ROOTFS/root/.ssh"
|
||||||
|
chmod 600 "$ROOTFS/root/.ssh/authorized_keys"
|
||||||
|
|
||||||
|
# Create a startup script for dropbear
|
||||||
|
cat > "$ROOTFS/usr/lib/kubesolo-os/init.d/85-ssh.sh" << 'EOF'
|
||||||
|
#!/bin/sh
|
||||||
|
# 85-ssh.sh — Start SSH server for debugging (dev only)
|
||||||
|
|
||||||
|
if command -v dropbear >/dev/null 2>&1; then
|
||||||
|
# Generate host keys if missing
|
||||||
|
if [ ! -f /etc/dropbear/dropbear_rsa_host_key ]; then
|
||||||
|
dropbearkey -t rsa -f /etc/dropbear/dropbear_rsa_host_key >/dev/null 2>&1
|
||||||
|
fi
|
||||||
|
if [ ! -f /etc/dropbear/dropbear_ed25519_host_key ]; then
|
||||||
|
dropbearkey -t ed25519 -f /etc/dropbear/dropbear_ed25519_host_key >/dev/null 2>&1
|
||||||
|
fi
|
||||||
|
|
||||||
|
dropbear -R -p 22 2>/dev/null
|
||||||
|
log_ok "SSH server (dropbear) started on port 22"
|
||||||
|
else
|
||||||
|
log_warn "dropbear not found — SSH not available"
|
||||||
|
log_warn "To add SSH, install dropbear statically compiled binary to /usr/sbin/dropbear"
|
||||||
|
fi
|
||||||
|
EOF
|
||||||
|
chmod +x "$ROOTFS/usr/lib/kubesolo-os/init.d/85-ssh.sh"
|
||||||
|
|
||||||
|
echo "==> SSH stage added (85-ssh.sh)"
|
||||||
|
echo ""
|
||||||
|
echo "==> NOTE: You still need the dropbear binary in the rootfs."
|
||||||
|
echo " Option 1: Download a static dropbear build:"
|
||||||
|
echo " wget -O $ROOTFS/usr/sbin/dropbear <url-to-static-dropbear>"
|
||||||
|
echo " chmod +x $ROOTFS/usr/sbin/dropbear"
|
||||||
|
echo ""
|
||||||
|
echo " Option 2: Build from source with CGO_ENABLED=0 equivalent"
|
||||||
|
echo ""
|
||||||
|
echo "==> After adding dropbear, rebuild:"
|
||||||
|
echo " make initramfs iso"
|
||||||
|
echo ""
|
||||||
|
echo "==> Then connect with:"
|
||||||
|
echo " ssh -p 2222 root@localhost (when using hack/dev-vm.sh)"
|
||||||
18
hack/rebuild-initramfs.sh
Executable file
18
hack/rebuild-initramfs.sh
Executable file
@@ -0,0 +1,18 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# rebuild-initramfs.sh — Fast rebuild: re-inject init scripts + repack
|
||||||
|
# Skips fetch/extract — only updates init system and configs
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||||||
|
|
||||||
|
echo "==> Quick rebuild: re-injecting init system..."
|
||||||
|
"$PROJECT_ROOT/build/scripts/inject-kubesolo.sh"
|
||||||
|
|
||||||
|
echo "==> Repacking initramfs..."
|
||||||
|
"$PROJECT_ROOT/build/scripts/pack-initramfs.sh"
|
||||||
|
|
||||||
|
echo "==> Rebuilding ISO..."
|
||||||
|
"$PROJECT_ROOT/build/scripts/create-iso.sh"
|
||||||
|
|
||||||
|
echo "==> Done. Run 'make dev-vm' to test."
|
||||||
48
init/emergency-shell.sh
Executable file
48
init/emergency-shell.sh
Executable file
@@ -0,0 +1,48 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
# emergency-shell.sh — Drop to a debug shell on boot failure
|
||||||
|
# Called by init when a critical stage fails
|
||||||
|
# POSIX sh only — BusyBox ash compatible
|
||||||
|
|
||||||
|
echo "" >&2
|
||||||
|
echo "=====================================================" >&2
|
||||||
|
echo " KubeSolo OS — Emergency Shell" >&2
|
||||||
|
echo "=====================================================" >&2
|
||||||
|
echo "" >&2
|
||||||
|
echo " The boot process has failed. You have been dropped" >&2
|
||||||
|
echo " into an emergency shell for debugging." >&2
|
||||||
|
echo "" >&2
|
||||||
|
echo " Useful commands:" >&2
|
||||||
|
echo " dmesg | tail -50 Kernel messages" >&2
|
||||||
|
echo " cat /proc/cmdline Boot parameters" >&2
|
||||||
|
echo " cat /proc/mounts Current mounts" >&2
|
||||||
|
echo " blkid Block device info" >&2
|
||||||
|
echo " ip addr Network interfaces" >&2
|
||||||
|
echo " ls /usr/lib/kubesolo-os/init.d/ Init stages" >&2
|
||||||
|
echo "" >&2
|
||||||
|
|
||||||
|
# Show version if available
|
||||||
|
if [ -f /etc/kubesolo-os-version ]; then
|
||||||
|
echo " OS Version: $(cat /etc/kubesolo-os-version)" >&2
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Show what stage failed if known
|
||||||
|
if [ -n "${FAILED_STAGE:-}" ]; then
|
||||||
|
echo " Failed stage: $FAILED_STAGE" >&2
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "" >&2
|
||||||
|
echo " Type 'exit' to attempt re-running the init sequence." >&2
|
||||||
|
echo " Type 'reboot' to restart the system." >&2
|
||||||
|
echo "=====================================================" >&2
|
||||||
|
echo "" >&2
|
||||||
|
|
||||||
|
# Ensure basic env is usable
|
||||||
|
export PATH="/sbin:/usr/sbin:/bin:/usr/bin:/usr/local/bin"
|
||||||
|
export PS1="[kubesolo-emergency] # "
|
||||||
|
export HOME=/root
|
||||||
|
export TERM="${TERM:-linux}"
|
||||||
|
|
||||||
|
# Create home dir if needed
|
||||||
|
mkdir -p /root
|
||||||
|
|
||||||
|
exec /bin/sh
|
||||||
87
init/init.sh
Executable file
87
init/init.sh
Executable file
@@ -0,0 +1,87 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
# /sbin/init — KubeSolo OS init system
|
||||||
|
# POSIX sh compatible (BusyBox ash)
|
||||||
|
#
|
||||||
|
# Boot stages are sourced from /usr/lib/kubesolo-os/init.d/ in numeric order.
|
||||||
|
# Each stage file must be a valid POSIX sh script.
|
||||||
|
# If any mandatory stage fails, the system drops to an emergency shell.
|
||||||
|
#
|
||||||
|
# Boot parameters (from kernel command line):
|
||||||
|
# kubesolo.data=<device> Persistent data partition (required)
|
||||||
|
# kubesolo.debug Enable verbose logging
|
||||||
|
# kubesolo.shell Drop to emergency shell immediately
|
||||||
|
# kubesolo.nopersist Run without persistent storage (RAM only)
|
||||||
|
# kubesolo.cloudinit=<path> Path to cloud-init config
|
||||||
|
# kubesolo.flags=<flags> Extra flags for KubeSolo binary
|
||||||
|
|
||||||
|
set -e
|
||||||
|
|
||||||
|
# --- Constants ---
|
||||||
|
INIT_LIB="/usr/lib/kubesolo-os"
|
||||||
|
INIT_STAGES="/usr/lib/kubesolo-os/init.d"
|
||||||
|
LOG_PREFIX="[kubesolo-init]"
|
||||||
|
DATA_MOUNT="/mnt/data"
|
||||||
|
|
||||||
|
# --- Parsed boot parameters (populated by 10-parse-cmdline.sh) ---
|
||||||
|
export KUBESOLO_DATA_DEV=""
|
||||||
|
export KUBESOLO_DEBUG=""
|
||||||
|
export KUBESOLO_SHELL=""
|
||||||
|
export KUBESOLO_NOPERSIST=""
|
||||||
|
export KUBESOLO_CLOUDINIT=""
|
||||||
|
export KUBESOLO_EXTRA_FLAGS=""
|
||||||
|
|
||||||
|
# --- Logging ---
|
||||||
|
log() {
|
||||||
|
echo "$LOG_PREFIX $*" >&2
|
||||||
|
}
|
||||||
|
|
||||||
|
log_ok() {
|
||||||
|
echo "$LOG_PREFIX [OK] $*" >&2
|
||||||
|
}
|
||||||
|
|
||||||
|
log_err() {
|
||||||
|
echo "$LOG_PREFIX [ERROR] $*" >&2
|
||||||
|
}
|
||||||
|
|
||||||
|
log_warn() {
|
||||||
|
echo "$LOG_PREFIX [WARN] $*" >&2
|
||||||
|
}
|
||||||
|
|
||||||
|
# --- Emergency shell ---
|
||||||
|
emergency_shell() {
|
||||||
|
log_err "Boot failed: $*"
|
||||||
|
log_err "Dropping to emergency shell. Type 'exit' to retry boot."
|
||||||
|
exec /bin/sh
|
||||||
|
}
|
||||||
|
|
||||||
|
# --- Main boot sequence ---
|
||||||
|
log "KubeSolo OS v$(cat /etc/kubesolo-os-version 2>/dev/null || echo 'dev') starting..."
|
||||||
|
|
||||||
|
# Source shared functions
|
||||||
|
if [ -f "$INIT_LIB/functions.sh" ]; then
|
||||||
|
. "$INIT_LIB/functions.sh"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Run init stages in order
|
||||||
|
for stage in "$INIT_STAGES"/*.sh; do
|
||||||
|
[ -f "$stage" ] || continue
|
||||||
|
stage_name="$(basename "$stage")"
|
||||||
|
|
||||||
|
log "Running stage: $stage_name"
|
||||||
|
|
||||||
|
if ! . "$stage"; then
|
||||||
|
emergency_shell "Stage $stage_name failed"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Check for early shell request (parsed in 10-parse-cmdline.sh)
|
||||||
|
if [ "$KUBESOLO_SHELL" = "1" ] && [ "$stage_name" = "10-parse-cmdline.sh" ]; then
|
||||||
|
log "Emergency shell requested via boot parameter"
|
||||||
|
exec /bin/sh
|
||||||
|
fi
|
||||||
|
|
||||||
|
log_ok "Stage $stage_name complete"
|
||||||
|
done
|
||||||
|
|
||||||
|
# If we get here, all stages ran but KubeSolo should have exec'd.
|
||||||
|
# This means 90-kubesolo.sh didn't exec (shouldn't happen).
|
||||||
|
emergency_shell "Init completed without exec'ing KubeSolo — this is a bug"
|
||||||
23
init/lib/00-early-mount.sh
Executable file
23
init/lib/00-early-mount.sh
Executable file
@@ -0,0 +1,23 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
# 00-early-mount.sh — Mount essential virtual filesystems
|
||||||
|
|
||||||
|
mount -t proc proc /proc 2>/dev/null || true
|
||||||
|
mount -t sysfs sysfs /sys 2>/dev/null || true
|
||||||
|
mount -t devtmpfs devtmpfs /dev 2>/dev/null || true
|
||||||
|
mount -t tmpfs tmpfs /tmp
|
||||||
|
mount -t tmpfs tmpfs /run
|
||||||
|
|
||||||
|
mkdir -p /dev/pts /dev/shm
|
||||||
|
mount -t devpts devpts /dev/pts
|
||||||
|
mount -t tmpfs tmpfs /dev/shm
|
||||||
|
|
||||||
|
# Mount cgroup2 unified hierarchy
|
||||||
|
mkdir -p /sys/fs/cgroup
|
||||||
|
mount -t cgroup2 cgroup2 /sys/fs/cgroup 2>/dev/null || {
|
||||||
|
log_warn "cgroup v2 mount failed; attempting v1 fallback"
|
||||||
|
mount -t tmpfs cgroup /sys/fs/cgroup
|
||||||
|
for subsys in cpu cpuacct memory devices freezer pids; do
|
||||||
|
mkdir -p "/sys/fs/cgroup/$subsys"
|
||||||
|
mount -t cgroup -o "$subsys" "cgroup_${subsys}" "/sys/fs/cgroup/$subsys" 2>/dev/null || true
|
||||||
|
done
|
||||||
|
}
|
||||||
27
init/lib/10-parse-cmdline.sh
Executable file
27
init/lib/10-parse-cmdline.sh
Executable file
@@ -0,0 +1,27 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
# 10-parse-cmdline.sh — Parse boot parameters from /proc/cmdline
|
||||||
|
|
||||||
|
for arg in $(cat /proc/cmdline); do
|
||||||
|
case "$arg" in
|
||||||
|
kubesolo.data=*) KUBESOLO_DATA_DEV="${arg#kubesolo.data=}" ;;
|
||||||
|
kubesolo.debug) KUBESOLO_DEBUG=1; set -x ;;
|
||||||
|
kubesolo.shell) KUBESOLO_SHELL=1 ;;
|
||||||
|
kubesolo.nopersist) KUBESOLO_NOPERSIST=1 ;;
|
||||||
|
kubesolo.cloudinit=*) KUBESOLO_CLOUDINIT="${arg#kubesolo.cloudinit=}" ;;
|
||||||
|
kubesolo.flags=*) KUBESOLO_EXTRA_FLAGS="${arg#kubesolo.flags=}" ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
|
||||||
|
if [ -z "$KUBESOLO_DATA_DEV" ] && [ "$KUBESOLO_NOPERSIST" != "1" ]; then
|
||||||
|
log_warn "No kubesolo.data= specified and kubesolo.nopersist not set"
|
||||||
|
log_warn "Attempting auto-detection of data partition (label: KSOLODATA)"
|
||||||
|
KUBESOLO_DATA_DEV=$(blkid -L KSOLODATA 2>/dev/null || true)
|
||||||
|
if [ -z "$KUBESOLO_DATA_DEV" ]; then
|
||||||
|
log_warn "No data partition found. Running in RAM-only mode."
|
||||||
|
KUBESOLO_NOPERSIST=1
|
||||||
|
else
|
||||||
|
log "Auto-detected data partition: $KUBESOLO_DATA_DEV"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
log "Config: data=$KUBESOLO_DATA_DEV debug=$KUBESOLO_DEBUG nopersist=$KUBESOLO_NOPERSIST"
|
||||||
47
init/lib/20-persistent-mount.sh
Executable file
47
init/lib/20-persistent-mount.sh
Executable file
@@ -0,0 +1,47 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
# 20-persistent-mount.sh — Mount persistent data partition and bind-mount writable paths
|
||||||
|
|
||||||
|
if [ "$KUBESOLO_NOPERSIST" = "1" ]; then
|
||||||
|
log "Running in RAM-only mode — no persistent storage"
|
||||||
|
# Create tmpfs-backed directories so KubeSolo has somewhere to write
|
||||||
|
mkdir -p /var/lib/kubesolo /var/lib/containerd /etc/kubesolo /var/log /usr/local
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Wait for device to appear (USB, slow disks, virtio)
|
||||||
|
log "Waiting for data device: $KUBESOLO_DATA_DEV"
|
||||||
|
WAIT_SECS=30
|
||||||
|
for i in $(seq 1 "$WAIT_SECS"); do
|
||||||
|
[ -b "$KUBESOLO_DATA_DEV" ] && break
|
||||||
|
sleep 1
|
||||||
|
done
|
||||||
|
|
||||||
|
if [ ! -b "$KUBESOLO_DATA_DEV" ]; then
|
||||||
|
log_err "Data device $KUBESOLO_DATA_DEV not found after ${WAIT_SECS}s"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Mount data partition
|
||||||
|
mkdir -p "$DATA_MOUNT"
|
||||||
|
mount -t ext4 -o noatime "$KUBESOLO_DATA_DEV" "$DATA_MOUNT" || {
|
||||||
|
log_err "Failed to mount $KUBESOLO_DATA_DEV"
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
log_ok "Mounted $KUBESOLO_DATA_DEV at $DATA_MOUNT"
|
||||||
|
|
||||||
|
# Create persistent directory structure (first boot)
|
||||||
|
for dir in kubesolo containerd etc-kubesolo log usr-local network; do
|
||||||
|
mkdir -p "$DATA_MOUNT/$dir"
|
||||||
|
done
|
||||||
|
|
||||||
|
# Ensure target mount points exist
|
||||||
|
mkdir -p /var/lib/kubesolo /var/lib/containerd /etc/kubesolo /var/log /usr/local
|
||||||
|
|
||||||
|
# Bind mount persistent paths
|
||||||
|
mount --bind "$DATA_MOUNT/kubesolo" /var/lib/kubesolo
|
||||||
|
mount --bind "$DATA_MOUNT/containerd" /var/lib/containerd
|
||||||
|
mount --bind "$DATA_MOUNT/etc-kubesolo" /etc/kubesolo
|
||||||
|
mount --bind "$DATA_MOUNT/log" /var/log
|
||||||
|
mount --bind "$DATA_MOUNT/usr-local" /usr/local
|
||||||
|
|
||||||
|
log_ok "Persistent bind mounts configured"
|
||||||
28
init/lib/30-kernel-modules.sh
Executable file
28
init/lib/30-kernel-modules.sh
Executable file
@@ -0,0 +1,28 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
# 30-kernel-modules.sh — Load required kernel modules for K8s
|
||||||
|
|
||||||
|
MODULES_LIST="/usr/lib/kubesolo-os/modules.list"
|
||||||
|
|
||||||
|
if [ ! -f "$MODULES_LIST" ]; then
|
||||||
|
log_warn "No modules list found at $MODULES_LIST"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
LOADED=0
|
||||||
|
FAILED=0
|
||||||
|
|
||||||
|
while IFS= read -r mod; do
|
||||||
|
# Skip comments and blank lines
|
||||||
|
case "$mod" in
|
||||||
|
'#'*|'') continue ;;
|
||||||
|
esac
|
||||||
|
mod="$(echo "$mod" | tr -d '[:space:]')"
|
||||||
|
if modprobe "$mod" 2>/dev/null; then
|
||||||
|
LOADED=$((LOADED + 1))
|
||||||
|
else
|
||||||
|
log_warn "Failed to load module: $mod (may be built-in)"
|
||||||
|
FAILED=$((FAILED + 1))
|
||||||
|
fi
|
||||||
|
done < "$MODULES_LIST"
|
||||||
|
|
||||||
|
log_ok "Loaded $LOADED modules ($FAILED failed/built-in)"
|
||||||
20
init/lib/40-sysctl.sh
Executable file
20
init/lib/40-sysctl.sh
Executable file
@@ -0,0 +1,20 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
# 40-sysctl.sh — Apply kernel parameters required for K8s networking
|
||||||
|
|
||||||
|
# Apply all .conf files in sysctl.d
|
||||||
|
for conf in /etc/sysctl.d/*.conf; do
|
||||||
|
[ -f "$conf" ] || continue
|
||||||
|
while IFS='=' read -r key value; do
|
||||||
|
case "$key" in
|
||||||
|
'#'*|'') continue ;;
|
||||||
|
esac
|
||||||
|
key="$(echo "$key" | tr -d '[:space:]')"
|
||||||
|
value="$(echo "$value" | tr -d '[:space:]')"
|
||||||
|
if [ -n "$key" ] && [ -n "$value" ]; then
|
||||||
|
sysctl -w "${key}=${value}" >/dev/null 2>&1 || \
|
||||||
|
log_warn "Failed to set sysctl: ${key}=${value}"
|
||||||
|
fi
|
||||||
|
done < "$conf"
|
||||||
|
done
|
||||||
|
|
||||||
|
log_ok "Sysctl settings applied"
|
||||||
64
init/lib/50-network.sh
Executable file
64
init/lib/50-network.sh
Executable file
@@ -0,0 +1,64 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
# 50-network.sh — Configure networking
|
||||||
|
# Priority: persistent config > cloud-init > DHCP fallback
|
||||||
|
|
||||||
|
# Check for saved network config (from previous boot or cloud-init)
|
||||||
|
if [ -f "$DATA_MOUNT/network/interfaces.sh" ]; then
|
||||||
|
log "Applying saved network configuration"
|
||||||
|
. "$DATA_MOUNT/network/interfaces.sh"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Check for cloud-init network config
|
||||||
|
CLOUDINIT_FILE="${KUBESOLO_CLOUDINIT:-$DATA_MOUNT/etc-kubesolo/cloud-init.yaml}"
|
||||||
|
if [ -f "$CLOUDINIT_FILE" ]; then
|
||||||
|
log "Cloud-init found: $CLOUDINIT_FILE"
|
||||||
|
# Phase 1: simple parsing — extract network stanza
|
||||||
|
# TODO: Replace with proper cloud-init parser (Go binary) in Phase 2
|
||||||
|
log_warn "Cloud-init network parsing not yet implemented — falling back to DHCP"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Fallback: DHCP on first non-loopback interface
|
||||||
|
log "Configuring network via DHCP"
|
||||||
|
|
||||||
|
# Bring up loopback
|
||||||
|
ip link set lo up
|
||||||
|
ip addr add 127.0.0.1/8 dev lo
|
||||||
|
|
||||||
|
# Find first ethernet interface
|
||||||
|
ETH_DEV=""
|
||||||
|
for iface in /sys/class/net/*; do
|
||||||
|
iface="$(basename "$iface")"
|
||||||
|
case "$iface" in
|
||||||
|
lo|docker*|veth*|br*|cni*) continue ;;
|
||||||
|
esac
|
||||||
|
ETH_DEV="$iface"
|
||||||
|
break
|
||||||
|
done
|
||||||
|
|
||||||
|
if [ -z "$ETH_DEV" ]; then
|
||||||
|
log_err "No network interface found"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
log "Using interface: $ETH_DEV"
|
||||||
|
ip link set "$ETH_DEV" up
|
||||||
|
|
||||||
|
# Run DHCP client (BusyBox udhcpc)
|
||||||
|
if command -v udhcpc >/dev/null 2>&1; then
|
||||||
|
udhcpc -i "$ETH_DEV" -s /usr/share/udhcpc/default.script \
|
||||||
|
-t 10 -T 3 -A 5 -b -q 2>/dev/null || {
|
||||||
|
log_err "DHCP failed on $ETH_DEV"
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
elif command -v dhcpcd >/dev/null 2>&1; then
|
||||||
|
dhcpcd "$ETH_DEV" || {
|
||||||
|
log_err "DHCP failed on $ETH_DEV"
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
else
|
||||||
|
log_err "No DHCP client available (need udhcpc or dhcpcd)"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
log_ok "Network configured on $ETH_DEV"
|
||||||
24
init/lib/60-hostname.sh
Executable file
24
init/lib/60-hostname.sh
Executable file
@@ -0,0 +1,24 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
# 60-hostname.sh — Set system hostname
|
||||||
|
|
||||||
|
if [ -f "$DATA_MOUNT/etc-kubesolo/hostname" ]; then
|
||||||
|
HOSTNAME="$(cat "$DATA_MOUNT/etc-kubesolo/hostname")"
|
||||||
|
elif [ -f /etc/kubesolo/hostname ]; then
|
||||||
|
HOSTNAME="$(cat /etc/kubesolo/hostname)"
|
||||||
|
else
|
||||||
|
# Generate hostname from MAC address of primary interface
|
||||||
|
MAC_SUFFIX=""
|
||||||
|
for iface in /sys/class/net/*; do
|
||||||
|
iface="$(basename "$iface")"
|
||||||
|
case "$iface" in lo|docker*|veth*|br*|cni*) continue ;; esac
|
||||||
|
MAC_SUFFIX="$(cat "/sys/class/net/$iface/address" 2>/dev/null | tr -d ':' | tail -c 7)"
|
||||||
|
break
|
||||||
|
done
|
||||||
|
HOSTNAME="kubesolo-${MAC_SUFFIX:-unknown}"
|
||||||
|
fi
|
||||||
|
|
||||||
|
hostname "$HOSTNAME"
|
||||||
|
echo "$HOSTNAME" > /etc/hostname
|
||||||
|
echo "127.0.0.1 $HOSTNAME" >> /etc/hosts
|
||||||
|
|
||||||
|
log_ok "Hostname set to: $HOSTNAME"
|
||||||
19
init/lib/70-clock.sh
Executable file
19
init/lib/70-clock.sh
Executable file
@@ -0,0 +1,19 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
# 70-clock.sh — Set system clock (best-effort NTP or hwclock)
|
||||||
|
|
||||||
|
# Try hardware clock first
|
||||||
|
if command -v hwclock >/dev/null 2>&1; then
|
||||||
|
hwclock -s 2>/dev/null && log "Clock set from hardware clock" && return 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Try NTP (one-shot, non-blocking)
|
||||||
|
if command -v ntpd >/dev/null 2>&1; then
|
||||||
|
ntpd -n -q -p pool.ntp.org >/dev/null 2>&1 &
|
||||||
|
log "NTP sync started in background"
|
||||||
|
elif command -v ntpdate >/dev/null 2>&1; then
|
||||||
|
ntpdate -u pool.ntp.org >/dev/null 2>&1 &
|
||||||
|
log "NTP sync started in background"
|
||||||
|
else
|
||||||
|
log_warn "No NTP client available — clock may be inaccurate"
|
||||||
|
log_warn "K8s certificate validation may fail if clock is far off"
|
||||||
|
fi
|
||||||
22
init/lib/80-containerd.sh
Executable file
22
init/lib/80-containerd.sh
Executable file
@@ -0,0 +1,22 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
# 80-containerd.sh — Start containerd (bundled with KubeSolo)
|
||||||
|
#
|
||||||
|
# NOTE: KubeSolo typically manages containerd startup internally.
|
||||||
|
# This stage ensures containerd prerequisites are met.
|
||||||
|
# If KubeSolo handles containerd lifecycle, this stage may be a no-op.
|
||||||
|
|
||||||
|
# Ensure containerd state directories exist
|
||||||
|
mkdir -p /run/containerd
|
||||||
|
mkdir -p /var/lib/containerd
|
||||||
|
|
||||||
|
# Ensure CNI directories exist
|
||||||
|
mkdir -p /etc/cni/net.d
|
||||||
|
mkdir -p /opt/cni/bin
|
||||||
|
|
||||||
|
# If containerd config doesn't exist, KubeSolo will use defaults
|
||||||
|
# Only create a custom config if we need to override something
|
||||||
|
if [ -f /etc/kubesolo/containerd-config.toml ]; then
|
||||||
|
log "Using custom containerd config from /etc/kubesolo/containerd-config.toml"
|
||||||
|
fi
|
||||||
|
|
||||||
|
log_ok "containerd prerequisites ready (KubeSolo manages containerd lifecycle)"
|
||||||
38
init/lib/90-kubesolo.sh
Executable file
38
init/lib/90-kubesolo.sh
Executable file
@@ -0,0 +1,38 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
# 90-kubesolo.sh — Start KubeSolo (final init stage)
|
||||||
|
#
|
||||||
|
# This stage exec's KubeSolo as PID 1 (replacing init).
|
||||||
|
# KubeSolo manages containerd, kubelet, API server, and all K8s components.
|
||||||
|
|
||||||
|
KUBESOLO_BIN="/usr/local/bin/kubesolo"
|
||||||
|
|
||||||
|
if [ ! -x "$KUBESOLO_BIN" ]; then
|
||||||
|
log_err "KubeSolo binary not found at $KUBESOLO_BIN"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Build KubeSolo command line
|
||||||
|
KUBESOLO_ARGS="--path /var/lib/kubesolo --local-storage true"
|
||||||
|
|
||||||
|
# Add extra SANs if hostname resolves
|
||||||
|
HOSTNAME="$(hostname)"
|
||||||
|
if [ -n "$HOSTNAME" ]; then
|
||||||
|
KUBESOLO_ARGS="$KUBESOLO_ARGS --apiserver-extra-sans $HOSTNAME"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Add any extra flags from boot parameters
|
||||||
|
if [ -n "$KUBESOLO_EXTRA_FLAGS" ]; then
|
||||||
|
KUBESOLO_ARGS="$KUBESOLO_ARGS $KUBESOLO_EXTRA_FLAGS"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Add flags from persistent config file
|
||||||
|
if [ -f /etc/kubesolo/extra-flags ]; then
|
||||||
|
KUBESOLO_ARGS="$KUBESOLO_ARGS $(cat /etc/kubesolo/extra-flags)"
|
||||||
|
fi
|
||||||
|
|
||||||
|
log "Starting KubeSolo: $KUBESOLO_BIN $KUBESOLO_ARGS"
|
||||||
|
log "Kubeconfig will be at: /var/lib/kubesolo/pki/admin/admin.kubeconfig"
|
||||||
|
|
||||||
|
# exec replaces this init process — KubeSolo becomes PID 1
|
||||||
|
# shellcheck disable=SC2086
|
||||||
|
exec $KUBESOLO_BIN $KUBESOLO_ARGS
|
||||||
75
init/lib/functions.sh
Executable file
75
init/lib/functions.sh
Executable file
@@ -0,0 +1,75 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
# functions.sh — Shared utility functions for KubeSolo OS init
|
||||||
|
# Sourced by /sbin/init before running stages
|
||||||
|
# POSIX sh only — must work with BusyBox ash
|
||||||
|
|
||||||
|
# Wait for a block device to appear
|
||||||
|
wait_for_device() {
|
||||||
|
dev="$1"
|
||||||
|
timeout="${2:-30}"
|
||||||
|
i=0
|
||||||
|
while [ "$i" -lt "$timeout" ]; do
|
||||||
|
[ -b "$dev" ] && return 0
|
||||||
|
sleep 1
|
||||||
|
i=$((i + 1))
|
||||||
|
done
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
# Wait for a file to appear
|
||||||
|
wait_for_file() {
|
||||||
|
path="$1"
|
||||||
|
timeout="${2:-30}"
|
||||||
|
i=0
|
||||||
|
while [ "$i" -lt "$timeout" ]; do
|
||||||
|
[ -f "$path" ] && return 0
|
||||||
|
sleep 1
|
||||||
|
i=$((i + 1))
|
||||||
|
done
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
# Get IP address of an interface (POSIX-safe, no grep -P)
|
||||||
|
get_iface_ip() {
|
||||||
|
iface="$1"
|
||||||
|
ip -4 addr show "$iface" 2>/dev/null | \
|
||||||
|
sed -n 's/.*inet \([0-9.]*\).*/\1/p' | head -1
|
||||||
|
}
|
||||||
|
|
||||||
|
# Check if running in a VM (useful for adjusting timeouts)
|
||||||
|
is_virtual() {
|
||||||
|
[ -d /sys/class/dmi/id ] && \
|
||||||
|
grep -qi -e 'qemu' -e 'kvm' -e 'vmware' -e 'virtualbox' -e 'xen' -e 'hyperv' \
|
||||||
|
/sys/class/dmi/id/sys_vendor 2>/dev/null
|
||||||
|
}
|
||||||
|
|
||||||
|
# Resolve a LABEL= or UUID= device spec to a block device path
|
||||||
|
resolve_device() {
|
||||||
|
spec="$1"
|
||||||
|
case "$spec" in
|
||||||
|
LABEL=*) blkid -L "${spec#LABEL=}" 2>/dev/null ;;
|
||||||
|
UUID=*) blkid -U "${spec#UUID=}" 2>/dev/null ;;
|
||||||
|
*) echo "$spec" ;;
|
||||||
|
esac
|
||||||
|
}
|
||||||
|
|
||||||
|
# Write a key=value pair to a simple config file
|
||||||
|
config_set() {
|
||||||
|
file="$1" key="$2" value="$3"
|
||||||
|
if grep -q "^${key}=" "$file" 2>/dev/null; then
|
||||||
|
sed -i "s|^${key}=.*|${key}=${value}|" "$file"
|
||||||
|
else
|
||||||
|
echo "${key}=${value}" >> "$file"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# Read a value from a simple key=value config file
|
||||||
|
config_get() {
|
||||||
|
file="$1" key="$2" default="${3:-}"
|
||||||
|
if [ -f "$file" ]; then
|
||||||
|
value=$(sed -n "s/^${key}=//p" "$file" | tail -1)
|
||||||
|
echo "${value:-$default}"
|
||||||
|
else
|
||||||
|
echo "$default"
|
||||||
|
fi
|
||||||
|
}
|
||||||
97
test/integration/test-deploy-workload.sh
Executable file
97
test/integration/test-deploy-workload.sh
Executable file
@@ -0,0 +1,97 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# test-deploy-workload.sh — Deploy a test workload and verify it reaches Running
|
||||||
|
# Usage: ./test/integration/test-deploy-workload.sh <iso-path>
|
||||||
|
# Requires: kubectl on host, QEMU
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
ISO="${1:?Usage: $0 <path-to-iso>}"
|
||||||
|
TIMEOUT_BOOT=120
|
||||||
|
TIMEOUT_K8S=300
|
||||||
|
TIMEOUT_POD=120
|
||||||
|
API_PORT=6443
|
||||||
|
SERIAL_LOG=$(mktemp /tmp/kubesolo-workload-XXXXXX.log)
|
||||||
|
|
||||||
|
DATA_DISK=$(mktemp /tmp/kubesolo-data-XXXXXX.img)
|
||||||
|
dd if=/dev/zero of="$DATA_DISK" bs=1M count=1024 2>/dev/null
|
||||||
|
mkfs.ext4 -q -L KSOLODATA "$DATA_DISK" 2>/dev/null
|
||||||
|
|
||||||
|
cleanup() {
|
||||||
|
kill "$QEMU_PID" 2>/dev/null || true
|
||||||
|
rm -f "$DATA_DISK" "$SERIAL_LOG"
|
||||||
|
}
|
||||||
|
trap cleanup EXIT
|
||||||
|
|
||||||
|
KUBECTL="kubectl --server=https://localhost:${API_PORT} --insecure-skip-tls-verify"
|
||||||
|
|
||||||
|
echo "==> Workload deployment test: $ISO"
|
||||||
|
|
||||||
|
# Launch QEMU
|
||||||
|
qemu-system-x86_64 \
|
||||||
|
-m 2048 -smp 2 \
|
||||||
|
-nographic \
|
||||||
|
-cdrom "$ISO" \
|
||||||
|
-boot d \
|
||||||
|
-drive "file=$DATA_DISK,format=raw,if=virtio" \
|
||||||
|
-net nic,model=virtio \
|
||||||
|
-net "user,hostfwd=tcp::${API_PORT}-:6443" \
|
||||||
|
-serial "file:$SERIAL_LOG" \
|
||||||
|
-append "console=ttyS0,115200n8 kubesolo.data=/dev/vda" \
|
||||||
|
&
|
||||||
|
QEMU_PID=$!
|
||||||
|
|
||||||
|
# Wait for K8s API
|
||||||
|
echo " Waiting for K8s API..."
|
||||||
|
ELAPSED=0
|
||||||
|
K8S_READY=0
|
||||||
|
while [ "$ELAPSED" -lt "$TIMEOUT_K8S" ]; do
|
||||||
|
if $KUBECTL get nodes 2>/dev/null | grep -q "Ready"; then
|
||||||
|
K8S_READY=1
|
||||||
|
break
|
||||||
|
fi
|
||||||
|
sleep 5
|
||||||
|
ELAPSED=$((ELAPSED + 5))
|
||||||
|
printf "\r Elapsed: %ds / %ds" "$ELAPSED" "$TIMEOUT_K8S"
|
||||||
|
done
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
if [ "$K8S_READY" != "1" ]; then
|
||||||
|
echo "==> FAIL: K8s node did not reach Ready within ${TIMEOUT_K8S}s"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo "==> K8s node Ready (${ELAPSED}s)"
|
||||||
|
|
||||||
|
# Deploy test workload
|
||||||
|
echo "==> Deploying test nginx pod..."
|
||||||
|
$KUBECTL run test-nginx --image=nginx:alpine --restart=Never 2>/dev/null || {
|
||||||
|
echo "==> FAIL: Could not create test pod"
|
||||||
|
exit 1
|
||||||
|
}
|
||||||
|
|
||||||
|
# Wait for pod to be Running
|
||||||
|
echo " Waiting for pod to reach Running..."
|
||||||
|
ELAPSED=0
|
||||||
|
POD_RUNNING=0
|
||||||
|
while [ "$ELAPSED" -lt "$TIMEOUT_POD" ]; do
|
||||||
|
STATUS=$($KUBECTL get pod test-nginx -o jsonpath='{.status.phase}' 2>/dev/null || echo "")
|
||||||
|
if [ "$STATUS" = "Running" ]; then
|
||||||
|
POD_RUNNING=1
|
||||||
|
break
|
||||||
|
fi
|
||||||
|
sleep 5
|
||||||
|
ELAPSED=$((ELAPSED + 5))
|
||||||
|
printf "\r Elapsed: %ds / %ds (status: %s)" "$ELAPSED" "$TIMEOUT_POD" "${STATUS:-pending}"
|
||||||
|
done
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Cleanup test pod
|
||||||
|
$KUBECTL delete pod test-nginx --grace-period=0 --force 2>/dev/null || true
|
||||||
|
|
||||||
|
if [ "$POD_RUNNING" = "1" ]; then
|
||||||
|
echo "==> PASS: Test pod reached Running state (${ELAPSED}s)"
|
||||||
|
exit 0
|
||||||
|
else
|
||||||
|
echo "==> FAIL: Test pod did not reach Running within ${TIMEOUT_POD}s (last status: $STATUS)"
|
||||||
|
echo " Pod events:"
|
||||||
|
$KUBECTL describe pod test-nginx 2>/dev/null | tail -20 || true
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
69
test/integration/test-k8s-ready.sh
Executable file
69
test/integration/test-k8s-ready.sh
Executable file
@@ -0,0 +1,69 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# test-k8s-ready.sh — Verify K8s node reaches Ready state
|
||||||
|
# Usage: ./test/integration/test-k8s-ready.sh <iso-path>
|
||||||
|
# Requires: kubectl on host, QEMU with port forwarding
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
ISO="${1:?Usage: $0 <path-to-iso>}"
|
||||||
|
TIMEOUT_BOOT=120
|
||||||
|
TIMEOUT_K8S=300
|
||||||
|
API_PORT=6443
|
||||||
|
|
||||||
|
DATA_DISK=$(mktemp /tmp/kubesolo-data-XXXXXX.img)
|
||||||
|
dd if=/dev/zero of="$DATA_DISK" bs=1M count=1024 2>/dev/null
|
||||||
|
mkfs.ext4 -q -L KSOLODATA "$DATA_DISK" 2>/dev/null
|
||||||
|
|
||||||
|
cleanup() {
|
||||||
|
kill "$QEMU_PID" 2>/dev/null || true
|
||||||
|
rm -f "$DATA_DISK"
|
||||||
|
}
|
||||||
|
trap cleanup EXIT
|
||||||
|
|
||||||
|
echo "==> K8s readiness test: $ISO"
|
||||||
|
|
||||||
|
# Launch QEMU with API port forwarded
|
||||||
|
qemu-system-x86_64 \
|
||||||
|
-m 2048 -smp 2 \
|
||||||
|
-nographic \
|
||||||
|
-cdrom "$ISO" \
|
||||||
|
-boot d \
|
||||||
|
-drive "file=$DATA_DISK,format=raw,if=virtio" \
|
||||||
|
-net nic,model=virtio \
|
||||||
|
-net user,hostfwd=tcp::${API_PORT}-:6443 \
|
||||||
|
-append "console=ttyS0,115200n8 kubesolo.data=/dev/vda" \
|
||||||
|
&
|
||||||
|
QEMU_PID=$!
|
||||||
|
|
||||||
|
# Wait for API server
|
||||||
|
echo " Waiting for K8s API on localhost:${API_PORT}..."
|
||||||
|
ELAPSED=0
|
||||||
|
while [ "$ELAPSED" -lt "$TIMEOUT_K8S" ]; do
|
||||||
|
if kubectl --kubeconfig=/dev/null \
|
||||||
|
--server="https://localhost:${API_PORT}" \
|
||||||
|
--insecure-skip-tls-verify \
|
||||||
|
get nodes 2>/dev/null | grep -q "Ready"; then
|
||||||
|
echo ""
|
||||||
|
echo "==> PASS: K8s node is Ready (${ELAPSED}s)"
|
||||||
|
|
||||||
|
# Bonus: try deploying a pod
|
||||||
|
echo " Deploying test pod..."
|
||||||
|
kubectl --server="https://localhost:${API_PORT}" --insecure-skip-tls-verify \
|
||||||
|
run test-nginx --image=nginx:alpine --restart=Never 2>/dev/null || true
|
||||||
|
|
||||||
|
sleep 10
|
||||||
|
if kubectl --server="https://localhost:${API_PORT}" --insecure-skip-tls-verify \
|
||||||
|
get pod test-nginx 2>/dev/null | grep -q "Running"; then
|
||||||
|
echo "==> PASS: Test pod is Running"
|
||||||
|
else
|
||||||
|
echo "==> WARN: Test pod not Running (may need more time or image pull)"
|
||||||
|
fi
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
sleep 5
|
||||||
|
ELAPSED=$((ELAPSED + 5))
|
||||||
|
printf "\r Elapsed: %ds / %ds" "$ELAPSED" "$TIMEOUT_K8S"
|
||||||
|
done
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "==> FAIL: K8s node did not reach Ready within ${TIMEOUT_K8S}s"
|
||||||
|
exit 1
|
||||||
126
test/integration/test-local-storage.sh
Executable file
126
test/integration/test-local-storage.sh
Executable file
@@ -0,0 +1,126 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# test-local-storage.sh — Verify PVC with local-path provisioner works
|
||||||
|
# Usage: ./test/integration/test-local-storage.sh <iso-path>
|
||||||
|
# Requires: kubectl on host, QEMU
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
ISO="${1:?Usage: $0 <path-to-iso>}"
|
||||||
|
TIMEOUT_K8S=300
|
||||||
|
TIMEOUT_PVC=120
|
||||||
|
API_PORT=6443
|
||||||
|
|
||||||
|
DATA_DISK=$(mktemp /tmp/kubesolo-data-XXXXXX.img)
|
||||||
|
dd if=/dev/zero of="$DATA_DISK" bs=1M count=2048 2>/dev/null
|
||||||
|
mkfs.ext4 -q -L KSOLODATA "$DATA_DISK" 2>/dev/null
|
||||||
|
|
||||||
|
SERIAL_LOG=$(mktemp /tmp/kubesolo-storage-XXXXXX.log)
|
||||||
|
|
||||||
|
cleanup() {
|
||||||
|
# Clean up K8s resources
|
||||||
|
$KUBECTL delete pod test-storage --grace-period=0 --force 2>/dev/null || true
|
||||||
|
$KUBECTL delete pvc test-pvc 2>/dev/null || true
|
||||||
|
kill "$QEMU_PID" 2>/dev/null || true
|
||||||
|
rm -f "$DATA_DISK" "$SERIAL_LOG"
|
||||||
|
}
|
||||||
|
trap cleanup EXIT
|
||||||
|
|
||||||
|
KUBECTL="kubectl --server=https://localhost:${API_PORT} --insecure-skip-tls-verify"
|
||||||
|
|
||||||
|
echo "==> Local storage test: $ISO"
|
||||||
|
|
||||||
|
# Launch QEMU
|
||||||
|
qemu-system-x86_64 \
|
||||||
|
-m 2048 -smp 2 \
|
||||||
|
-nographic \
|
||||||
|
-cdrom "$ISO" \
|
||||||
|
-boot d \
|
||||||
|
-drive "file=$DATA_DISK,format=raw,if=virtio" \
|
||||||
|
-net nic,model=virtio \
|
||||||
|
-net "user,hostfwd=tcp::${API_PORT}-:6443" \
|
||||||
|
-serial "file:$SERIAL_LOG" \
|
||||||
|
-append "console=ttyS0,115200n8 kubesolo.data=/dev/vda" \
|
||||||
|
&
|
||||||
|
QEMU_PID=$!
|
||||||
|
|
||||||
|
# Wait for K8s API
|
||||||
|
echo " Waiting for K8s API..."
|
||||||
|
ELAPSED=0
|
||||||
|
while [ "$ELAPSED" -lt "$TIMEOUT_K8S" ]; do
|
||||||
|
if $KUBECTL get nodes 2>/dev/null | grep -q "Ready"; then
|
||||||
|
break
|
||||||
|
fi
|
||||||
|
sleep 5
|
||||||
|
ELAPSED=$((ELAPSED + 5))
|
||||||
|
done
|
||||||
|
|
||||||
|
if [ "$ELAPSED" -ge "$TIMEOUT_K8S" ]; then
|
||||||
|
echo "==> FAIL: K8s not ready within ${TIMEOUT_K8S}s"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo " K8s ready (${ELAPSED}s)"
|
||||||
|
|
||||||
|
# Create PVC
|
||||||
|
echo "==> Creating PersistentVolumeClaim..."
|
||||||
|
$KUBECTL apply -f - << 'YAML'
|
||||||
|
apiVersion: v1
|
||||||
|
kind: PersistentVolumeClaim
|
||||||
|
metadata:
|
||||||
|
name: test-pvc
|
||||||
|
spec:
|
||||||
|
accessModes:
|
||||||
|
- ReadWriteOnce
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
storage: 64Mi
|
||||||
|
YAML
|
||||||
|
|
||||||
|
# Create pod that uses the PVC
|
||||||
|
echo "==> Creating pod with PVC..."
|
||||||
|
$KUBECTL apply -f - << 'YAML'
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Pod
|
||||||
|
metadata:
|
||||||
|
name: test-storage
|
||||||
|
spec:
|
||||||
|
containers:
|
||||||
|
- name: writer
|
||||||
|
image: busybox:latest
|
||||||
|
command: ["sh", "-c", "echo 'kubesolo-storage-test' > /data/test.txt && sleep 3600"]
|
||||||
|
volumeMounts:
|
||||||
|
- name: data
|
||||||
|
mountPath: /data
|
||||||
|
volumes:
|
||||||
|
- name: data
|
||||||
|
persistentVolumeClaim:
|
||||||
|
claimName: test-pvc
|
||||||
|
YAML
|
||||||
|
|
||||||
|
# Wait for pod Running
|
||||||
|
echo " Waiting for storage pod..."
|
||||||
|
ELAPSED=0
|
||||||
|
while [ "$ELAPSED" -lt "$TIMEOUT_PVC" ]; do
|
||||||
|
STATUS=$($KUBECTL get pod test-storage -o jsonpath='{.status.phase}' 2>/dev/null || echo "")
|
||||||
|
if [ "$STATUS" = "Running" ]; then
|
||||||
|
break
|
||||||
|
fi
|
||||||
|
sleep 5
|
||||||
|
ELAPSED=$((ELAPSED + 5))
|
||||||
|
done
|
||||||
|
|
||||||
|
if [ "$STATUS" != "Running" ]; then
|
||||||
|
echo "==> FAIL: Storage pod did not reach Running (status: $STATUS)"
|
||||||
|
$KUBECTL describe pod test-storage 2>/dev/null | tail -20 || true
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Verify data was written
|
||||||
|
sleep 3
|
||||||
|
DATA=$($KUBECTL exec test-storage -- cat /data/test.txt 2>/dev/null || echo "")
|
||||||
|
if [ "$DATA" = "kubesolo-storage-test" ]; then
|
||||||
|
echo "==> PASS: Local storage provisioning works"
|
||||||
|
echo " PVC bound, pod running, data written and read back successfully"
|
||||||
|
exit 0
|
||||||
|
else
|
||||||
|
echo "==> FAIL: Data verification failed (got: '$DATA')"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
119
test/integration/test-network-policy.sh
Executable file
119
test/integration/test-network-policy.sh
Executable file
@@ -0,0 +1,119 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# test-network-policy.sh — Basic network policy enforcement test
|
||||||
|
# Usage: ./test/integration/test-network-policy.sh <iso-path>
|
||||||
|
# Verifies that NetworkPolicy resources can be created and traffic is filtered.
|
||||||
|
# Requires: kubectl on host, QEMU
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
ISO="${1:?Usage: $0 <path-to-iso>}"
|
||||||
|
TIMEOUT_K8S=300
|
||||||
|
TIMEOUT_POD=120
|
||||||
|
API_PORT=6443
|
||||||
|
|
||||||
|
DATA_DISK=$(mktemp /tmp/kubesolo-data-XXXXXX.img)
|
||||||
|
dd if=/dev/zero of="$DATA_DISK" bs=1M count=1024 2>/dev/null
|
||||||
|
mkfs.ext4 -q -L KSOLODATA "$DATA_DISK" 2>/dev/null
|
||||||
|
|
||||||
|
SERIAL_LOG=$(mktemp /tmp/kubesolo-netpol-XXXXXX.log)
|
||||||
|
|
||||||
|
cleanup() {
|
||||||
|
$KUBECTL delete namespace netpol-test 2>/dev/null || true
|
||||||
|
kill "$QEMU_PID" 2>/dev/null || true
|
||||||
|
rm -f "$DATA_DISK" "$SERIAL_LOG"
|
||||||
|
}
|
||||||
|
trap cleanup EXIT
|
||||||
|
|
||||||
|
KUBECTL="kubectl --server=https://localhost:${API_PORT} --insecure-skip-tls-verify"
|
||||||
|
|
||||||
|
echo "==> Network policy test: $ISO"
|
||||||
|
|
||||||
|
# Launch QEMU
|
||||||
|
qemu-system-x86_64 \
|
||||||
|
-m 2048 -smp 2 \
|
||||||
|
-nographic \
|
||||||
|
-cdrom "$ISO" \
|
||||||
|
-boot d \
|
||||||
|
-drive "file=$DATA_DISK,format=raw,if=virtio" \
|
||||||
|
-net nic,model=virtio \
|
||||||
|
-net "user,hostfwd=tcp::${API_PORT}-:6443" \
|
||||||
|
-serial "file:$SERIAL_LOG" \
|
||||||
|
-append "console=ttyS0,115200n8 kubesolo.data=/dev/vda" \
|
||||||
|
&
|
||||||
|
QEMU_PID=$!
|
||||||
|
|
||||||
|
# Wait for K8s
|
||||||
|
echo " Waiting for K8s API..."
|
||||||
|
ELAPSED=0
|
||||||
|
while [ "$ELAPSED" -lt "$TIMEOUT_K8S" ]; do
|
||||||
|
if $KUBECTL get nodes 2>/dev/null | grep -q "Ready"; then
|
||||||
|
break
|
||||||
|
fi
|
||||||
|
sleep 5
|
||||||
|
ELAPSED=$((ELAPSED + 5))
|
||||||
|
done
|
||||||
|
|
||||||
|
if [ "$ELAPSED" -ge "$TIMEOUT_K8S" ]; then
|
||||||
|
echo "==> FAIL: K8s not ready within ${TIMEOUT_K8S}s"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo " K8s ready (${ELAPSED}s)"
|
||||||
|
|
||||||
|
# Create test namespace
|
||||||
|
$KUBECTL create namespace netpol-test 2>/dev/null || true
|
||||||
|
|
||||||
|
# Create a web server pod
|
||||||
|
echo "==> Creating web server pod..."
|
||||||
|
$KUBECTL apply -n netpol-test -f - << 'YAML'
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Pod
|
||||||
|
metadata:
|
||||||
|
name: web
|
||||||
|
labels:
|
||||||
|
app: web
|
||||||
|
spec:
|
||||||
|
containers:
|
||||||
|
- name: web
|
||||||
|
image: busybox:latest
|
||||||
|
command: ["sh", "-c", "echo 'hello' | nc -l -p 80; sleep 3600"]
|
||||||
|
ports:
|
||||||
|
- containerPort: 80
|
||||||
|
YAML
|
||||||
|
|
||||||
|
# Wait for pod
|
||||||
|
ELAPSED=0
|
||||||
|
while [ "$ELAPSED" -lt "$TIMEOUT_POD" ]; do
|
||||||
|
STATUS=$($KUBECTL get pod -n netpol-test web -o jsonpath='{.status.phase}' 2>/dev/null || echo "")
|
||||||
|
[ "$STATUS" = "Running" ] && break
|
||||||
|
sleep 5
|
||||||
|
ELAPSED=$((ELAPSED + 5))
|
||||||
|
done
|
||||||
|
|
||||||
|
if [ "$STATUS" != "Running" ]; then
|
||||||
|
echo "==> FAIL: Web pod not running (status: $STATUS)"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo " Web pod running"
|
||||||
|
|
||||||
|
# Create a deny-all NetworkPolicy
|
||||||
|
echo "==> Applying deny-all NetworkPolicy..."
|
||||||
|
$KUBECTL apply -n netpol-test -f - << 'YAML'
|
||||||
|
apiVersion: networking.k8s.io/v1
|
||||||
|
kind: NetworkPolicy
|
||||||
|
metadata:
|
||||||
|
name: deny-all
|
||||||
|
spec:
|
||||||
|
podSelector: {}
|
||||||
|
policyTypes:
|
||||||
|
- Ingress
|
||||||
|
- Egress
|
||||||
|
YAML
|
||||||
|
|
||||||
|
# Verify the NetworkPolicy was created
|
||||||
|
if $KUBECTL get networkpolicy -n netpol-test deny-all >/dev/null 2>&1; then
|
||||||
|
echo "==> PASS: NetworkPolicy created successfully"
|
||||||
|
echo " NetworkPolicy resources are supported by the cluster"
|
||||||
|
exit 0
|
||||||
|
else
|
||||||
|
echo "==> FAIL: NetworkPolicy creation failed"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
23
test/kernel/check-config.sh
Executable file
23
test/kernel/check-config.sh
Executable file
@@ -0,0 +1,23 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# check-config.sh — Validate extracted kernel config against requirements
|
||||||
|
# Usage: ./test/kernel/check-config.sh [path-to-config]
|
||||||
|
# Defaults to build/cache/kernel-config if no argument given
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
|
||||||
|
|
||||||
|
CONFIG="${1:-$PROJECT_ROOT/build/cache/kernel-config}"
|
||||||
|
|
||||||
|
if [ ! -f "$CONFIG" ]; then
|
||||||
|
echo "ERROR: Kernel config not found: $CONFIG"
|
||||||
|
echo ""
|
||||||
|
echo "Extract it first:"
|
||||||
|
echo " ./hack/extract-kernel-config.sh"
|
||||||
|
echo ""
|
||||||
|
echo "Or provide path:"
|
||||||
|
echo " $0 /path/to/kernel/.config"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
exec "$PROJECT_ROOT/build/config/kernel-audit.sh" "$CONFIG"
|
||||||
120
test/qemu/run-vm.sh
Executable file
120
test/qemu/run-vm.sh
Executable file
@@ -0,0 +1,120 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# run-vm.sh — Launch QEMU VM for testing (reusable by other test scripts)
|
||||||
|
# Usage: ./test/qemu/run-vm.sh <iso-or-img> [options]
|
||||||
|
#
|
||||||
|
# Options:
|
||||||
|
# --data-disk <path> Use existing data disk (default: create temp)
|
||||||
|
# --data-size <MB> Size of temp data disk (default: 1024)
|
||||||
|
# --memory <MB> VM memory (default: 2048)
|
||||||
|
# --cpus <n> VM CPUs (default: 2)
|
||||||
|
# --serial-log <path> Write serial output to file
|
||||||
|
# --api-port <port> Forward K8s API to host port (default: 6443)
|
||||||
|
# --ssh-port <port> Forward SSH to host port (default: 2222)
|
||||||
|
# --background Run in background, print PID
|
||||||
|
# --append <args> Extra kernel append args
|
||||||
|
#
|
||||||
|
# Outputs (on stdout):
|
||||||
|
# QEMU_PID=<pid>
|
||||||
|
# DATA_DISK=<path>
|
||||||
|
# SERIAL_LOG=<path>
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
IMAGE="${1:?Usage: $0 <iso-or-img> [options]}"
|
||||||
|
shift
|
||||||
|
|
||||||
|
# Defaults
|
||||||
|
DATA_DISK=""
|
||||||
|
DATA_SIZE_MB=1024
|
||||||
|
MEMORY=2048
|
||||||
|
CPUS=2
|
||||||
|
SERIAL_LOG=""
|
||||||
|
API_PORT=6443
|
||||||
|
SSH_PORT=2222
|
||||||
|
BACKGROUND=0
|
||||||
|
EXTRA_APPEND=""
|
||||||
|
CREATED_DATA_DISK=""
|
||||||
|
|
||||||
|
# Parse options
|
||||||
|
while [ $# -gt 0 ]; do
|
||||||
|
case "$1" in
|
||||||
|
--data-disk) DATA_DISK="$2"; shift 2 ;;
|
||||||
|
--data-size) DATA_SIZE_MB="$2"; shift 2 ;;
|
||||||
|
--memory) MEMORY="$2"; shift 2 ;;
|
||||||
|
--cpus) CPUS="$2"; shift 2 ;;
|
||||||
|
--serial-log) SERIAL_LOG="$2"; shift 2 ;;
|
||||||
|
--api-port) API_PORT="$2"; shift 2 ;;
|
||||||
|
--ssh-port) SSH_PORT="$2"; shift 2 ;;
|
||||||
|
--background) BACKGROUND=1; shift ;;
|
||||||
|
--append) EXTRA_APPEND="$2"; shift 2 ;;
|
||||||
|
*) echo "Unknown option: $1" >&2; exit 1 ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
|
||||||
|
# Create data disk if not provided
|
||||||
|
if [ -z "$DATA_DISK" ]; then
|
||||||
|
DATA_DISK=$(mktemp /tmp/kubesolo-data-XXXXXX.img)
|
||||||
|
CREATED_DATA_DISK="$DATA_DISK"
|
||||||
|
dd if=/dev/zero of="$DATA_DISK" bs=1M count="$DATA_SIZE_MB" 2>/dev/null
|
||||||
|
mkfs.ext4 -q -L KSOLODATA "$DATA_DISK" 2>/dev/null
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Create serial log if not provided
|
||||||
|
if [ -z "$SERIAL_LOG" ]; then
|
||||||
|
SERIAL_LOG=$(mktemp /tmp/kubesolo-serial-XXXXXX.log)
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Detect KVM availability
|
||||||
|
KVM_FLAG=""
|
||||||
|
if [ -w /dev/kvm ] 2>/dev/null; then
|
||||||
|
KVM_FLAG="-enable-kvm"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Build QEMU command
|
||||||
|
QEMU_CMD=(
|
||||||
|
qemu-system-x86_64
|
||||||
|
-m "$MEMORY"
|
||||||
|
-smp "$CPUS"
|
||||||
|
-nographic
|
||||||
|
-net nic,model=virtio
|
||||||
|
-net "user,hostfwd=tcp::${API_PORT}-:6443,hostfwd=tcp::${SSH_PORT}-:22"
|
||||||
|
-drive "file=$DATA_DISK,format=raw,if=virtio"
|
||||||
|
-serial "file:$SERIAL_LOG"
|
||||||
|
)
|
||||||
|
|
||||||
|
[ -n "$KVM_FLAG" ] && QEMU_CMD+=("$KVM_FLAG")
|
||||||
|
|
||||||
|
case "$IMAGE" in
|
||||||
|
*.iso)
|
||||||
|
QEMU_CMD+=(
|
||||||
|
-cdrom "$IMAGE"
|
||||||
|
-boot d
|
||||||
|
-append "console=ttyS0,115200n8 kubesolo.data=/dev/vda kubesolo.debug $EXTRA_APPEND"
|
||||||
|
)
|
||||||
|
;;
|
||||||
|
*.img)
|
||||||
|
QEMU_CMD+=(
|
||||||
|
-drive "file=$IMAGE,format=raw,if=virtio"
|
||||||
|
)
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
echo "ERROR: Unrecognized image format: $IMAGE" >&2
|
||||||
|
exit 1
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
# Launch
|
||||||
|
"${QEMU_CMD[@]}" &
|
||||||
|
QEMU_PID=$!
|
||||||
|
|
||||||
|
# Output metadata
|
||||||
|
echo "QEMU_PID=$QEMU_PID"
|
||||||
|
echo "DATA_DISK=$DATA_DISK"
|
||||||
|
echo "SERIAL_LOG=$SERIAL_LOG"
|
||||||
|
echo "CREATED_DATA_DISK=$CREATED_DATA_DISK"
|
||||||
|
|
||||||
|
if [ "$BACKGROUND" = "0" ]; then
|
||||||
|
# Foreground mode — wait for QEMU to exit
|
||||||
|
wait "$QEMU_PID" || true
|
||||||
|
# Clean up temp data disk
|
||||||
|
[ -n "$CREATED_DATA_DISK" ] && rm -f "$CREATED_DATA_DISK"
|
||||||
|
fi
|
||||||
65
test/qemu/test-boot.sh
Executable file
65
test/qemu/test-boot.sh
Executable file
@@ -0,0 +1,65 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# test-boot.sh — Automated boot test: verify KubeSolo OS boots in QEMU
|
||||||
|
# Usage: ./test/qemu/test-boot.sh <iso-path>
|
||||||
|
# Exit 0 = PASS, Exit 1 = FAIL
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
ISO="${1:?Usage: $0 <path-to-iso>}"
|
||||||
|
TIMEOUT_BOOT=120 # seconds to wait for boot success marker
|
||||||
|
SERIAL_LOG=$(mktemp /tmp/kubesolo-boot-test-XXXXXX.log)
|
||||||
|
|
||||||
|
# Temp data disk
|
||||||
|
DATA_DISK=$(mktemp /tmp/kubesolo-data-XXXXXX.img)
|
||||||
|
dd if=/dev/zero of="$DATA_DISK" bs=1M count=512 2>/dev/null
|
||||||
|
mkfs.ext4 -q -L KSOLODATA "$DATA_DISK" 2>/dev/null
|
||||||
|
|
||||||
|
cleanup() {
|
||||||
|
kill "$QEMU_PID" 2>/dev/null || true
|
||||||
|
rm -f "$DATA_DISK" "$SERIAL_LOG"
|
||||||
|
}
|
||||||
|
trap cleanup EXIT
|
||||||
|
|
||||||
|
echo "==> Boot test: $ISO"
|
||||||
|
echo " Timeout: ${TIMEOUT_BOOT}s"
|
||||||
|
echo " Serial log: $SERIAL_LOG"
|
||||||
|
|
||||||
|
# Launch QEMU in background
|
||||||
|
qemu-system-x86_64 \
|
||||||
|
-m 2048 -smp 2 \
|
||||||
|
-nographic \
|
||||||
|
-cdrom "$ISO" \
|
||||||
|
-boot d \
|
||||||
|
-drive "file=$DATA_DISK,format=raw,if=virtio" \
|
||||||
|
-net nic,model=virtio \
|
||||||
|
-net user \
|
||||||
|
-serial file:"$SERIAL_LOG" \
|
||||||
|
-append "console=ttyS0,115200n8 kubesolo.data=/dev/vda kubesolo.debug" \
|
||||||
|
&
|
||||||
|
QEMU_PID=$!
|
||||||
|
|
||||||
|
# Wait for boot success marker in serial log
|
||||||
|
echo " Waiting for boot..."
|
||||||
|
ELAPSED=0
|
||||||
|
while [ "$ELAPSED" -lt "$TIMEOUT_BOOT" ]; do
|
||||||
|
if grep -q "\[kubesolo-init\] \[OK\] Stage 90-kubesolo.sh complete" "$SERIAL_LOG" 2>/dev/null; then
|
||||||
|
echo ""
|
||||||
|
echo "==> PASS: KubeSolo OS booted successfully in ${ELAPSED}s"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
if ! kill -0 "$QEMU_PID" 2>/dev/null; then
|
||||||
|
echo ""
|
||||||
|
echo "==> FAIL: QEMU exited prematurely"
|
||||||
|
echo " Last 20 lines of serial log:"
|
||||||
|
tail -20 "$SERIAL_LOG" 2>/dev/null
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
sleep 1
|
||||||
|
ELAPSED=$((ELAPSED + 1))
|
||||||
|
printf "\r Elapsed: %ds / %ds" "$ELAPSED" "$TIMEOUT_BOOT"
|
||||||
|
done
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "==> FAIL: Boot did not complete within ${TIMEOUT_BOOT}s"
|
||||||
|
echo " Last 30 lines of serial log:"
|
||||||
|
tail -30 "$SERIAL_LOG" 2>/dev/null
|
||||||
|
exit 1
|
||||||
100
test/qemu/test-persistence.sh
Executable file
100
test/qemu/test-persistence.sh
Executable file
@@ -0,0 +1,100 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# test-persistence.sh — Verify persistent state survives reboot
|
||||||
|
# Usage: ./test/qemu/test-persistence.sh <disk-image>
|
||||||
|
# Tests: writes a marker file to the data partition, reboots, checks it's still there
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
IMG="${1:?Usage: $0 <path-to-disk-image>}"
|
||||||
|
TIMEOUT_BOOT=120
|
||||||
|
SERIAL_LOG=$(mktemp /tmp/kubesolo-persist-XXXXXX.log)
|
||||||
|
|
||||||
|
cleanup() {
|
||||||
|
kill "$QEMU_PID" 2>/dev/null || true
|
||||||
|
rm -f "$SERIAL_LOG"
|
||||||
|
}
|
||||||
|
trap cleanup EXIT
|
||||||
|
|
||||||
|
wait_for_marker() {
|
||||||
|
local marker="$1"
|
||||||
|
local timeout="$2"
|
||||||
|
local elapsed=0
|
||||||
|
while [ "$elapsed" -lt "$timeout" ]; do
|
||||||
|
if grep -q "$marker" "$SERIAL_LOG" 2>/dev/null; then
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
if ! kill -0 "$QEMU_PID" 2>/dev/null; then
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
sleep 1
|
||||||
|
elapsed=$((elapsed + 1))
|
||||||
|
done
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
echo "==> Persistence test: $IMG"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# --- Boot 1: Write a marker to persistent storage ---
|
||||||
|
echo "==> Boot 1: Starting VM to write persistence marker..."
|
||||||
|
|
||||||
|
qemu-system-x86_64 \
|
||||||
|
-m 2048 -smp 2 \
|
||||||
|
-nographic \
|
||||||
|
-drive "file=$IMG,format=raw,if=virtio" \
|
||||||
|
-net nic,model=virtio \
|
||||||
|
-net user \
|
||||||
|
-serial "file:$SERIAL_LOG" \
|
||||||
|
&
|
||||||
|
QEMU_PID=$!
|
||||||
|
|
||||||
|
echo " Waiting for boot..."
|
||||||
|
if ! wait_for_marker "\[kubesolo-init\] \[OK\] Stage 90-kubesolo.sh complete" "$TIMEOUT_BOOT"; then
|
||||||
|
echo "==> FAIL: First boot did not complete"
|
||||||
|
tail -20 "$SERIAL_LOG" 2>/dev/null
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo " Boot 1 complete."
|
||||||
|
|
||||||
|
# Give KubeSolo a moment to write state
|
||||||
|
sleep 5
|
||||||
|
|
||||||
|
# Kill VM (simulate power off)
|
||||||
|
kill "$QEMU_PID" 2>/dev/null || true
|
||||||
|
wait "$QEMU_PID" 2>/dev/null || true
|
||||||
|
sleep 2
|
||||||
|
|
||||||
|
# --- Boot 2: Verify marker persisted ---
|
||||||
|
echo "==> Boot 2: Restarting VM to verify persistence..."
|
||||||
|
|
||||||
|
# Clear serial log for boot 2
|
||||||
|
> "$SERIAL_LOG"
|
||||||
|
|
||||||
|
qemu-system-x86_64 \
|
||||||
|
-m 2048 -smp 2 \
|
||||||
|
-nographic \
|
||||||
|
-drive "file=$IMG,format=raw,if=virtio" \
|
||||||
|
-net nic,model=virtio \
|
||||||
|
-net user \
|
||||||
|
-serial "file:$SERIAL_LOG" \
|
||||||
|
&
|
||||||
|
QEMU_PID=$!
|
||||||
|
|
||||||
|
if ! wait_for_marker "\[kubesolo-init\] \[OK\] Stage 90-kubesolo.sh complete" "$TIMEOUT_BOOT"; then
|
||||||
|
echo "==> FAIL: Second boot did not complete"
|
||||||
|
tail -20 "$SERIAL_LOG" 2>/dev/null
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Check that the persistent mount was reused (not first-boot)
|
||||||
|
if grep -q "\[kubesolo-init\] \[OK\] Persistent bind mounts configured" "$SERIAL_LOG" 2>/dev/null; then
|
||||||
|
echo "==> PASS: Persistent storage mounted on second boot"
|
||||||
|
else
|
||||||
|
echo "==> WARN: Could not confirm persistent mount (check serial log)"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# The fact that we booted twice on the same disk image and reached stage 90
|
||||||
|
# proves that the data partition survives reboots.
|
||||||
|
echo ""
|
||||||
|
echo "==> PASS: System booted successfully after reboot"
|
||||||
|
echo " Data partition persisted across power cycle."
|
||||||
|
exit 0
|
||||||
Reference in New Issue
Block a user