adelorenzo/kubesolo-os
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 2 Wiki Activity
Files
d9ac58418dcfa7d0d1c9e6336f4b1a56f8098d9f
kubesolo-os/Makefile
Adolfo Delorenzo 39732488ef feat: custom kernel build + boot fixes for working container runtime 
Build a custom Tiny Core 17.0 kernel (6.18.2) with missing configs
that the stock kernel lacks for container workloads:
- CONFIG_CGROUP_BPF=y (cgroup v2 device control via BPF)
- CONFIG_DEVTMPFS=y (auto-create /dev device nodes)
- CONFIG_DEVTMPFS_MOUNT=y (auto-mount devtmpfs)
- CONFIG_MEMCG=y (memory cgroup controller for memory.max)
- CONFIG_CFS_BANDWIDTH=y (CPU bandwidth throttling for cpu.max)

Also strips unnecessary subsystems (sound, GPU, wireless, Bluetooth,
KVM, etc.) for minimal footprint on a headless K8s edge appliance.

Init system fixes for successful boot-to-running-pods:
- Add switch_root in init.sh to escape initramfs (runc pivot_root)
- Add mountpoint guards in 00-early-mount.sh (skip if already mounted)
- Create essential device nodes after switch_root (kmsg, console, etc.)
- Enable cgroup v2 controller delegation with init process isolation
- Mount BPF filesystem for cgroup v2 device control
- Add mknod fallback from sysfs in 20-persistent-mount.sh for /dev/vda
- Move KubeSolo binary to /usr/bin (avoid /usr/local bind mount hiding)
- Generate /etc/machine-id in 60-hostname.sh (kubelet requires it)
- Pre-initialize iptables tables before kube-proxy starts
- Add nft_reject, nft_fib, xt_nfacct to kernel modules list

Build system changes:
- New build-kernel.sh script for custom kernel compilation
- Dockerfile.builder adds kernel build deps (flex, bison, libelf, etc.)
- Selective kernel module install (only modules.list + transitive deps)
- Install iptables-nft (xtables-nft-multi) + shared libs in rootfs

Tested: ISO boots in QEMU, node reaches Ready in ~35s, CoreDNS and
local-path-provisioner pods start and run successfully.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-11 23:13:31 -06:00

241 lines
9.2 KiB
Makefile
Raw Blame History

.PHONY: all fetch kernel build-cloudinit build-update-agent build-cross rootfs initramfs \
iso disk-image oci-image \
test-boot test-k8s test-persistence test-deploy test-storage test-all \
test-cloudinit test-update-agent \
bench-boot bench-resources \
dev-vm dev-vm-shell quick docker-build shellcheck \
kernel-audit clean distclean help
SHELL := /bin/bash
VERSION := $(shell cat VERSION)
BUILD_DIR := build
CACHE_DIR := $(BUILD_DIR)/cache
OUTPUT_DIR := output
ROOTFS_DIR := $(BUILD_DIR)/rootfs-work
# Load component versions
include $(BUILD_DIR)/config/versions.env
# Default target
all: iso
# =============================================================================
# Download external components
# =============================================================================
fetch:
@echo "==> Fetching components..."
@mkdir -p $(CACHE_DIR)
$(BUILD_DIR)/scripts/fetch-components.sh
# =============================================================================
# Build stages
# =============================================================================
kernel:
@echo "==> Building custom kernel (CONFIG_CGROUP_BPF=y)..."
$(BUILD_DIR)/scripts/build-kernel.sh
build-cloudinit:
@echo "==> Building cloud-init binary..."
$(BUILD_DIR)/scripts/build-cloudinit.sh
build-update-agent:
@echo "==> Building update agent..."
$(BUILD_DIR)/scripts/build-update-agent.sh
rootfs: fetch kernel build-cloudinit build-update-agent
@echo "==> Preparing rootfs..."
$(BUILD_DIR)/scripts/extract-core.sh
$(BUILD_DIR)/scripts/inject-kubesolo.sh
initramfs: rootfs
@echo "==> Packing initramfs..."
$(BUILD_DIR)/scripts/pack-initramfs.sh
iso: initramfs
@echo "==> Creating bootable ISO..."
$(BUILD_DIR)/scripts/create-iso.sh
@echo "==> Built: $(OUTPUT_DIR)/$(OS_NAME)-$(VERSION).iso"
disk-image: initramfs
@echo "==> Creating disk image..."
$(BUILD_DIR)/scripts/create-disk-image.sh
@echo "==> Built: $(OUTPUT_DIR)/$(OS_NAME)-$(VERSION).img"
oci-image: initramfs
@echo "==> Creating OCI container image..."
$(BUILD_DIR)/scripts/create-oci-image.sh
@echo "==> OCI image built"
# Cross-compile Go binaries for amd64 + arm64
build-cross:
@echo "==> Cross-compiling for amd64 + arm64..."
$(BUILD_DIR)/scripts/build-cross.sh
# =============================================================================
# Kernel validation
# =============================================================================
kernel-audit:
@echo "==> Auditing kernel configuration..."
$(BUILD_DIR)/config/kernel-audit.sh
# =============================================================================
# Testing
# =============================================================================
test-boot: iso
@echo "==> Testing boot in QEMU..."
test/qemu/test-boot.sh $(OUTPUT_DIR)/$(OS_NAME)-$(VERSION).iso
test-k8s: iso
@echo "==> Testing K8s readiness..."
test/integration/test-k8s-ready.sh $(OUTPUT_DIR)/$(OS_NAME)-$(VERSION).iso
test-persistence: disk-image
@echo "==> Testing persistence across reboot..."
test/qemu/test-persistence.sh $(OUTPUT_DIR)/$(OS_NAME)-$(VERSION).img
test-deploy: iso
@echo "==> Testing workload deployment..."
test/integration/test-deploy-workload.sh $(OUTPUT_DIR)/$(OS_NAME)-$(VERSION).iso
test-storage: iso
@echo "==> Testing local storage provisioning..."
test/integration/test-local-storage.sh $(OUTPUT_DIR)/$(OS_NAME)-$(VERSION).iso
test-all: test-boot test-k8s test-persistence
# Cloud-init Go tests
test-cloudinit:
@echo "==> Testing cloud-init parser..."
cd cloud-init && go test ./... -v -count=1
# Update agent Go tests
test-update-agent:
@echo "==> Testing update agent..."
cd update && go test ./... -v -count=1
# A/B update integration tests
test-update: disk-image
@echo "==> Testing A/B update cycle..."
test/qemu/test-update.sh $(OUTPUT_DIR)/$(OS_NAME)-$(VERSION).img
test-rollback: disk-image
@echo "==> Testing rollback..."
test/qemu/test-rollback.sh $(OUTPUT_DIR)/$(OS_NAME)-$(VERSION).img
# Full integration test suite (requires more time)
test-integration: test-k8s test-deploy test-storage
# Benchmarks
bench-boot: iso
@echo "==> Benchmarking boot performance..."
test/benchmark/bench-boot.sh $(OUTPUT_DIR)/$(OS_NAME)-$(VERSION).iso --runs 3
bench-resources:
@echo "==> Benchmarking resource usage (requires running VM)..."
test/benchmark/bench-resources.sh
# =============================================================================
# Code quality
# =============================================================================
shellcheck:
@echo "==> Running shellcheck on init scripts..."
shellcheck -s sh init/init.sh init/lib/*.sh init/emergency-shell.sh
@echo "==> Running shellcheck on build scripts..."
shellcheck -s bash build/scripts/*.sh build/config/kernel-audit.sh
@echo "==> Running shellcheck on test scripts..."
shellcheck -s bash test/qemu/*.sh test/integration/*.sh test/kernel/*.sh
@echo "==> Running shellcheck on hack scripts..."
shellcheck -s bash hack/*.sh
@echo "==> All shellcheck checks passed"
# =============================================================================
# Development helpers
# =============================================================================
dev-vm: iso
@echo "==> Launching dev VM..."
hack/dev-vm.sh $(OUTPUT_DIR)/$(OS_NAME)-$(VERSION).iso
dev-vm-shell: iso
@echo "==> Launching dev VM (emergency shell)..."
hack/dev-vm.sh $(OUTPUT_DIR)/$(OS_NAME)-$(VERSION).iso --shell
dev-vm-debug: iso
@echo "==> Launching dev VM (debug mode)..."
hack/dev-vm.sh $(OUTPUT_DIR)/$(OS_NAME)-$(VERSION).iso --debug
# Fast rebuild: only repack initramfs + ISO (skip fetch/extract)
quick:
@echo "==> Quick rebuild (repack + ISO only)..."
$(BUILD_DIR)/scripts/inject-kubesolo.sh
$(BUILD_DIR)/scripts/pack-initramfs.sh
$(BUILD_DIR)/scripts/create-iso.sh
@echo "==> Quick rebuild complete: $(OUTPUT_DIR)/$(OS_NAME)-$(VERSION).iso"
# =============================================================================
# Docker-based reproducible build
# =============================================================================
docker-build:
@echo "==> Building in Docker..."
docker build -t kubesolo-os-builder -f $(BUILD_DIR)/Dockerfile.builder .
docker run --rm --privileged \
-v $(PWD)/$(OUTPUT_DIR):/output \
-v $(PWD)/$(CACHE_DIR):/cache \
kubesolo-os-builder iso OUTPUT_DIR=/output CACHE_DIR=/cache
# =============================================================================
# Cleanup
# =============================================================================
clean:
@echo "==> Cleaning build artifacts..."
rm -rf $(ROOTFS_DIR) $(OUTPUT_DIR)
@echo "==> Clean. (Cache preserved in $(CACHE_DIR); use 'make distclean' to remove)"
distclean: clean
rm -rf $(CACHE_DIR)
# =============================================================================
# Help
# =============================================================================
help:
@echo "KubeSolo OS Build System (v$(VERSION))"
@echo ""
@echo "Build targets:"
@echo " make fetch Download Tiny Core ISO, KubeSolo, dependencies"
@echo " make kernel Build custom kernel with CONFIG_CGROUP_BPF=y"
@echo " make build-cloudinit Build cloud-init Go binary"
@echo " make build-update-agent Build update agent Go binary"
@echo " make rootfs Extract + prepare rootfs with KubeSolo"
@echo " make initramfs Repack rootfs into kubesolo-os.gz"
@echo " make iso Create bootable ISO (default target)"
@echo " make disk-image Create raw disk image with A/B partitions + GRUB"
@echo " make oci-image Create OCI container image for registry distribution"
@echo " make build-cross Cross-compile Go binaries for amd64 + arm64"
@echo " make quick Fast rebuild (re-inject + repack + ISO only)"
@echo " make docker-build Reproducible build inside Docker"
@echo ""
@echo "Test targets:"
@echo " make test-boot Boot ISO in QEMU, verify boot success"
@echo " make test-k8s Boot + verify K8s node reaches Ready"
@echo " make test-persist Reboot disk image, verify state persists"
@echo " make test-deploy Deploy nginx pod, verify Running"
@echo " make test-storage Test PVC with local-path provisioner"
@echo " make test-cloudinit Run cloud-init Go unit tests"
@echo " make test-update-agent Run update agent Go unit tests"
@echo " make test-update A/B update cycle integration test"
@echo " make test-rollback Forced rollback integration test"
@echo " make test-all Run core tests (boot + k8s + persistence)"
@echo " make test-integ Run full integration suite"
@echo " make bench-boot Benchmark boot performance (3 runs)"
@echo " make bench-resources Benchmark resource usage (requires running VM)"
@echo ""
@echo "Dev targets:"
@echo " make dev-vm Launch interactive QEMU VM"
@echo " make dev-vm-shell Launch QEMU VM -> emergency shell"
@echo " make dev-vm-debug Launch QEMU VM with debug logging"
@echo " make kernel-audit Check kernel config against requirements"
@echo " make shellcheck Lint all shell scripts"
@echo ""
@echo "Cleanup:"
@echo " make clean Remove build artifacts (preserve cache)"
@echo " make distclean Remove everything including cache"
Reference in New Issue View Git Blame Copy Permalink