feat: custom kernel build + boot fixes for working container runtime

Build a custom Tiny Core 17.0 kernel (6.18.2) with missing configs
that the stock kernel lacks for container workloads:
- CONFIG_CGROUP_BPF=y (cgroup v2 device control via BPF)
- CONFIG_DEVTMPFS=y (auto-create /dev device nodes)
- CONFIG_DEVTMPFS_MOUNT=y (auto-mount devtmpfs)
- CONFIG_MEMCG=y (memory cgroup controller for memory.max)
- CONFIG_CFS_BANDWIDTH=y (CPU bandwidth throttling for cpu.max)

Also strips unnecessary subsystems (sound, GPU, wireless, Bluetooth,
KVM, etc.) for minimal footprint on a headless K8s edge appliance.

Init system fixes for successful boot-to-running-pods:
- Add switch_root in init.sh to escape initramfs (runc pivot_root)
- Add mountpoint guards in 00-early-mount.sh (skip if already mounted)
- Create essential device nodes after switch_root (kmsg, console, etc.)
- Enable cgroup v2 controller delegation with init process isolation
- Mount BPF filesystem for cgroup v2 device control
- Add mknod fallback from sysfs in 20-persistent-mount.sh for /dev/vda
- Move KubeSolo binary to /usr/bin (avoid /usr/local bind mount hiding)
- Generate /etc/machine-id in 60-hostname.sh (kubelet requires it)
- Pre-initialize iptables tables before kube-proxy starts
- Add nft_reject, nft_fib, xt_nfacct to kernel modules list

Build system changes:
- New build-kernel.sh script for custom kernel compilation
- Dockerfile.builder adds kernel build deps (flex, bison, libelf, etc.)
- Selective kernel module install (only modules.list + transitive deps)
- Install iptables-nft (xtables-nft-multi) + shared libs in rootfs

Tested: ISO boots in QEMU, node reaches Ready in ~35s, CoreDNS and
local-path-provisioner pods start and run successfully.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-02-11 23:13:31 -06:00
parent 456aa8eb5b
commit 39732488ef
13 changed files with 794 additions and 77 deletions

View File

@@ -1,4 +1,4 @@
.PHONY: all fetch build-cloudinit build-update-agent build-cross rootfs initramfs \
.PHONY: all fetch kernel build-cloudinit build-update-agent build-cross rootfs initramfs \
iso disk-image oci-image \
test-boot test-k8s test-persistence test-deploy test-storage test-all \
test-cloudinit test-update-agent \
@@ -30,6 +30,10 @@ fetch:
# =============================================================================
# Build stages
# =============================================================================
kernel:
@echo "==> Building custom kernel (CONFIG_CGROUP_BPF=y)..."
$(BUILD_DIR)/scripts/build-kernel.sh
build-cloudinit:
@echo "==> Building cloud-init binary..."
$(BUILD_DIR)/scripts/build-cloudinit.sh
@@ -38,7 +42,7 @@ build-update-agent:
@echo "==> Building update agent..."
$(BUILD_DIR)/scripts/build-update-agent.sh
rootfs: fetch build-cloudinit build-update-agent
rootfs: fetch kernel build-cloudinit build-update-agent
@echo "==> Preparing rootfs..."
$(BUILD_DIR)/scripts/extract-core.sh
$(BUILD_DIR)/scripts/inject-kubesolo.sh
@@ -176,7 +180,7 @@ docker-build:
docker run --rm --privileged \
-v $(PWD)/$(OUTPUT_DIR):/output \
-v $(PWD)/$(CACHE_DIR):/cache \
kubesolo-os-builder make iso OUTPUT_DIR=/output CACHE_DIR=/cache
kubesolo-os-builder iso OUTPUT_DIR=/output CACHE_DIR=/cache
# =============================================================================
# Cleanup
@@ -197,6 +201,7 @@ help:
@echo ""
@echo "Build targets:"
@echo " make fetch Download Tiny Core ISO, KubeSolo, dependencies"
@echo " make kernel Build custom kernel with CONFIG_CGROUP_BPF=y"
@echo " make build-cloudinit Build cloud-init Go binary"
@echo " make build-update-agent Build update agent Go binary"
@echo " make rootfs Extract + prepare rootfs with KubeSolo"