Adolfo Delorenzo
31eee77397
Some checks failed
ARM64 Build / Build generic ARM64 disk image (push) Failing after 5s
CI / Go Tests (push) Successful in 3m51s
CI / Shellcheck (push) Successful in 1m5s
CI / Build Go Binaries (amd64, linux, linux-amd64) (push) Successful in 2m48s
CI / Build Go Binaries (arm64, linux, linux-arm64) (push) Successful in 2m50s
Fourth round of the v0.3 nftables-on-arm64 debug saga. After the
NF_TABLES_IPV4 family fix from 7e46f8f, KubeSolo + containerd + a
CoreDNS pod all reach Running state, but kube-proxy fails to install
Service rules:
add rule ip kube-proxy service-2QRHZV4L-default/kubernetes/tcp/https
numgen random mod 1 vmap { 0 : goto ... }
^^^^^^^^^^^^^^^^^^^
Error: Could not process rule: No such file or directory
The caret points at `numgen random mod 1`. That's the nftables
NUMGEN expression — kube-proxy's nftables backend uses it for random
endpoint load-balancing across Service endpoints. Without
CONFIG_NFT_NUMGEN compiled into the kernel, every Service sync fails
and kube-dns / any ClusterIP is unreachable.
Cascade: kube-proxy sync fail -> kube-dns Service has no DNAT ->
CoreDNS readiness probe never goes Ready -> KubeSolo's coredns
deploy step times out after 15 attempts -> FTL -> kernel panic.
Fix: add NFT_NUMGEN to kernel-container.fragment, plus the small
family of expression modules kube-proxy and CNI plugins commonly use
so we don't repeat this debug loop for the next missing one:
CONFIG_NFT_NUMGEN=m random / inc LB
CONFIG_NFT_HASH=m consistent-hash LB (sessionAffinity=ClientIP)
CONFIG_NFT_OBJREF=m named objects (counters, quotas) refs in rules
CONFIG_NFT_LIMIT=m rate-limit expression
CONFIG_NFT_LOG=m log expression (used by some CNI debug rules)
All =m so init's stage-30 loads them from modules.list / modules-arm64.list
alongside the existing nft_nat / nft_masq / nft_compat.
This needs another kernel rebuild (rm -rf build/cache/kernel-arm64-generic,
sudo make kernel-arm64) on the Odroid. After that we should have a fully
working KubeSolo OS v0.3 on ARM64 generic — at which point the only thing
left is to tag v0.3.1 and verify the rewritten release.yaml workflow
publishes both arches automatically.
Note on runc-PATH log noise: containerd-shim-runc-v2 -info probes for
runc in $PATH and fails because KubeSolo's runc lives at
/var/lib/kubesolo/containerd/runc. This is cosmetic — actual container
creation uses an absolute path from the containerd config and works
fine (CoreDNS container did start successfully). Will polish in v0.3.2.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
131 lines
3.7 KiB
Plaintext
131 lines
3.7 KiB
Plaintext
# KubeSolo OS — Shared kernel config fragment for container workloads
|
|
#
|
|
# Applied on top of:
|
|
# - Tiny Core stock config (x86_64) via build-kernel.sh
|
|
# - mainline kernel.org arm64 defconfig via build-kernel-arm64.sh
|
|
# - bcm2711_defconfig / bcm2712_defconfig via build-kernel-rpi.sh
|
|
#
|
|
# All entries here are architecture-agnostic.
|
|
# Apply this fragment twice with `make olddefconfig` between passes — TC's stock
|
|
# config has CONFIG_SECURITY disabled, which causes a single-pass olddefconfig
|
|
# to strip the security subtree before its dependencies (SYSFS, MULTIUSER) are
|
|
# resolved.
|
|
|
|
# cgroup v2 (mandatory for containerd/runc)
|
|
CONFIG_CGROUPS=y
|
|
CONFIG_CGROUP_CPUACCT=y
|
|
CONFIG_CGROUP_DEVICE=y
|
|
CONFIG_CGROUP_FREEZER=y
|
|
CONFIG_CGROUP_SCHED=y
|
|
CONFIG_CGROUP_PIDS=y
|
|
CONFIG_MEMCG=y
|
|
CONFIG_CGROUP_BPF=y
|
|
CONFIG_CFS_BANDWIDTH=y
|
|
|
|
# BPF (required for cgroup v2 device control)
|
|
CONFIG_BPF=y
|
|
CONFIG_BPF_SYSCALL=y
|
|
|
|
# Namespaces (mandatory for containers)
|
|
CONFIG_NAMESPACES=y
|
|
CONFIG_NET_NS=y
|
|
CONFIG_PID_NS=y
|
|
CONFIG_USER_NS=y
|
|
CONFIG_UTS_NS=y
|
|
CONFIG_IPC_NS=y
|
|
|
|
# Device management
|
|
CONFIG_DEVTMPFS=y
|
|
CONFIG_DEVTMPFS_MOUNT=y
|
|
|
|
# Filesystem
|
|
CONFIG_OVERLAY_FS=y
|
|
CONFIG_SQUASHFS=y
|
|
CONFIG_EXT4_FS=y
|
|
CONFIG_VFAT_FS=y
|
|
|
|
# Networking
|
|
CONFIG_BRIDGE=m
|
|
CONFIG_NETFILTER=y
|
|
CONFIG_NF_CONNTRACK=m
|
|
CONFIG_NF_NAT=m
|
|
CONFIG_NF_TABLES=m
|
|
CONFIG_VETH=m
|
|
CONFIG_VXLAN=m
|
|
|
|
# nftables address-family handlers. These are BOOL Kconfigs (not tristate)
|
|
# so they have to be built into the kernel — there's no module to modprobe
|
|
# at runtime. Without them, `nft add table ip ...` returns EOPNOTSUPP and
|
|
# KubeSolo v1.1.4+'s pod-masquerade setup fails at boot.
|
|
CONFIG_NF_TABLES_IPV4=y
|
|
CONFIG_NF_TABLES_IPV6=y
|
|
CONFIG_NF_TABLES_INET=y
|
|
CONFIG_NF_TABLES_NETDEV=y
|
|
|
|
# nftables expression modules used by KubeSolo's masquerade ruleset, the
|
|
# kube-proxy nft backend (Kubernetes 1.34+), and the xtables compat path.
|
|
# Listed in modules.list / modules-arm64.list so init loads them at boot.
|
|
CONFIG_NFT_NAT=m
|
|
CONFIG_NFT_MASQ=m
|
|
CONFIG_NFT_CT=m
|
|
CONFIG_NFT_REDIR=m
|
|
CONFIG_NFT_REJECT=m
|
|
CONFIG_NFT_REJECT_INET=m
|
|
CONFIG_NFT_COMPAT=m
|
|
CONFIG_NFT_FIB=m
|
|
CONFIG_NFT_FIB_IPV4=m
|
|
CONFIG_NFT_FIB_IPV6=m
|
|
# numgen drives kube-proxy's random / round-robin endpoint LB:
|
|
# `numgen random mod N vmap { ... }` in service rules.
|
|
# Without it kube-proxy's nft sync fails with ENOENT on every service.
|
|
CONFIG_NFT_NUMGEN=m
|
|
# hash drives consistent-hash LB (sessionAffinity=ClientIP, etc.).
|
|
CONFIG_NFT_HASH=m
|
|
# objref / limit / log are used by various policy expressions kube-proxy and
|
|
# CNI plugins emit. Including them pre-empts a future "could not process
|
|
# rule" debug loop.
|
|
CONFIG_NFT_OBJREF=m
|
|
CONFIG_NFT_LIMIT=m
|
|
CONFIG_NFT_LOG=m
|
|
|
|
# IPv4 NAT bits NFT_MASQ depends on. Auto-selected on most kernels but we
|
|
# pin them explicitly so olddefconfig doesn't strip them when the fragment
|
|
# is applied on top of a minimal defconfig.
|
|
CONFIG_NF_NAT_MASQUERADE=y
|
|
|
|
# Security: AppArmor + Audit
|
|
CONFIG_AUDIT=y
|
|
CONFIG_AUDITSYSCALL=y
|
|
CONFIG_SECURITY=y
|
|
CONFIG_SECURITYFS=y
|
|
CONFIG_SECURITY_NETWORK=y
|
|
CONFIG_SECURITY_APPARMOR=y
|
|
CONFIG_DEFAULT_SECURITY_APPARMOR=y
|
|
CONFIG_LSM=lockdown,yama,apparmor
|
|
|
|
# Security: seccomp
|
|
CONFIG_SECCOMP=y
|
|
CONFIG_SECCOMP_FILTER=y
|
|
|
|
# Crypto (image verification)
|
|
CONFIG_CRYPTO_SHA256=y
|
|
|
|
# Disable unnecessary subsystems for headless edge appliance
|
|
# CONFIG_SOUND is not set
|
|
# CONFIG_DRM is not set
|
|
# CONFIG_KVM is not set
|
|
# CONFIG_MEDIA_SUPPORT is not set
|
|
# CONFIG_WIRELESS is not set
|
|
# CONFIG_WLAN is not set
|
|
# CONFIG_CFG80211 is not set
|
|
# CONFIG_BT is not set
|
|
# CONFIG_NFC is not set
|
|
# CONFIG_INFINIBAND is not set
|
|
# CONFIG_PCMCIA is not set
|
|
# CONFIG_HAMRADIO is not set
|
|
# CONFIG_ISDN is not set
|
|
# CONFIG_ATM is not set
|
|
# CONFIG_INPUT_JOYSTICK is not set
|
|
# CONFIG_INPUT_TABLET is not set
|
|
# CONFIG_FPGA is not set
|