Second nft crash report from QEMU virt:
failed to set up pod masquerade
nft add table ip kubesolo-masq:
signal: aborted (output: *** stack smashing detected ***: terminated)
Root cause: two glibcs are visible to dynamically-linked binaries in the
rootfs. piCore64 ships glibc at /lib/libc.so.6; we copy the build host's
glibc (for the iptables-nft / nft / xtables-modules family) to
/lib/$LIB_ARCH/libc.so.6. The dynamic linker can resolve one binary's
NEEDED libc.so.6 to piCore's and another (via transitive load through
e.g. libnftables.so.1) to ours. Each libc has its own __stack_chk_guard
global; stack frames whose canary was written by code from libc-A and
checked by code from libc-B trip "stack smashing detected" → SIGABRT.
This didn't fire before nft was added because no host-installed dyn
binary actually got invoked before kubesolo crashed at first-boot
preflight.
Three layered fixes in inject-kubesolo.sh:
1. Bundle the full glibc family (was just libc.so.6 + ld). Now also
libpthread, libdl, libm, libresolv, librt, libanl, libgcc_s. Without
these, transitively-loaded host libs could pull them in from piCore's
/lib and re-introduce the split.
2. After bundling, delete piCore's duplicates from /lib/ where our copy
exists in /lib/$LIB_ARCH/. The dynamic linker's search now has
exactly one match per soname.
3. Write /etc/ld.so.conf giving /lib/$LIB_ARCH precedence over /lib, and
run `ldconfig -r "$ROOTFS"` to bake an explicit /etc/ld.so.cache.
The runtime linker uses the cache (when present) instead of falling
back to compiled-in default paths, making lookup order deterministic.
Also done (followups from previous commit):
- build/Dockerfile.builder gains nftables so docker-build picks up nft.
- .gitea/workflows/release.yaml's amd64 build job installs iptables +
nftables (previously only listed iptables-related libs but not the
CLIs themselves).
Verified by shellcheck. End-to-end QEMU verification on the Odroid next.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 7 of v0.3. The update agent can now pull update artifacts from any
OCI-compliant registry (ghcr.io, quay.io, harbor, zot, etc.) alongside the
existing HTTP latest.json protocol. Multi-arch artifacts are resolved
through manifest indexes so the same tag (e.g. "stable") yields the
right kernel + initramfs for runtime.GOARCH.
New package update/pkg/oci (~280 LOC, 9 tests):
- Client wraps oras-go/v2's remote.Repository. NewClient parses
host/path references; WithPlainHTTP toggle for httptest.
- FetchMetadata resolves a tag and returns image.UpdateMetadata from
manifest annotations (io.kubesolo.os.{version,channel,architecture,
min_compatible_version,release_notes,release_date}). No blobs fetched.
- Pull resolves the tag, walks index → arch-specific manifest, downloads
kernel + initramfs layers identified by their custom media types
(application/vnd.kubesolo.os.kernel.v1+octet-stream and
application/vnd.kubesolo.os.initramfs.v1+gzip), verifies their digests
against the manifest, returns the same image.StagedImage shape the
HTTP client produces.
- Cross-arch single-arch manifests are refused via the AnnotArch check
(defense in depth on top of the gates in cmd/apply.go).
- Tests use a hand-rolled httptest registry implementing /v2/probe,
manifest fetch by tag-or-digest, blob fetch by digest. Cover index
arch-selection, single-arch manifests, missing-arch error, tampered
blob rejection (digest mismatch), and reference parsing.
Dependencies added: oras.land/oras-go/v2 v2.6.0 plus its transitive
opencontainers/{go-digest,image-spec} and golang.org/x/sync. All small
and well-maintained; total binary size impact is negligible relative to
the existing 6.1 MB update agent.
cmd/apply.go:
- New --registry and --tag flags; mutually exclusive with --server.
- applyMetadataGates extracted as a helper, called from both transports
so channel/arch/min-version policy is enforced identically regardless
of how metadata was fetched.
- State transitions identical to the HTTP path: Checking → Downloading
→ Staged, with RecordError on any failure.
cmd/opts.go: --registry, --tag CLI flags. update.conf "server=" already
accepts either an HTTP URL or an OCI ref; the agent distinguishes by
which CLI/conf field carries the value.
build/scripts/push-oci-artifact.sh: new tool that publishes a single-arch
update artifact via the oras CLI with our custom media types and
annotations. After running for each arch, the operator composes the
multi-arch index with `oras manifest index create`. Documented inline.
build/Dockerfile.builder: installs oras 1.2.3 from upstream releases so
the Gitea Actions build container can run the new script.
Signature verification on the OCI path is intentionally deferred — the
artifact format is digest-verified end-to-end via oras-go, and Ed25519
signature consumption via OCI referrers is a follow-up. Plain HTTP
clients keep their existing signature path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
piCore64 v15.0.0 ships BusyBox built with ARM instructions that QEMU virt
cannot emulate even under -cpu max — applets like mkdir, uname, readlink
SIGILL on first invocation (el0_undef in the panic trace). mount works
because piCore's busybox.suid happens to use a different code path.
Fix: when building the arm64 rootfs, replace piCore's bin/busybox and
bin/busybox.suid with /bin/busybox from the build host (Ubuntu's
busybox-static, statically linked, built for generic ARMv8-A).
Also add busybox-static to Dockerfile.builder so the Docker-based build
flow has the same fallback available.
Long-term: source a known-good ARM64 BusyBox build (Alpine, or our own
from upstream BusyBox) so we don't depend on the build host's package
manager. Tracked as future work.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Produces a UEFI-bootable raw disk image for generic ARM64 hosts (QEMU virt,
Ampere/Graviton cloud, ARM64 SBCs with UEFI). Reuses the existing 4-partition
A/B layout from x86 (EFI 256 MB FAT32 + System A 512 MB ext4 + System B 512 MB
ext4 + Data ext4 remainder).
Changes:
- build/scripts/create-disk-image.sh: TARGET_ARCH env var (amd64 default,
arm64). Selects kernel source path, grub-mkimage target (x86_64-efi vs
arm64-efi), EFI binary name (bootx64.efi vs BOOTAA64.EFI), grub.cfg variant,
and whether to also install BIOS GRUB (x86 only).
- build/grub/grub-arm64.cfg: ARM64 variant of grub.cfg. Identical A/B logic;
console=ttyAMA0+ttyS0 to cover QEMU virt PL011, Ampere PL011, and Graviton
16550-compat.
- build/Dockerfile.builder: add grub-efi-amd64-bin, grub-efi-arm64-bin,
grub-pc-bin, grub-common, grub2-common so the builder container can produce
EFI images for both architectures.
- hack/dev-vm-arm64.sh: split into kernel mode (direct -kernel/-initrd, fast
iteration) and --disk mode (UEFI firmware + GRUB + disk image, full
integration test). Probes common UEFI firmware paths on Ubuntu/Fedora/macOS.
Default kernel path now points at kernel-arm64-generic/Image with fallback
to the renamed custom-kernel-rpi/Image.
- test/qemu/test-boot-arm64-disk.sh: new CI test for the full UEFI -> GRUB ->
kernel -> stage-90 boot chain. Uses a scratch copy of the disk so grubenv
writes don't mutate the source artifact.
- Makefile: new disk-image-arm64 target (depends on rootfs-arm64 + kernel-arm64),
new test-boot-arm64-disk target, .PHONY + help updates.
Phase 3 scaffold is in place. First real end-to-end ARM64 build runs in the
next step on the Odroid runner — that's where we find out what's actually
broken.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Add kpartx for reliable loop partition mapping in Docker containers
- Fix piCore64 download URL (changed from .img.gz to .zip format)
- Fix piCore64 boot partition mount (initramfs on p1, not p2)
- Fix tar --wildcards for RPi firmware extraction
- Add MIT license (same as KubeSolo)
- Add kpartx and unzip to Docker builder image
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Dockerfile.builder: Go 1.24.0 → 1.25.5 (go.mod requires it)
- test-boot.sh: use direct kernel boot via ISO extraction instead of
broken -cdrom + -append; fix boot marker to "KubeSolo is running"
(Stage 90 blocks on wait, never emits "complete")
- test-security-hardening.sh: same direct kernel boot and marker fixes
- run-vm.sh, dev-vm.sh, dev-vm-arm64.sh: quote QEMU -net args to
silence shellcheck SC2054
- fetch-components.sh, fetch-rpi-firmware.sh, dev-vm-arm64.sh: fix
trap quoting (SC2064)
Validated: full Docker build, 94 Go tests pass, QEMU boot (73s),
security hardening test (6/6 pass, 1 AppArmor skip pending kernel
rebuild).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Security hardening: bind kubeconfig server to localhost, mount hardening
(noexec/nosuid/nodev on tmpfs), sysctl network hardening, kernel module
loading lock after boot, SHA256 checksum verification for downloads,
kernel AppArmor + Audit support, complain-mode AppArmor profiles for
containerd and kubelet, and security integration test.
ARM64 Raspberry Pi support: piCore64 base extraction, RPi kernel build
from raspberrypi/linux fork, RPi firmware fetch, SD card image with 4-
partition GPT and tryboot A/B mechanism, BootEnv Go interface abstracting
GRUB vs RPi boot environments, architecture-aware build scripts, QEMU
aarch64 dev VM and boot test.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
