Why Cross-Compiled Kernels Work on Raspberry Pi—And Why Modules Sometimes Fail

Cross-compiled kernels and modules can work seamlessly on a Raspberry Pi system due to several critical factors:

1. Architecture Compatibility

The cross-compiler (e.g., aarch64-linux-gnu-gcc) generates binary code specifically for the ARM64 architecture used by Raspberry Pi. This ensures the compiled kernel and modules can execute on the target hardware, even if compiled on a different system.

2. Binary Interface Consistency

When the kernel and its modules are compiled with the same compiler version and flags, they share a consistent Application Binary Interface (ABI). This ABI alignment ensures components communicate correctly at the binary level, with matching function calls, data structures, and memory layouts.

3. Hardware Abstraction

The Linux kernel abstracts hardware specifics, allowing the same compiled code to run on different physical devices as long as they share the same architecture (ARM64 in this case). This abstraction layer handles device-specific details, enabling portability.

4. Compiler Optimization

Cross-compilers are tailored to generate optimized machine code for the target architecture. For example, they might leverage ARM64-specific instructions or memory alignment strategies, ensuring efficient execution even when compiled on a different host system.

Why Your Cross-Compiled Kernel Works but Locally Built Modules Don’t

The core issue is compiler inconsistency.

• Your kernel was cross-compiled using aarch64-linux-gnu-gcc (Ubuntu 11.4.0) on a separate system.

• When building modules locally on the Raspberry Pi, you’re using aarch64-linux-gnu-gcc (Debian 12.2.0)—a different compiler version.

Even though both compilers target ARM64, their internal differences impact:

• Symbol versioning: Checksums for kernel functions/data structures.

• Function calling conventions: How arguments are passed between functions.

• Memory layout: Padding and alignment of data structures.

• Compiler optimizations: Code generation strategies (e.g., loop unrolling).

The kernel enforces strict ABI consistency to prevent instability. If these details don’t match exactly, modules are rejected with errors like “Invalid module format” or “disagrees about version of symbol module_layout”.

Technical Checks During Module Loading

When a module is loaded, the kernel verifies:

• Vermagic: A metadata string encoding the kernel version, configuration, and compiler details.

• Module CRC: Checksums of symbols (e.g., functions, variables) to ensure they match the kernel’s expectations.

• Module Layout: Memory organization of data structures, which must align with the kernel’s own layout.

Mismatches in any of these areas trigger load failures to protect system stability.

The Solution

For modules to work with your kernel:

• Compile them with the exact same compiler version and flags used for the kernel.

• Example: If the kernel was built with Ubuntu’s GCC 11.4.0, use that compiler for modules—not Debian’s GCC 12.2.0.

• Best practice: Rebuild modules on the same system where the kernel was compiled, or reuse pre-built modules from that environment.

Why Aren’t Other Raspberry Pi OS Components Affected?

Userspace applications (e.g., system utilities, desktop apps) work fine despite compiler differences because they interact with the kernel through stable interfaces:

1. System Call InterfacePrograms like ls or grep use standardized system calls (e.g., open(), read()), which are designed to remain stable across compiler versions.

2. Standard C Library (libc)Most apps rely on libc as an intermediary, which provides a consistent API/ABI regardless of how the kernel was compiled.

3. Dynamic LinkingShared libraries (e.g., libssl.so) abstract compiler-specific details, enabling compatibility across environments.

4. Userspace ABI StabilityThe kernel guarantees backward compatibility for userspace programs. This “stable ABI, unstable API” principle allows the kernel’s internal code to evolve while keeping user-facing interfaces consistent.

Key Differences: Kernel Modules vs. Userspace Apps

• Kernel Modules:

• Run in kernel space with full privileges.

• Directly access kernel internals (structures, functions).

• Require exact ABI matching with the kernel.

• Userspace Apps:

• Run in isolated memory spaces.

• Interact with the kernel via stable system calls.

• Depend on standardized interfaces (libc, dynamic linking).

Summary

• Cross-compiled kernels work because they target the correct architecture and leverage hardware abstraction.

• Modules fail with compiler mismatches due to strict ABI checks for symbol versions, memory layouts, and optimizations.

• Userspace apps remain unaffected because they use stable, version-agnostic interfaces.

This dichotomy exists by design: the kernel protects its internal integrity while ensuring userspace compatibility. For developers, this means always matching compiler versions when building modules—and enjoying the flexibility of cross-compilation for the rest of the system!