Cross-compiling Complex Rust Programs for OpenWrt Targets

We made Rust our language of choice for Althea because we wanted to have our cake and eat it to. The ability to use a rich ecosystem of existing libraries, have strong guarantees on safety, and small and efficient binaries that could run on our target OpenWrt devices.

After overcoming several different compilation and size issues we’ve mostly achieved our goal. While Rust binaries are still very big by normal embedded device standards (on the order of 2mb) they are small enough that we can ship full featured applications developed without restrictions and have them run well.

This guide is about issues with advanced syntax and building more complex crates, if you want to cross compile a simple ‘hello world’ program start with the excellent rust-cross guide.

proc_macro

High level abstractions with low level results are littered through out the Rust language. In this example we’ve defined serialization for the base types and then can simply automatically derive it for all objects containing those types.

#[derive(Debug, Serialize, Deserialize, PartialEq)]
   struct MyStruct {
       addr: EthAddress,
       sig: EthSignature,
       key: EthPrivateKey
   }

This compiles perfectly on normal x86 targets, works perfectly too. Everything seems perfect until:

[justin@aperturescience.robot serde]$ cargo build --target mips-unknown-linux-musl   
Compiling proc-macro2 v0.2.1   
Compiling serde_test v1.0.27 (file:///home/justin/repos/serde/serde_test)   Compiling quote v0.4.2
error[E0463]: can't find crate for `proc_macro`  --> /home/justin/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.2.1/src/lib.rs:27:1   
|27 | extern crate proc_macro;   
| ^^^^^^^^^^^^^^^^^^^^^^^^ can't find crate
error: aborting due to previous error
error: Could not compile `proc-macro2`.
warning: build failed, waiting for other jobs to finish...
error: build failed

What is proc_macro and why doesn’t it exist when compiling for targets that aren’t x86? proc_macro is a compiler plugin system. Since Rust allows crates like Serde to provide syntactic sugar that gets turned into actual code the compiler structure has to accommodate code generation from a different source than itself.

When you go to compile a Rust program with #[derive(Serialize)] or some other property provided by a crate like Serde you’re building a proc_macro crate, these special crates generate a compiler plugin that’s used at compile time to build the code that #[derive(Serialize)] translates to.

So trying to crosscompile a proc_macro crate doesn’t really make any sense. You only need the plugin to generate code and all of it’s functionality is useless on the cross architecture you’re building for. Therefore Cargo just tells you that this useless version of proc_macro doesn’t exist.

This is caused by having a specific type of useless dependency. If you’ve tried to extend or define other proc macro traits you may have a extern crate quote and or extern crate syn somewhere that’s no longer needed.

This can happen if you wrote some derivations that used proc-macro functionality and then re-factored them out without removing the dependency import. Since you aren’t defining anything that actually needs a macro the compiler will let you leave off the required tag in the Cargo.toml.

Inside of this edge case you will get the above error, it can be corrected by either removing the creates quote and syn if they are no longer needed or properly defining and using a macro in the crate that imports them.

Big thanks to Manish Goregaokar for helping improve this section of the article.

Linking

With the proc_macro issue gone we can try again. Only to see this:

error: linking with `/home/justin/repos/althea-firmware/build/staging_dir/toolchain-mips_24kc_gcc-5.5.0_musl/bin/mips-openwrt-linux-gcc` 
failed: exit code: 1 |  = note: "/home/justin/repos/althea-firmware/build/staging_dir/toolchain-mips_24kc_gcc-5.5.0_musl/bin/mips-openwrt-linux-gcc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-L"

Followed by a horrifyingly large number of libs. But the key line is at the bottom.

/home/justin/repos/althea_rs/target/mips-unknown-linux-musl/debug/deps/libminiz_sys-9df090e5aeaaf6af.rlib: error adding symbols: File in wrong format

Oh no, an x86 library has found it’s way into our mips linking operation! Cargo was building and linking for the right platform but somehow those flags got ignored for some subset of the libraries.

Crates like cc-rs are used to link external C or C++ libraries into Rust programs at compile time. You might notice that the popular rust-openssl crate uses this method to link against the system OpenSSL.

It’s not enough to tell Cargo what to compile for, you must also let cc-rs know that it’s cross compiling and rust-openssl needs to know where the target OpenSSL headers are located.

#!/bin/bashexport CARGO_TARGET_MIPS_UNKNOWN_LINUX_MUSL_LINKER=/home/justin/repos/althea-firmware/build/staging_dir/toolchain-mips_24kc_gcc-5.5.0_musl/bin/mips-openwrt-linux-gccexport TARGET_CC=/home/justin/repos/althea-firmware/build/staging_dir/toolchain-mips_24kc_gcc-5.5.0_musl/bin/mips-openwrt-linux-gccexport HOST_CC=gccexport MIPS_UNKNOWN_LINUX_MUSL_OPENSSL_DIR=/home/justin/repos/althea-firmware/build/staging_dir/target-mips_24kc_musl/usr/export PKG_CONFIG_ALLOW_CROSS=1cargo build --target mips-unknown-linux-musl --release

Notice how in this build script we not only specify CARGO_TARGET but also TARGET_CC and HOST_CC which are read by cc-rs to determine what to compile for where. Finally we specify the OPENSSL_DIR for our target architecture. This script can be simplified some, but is useful to illustrate all the moving parts clearly.

Building natively

Now that we have our binary compiled how do we get it onto a OpenWRT device? The easiest and best integrated way is to build as part of a build system feed using a makefile.

You can find ours in the althea-packages repo with the final and most minimal set of flags you need to compile.

define Build/Compile	
(\		
cd $(PKG_BUILD_DIR) && \\		
PKG_CONFIG_ALLOW_CROSS=1 \		
OPENSSL_DIR=$(STAGING_DIR)/usr/ \		
RUSTFLAGS="-C linker=$(TARGET_CC)" \		
TARGET=$(RUST_TRIPLE) \		
CC=$(TARGET_CC) \		
CFLAGS="$(TARGET_CFLAGS)" \		
TARGET_CFLAGS="$(TARGET_CFLAGS)" \		
CXX=$(TARGET_CXX) \		
CXXFLAGS="$(TARGET_CXXFLAGS)" \		
TARGET_CXXFLAGS="$(TARGET_CXXFLAGS)" \\		

cargo build --all --release --target $(RUST_TRIPLE) \	
)	$(STRIP) $(RUST_BIN_PATHS)
endef

Be sure to include the binary stripping step and use some of the less invasive options to shrink binary size in this guide.

We can’t guarantee that these steps will work for all Crates, for example our makefile may need HOST_CC defined if cc-rs required it. But we wanted to document how we managed to get no compromises Rust code onto OpenWRT. Fulfilling the promise of a high level development environment with low level outcomes.