Building OpenWRT on a PaX-Enabled Linux Host

I just went through the process of troubleshooting why OpenWRT will not build on my laptop which uses a PaX-enabled Linux kernel. It wasn't super-fun figuring out why things didn't work, so I thought I would share how I fixed it so that it might save you some time.

For completeness, my system is currently using Linux kernel 4.3.3-r1 from Gentoo with PaX and Grsecurity enabled, and it has a hardened userland. Here are the PaX options that my kernel has configured. This might not be ideal, but I have to use this system too.

$ grep _PAX_ .config
CONFIG_PAX_USERCOPY_SLABS=y
# CONFIG_PAX_SOFTMODE is not set
# CONFIG_PAX_PT_PAX_FLAGS is not set
CONFIG_PAX_XATTR_PAX_FLAGS=y
# CONFIG_PAX_NO_ACL_FLAGS is not set
CONFIG_PAX_HAVE_ACL_FLAGS=y
# CONFIG_PAX_HOOK_ACL_FLAGS is not set
CONFIG_PAX_NOEXEC=y
CONFIG_PAX_PAGEEXEC=y
CONFIG_PAX_EMUTRAMP=y
CONFIG_PAX_MPROTECT=y
# CONFIG_PAX_MPROTECT_COMPAT is not set
# CONFIG_PAX_ELFRELOCS is not set
# CONFIG_PAX_KERNEXEC is not set
CONFIG_PAX_KERNEXEC_PLUGIN_METHOD=""
CONFIG_PAX_ASLR=y
# CONFIG_PAX_RANDKSTACK is not set
CONFIG_PAX_RANDUSTACK=y
CONFIG_PAX_RANDMMAP=y
CONFIG_PAX_MEMORY_SANITIZE=y
CONFIG_PAX_MEMORY_STACKLEAK=y
CONFIG_PAX_MEMORY_STRUCTLEAK=y
# CONFIG_PAX_MEMORY_UDEREF is not set
CONFIG_PAX_REFCOUNT=y
CONFIG_PAX_USERCOPY=y
# CONFIG_PAX_USERCOPY_DEBUG is not set
CONFIG_PAX_SIZE_OVERFLOW=y
CONFIG_PAX_LATENT_ENTROPY=y
$

Normally, when certain binaries need PaX features disabled you would use the setfattr command. For example, to disable MPROTECT on /usr/bin/perl:

sudo setfattr -n user.pax.flags -v "m" /usr/bin/perl

This is fine for binaries that exist and are rarely overwritten. That breaks-down however when building OpenWRT since the build process builds utilities and then executes them later in the build process. For this, we need to use paxd so that the PaX markings can be set as soon as the binaries are created.

paxd is dead-easy to setup. It uses the file /etc/paxd.conf to list the flag and file combinations it should watch for and set.

When building OpenWRT, it was MPROTECT and a small number of utilities that caused me issues. I used paxd to disable MPROTECT on the system utilities and OpenWRT-built files. My /etc/paxd.conf looks like this:

m /usr/bin/perl
m /usr/bin/wget
m /usr/bin/openssl
m /usr/libexec/git-core/git-remote-http

m /home/userid/openwrt/build_dir/host/openssl-1.0.2a/apps/openssl
m /home/userid/openwrt/staging_dir/host/bin/qemu-img
m /home/userid/openwrt/staging_dir/host/bin/mkimage

If these paths change, or if your build needs other flags modified, you will have to modify your paxd.conf accordingly. If flags are missing, you'll likely see something like this in the kernel log.

Jan 4 12:50:26 hostname kernel: [137577.409937] grsec: denied marking stack executable as requested by PT_GNU_STACK marking in /home/userid/openwrt/staging_dir/host/lib/libcrypto.so.1.0.0 by /home/userid/openwrt/staging_dir/host/bin/mkimage[mkimage:26986] uid/euid:1001/1001 gid/egid:105/105, parent /usr/bin/gmake[make:25045] uid/euid:1001/1001 gid/egid:105/105

Jan 4 12:50:26 hostname kernel: [137577.409957] grsec: denied RWX mprotect of by /home/userid/openwrt/staging_dir/host/bin/mkimage[mkimage:26986] uid/euid:1001/1001 gid/egid:105/105, parent /usr/bin/gmake[make:25045] uid/euid:1001/1001 gid/egid:105/105