During my recent personal research, I came across an embedded device running a custom Linux-based system. One of my goals, when evaluating its security, was to programmatically monitor the system’s processes and hook various custom libraries for logging their actions.

Objective

#

Through my preceding research, I have identified a target library (target.so) that was being referenced by various other ELFs. Some of these processes were already running, while others could be invoked by dynamic events or run periodically at unknown intervals.

Since the function I wanted to hook was pretty simple, one of my early ideas was to just modify /etc/ld.so.preload to include my hook shared library in all future processes. This turned out to be problematic as most of the system’s partitions were read-only, enforced at the hardware level. Of course, this big limitation also prevented me from setting any additional environment variables like LD_PRELOAD.

Enter Frida

#

The next obvious choice was Frida, an extremely powerful dynamic instrumentation toolkit. Although maybe a bit overkill for my purpose, Frida would allow me to easily reproduce my results in the future and provide me the ability to create more complex hooks with ease.

At the same time, this was also the point where many major frustrations arose.

The problems

#

Even though my target’s device CPU supported the hard-float ABI (armhf), the system’s developers opted to compile all binaries in soft-float ABI (armel). This might seem like an obvious point of caution to people more experienced with embedded systems, but personally, apart from maybe size constraints, it doesn’t make much sense.

When I first gained a root shell on the device, I checked to see exactly what kind of system I’m dealing with. One of my data points was the CPU model or supported features.

root@target:/# cat /proc/cpuinfo
processor       : 0
model name      : ARMv7 Processor rev 5 (v7l)
BogoMIPS        : 100.00
Features        : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae 
CPU implementer : 0x41
CPU architecture: 7
CPU variant     : 0x0
CPU part        : 0xc07
CPU revision    : 5

As we can see, the CPU clearly supports armhf. As a result, for simpler tools like dropbear or gdbserver, choosing a musl-based cross-compiler with armhf support and statically compiling, proved sufficient.

One such compiler is conveniently provided by Void Linux: https://pkgs.org/search/?q=musleabihf

Frida Server

#

Unfortunately, running a frida server on the device wasn’t as simple.

During my initial attempts, I thought I could just download the precompiled frida-server-armhf variant from GitHub and proceed with my hooking efforts. The binary could run on the target, after replacing the interpreter path with the correct ld path found on the device.

patchelf --set-interpreter /lib/ld-linux.so.3 frida-server-16.6.6-linux-armhf

The screenshot below depicts the subsequent problems encountered. Despite frida-server successfully starting and being able to communicate with a Frida client, it was unable to actually attach to any processes, throwing the error «Failed to attach: unable to load library».

Analyzing Frida’s loader, we can see this error message is emitted when frida-server fails to dynamically load the friga-agent shared object. The root cause of the libc->dlopen failure stems from the inability of the dynamic loader (ld) to load a hard-float ABI object, as mentioned previously.

    ctx->agent_handle = libc->dlopen (agent_path, libc->dlopen_flags, pretend_caller_addr);
    if (ctx->agent_handle == NULL)
      goto dlopen_failed;

    if (agent_codefd != -1)
    {
      libc->close (agent_codefd);
      agent_codefd = -1;
    }

    ctx->agent_entrypoint_impl = libc->dlsym (ctx->agent_handle, ctx->agent_entrypoint, pretend_caller_addr);
    if (ctx->agent_entrypoint_impl == NULL)
      goto dlsym_failed;
  }

Source: https://github.com/frida/frida-core/blob/9594e1a8b81f151f49dedafeca1d85a33654137c/src/linux/helpers/loader.c#L116

Compiling Frida

#

While probably other tricks exist, to avoid compiling the whole project, it’s safer to assume a compilation will have higher chances of success than anything else.

In order to compile such a project on my x86-64 PC, I required a toolchain that:

• Can compile for ARM soft-float ABI, as most cross-toolchains support by default only armhf.
• Builds against GNU libc, and especially as close to libc version 2.24 as possible.
• Can be run on an environment modern enough, to support all Frida building requirements¹, especially Node 18.

For my use case, I discovered two methods that both resulted in a fully capable frida-server running on the embedded device.

Correct way: Crosstool-NG

#

The reason I’ve labeled this as the correct way, stems from the ability of crosstool-NG to build toolchains for almost any kind of system configuration. Despite this, I avoided this method when I was rushing to get something running, due to my unfamiliarity with the project.
After I achieved my research goals, I revisited the project and took some time to build Frida using it.

Due to Frida requiring Node as a build dependency, I’ve chosen to run all subsequent compilation steps in a Docker container, using as base the node:18 image.

Generating the toolchain

#

Having installed crosstool-NG in the container, we need to configure our toolchain.

As a safe starting point, I’ve chosen the arm-unknown-linux-gnueabi template. Below, we can see the default configuration of this template, which we will need to tweak to match our needs.

root@dad2bad6b3a5:/crosstool# ct-ng show-arm-unknown-linux-gnueabi
[G...]   arm-unknown-linux-gnueabi
    Languages       : C,C++
    OS              : linux-6.13
    Binutils        : binutils-2.43.1
    Compiler        : gcc-14.2.0
    Linkers         :
    C library       : glibc-2.41
    Debug tools     : duma-2_5_21 gdb-16.2 ltrace-0.7.3 strace-6.13
    Companion libs  : expat-2.5.0 gettext-0.23.1 gmp-6.3.0 isl-0.26 libelf-0.8.13 libiconv-1.16 mpc-1.3.1 mpfr-4.2.1 ncurses-6.4 zlib-1.3.1 zstd-1.5.6
    Companion tools :

In order to tweak this configuration, we first need to load it to our current directory with the command: ct-ng arm-unknown-linux-gnueabi.

Then we can start changing its options to our target values with the command: ct-ng menuconfig.

For my case, I tried to keep things simple, not to risk breaking the toolchain. I only selected glibc version 2.24 and Linux kernel version 4.19, all other important settings like soft-float ABI were already preconfigured from the selected template.

After 10 minutes of compiling, a new directory will be created housing the toolchain libraries and executables.

Building the Frida server

#

Finally, all we have to do is add the toolchain into our PATH and follow the instructions from Frida’s building documentation¹.

This can be summed up in 3 steps:

1. Clone https://github.com/frida/frida.git in a folder.
2. Run the ./configure script specifying the components to enable/disable and the suffix of our cross-compiler under the --host= switch.
3. Compile everything into a single binary with make.

More frustrations

#

While I wish I wouldn’t have to write this section, it may be useful in case anyone else faces these problems in the future.

Remember earlier, I’ve chosen for my toolchain the latest version of GCC compiler 14.2. Turns out, with every GCC release greater than version 6, the resulting frida-server binary depended on the libatomic.so library, which didn’t exist on my embedded device.

This library requirement presumably originated from the openssl dependency of Frida. Reading the code responsible for adding support for atomic operations, let me to believe that defining the preprocessor variable __STDC_NO_ATOMICS__ would make openssl fallback to using locks, removing the libatomic requirement, albeit making the code somewhat slower.

The only way I’ve found to add a preprocessor variable, without patching Frida’s build scripts, was to append meson options to the ./configure script. Environment variables like CFLAGS or CXXFLAGS weren’t picked up.
Example: ./configure --switch1 --switch2 -- -Dc_args="-D__STDC_NO_ATOMICS__=1"

Unfortunately, this wasn’t enough; while I could see __STDC_NO_ATOMICS__ being defined in openssl’s compilation command line, the resulting ELF still depended on libatomic. To overcome this problem, I decided to use GCC version 6, the same GCC version that the libc found on my target device was compiled with.

Crosstool-NG - Dockerfile

#

The following Dockerfile includes all previously described steps for compiling a frida-server for my target device.
Running the command docker build -o . . should save a frida-server binary in the current directory.

FROM node:18 AS builder

# crosstool-ng deps
RUN apt-get update && \
    apt-get install -y gcc g++ gperf bison flex texinfo help2man make libncurses5-dev \
    python3-dev autoconf automake libtool libtool-bin gawk wget bzip2 xz-utils unzip \
    patch libstdc++6 rsync git meson ninja-build

# install crosstool-ng
ARG crosstool_ver=1.27.0

WORKDIR /crosstool

RUN wget https://github.com/crosstool-ng/crosstool-ng/releases/download/crosstool-ng-${crosstool_ver}/crosstool-ng-${crosstool_ver}.tar.xz && \
    tar -xvf crosstool-ng-${crosstool_ver}.tar.xz && \
    cd crosstool-ng-${crosstool_ver} && \
    ./configure --prefix=/ && \
    make && make install


# build the cross-toolchain
WORKDIR /toolchain

COPY <<EOF ./defconfig
CT_CONFIG_VERSION="4"
CT_EXPERIMENTAL=y
CT_ALLOW_BUILD_AS_ROOT=y
CT_ALLOW_BUILD_AS_ROOT_SURE=y
# CT_REMOVE_DOCS is not set
CT_ARCH_ARM=y
CT_ARCH_FLOAT_SW=y
CT_TARGET_VENDOR="frida"
CT_KERNEL_LINUX=y
CT_LINUX_V_4_19=y
CT_BINUTILS_LINKER_LD_GOLD=y
CT_BINUTILS_GOLD_THREADS=y
CT_BINUTILS_LD_WRAPPER=y
CT_BINUTILS_PLUGINS=y
CT_GLIBC_V_2_24=y
CT_GCC_V_6=y
# CT_CC_GCC_SJLJ_EXCEPTIONS is not set
CT_CC_LANG_CXX=y
CT_COMP_LIBS_EXPAT=y
CT_COMP_LIBS_LIBELF=y
EOF

RUN ct-ng defconfig && ct-ng build.24

ENV PATH="${PATH}:/root/x-tools/arm-frida-linux-gnueabi/bin/"

# build frida-server
WORKDIR /build

RUN git clone --depth=1 https://github.com/frida/frida.git && \
    cd frida/ && mkdir build && cd build/ && \
    ../configure --host=arm-frida-linux-gnueabi --without-prebuilds=sdk:host --enable-server \
    --disable-frida-tools --disable-graft-tool --disable-gadget --disable-inject --disable-frida-python && \
    make -j24

# export just frida-server
FROM scratch

COPY --from=builder /build/frida/build/subprojects/frida-core/server/frida-server /

Hacky way: Debian

#

As mentioned previously, during my research rush, I didn’t have the time to fully investigate crosstool-ng. I needed something that could help me get Frida compiled and running on the device as fast as possible. The idea was to think like the system’s developers, meaning the combination of glibc and GCC versions, would probably not be something very exotic.

A quick search on the awesome website https://pkgs.org, revealed that indeed the biggest Linux distributions offered pre-built armel compilers; the only problem was their included glibc version.

Taking a look at Debian’s package search results for libc6-armel-cross, we can see that Debian buster offers glibc version 2.28, close but not exactly my target’s 2.24.

I needed to pick, hopefully, one version before Debian stretch. The only reliable source of versioning for such old packages I found to be https://distrowatch.com.

Loading the full package list from distrowatch’s page, allowed me to confirm my hypothesis! Debian stretch contained exactly the package I was looking for!

Hacky way - Dockerfile

#

The only thing left to do now, is to force install such old packages into the much newer Debian bookworm that node:18 is based upon.

One way to solve this problem, was to download these very old packages (along with their dependencies) from debian:stretch and force install them onto the node:18 container. This is essentially based on the fact that programs built for older glibc versions should be able to run on newer glibc versions, following glibc backward-compatible guarantees.

This forceful installation added the arm-linux-gnueabi-* family of tools and libraries in the standard system paths, as if they were installed with apt.

The build process remains largely the same as before, with the only minor difference being that a plain ./configure didn’t work for this case. Again, more openssl errors were generated, but these were quickly avoided by using ./releng/deps.py build before retrying to configure the project.

FROM debian:stretch AS stretch

RUN sed -i -re 's/deb.debian.org|security.debian.org/archive.debian.org/g' /etc/apt/sources.list && \
    sed -i -re 's/-updates/-proposed-updates/g' /etc/apt/sources.list

RUN apt-get update && \
    apt-get install -y --download-only g++-arm-linux-gnueabi

FROM node:18 AS builder

RUN apt-get update && \
    apt-get install -y python3 git build-essential

WORKDIR /oldgcc

COPY --from=stretch /var/cache/apt/archives/*.deb ./

RUN dpkg -i --force-all *.deb

WORKDIR /build

RUN git clone --depth=1 https://github.com/frida/frida.git && \
    cd frida/ && mkdir build && cd build/ && \
    ../configure --enable-server --disable-frida-tools --disable-graft-tool --disable-gadget --disable-inject --disable-frida-python --host=arm-linux-gnueabi; \
    ../releng/deps.py build --bundle=sdk --host=arm-linux-gnueabi --exclude v8 && \
    ../configure --enable-server --disable-frida-tools --disable-graft-tool --disable-gadget --disable-inject --disable-frida-python --host=arm-linux-gnueabi && \
    make -j24

FROM scratch

COPY --from=builder /build/frida/build/subprojects/frida-core/server/frida-server /

Conclusions

#

Frida is a very powerful tool, and it’s unfortunate there aren’t many available blogs for using it beyond mobile applications. I hope this article can help people, in similar situations, solve their Frida compilation problems.

All the build processes and Dockerfiles, have been last tested and confirmed to be working as of March 2025.