After my big “rewrite it in Rust” campaign for Kitchen Server, then immediately letting it languish for a month to go work on another project, I finally got around to addressing an unfortunate revelation. In switching away from Python, I gave up its automagical cross-platform capabilities and learned that Docker isn’t quite as automagical as I previously thought. Where before I could basically target a Linux environment within the container and tell Docker to make it cross-compatible, I learned that Docker doesn’t actually emulate the platform in question to the extent that it can run x86 binaries on my ArmV8 Raspberry Pi.

I guess this was always a learning project for me, right? Toward that end, this post serves as a collection of my thoughts – a retrospective for myself, and maybe a guide for someone else.

But good news! docker buildx has tools to pass platform architectures to the Dockerfile as variables. So the solution should just be to pass those variables to Rust when we call cargo build, right?

Problem one: Docker’s platform variables are a different format from Rust’s target triples. Docker provides something like linux/arm64, which should trigger a build with --target aarch64-unknown-linux-musl (that’s architecture, vendor, OS, and library). But surely there’s a way for the Dockerfile to do a little substitution, right?

Not as far as I can tell. There’s no if statements or any way to dynamically change variable values during runtime. This is probably for the best, but the Dockerfile standard is not Turing complete. So to get dynamic behavior, I actually got as far as running a Bash script to determine the target triple from the platform, and calling cargo build from within that script.

Then I hit problem two: the Docker image I use to cross-compile my code uses image tags that are also a different format (more in line with the Rust standard). This was a bigger problem because of the first line of the Dockerfile is

FROM --platform=$BUILDPLATFORM messense/rust-musl-cross:??? AS build

Again, Dockerfile is not Turing complete, and I had no way to translate e.g. linux/amd64 to the required x86_64-musl tag.

At this point I decided to acknowledge the whole approach smelled anyway. I was running CI with Docker in Docker so I could build my program in a different Docker container which was then immediately discarded. I was burning precious CI minutes compiling everything from scratch every time, which was particularly silly because I was already caching my builds in [Tagalyzer](), my other project.

Back to the drawing board. The good news is that for all the dragons I’ve heard of in cross-compilation, it was actually surprisingly easy to get a working plan:

1. Compile the binaries in separate jobs, storing them as artifacts
2. Take the build phase out of the Dockerfile, copying the binaries from the file system instead of the build image
3. …which in turn requires that the binaries are in directories accessible dynamically from within the Dockerfile.

Fortunately, Docker’s aberrant nomenclature actually came in handy here: linux/amd64 actually makes a valid directory structure. Here’s a simplified version of the solution:

build-amd64-bin:
  stage: build
  image: "$RUST_IMAGE:x86_64-musl" # Equivalent to linux/amd64
  script:
    - cargo build --release --target x86_64-unknown-linux-musl
    # No testing in this pipeline, so just build the release binary
  artifacts:
    paths:
      - pie_chart/target/x86_64-unknown-linux-musl/release/
      # And save it for future jobs

build-aarch64-bin: # All the cool kids mix naming conventions
  stage: build
  image: "$RUST_IMAGE:aarch64-musl"
  # -- snip compilation script --

deploy-docker-image:
  stage: deploy
  needs: [build-amd64-bin, build-aarch64-bin]
  image: $DOCKER_IMAGE # Docker in Docker
  services:
    - $DOCKER_IMAGE # So we don't have to also install Docker
  script:
    # Set up --platform style file structure
    - mkdir -p linux/amd64/
    - mkdir linux/arm64/
    # Copy binaries over from Rust's target triple dir to the platform dir
    - cp target/x86_64-unknown-linux-musl/release/pie_chart linux/amd64/
    - cp target/aarch64-unknown-linux-musl/release/pie_chart linux/arm64
    # And finally build the images
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - docker buildx create # -- snip args --
    - docker buildx build # -- snip args --

FROM scratch
ARG TARGETPLATFORM
COPY [ "${TARGETPLATFORM}/pie_chart", "/pie_chart/pie_chart" ]
# E.g. "linux/amd64/[binary]"
ENTRYPOINT [ "/pie_chart/pie_chart" ]
EXPOSE 8080

If you want to poke around the full version, the repository at that commit is here.

The last hiccup I encountered: for some reason, neither the Rust compiler getting a musl target or the musl-specific compilation image recognized that the GNU linker (ld) was incorrect. Fortunately, the fix was relatively easy: adding a ./.cargo/config.toml file with LLVM’s linker, lld:

[target.aarch64-unknown-linux-musl]
linker = "rust-lld"

[target.x86_64-unknown-linux-musl]
linker = "rust-lld"

(I’m including that partly for anyone trying to follow my footsteps.)

Results

Something that’s fascinated me a lot lately is how I can write some data to a text file on my MacBook, upload it somewhere, and then someone else’s Linux/AMD computers will know how to download a pre-packaged runtime environment, run my code, and spit out a new pre-packaged runtime environment that I can then download to the Raspberry Pi sitting in my coat closet and run the software.

I’m far from truly understanding all of these steps, but this has been an adventure in removing the magic from “automagical”.