Makefile & Container Fusion

February 3, 2021

Makefiles always make me feel comfortable whenever they are delivered with projects. The ease of following convention make build never ceases to amaze me. However, there is still one fear I have, system dependencies. Why don’t we get rid of that fear with fusion of makefiles and containers?

Make

Make¹ is a great utility to automate many task for your projects. Instead of bulky shell scripts, I very much prefer makefiles² for their ability to divide tasks into dependent sub tasks. These tasks are called targets in make context. You probably have seen make build and make install duo. Those are most conventional targets. Another thing is, make is time aware and can deduce only required tasks upon change. No need to execute targets all over.

While It’s possible to vendor³ all dependencies with projects, this is not practical. At one point, every system has a dependency to another. Makefiles may require system dependencies to meet their targets. These can be many things like binary files under /usr/bin or shared libraries. In some cases, these dependencies can be very specific to a version. We can guarantee these file dependencies with utilizing containers.

Containers

Containers⁴ enable you to isolate your environment in many aspects. Under the hood, containers leverage Linux namespaces⁵. In layman’s terms they let you to insulate your file systems, networks, processes and many more.

The one we are interested with today is file system insulation, which is thanks to mount namespaces⁶. With this at hand, we can setup a environment where we can guarantee that file dependencies would be satisfied.

The fusion

Like make build always makes me happy, make docker_build makes me even happier. The idea is to wrap make environment inside containers where It’s safe to run tasks with file dependencies. For this exercise, we will use Docker⁷ to containerize make’s file system.

Let’s have a simple C++ program that parses numbers from json files and does basic arithmetic operations on them respecting order precedence. To have some spice, we shall have numbers in any radix⁸.

Because our time is too precious to wait, we have used rapidjson⁹. And because we are always self-criticizing, we have added component tests with pytest-bdd¹⁰. What a perfection we are. A simple makefile to cover both build and component test stages would be like the following.

all: build

build: jsoncalc
jsoncalc: src/jsoncalc.cpp
	g++ -Wall -Wextra $^ -o $@

.PHONY:
component_test: jsoncalc
	python3 -m pytest \
		--basetemp=tmp_pytest \
		--verbose \
		test/steps

However nice it may look, this makefile assumes that there is g++ with rapidjson header files and python with pytest packages installed on the running system. This assumption list could have been much longer for real projects.

We can ensure these assumptions by introducing make & container fusion here. This can be realized with following makefile directives.

PROJECT_NAME ?= jsoncalc
IMAGE_NAME_TAG = $(PROJECT_NAME)

# Usage: docker_cmd <additional-run-options> <image-directory-under-container> <args>
define docker_cmd
	@# Strip whitespaces.
	image_name=$(shell expr $(2) : "\(\S\+\)") && \
			   docker build --tag "$(IMAGE_NAME_TAG):$${image_name}" "container/$${image_name}" && \
	docker run \
		$(DOCKER_OPTS) \
		$(1) \
		"$(IMAGE_NAME_TAG):$${image_name}" \
		$(3)
endef

# Array of variables to feed into docker environment.
DOCKER_ENV_VARS = \
	TERM

DOCKER_OPTS = \
	--interactive \
	--tty \
	--rm \
	--volume '$(CURDIR):$(CURDIR)' \
	--workdir '$(CURDIR)' \
	--user $(shell id --user):$(shell id --group) \
	$(foreach var, $(DOCKER_ENV_VARS), --env $(var)) \

.PHONY: docker_%
docker_%:
	$(call docker_cmd ,\
		$(DOCKER_EXTRA_OPTS) ,\
		$(subst docker_,,$@) ,\
		$(DOCKER_PRE_CMD) $(MAKE) $(subst docker_,,$@) \
	)

docker_component_test: docker_build

docker_cmd
You can think this of as a function inside Makefile with specified arguments. This function automatically deduces the path for Dockerfile from Its target name and builds required container image for relevant target. After building is complete, It runs the arguments supplied to is It as if It was ran under container.

E.g. make docker_build would look for containers/build/Dockerfile and build It. After that, It would spin up the built container and issue the command provided with args parameter under container.
```
[0] [00:01] [can@homebook jsoncalc] (master)
$ docker images | grep jsoncalc
jsoncalc                                 component_test   fed064a9f4b6   About an hour ago   134MB
jsoncalc                                 build            fbba4605e463   About an hour ago   246MB
```
docker_env_vars and docker_ops
These are for configurations of environment variables that should be forwarded from host to container and default docker options to access source/test files with same host uid/gid, respectively.
docker_%
This is a wildcard make target that captures every docker_<actual_target> calls. Essentially this uses the docker_cmd function to build the relevant container image and spin It up. What critical here is, It deduces real target from issued command and run the very same make command under the container. What good about this is this would be automatically applicable for any future targets defined inside the Makefile.

E.g. for make docker_build It would issue make build command under relevant container.
docker_component_test: docker_build
This is for to keep same target dependencies as component_test target already requires build target.

Action

Conclusion

The final version of this exercise can be found at here.

This can greatly aid software development/test cycles making them both faster and easier. The technique can be implemented for many scenarios such like including Selenium tests with X11 usage under containers.

As this prune most of project dependencies, CI/CD integration can be easy as running command make docker_build from your CI/CD solution. And this can be significant at onboarding process' where most developer/testers struggle with setting-up their environments for their project needs.