Chapter 8. Packages and dependencies

Erlang.mk can fetch and compile the dependencies that your project requires. Erlang.mk improves upon the concepts introduced by Rebar, so they should be familiar to many seasoned Erlang developers.

This chapter will explain how to add dependencies to your project, how they are fetched and built, how to lock the versions that were fetched, and how to bundle them directly in a single repository.

8.1. Adding dependencies to your project

List the applications you need in DEPS. Cowboy is in the built-in package index, so the name alone is enough:

DEPS = cowboy

The next time you run make, Erlang.mk will fetch and compile Cowboy. Erlang.mk will also ensure Cowboy is available whenever you use the shell, run tests and any other operations.

You can pin the version without repeating the repository:

DEPS = cowboy
dep_cowboy_commit = 2.19.0

For a project that is not in the index, set dep_$(NAME) to the fetch method and location. Hex packages look like this:

DEPS = jsx
dep_jsx = hex 3.1.0

Git, and the other methods, are listed in the next section.

Erlang.mk will fill in the application resource file with all applications found in DEPS. But not all dependencies are Erlang applications, and not all dependencies need to be a runtime dependency. That’s where the BUILD_DEPS variable comes in: it works just like DEPS, except the dependencies listed there will not be added as runtime dependencies.

For example, you could add a parse transform project like this to make it available only at build time:

BUILD_DEPS = erlando
dep_erlando = git https://github.com/proger/erlando master

Or you could depend on a C project directly, if you are building a NIF:

BUILD_DEPS = leveldb
dep_leveldb = git https://github.com/basho/leveldb 2.1.3

This dependency will be built before your application, so you could easily copy the resulting shared file into your priv/ directory as part of the build process. More information about that in the NIFs and port drivers chapter.

Another variable, LOCAL_DEPS, allows specifying runtime dependencies which are part of Erlang/OTP itself, but also dependencies that are included in the repository. Since they are already on your system, there is no need to fetch them. Do note that there is no way to choose the version, the application used will be the one already on your system.

You could depend on the Crypto application, for example:

LOCAL_DEPS = crypto

Erlang.mk comes with additional types of dependencies. It has TEST_DEPS for dependencies used only for testing:

TEST_DEPS = ct_helper
dep_ct_helper = git https://github.com/ninenines/ct_helper master

DOC_DEPS for dependencies used only when building documentation:

DOC_DEPS = edown
dep_edown = git https://github.com/uwiger/edown master

REL_DEPS for dependencies required to build the release, or to include extra applications in the release:

REL_DEPS = recon
dep_recon = hex 2.5.6

And SHELL_DEPS for dependencies to make available when running the make shell command:

SHELL_DEPS = observer_cli
dep_observer_cli = hex 2.0.0

All these will be documented in more details in their respective chapters.

Note that these additional types of dependencies will only be fetched after normal dependencies by default. You can force a dependency to be downloaded first by extending the target deps:: before including erlang.mk, for example:

deps
$(CURDIR)/deps/triq

Sometimes dependencies are allowed to be missing. However, your application may depend on an optional application being started. To ensure that an optional dependency is started before your application, the variable OPTIONAL_DEPS may be used:

OPTIONAL_DEPS = quicer

The top-level project can then decide whether to include this application by adding it to its BUILD_DEPS and including it in the release dependencies.

8.1.1. Fetch methods

Erlang.mk comes with a number of different fetch methods. There are fetch methods that will work everywhere, and fetch methods that will only work in a given environment.

The following table lists all existing methods:

Name Format Description

git

git repo commit

Clone the Git repository and checkout the given version

git-subfolder

git repo commit subfolder

Clone the Git repository, checkout the given version and use one of its subfolders as a dependency

git-submodule

git-submodule

Initialize and update the Git submodule

cp

cp path/to/repo

Recursively copy a local directory

ln

ln path/to/repo

Symbolically link a local directory

hex

hex version [pkg]

Download the given project version from hex.pm

fail

N/A

Always fail, reserved for internal use

default

N/A

Reserved

The git method has a repository and commit. You can use any valid commit, tag or branch in that repository for the commit value.

For example, to fetch Cowboy with tag 2.19.0 from Git:

dep_cowboy = git https://github.com/ninenines/cowboy 2.19.0

Git also comes with a concept of submodules. Erlang.mk can automatically initializes and updates submodules for dependencies, as long as they were added beforehand using git submodule add:

dep_cowboy = git-submodule

You can copy a directory from your machine using the cp method. It only takes the path to copy from:

dep_cowboy = cp $(HOME)/ninenines/cowboy

Finally, you can use a package from the Hex repository:

dep_cowboy = hex 2.19.0

If the package on Hex has a different name than the application, you can provide it after the version:

dep_uuid = hex 1.7.5 uuid_erl

8.1.2. Custom fetch methods

If none of the existing methods fit your use, you can simply define your own. Erlang.mk will consider all variables that are named as dep_fetch_$(METHOD) to be available fetch methods. You can do anything inside this variable, as long as you create a folder named $(DEPS_DIR)/$(call query_name,$1). Or in layman terms, if your dependency is Cowboy, this would become deps/cowboy.

To give an example, this is what the Git method does:

define dep_fetch_git
    git clone -q -n -- $(call query_repo_git,$1) $(DEPS_DIR)/$(call query_name,$1); \
    cd $(DEPS_DIR)/$(call query_name,$1) && git checkout -q $(call query_version_git,$1);
endef

Note that, like dependency information, this custom fetch method must be written before including erlang.mk.

8.2. Built-in package index

The index is only a handful of commonly used projects. make search lists them; make search q=cowboy filters that list.

If you need a different fetch method or repository for an indexed package, for example a fork, set dep_$(NAME) as usual:

DEPS = cowboy
dep_cowboy = git https://github.com/essen/cowboy 2.19.0

8.3. How deps are fetched and built

The order in which dependencies are fetched and built is well defined. This means that Erlang.mk will get the same applications regardless of the command or options being used.

In tree traversal terms, where the list of dependencies is a tree, Erlang.mk fetches everything using the pre-order traversal method. The steps can be summarized like this, starting from the root application:

  1. Fetch all dependencies for the application
  2. Build first dependency
  3. Build Nth dependency
  4. Build last dependency

Every time a dependency is built, these same steps are followed, recursively.

Do note that the first step, fetching all dependencies of an application, is not guaranteed to be ordered. The reason for this is that it is not possible to have the same dependency listed twice in a single application, and therefore there can be no conflicts. Remember, this step only fetches, at no point are different applications built in parallel.

What about conflicts between the dependencies of different applications? Simple. Since builds are ordered, this means that the first version of an application that is fetched will be the one that wins.

This means that if project A depends on projects B and C, in this order, and that both B and C depend on a different version of D, it will always be B’s version of D that wins, because we fetch the dependencies of B before fetching those from C.

Similarly, if project A depends on projects B, C and D, regardless of the order, and A, B and C depend on a different version of D, it will always be A’s version that wins, because we fetch all dependencies of A before fetching those from B or C.

Once a dependency is built, it will not be built again by default. Typically dependencies do not need to be recompiled and this speeds up building immensely. There are a few ways to force recompiling a dependency however:

  • The dependency directory is a symbolic link; the dependency will always be recompiled.
  • The dependency is built directly, for example with a command like make -C deps/cowlib, or make in the dependency’s directory.
  • The variable FULL is set, for example make FULL=1. This will force building of all dependencies. This can be added to your Makefile before including erlang.mk.
  • The file $ERLANG_MK_TMP/dep_built/$dep is removed.

8.4. Fetching and listing dependencies only

You can fetch all dependencies recursively without building anything, with the make fetch-deps command. It follows the same rules described in the section above.

You can list all dependencies recursively, again without building anything, with the make list-deps command. It will obviously need to fetch all dependencies exactly like make fetch-deps. Once everything is fetched, it prints a sorted list of absolute paths to the dependencies.

By default, fetch-deps and list-deps work on the BUILD_DEPS and DEPS lists only. To also fetch/list TEST_DEPS, DOC_DEPS, REL_DEPS and/or SHELL_DEPS, you have two possibilities:

  • You can use make fetch-test-deps, make fetch-doc-deps, make fetch-rel-deps and make fetch-shell-deps commands respectively. If you want to list them, you can use make list-test-deps, make list-doc-deps, make list-rel-deps and make list-shell-deps respectively.
  • You can use make fetch-deps or make list-deps with the Makefile variable DEP_TYPES set to a list of dependency types you want. The types are test, doc, rel and shell respectively. For example, you can list test and doc dependencies with make list-deps DEP_TYPES='test doc'.

Note that only first level ‘TEST_DEPS, `DOC_DEPS, REL_DEPS and SHELL_DEPS are included, not dependencies’ one. In other word, make list-test-deps lists the TEST_DEPS of your project, but not TEST_DEPS of the projects yours depend on.

No matter which method you use, BUILD_DEPS and DEPS are always included.

Internally, the make fetch-* commands store the complete list of dependencies in files named $(ERLANG_MK_RECURSIVE_DEPS_LIST), $(ERLANG_MK_RECURSIVE_TEST_DEPS_LIST), $(ERLANG_MK_RECURSIVE_DOC_DEPS_LIST), $(ERLANG_MK_RECURSIVE_REL_DEPS_LIST) and $(ERLANG_MK_RECURSIVE_SHELL_DEPS_LIST). Those files are simply printed by the make list-* commands.

make list-* commands are made for human beings. If you need the list of dependencies in a Makefile or a script, you should use the content of those files directly instead. The reason is that make fetch-* and make list-* may have unwanted content in their output, such as actual fetching of dependencies.

8.5. Dependency lock

Fetching git, git-subfolder or hex dependencies writes lock.mk in the project directory when that fetch finishes. The file records the fetches that happened. Dependencies of dependencies, and dependencies of local dependencies, are written to that same file. Local dependencies themselves are not listed. Run make lock so the lock is effective. It is not effective by default. Committing the locked lock.mk ensures that the dependencies that are fetched do not change. The file is not removed on distclean.

The file is only appended to. A git or git-subfolder fetch adds a block after whatever is already there, including comments and lines edited by hand. A hex fetch adds a block when dep_$(NAME)_checksum is not already set. The later := is the one that applies. To fully regenerate lock.mk, all dependencies must be fetched again.

ERLANG_MK_LOCK := 1 enables the lock. The line stands on its own. make lock fetches anything still missing, following the same rules as make fetch-deps, then writes that line at the top of the file. make unlock removes the line and keeps the entries. With the line absent, the blocks below do nothing.

Each recorded dependency is wrapped in ifdef ERLANG_MK_LOCK. git sets the repository and the full commit:

ifdef ERLANG_MK_LOCK
dep_NAME := git URL COMMIT
dep_NAME_commit := COMMIT
endif

git-subfolder adds the subfolder as the fourth word:

ifdef ERLANG_MK_LOCK
dep_NAME := git-subfolder URL COMMIT SUBFOLDER
dep_NAME_commit := COMMIT
endif

hex sets the version. dep_NAME_commit is that same version. dep_NAME_checksum is the SHA-256 of the package tarball. When the package name differs from the dependency name, it is the third word:

ifdef ERLANG_MK_LOCK
dep_NAME := hex VERSION
dep_NAME_commit := VERSION
dep_NAME_checksum := SHA256
endif
ifdef ERLANG_MK_LOCK
dep_NAME := hex VERSION PACKAGE
dep_NAME_commit := VERSION
dep_NAME_checksum := SHA256
endif

A git-submodule dependency is not recorded. Its commit is the gitlink in the current repository. Any other fetch method keeps the definition from the Makefile. A dependency with no lines in lock.mk is left alone.

With the lock enabled, Cowboy fetched at tag 2.19.0 and jsx 3.1.0 look like this. Each fetch appends its own block:

ERLANG_MK_LOCK := 1

ifdef ERLANG_MK_LOCK
dep_cowboy := git https://github.com/ninenines/cowboy 79e3fb02b31d47af6e69e8f3ba18fba291a3072a
dep_cowboy_commit := 79e3fb02b31d47af6e69e8f3ba18fba291a3072a
endif
ifdef ERLANG_MK_LOCK
dep_jsx := hex 3.1.0
dep_jsx_commit := 3.1.0
dep_jsx_checksum := 0c5cc8fdc11b53cc25cf65ac6705ad39e54ecc56d1c22e4adb8f5a53fb9427f3
endif

While the lock is enabled, those lines are what the next fetch uses. Changing a dep_$(NAME) line in the Makefile leaves the locked fetch unchanged. An existing dependency directory is not fetched again.

When dep_$(NAME)_checksum is set, a hex fetch compares the tarball with it before extracting. The fetch stops when they differ.

TEST_DEPS, DOC_DEPS, REL_DEPS and SHELL_DEPS are recorded when they are fetched. Pass DEP_TYPES to make lock to include them, the same way as for make fetch-deps.

Removing a dependency from the Makefile leaves its lines in lock.mk. Delete those lines to drop the pin.

A build started from the top-level project writes the top-level lock.mk. Running make yourself inside a dependency writes that dependency’s own lock.mk.

8.6. Querying dependencies

You can obtain information about all dependencies with the make query-deps family of commands:

  • make query-deps will list dependencies found in BUILD_DEPS and DEPS recursively.
  • make query-doc-deps will list documentation dependencies of the current project.
  • make query-rel-deps will list release dependencies of the current project.
  • make query-shell-deps will list shell dependencies of the current project.
  • make query-test-deps will list test dependencies of the current project.

By default the information printed will be the dependency name, fetch method, repository and version, prefixed by the current project’s name. But this output can be customized via the variable QUERY:

$ make query-deps QUERY="name fetch_method repo version extra absolute_path"

The following options are available:

name
The dependency name.
fetch_method
The dependency’s fetch method.
repo
The dependency’s repository.
version
The dependency’s version, tag or commit.
extra
Any additional information specific to the fetch method used.
absolute_path
The dependency’s location after it has been fetched.

Fields that have no value will print -. For example not all fetch methods have a value for the version.

The value for extra, when available, will be formatted with the name of the information printed prefixed. For example the hex fetch method will add package-name=uuid_erl for the uuid application.

8.7. Dependency tree

make tree prints the dependencies from BUILD_DEPS and DEPS as a tree. Each line shows the dependency name and its version:

myapp
└── cowboy 2.7.0
    ├── cowlib 2.8.0
    └── ranch 1.7.1

Dependencies are fetched first, the same way as make query-deps.

A dependency name is expanded the first time it appears. Erlang.mk keeps a single checkout per name, so a later occurrence keeps the version from that line and does not repeat the children of the first checkout.

make tree always prints the name and the version. Setting QUERY does not change this output.

8.8. Ignoring unwanted dependencies

Sometimes, you may want to ignore dependencies entirely. Not even fetch them. You may want to do this because a project you depend on depends on an application you do not need (like a dependency for building documentation or testing). Or maybe the dependency is already installed on your system.

To ignore a dependency, simply add it to the IGNORE_DEPS variable:

IGNORE_DEPS += edown proper

This will only ignore dependencies that are needed for building. It is therefore safe to write:

IGNORE_DEPS += edown proper
TEST_DEPS = proper

The PropEr application will be fetched as intended when running make tests or make check. It will however not be fetched when running make or make deps.

8.9. Dependencies directory

Dependencies are fetched in $(DEPS_DIR). By default this is the deps directory. You can change this default, but you should only do so if it was not defined previously. Erlang.mk uses this variable to tell dependencies where to fetch their own dependencies.

You will therefore need to use ?= instead of =. Of course, if you know you will never use this project as a dependency, = will work. But to avoid it biting you later on, do this:

DEPS_DIR ?= $(CURDIR)/libs

The $(CURDIR) part is important, otherwise dependencies of dependencies will be fetched in the wrong directory.

Erlang.mk will also export the REBAR_DEPS_DIR variable for compatibility with Rebar build tools, as long as they are recent enough.

8.10. Many applications in one repository

In addition to the dependencies that are fetched, Erlang.mk also allows you to have dependencies local to your repository. This kind of layout is sometimes called multi-application repositories, or repositories with multiple applications.

They work exactly the same as remote dependencies, except:

  • They are not fetched
  • They are not autopatched
  • They are not deleted on make distclean
  • They are not automatically added to the application resource file

To properly fill the application resource file and compile apps in the right order, you will need to define the LOCAL_DEPS variable for each relevant application, the same as for OTP applications. Apps can depend on each other in this way, and their compilation order will follow the same rules as regular dependencies in DEPS.

The top-level LOCAL_DEPS variable, if defined, will determine which apps (along with their dependencies) to build, and also which apps should be added to the top-level application resource file, if there is one. This may be useful, for example, for specifying a different set of apps to build for different releases. If LOCAL_DEPS is not defined, then all apps in the $(APPS_DIR) will be built, but none will be automatically added to the top-level application resource file.

If there is a conflict between a local dependency and a remote dependency, then the local dependency always wins; an error will be triggered when trying to fetch the conflicting remote dependency.

To start using dependencies local to the repository, simply create a folder named $(APPS_DIR). By default, this folder is the apps/ directory.

You can use Erlang.mk to bootstrap local dependencies by using the command make new-app or make new-lib. This command will create the necessary directories and bootstrap the application.

For example, to create a full fledged OTP application as a local dependency:

$ make new-app in=webchat

Or, the same as an OTP library:

$ make new-lib in=webchat

Templates also work with local dependencies, from the root directory of the project. You do need however to tell Erlang.mk to create the files in the correct application:

$ make new t=gen_server n=my_server in=webchat

8.11. Repositories with no application at the root level

It’s possible to use Erlang.mk with only applications in $(APPS_DIR), and nothing at the root of the repository. Just create a folder, put the erlang.mk file in it, write a Makefile that includes it, and start creating your applications.

Similarly, it’s possible to have a repository with only dependencies found in $(DEPS_DIR). You just need to create a Makefile and specify the dependencies you want. This allows you to create a repository for handling the building of releases, for example.

8.12. Autopatch

Erlang.mk will automatically patch all the dependencies it fetches. It needs to do this to ensure that the dependencies become compatible with not only Erlang.mk, but also with the version of Erlang.mk that is currently used.

When fetching a dependency, the following operations are performed:

  • Fetch the dependency using the configured fetch method
  • If it contains a configure.ac or configure.in file, run autoreconf -Wall -vif -I m4
  • If it contains a configure script, run it
  • Run autopatch on the project

Autopatch first checks if there is any project-specific patch enabled. There are currently two: ELIXIR_PATCH for the elixir dependency and HUT_PATCH for the hut dependency.

Otherwise, autopatch performs different operations depending on the kind of project it finds the dependency to be.

  • Rebar projects are automatically converted to use Erlang.mk as their build tool. This essentially patches Rebar out, and fixes and converts the project to be compatible with Erlang.mk.
  • Erlang.mk projects have their Makefile patched, if necessary, to include the top-level project’s Erlang.mk. This is to ensure that functionality works across all dependencies, even if the dependency’s Erlang.mk is outdated. The patched Makefile can be safely committed if necessary.
  • Other Erlang projects get a small Erlang.mk Makefile generated automatically.
  • Projects with no source directory and no Makefile get an empty Makefile generated, for compatibility purposes.
  • Other projects with no Makefile are left untouched.

You can add additional commands to be run immediately before or after autopatch is done by extending the target autopatch-$(dep)::, for example this would remove a module:

autopatch-ranch::
    rm -f $(DEPS_DIR)/ranch/src/ranch_proxy_header.erl

A common use case for this feature is to apply a PATCH file on the dependency immediately after fetching it. It can also be used to add compiler options, for example:

autopatch-couchbeam::
    printf "\nERLC_OPTS += -DWITH_JIFFY\n" >> $(DEPS_DIR)/couchbeam/Makefile

The commands will run before autopatch when the target is defined before including erlang.mk, and after otherwise.

You can disable the replacing of the erlang.mk file by defining the NO_AUTOPATCH_ERLANG_MK variable:

NO_AUTOPATCH_ERLANG_MK = 1

You can also disable autopatch entirely for a few select projects using the NO_AUTOPATCH variable:

NO_AUTOPATCH = cowboy ranch cowlib

8.13. Dealing with duplicate modules

When there are duplicate modules found in both applications and their dependencies, some tasks may fail. Erlang expects modules to be unique in general.

When the duplicates are found in dependencies, you will need to remove one of the duplicates at fetch time. To do so, you can add a rule similar to this to your Makefile before including erlang.mk:

DEPS_DIR = $(CURDIR)/deps

deps:: $(DEPS_DIR)/cowlib
    $(verbose) rm -f $(DEPS_DIR)/cowlib/src/cow_ws.erl

This must be done from the application that has this dependency. Only define the DEPS_DIR variable if necessary.

8.14. Skipping deps

It is possible to temporarily skip all dependency operations. This is done by defining the SKIP_DEPS variable. Use cases include being somewhere with no connection to download them, or perhaps a peculiar setup.

A typical usage would be:

$ make SKIP_DEPS=1

When the variable is defined:

  • Dependencies will not be compiled or downloaded when required
  • The dependency directory $(DEPS_DIR) will not be removed on make distclean

This variable only applies to remote dependencies.