Improving Ninja’s “-d explain” output

By David Röthlisberger. Tweet your comments @drothlis.

Published 21 Feb 2022.

Ninja’s “-d explain” command-line option is supposed to explain what caused a build target to be re-built, but it’s next to useless in a large build: It says what’s dirty but it’s hard to relate which dirty input caused which output to be re-built. At the beginning of your build you get pages & pages of “these things are dirty” and then it starts the actual build. Also, this “explain” output is printed before ninja has realised that a lot of those things aren’t actually dirty (thanks to “restat”).

I wrote pull request #2067 to make the explain output more useful. I have been using this for 6 weeks now and it works well. The implementation is dead simple: Instead of printing the explain string immediately, the EXPLAIN macro saves it to a map, and we print it later. So Ninja will use a bit more memory, but only if you use -d explain.

At my day job, a no-op build with -d explain used to print 1,800 lines of explain output alone (we use restat a lot!). With my patch, the entire output for the same no-op build is 93 lines:

$ /usr/bin/ninja -d explain
ninja explain: output FORCE of phony edge with no inputs doesn't exist
ninja explain: FORCE is dirty
ninja explain: _build/git-commit-sha/node/buildtools/fakeroot-HEAD is dirty
ninja explain: _build/git-archive/node/buildtools/fakeroot-HEAD-.tar is dirty
ninja explain: _build/fakeroot/x86_64-linux-gnu/libfakeroot.so is dirty
ninja explain: FORCE is dirty
ninja explain: _build/git-commit-sha/node/buildtools/bubblewrap-HEAD is dirty
ninja explain: _build/git-archive/node/buildtools/bubblewrap-HEAD-.tar is dirty
ninja explain: _build/bwrap/prefix/bin/bwrap is dirty
ninja explain: _build/bwrap/bin/bwrap is dirty
ninja explain: _build/ostree/refs/heads/ostree_mod/0b53ddb00974bb288b9ede19cd6cb623b8847f90e089e99f46bde97d39abe95c is dirty
ninja explain: _build/ostree/refs/heads/ostree_combine/2c053c92e1f7ac71a8ec1bec4807d14cdd184ccd43dc1d87f6ee134fe86062a2 is dirty
ninja explain: _build/fakeroot/x86_64-linux-gnu/libfakeroot.so is dirty
ninja explain: _build/bwrap/bin/bwrap is dirty
ninja explain: _build/ostree/refs/heads/ostree_mod/8b002865c513b2fa06a1cb717f013d6fe24e807c7ba1079e4e6c902dbe670e4b is dirty
ninja explain: _build/fakeroot/x86_64-linux-gnu/libfakeroot.so is dirty
ninja explain: _build/bwrap/bin/bwrap is dirty
ninja explain: _build/ostree/refs/heads/ostree_mod/37bee7247435b8ad80625aca66a49096d84504a16fff2db92a65f9d5ecea4897 is dirty
ninja explain: _build/ostree/refs/heads/ostree_combine/8cbd6b872c5d8465c4d51acc2b65d2971b48d68c8b31097f7755d42efe2abd01 is dirty
ninja explain: _build/ostree/refs/heads/ostree_combine/c2140efbe94bbbcd62e4ed7f787efd681e2620107a954ff3bc0c51898d175c55 is dirty
ninja explain: FORCE is dirty
ninja explain: _build/git-commit-sha/supervisor/supervisor-HEAD is dirty
ninja explain: _build/ostree/refs/heads/git-archive/supervisor/supervisor/HEAD is dirty
ninja explain: _build/ostree/refs/heads/ostree_combine/2e2b8d4c67f9aaf90e766a202be322d2f6d0047cb7f80d7470273ed174d21a96 is dirty
ninja explain: _build/fakeroot/x86_64-linux-gnu/libfakeroot.so is dirty
ninja explain: _build/bwrap/bin/bwrap is dirty
ninja explain: _build/ostree/refs/heads/ostree_mod/182d7de4ecde445bd740030ec6a03f4f10da1737d45f658d7c99c21b97f3956e is dirty
ninja explain: _build/ostree_to_tarball/ostree_mod/182d7de4ecde445bd740030ec6a03f4f10da1737d45f658d7c99c21b97f3956e.tar is dirty
ninja explain: _build/docker/7495d8e235ffc5daf07b8db87d071b3ff6e1c6d64173c7e6a3bd81b3e327b514 is dirty
ninja explain: _build/docker/7495d8e235ffc5daf07b8db87d071b3ff6e1c6d64173c7e6a3bd81b3e327b514 is dirty
ninja explain: _build/docker/central-server is dirty
ninja explain: FORCE is dirty
ninja explain: _build/shell/git_describe___match_nothing___a-5bf1acd2e72fb863c1531045a08b6c42243c8c989d2aff504f11f2b4a1d882bf is dirty
ninja explain: _build/docker/7495d8e235ffc5daf07b8db87d071b3ff6e1c6d64173c7e6a3bd81b3e327b514 is dirty
ninja explain: _build/fakeroot/x86_64-linux-gnu/libfakeroot.so is dirty
ninja explain: _build/bwrap/bin/bwrap is dirty
ninja explain: _build/ostree/refs/heads/ostree_mod/a273c5737426bb562742ab6bc9fd66eafed2cb87db0075feeea107c028c786bc is dirty
ninja explain: _build/ostree/refs/heads/ostree_combine/925b9d6d064c3c26a626c29aa570d722d74b1001dc79ce144d90308005437b67 is dirty
ninja explain: _build/git-archive/node/buildtools/fakeroot-HEAD-.tar is dirty
ninja explain: _build/fakeroot/arm-linux-gnueabihf/libfakeroot.so is dirty
ninja explain: _build/bwrap/bin/bwrap is dirty
ninja explain: _build/ostree/refs/heads/ostree_mod/cb91d57b34c7a24649e5217b4c975e15592664ff599b9dfc99af98c6921c2868 is dirty
ninja explain: _build/fakeroot/x86_64-linux-gnu/libfakeroot.so is dirty
ninja explain: _build/bwrap/bin/bwrap is dirty
ninja explain: _build/ostree/refs/heads/ostree_mod/9a6d264af51a1e735deea66cf89185c742145f6d714e558e803e548b508ec24f is dirty
ninja explain: _build/ostree/refs/heads/ostree_combine/7b6b8f7014c83dd5a1b8e81082b474a834be786e6b8bcb5c38093e5cb16cef8f is dirty
ninja explain: _build/fakeroot/x86_64-linux-gnu/libfakeroot.so is dirty
ninja explain: _build/bwrap/bin/bwrap is dirty
ninja explain: _build/ostree/refs/heads/ostree_mod/c5b673c8b1bd36d129136d19ce88ae23289731a058139ecaea0a9457914368ce is dirty
ninja explain: _build/fakeroot/x86_64-linux-gnu/libfakeroot.so is dirty
ninja explain: _build/bwrap/bin/bwrap is dirty
ninja explain: _build/ostree/refs/heads/ostree_mod/cffa956f3d5db373dce1bc765a467c160b9879f10aed9e15e216709573222d24 is dirty
ninja explain: _build/ostree/refs/heads/ostree_combine/c771073442a91db3b6f3597a7552fb8672282b3c1d6955adca8d1490823f39d9 is dirty
ninja explain: _build/fakeroot/x86_64-linux-gnu/libfakeroot.so is dirty
ninja explain: _build/bwrap/bin/bwrap is dirty
ninja explain: _build/ostree/refs/heads/ostree_mod/896e6e67884fd5eef1a6bb66c17e1934021608f18187366e3e653d48b95a75b1 is dirty
ninja explain: _build/ostree/refs/heads/ostree_combine/86f19041d0db40c8aa49d405e1924da57e4e06afd3d3701596c183d318d00d8c is dirty
ninja explain: _build/fakeroot/arm-linux-gnueabihf/libfakeroot.so is dirty
[... skipping 1,700 lines...]
ninja explain: _build/fakeroot/x86_64-linux-gnu/libfakeroot.so is dirty
ninja explain: _build/bwrap/bin/bwrap is dirty
ninja explain: _build/ostree/refs/heads/ostree_mod/f87fba9ecfbbcf57e6a3efa61143b3a3223739e4b74fe602a2f0e087982489dd is dirty
ninja explain: _build/ostree/refs/heads/ostree_combine/fd1d807f90be92d9b68b62c52801f2a63b687d0806db6f1b5b1a44b6c8922389 is dirty
ninja explain: _build/ostree/refs/heads/ostree_combine/f461481b3d6f0a4057370d27207c458f400c15e76bd3126b4765c63d99fcb7b6 is dirty
ninja explain: _build/shell/git_describe___match_nothing___a-5bf1acd2e72fb863c1531045a08b6c42243c8c989d2aff504f11f2b4a1d882bf is dirty
ninja explain: _build/ostree_tag/_build/ostree/stbt-node-nano-rootfs is dirty
ninja explain: _build/ostree_publish/refs/heads/_build/ostree_tag/_build/ostree/stbt-node-nano-rootfs is dirty
ninja explain: _build/ostree_publish/refs/heads/_build/ostree_tag/_build/ostree/stbt-node-bootstrap is dirty
ninja explain: _build/ostree_publish/refs/heads/_build/ostree_tag/_build/ostree/stbt-node-rootfs is dirty
[1/833] git -C "$repo" rev-parse "$commit"
[2/829] git -C "$repo" rev-parse "$commit"
[3/205] git -C "$repo" rev-parse "$commit"
[4/197] git -C "$repo" rev-parse "$commit"
[5/184] git -C "$repo" rev-parse "$commit"
[6/173] git -C "$repo" rev-parse "$commit"
[7/161] git -C "$repo" rev-parse "$commit"
[8/149] git -C "$repo" rev-parse "$commit"
[9/133] git -C "$repo" rev-parse "$commit"
[10/120] git -C "$repo" rev-parse "$commit"
[11/113] git -C "$repo" rev-parse "$commit"
[12/106] git -C "$repo" rev-parse "$commit"
[13/99] git -C "$repo" rev-parse "$commit"
[14/89] git -C "$repo" rev-parse "$commit"
[15/80] git -C "$repo" rev-parse "$commit"
[16/74] git -C "$repo" rev-parse "$commit"
[17/68] git -C "$repo" rev-parse "$commit"
[18/55] git -C "$repo" rev-parse "$commit"
[19/54] git -C "$repo" rev-parse "$commit"
[20/53] git -C "$repo" rev-parse "$commit"
[21/46] git -C "$repo" rev-parse "$commit"
[22/45] git -C "$repo" rev-parse "$commit"
[23/33] git describe --match=nothing --always --dirty --abbrev=40

$ ./ninja -d explain
ninja explain: FORCE is dirty
[0/833] git -C "$repo" rev-parse "$commit"
ninja explain: FORCE is dirty
[0/833] git -C "$repo" rev-parse "$commit"
ninja explain: FORCE is dirty
[0/833] git -C "$repo" rev-parse "$commit"
ninja explain: FORCE is dirty
[0/833] git -C "$repo" rev-parse "$commit"
ninja explain: FORCE is dirty
[0/833] git -C "$repo" rev-parse "$commit"
ninja explain: FORCE is dirty
[0/833] git -C "$repo" rev-parse "$commit"
ninja explain: FORCE is dirty
[1/833] git -C "$repo" rev-parse "$commit"
ninja explain: FORCE is dirty
[1/830] git -C "$repo" rev-parse "$commit"
ninja explain: FORCE is dirty
[2/830] git -C "$repo" rev-parse "$commit"
ninja explain: FORCE is dirty
[2/205] git -C "$repo" rev-parse "$commit"
ninja explain: FORCE is dirty
[3/205] git -C "$repo" rev-parse "$commit"
ninja explain: FORCE is dirty
[3/192] git -C "$repo" rev-parse "$commit"
ninja explain: FORCE is dirty
[4/192] git -C "$repo" rev-parse "$commit"
ninja explain: FORCE is dirty
[4/185] git -C "$repo" rev-parse "$commit"
ninja explain: FORCE is dirty
[5/185] git -C "$repo" rev-parse "$commit"
ninja explain: FORCE is dirty
[5/173] git -C "$repo" rev-parse "$commit"
ninja explain: FORCE is dirty
[6/173] git -C "$repo" rev-parse "$commit"
ninja explain: FORCE is dirty
[6/161] git -C "$repo" rev-parse "$commit"
ninja explain: FORCE is dirty
[7/161] git -C "$repo" rev-parse "$commit"
ninja explain: FORCE is dirty
[7/149] git -C "$repo" rev-parse "$commit"
ninja explain: FORCE is dirty
[8/149] git -C "$repo" rev-parse "$commit"
ninja explain: FORCE is dirty
[8/133] git -C "$repo" rev-parse "$commit"
ninja explain: FORCE is dirty
[9/133] git -C "$repo" rev-parse "$commit"
ninja explain: FORCE is dirty
[9/120] git -C "$repo" rev-parse "$commit"
ninja explain: FORCE is dirty
[10/120] git -C "$repo" rev-parse "$commit"
ninja explain: FORCE is dirty
[10/113] git -C "$repo" rev-parse "$commit"
ninja explain: FORCE is dirty
[11/113] git -C "$repo" rev-parse "$commit"
ninja explain: FORCE is dirty
[11/106] git -C "$repo" rev-parse "$commit"
ninja explain: FORCE is dirty
[12/106] git -C "$repo" rev-parse "$commit"
ninja explain: FORCE is dirty
[12/99] git -C "$repo" rev-parse "$commit"
ninja explain: FORCE is dirty
[13/99] git -C "$repo" rev-parse "$commit"
ninja explain: FORCE is dirty
[13/95] git describe --match=nothing --always --dirty --abbrev=40
ninja explain: FORCE is dirty
[14/95] git -C "$repo" rev-parse "$commit"
ninja explain: FORCE is dirty
[14/86] git -C "$repo" rev-parse "$commit"
ninja explain: FORCE is dirty
[15/86] git -C "$repo" rev-parse "$commit"
ninja explain: FORCE is dirty
[15/74] git -C "$repo" rev-parse "$commit"
ninja explain: FORCE is dirty
[16/74] git -C "$repo" rev-parse "$commit"
ninja explain: FORCE is dirty
[16/61] git -C "$repo" rev-parse "$commit"
ninja explain: FORCE is dirty
[17/61] git -C "$repo" rev-parse "$commit"
ninja explain: FORCE is dirty
[17/55] git -C "$repo" rev-parse "$commit"
ninja explain: FORCE is dirty
[18/55] git -C "$repo" rev-parse "$commit"
ninja explain: FORCE is dirty
[19/48] git -C "$repo" rev-parse "$commit"
ninja explain: FORCE is dirty
[20/47] git -C "$repo" rev-parse "$commit"
ninja explain: FORCE is dirty
[21/46] git -C "$repo" rev-parse "$commit"
ninja explain: FORCE is dirty
[22/45] git -C "$repo" rev-parse "$commit"
ninja explain: FORCE is dirty
[23/33] git describe --match=nothing --always --dirty --abbrev=40

Ninja’s original output on the left (some of it; the total output was 1,800 lines).
On the right, Ninja’s total output with my patch.

Future work

I have an elaborate fantasy where ninja could serialise the build graph to disk (or maybe just the subset of the graph that actually needed rebuilding). This could be augmented with the reason why each rule was run, and possibly even with the command output for each rule. Then the existing ninja tools like “graph”, “browse”, and “query” could load this graph instead of the “build.ninja” file, so that you can inspect interactively what happened the last time you ran Ninja.

You could even use the same “build.ninja” syntax for this serialisation, to avoid adding lots of new code to Ninja.

Realistically I’ll never get around to implementing this, and my current pull request gets me close enough.