path: root/doc/manual.cli

// file      : doc/manual.cli
// copyright : Copyright (c) 2014-2018 Code Synthesis Ltd
// license   : MIT; see accompanying LICENSE file

"\name=build2-build-system-manual"
"\subject=build system"
"\title=Build System"

// NOTES
//
// - Maximum <pre> line is 70 characters.
//

// @@ backlink variable ref (build system core variables reference?)
// @@ how do we get code for unit tests (utility libraries)
// @@ installation of dependencies

/*
@@ explain unit tests implementation (utility libs by example)

@@ include includes once (also source)

- amalgamation (I think leave to its section, maybe mention and ref in search
  order)
- create: as common configuration?

@@ info (where? in scopes? could show some? separate section?)
@@ other meta-ops: create (anything else?)

@@ all tree output needs extra space (review with mc) (also dir/ suffix)

@@ Need to mention ixx/txx files somewhere since used in bdep-new-generated
   projects.

@@ establish a chapter for each module
*/

"
\h0#preface|Preface|

This document describes the \c{build2} build system. For the build system
driver command line interface refer to the \l{b(1)} man pages. For other tools
in the \c{build2} toolchain (package and project managers, etc) see the
\l{https://build2.org/doc.xhtml Documentation} index.

\h1#intro|Introduction|

The \c{build2} build system is a native, cross-platform build system with a
terse, mostly declarative domain-specific language, a conceptual model of
build, and a uniform interface with consistent behavior across all the
platforms and compilers.

Those familiar with \c{make} will see many similarities, though mostly
conceptual rather than syntactic. This is not surprising since \c{build2}
borrows the fundamental DAG-based build model from original \c{make} and many
of its conceptual extensions from GNU \c{make}. We believe, paraphrasing a
famous quote, that \i{those who do not understand \c{make} are condemned to
reinvent it, poorly.} So the goal of \c{build2} is to reinvent \c{make}
\i{well} while handling the demands and complexity of modern cross-platform
software development.

Like \c{make}, \c{build2} is an \i{honest} build system where you can expect
to understand what's going on underneath and be able to customize most of its
behavior to suit your needs. This is not to say that it's not an
\i{opinionated} build system and if you find yourself \"fighting\" some of its
fundamental design decisions, it would be wiser to look for alternatives.

We also believe the importance and complexity of the problem warranted the
design of a new purpose-built language and will hopefully justify the time it
takes for you to master it. In the end we hope \c{build2} will make creating
and maintain build infrastructure for your projects a pleasant task.

Also note that \c{build2} is not specific to C/C++ or even to compiled
languages and its build model is general enough to handle any DAG-based
operations. See the \l{#module-bash \c{bash} Module} for a good example.

While the build system is part of a larger, well-integrated build toolchain
that includes the package/project dependency managers, it does not depend on
them and its standalone usage is the only subject of this document.

We begin with a tutorial introduction that aims to show the essential elements
of the build system on real examples but without getting into too much
detail. Specifically, we want to quickly get to the point where we can build
useful executable and library projects.


\h#intro-hello|Hello, World|

Let's start with the customary \i{\"Hello, World\"} example: a single source
file from which we would like to build an executable:

\
$ tree hello/
hello/
└── hello.cxx

$ cat hello/hello.cxx

#include <iostream>

int main ()
{
  std::cout << \"Hello, World!\" << std::endl;
}
\

While this very basic program hardly resemble what most software projects look
today, it is useful for introducing key build system concepts without getting
overwhelmed. In this spirit we will also use the \c{build2} \i{simple project}
structure, which, similarly, should not be used for anything but quick
sketches.

\N|Simple projects have so many restrictions and limitations that they are
hardly usable for anything but, well, \i{really} simple projects.
Specifically, such projects cannot be imported by other projects nor can they
use build system modules that require bootstrapping. Which includes \c{test},
\c{install}, \c{dist}, and \c{config} modules. And without the \c{config}
module there is no support for persistent configurations. As a result, only
use a simple project if you are happy to always build in source and with the
default build configuration or willing to specify the output directory and/or
custom configuration on every invocation.|

To turn our \c{hello/} directory into a simple project all we need to do
is add a \c{buildfile}:

\
$ tree hello/
hello/
├── hello.cxx
└── buildfile

$ cat hello/buildfile

using cxx

exe{hello}: cxx{hello.cxx}
\

Let's start from the bottom: the second line is a \i{dependency declaration}.
On the left hand side of \c{:} we have a \i{target}, the \c{hello} executable,
and on the right hand side \- a \i{prerequisite}, the \c{hello.cxx} source
file. Those \c{exe} and \c{cxx} in \c{exe{...\}} and \c{cxx{...\}} are called
\i{target types}. In fact, for clarify, target type names are always
mentioned with the trailing \c{{\}}, for example, \"the \c{exe{\}} target type
denotes an executable\".

Notice that the dependency declaration does not specify \i{how} to build an
executable from a C++ source file \- this is the job of a \i{rule}. When the
build system needs to update a target, it tries to \i{match} a suitable rule
based on the types of the target and its prerequisites. The \c{build2} core
has a number of predefined fundamental rules with the rest coming from
\i{build system modules}. For example, the \c{cxx} module defines a number of
rules for compiling C++ source code as well as linking executables and
libraries.

It's now easy to guess what the first line of our \c{buildfile} does: it loads
the \c{cxx} module which defines the rules necessary to build our program (and
it also registers the \c{cxx{\}} target type).

Let's now try to build and run our program (\c{b} is the build system driver):

\
$ cd hello/

$ b
c++ cxx{hello}
ld exe{hello}

$ ls -1
buildfile
hello.cxx
hello
hello.d
hello.o
hello.o.d

$ ./hello
Hello, World!
\

Or, if we are on Windows and using Visual Studio, from the Visual Studio
development command prompt:

\
> cd hello

> b config.cxx=cl.exe
c++ cxx{hello}
ld exe{hello}

> dir /b
buildfile
hello.cxx
hello.exe
hello.exe.d
hello.exe.obj
hello.exe.obj.d

> .\hello.exe
Hello, World!
\

Let's discuss a few points about the build output. Firstly, to reduce the
noise, the commands being executed,

\
c++ cxx{hello}
ld exe{hello}
\

are by default shown abbreviated and with the same target type notation as we
used in the \c{buildfile}. If, however, you would like to see the actual
command lines, you can pass \c{-v} (to see even more, there is the \c{-V}
as well as the \c{--verbose} options; see \l{b(1)} for details). For example:

\
$ b -v
g++ -o hello.o -c hello.cxx
g++ -o hello hello.o
\

Most of the files produced by the build system should be self-explanatory: we
have the object file (\c{hello.o}, \c{hello.obj}) and executable (\c{hello},
\c{hello.exe}). For each of them we also have the corresponding \c{.d} files
which store the \i{auxiliary dependency information}, things like compile
options, header dependencies, etc.

To remove the build system output we use the \c{clean} \i{operation} (if no
operation is specified, the default is \c{update}):

\
$ b clean
rm exe{hello}
rm obje{hello}

$ ls -1
buildfile
hello.cxx
\

One of the main reasons behind the \i{target type} concept is the
platform/compiler-specified variances in file names as illustrated by the
above listings. In our \c{buildfile} we refer to the executable target as
\c{exe{hello\}}, not as \c{hello.exe} or \c{hello$EXT}. The actual file
extension, if any, will be determined based on the compiler's target platform
by the rule doing the linking. In this sense, target types are a
platform-independent replacement of file extensions (though they do have other
benefits, such as allowing non-file targets as well as being hierarchical).

Let's revisit the dependency declaration line from our \c{buildfile}:

\
exe{hello}: cxx{hello.cxx}
\

In the light of target types replacing file extensions this looks
tautological: why do we need to specify both the \c{cxx{\}} target type
\i{and} the \c{.cxx} file extension? In fact, we don't if we specify the
default file extension for the \c{cxx{\}} target type. Here is our updated
\c{buildfile} in its entirety:

\
using cxx

cxx{*}: extension = cxx

exe{hello}: cxx{hello}
\

Let's unpack the new line. What we have here is a \i{target
type/patter-specific variable}. It only applies to targets of the \c{cxx{\}}
type whose names match the \c{*} wildcard pattern. The \c{extension} variable
name is reserved by the \c{build2} core for specifying default target type
extensions.

Let's see how all these pieces fit together. When the build systems needs to
update \c{exe{hello\}}, it searches for a suitable rule. A rule from the
\c{cxx} module matches since it knows how to build a target of type \c{exe{\}}
from a prerequisite of type \c{cxx{\}}. When the matched rule is \i{applied},
it searches for a target for the \c{cxx{hello\}} prerequisite. During this
search, the \c{extension} variable is looked up and its value is used to end
up with the \c{hello.cxx} file.

Our new dependency declaration,

\
exe{hello}: cxx{hello}
\

has the canonical style: no extensions, only target types. Sometimes explicit
extension specification is still necessary, for example, if your project uses
multiple extensions for the same file type. But if unnecessary, it should be
omitted for brevity.

\N|If you prefer the \c{.cpp} file extension and your source file is called
\c{hello.cpp}, then the only line in our \c{buildfile} that needs changing is
the \c{extension} variable assignment:

\
cxx{*}: extension = cpp
\

|

Let's say our \c{hello} program got complicated enough to warrant moving some
functionality into a separate source/header module (or a real C++ module).
For example:

\
$ tree hello/
hello/
├── hello.cxx
├── utility.hxx
├── utility.cxx
└── buildfile
\

This is what our updated \c{buildfile} could look like:

\
using cxx

hxx{*}: extension = hxx
cxx{*}: extension = cxx

exe{hello}: cxx{hello} hxx{utility} cxx{utility}
\

Nothing really new here: we've specified the default extension for the
\c{hxx{\}} target type and listed the new header and source file as
prerequisites. If you have experience with other build systems, then
explicitly listing headers might seem strange to you. In \c{build2} you
have to explicitly list all the prerequisites of a target that should
end up in a distribution of your project.

\N|You don't have to list \i{all} headers that you include, only the ones
belonging to your project. In other words, \c{build2} performs automatic
header dependency extraction like all modern C/C++ build systems.|

In real projects with a substantial number of source files repeating target
types and names will quickly become noisy. To tidy things up we can use
\i{name generation}. Here are a few examples of dependency declarations
equivalent to the above:

\
exe{hello}: cxx{hello utility} hxx{utility}
exe{hello}: cxx{hello} {hxx cxx}{utility}
\

The last form is probably the best choice if your project contains a large
number of header/source pairs. Here is a more realistic example:

\
exe{hello}: {    cxx}{hello}               \
            {hxx    }{forward types}       \
            {hxx cxx}{format print utility}
\

Manually listing a prerequisite every time we add a new source file to our
project is both tedious and error prone. Instead, we can automate our
dependency declarations with wildcard \i{name patterns}. For example:

\
exe{hello}: {hxx cxx}{*}
\

Based on the previous discussion of default extensions you can probably guess
how this works: for each target type the value of the \c{extension} variable
is added to the pattern and files matching the result become the
prerequisites. So, in our case, we will end up with files matching the
\c{*.hxx} and \c{*.cxx} wildcard patterns.

In more complex projects it is often convenient to organize source code into
subdirectories. To handle such project we can use the recursive wildcard:

\
exe{hello}: {hxx cxx}{**}
\

\N|Using wildcards is somewhat controversial. Patterns definitely make
development more pleasant and less error prone: you don't need to update your
\c{buildfile} every time you add, remove, or rename a source file and you
won't forget to explicitly list headers, a mistake that is often only detected
when trying to build a distribution of a project. On the other hand, there is
a possibility of including stray source files into your build without
noticing. And, for more complex projects, name patterns can become equally
complex (see \l{#name-patterns Name Patterns} for details). Note, however,
that on modern hardware the performance of wildcard search hardly warrants a
consideration.

In our experience, at least when combined with modern version control systems
like \c{git(1)}, stray source files are rarely an issue and generally the
benefits of wildcards outweigh their drawbacks. But, in the end, whether to
use them or not is a personal choice and, as shown above, \c{build2} supports
both approaches.|

And that's about all there is to our \c{hello} example. To summarize, we've
seen that to build a simple project we need just a single \c{buildfile} which
itself doesn't contain much more than a dependency declaration for what we
want to build. But we've also learned that simple projects are only really
meant for quick sketches. So let's convert our \c{hello} example to the
\i{standard project} structure which is what we will be using in most of our
real projects.


\h#intro-proj-struct|Project Structure|

A \c{build2} \i{standard project} has the following overall layout:

\
hello/
├── build/
│   ├── bootstrap.build
│   └── root.build
├── ...
└── buildfile
\

Specifically, the project's root directory should contain the \c{build/}
subdirectory as well as the root \c{buildfile}. The \c{build/} subdirectory
contains project-wide build system information.

\N|The \l{bdep-new(1)} command is an easy way to create the standard layout
executable (\c{-t\ exe}) and library (\c{-t\ lib}) projects. To change the C++
file extensions to \c{.hpp/.cpp}, pass \c{-l c++,cpp}. For example:

\
$ bdep new --no-init -t exe -l c++,cpp hello
\

|

To support lazy loading of subprojects (discussed later), reading of the
project's build information is split into two phases: bootstrapping and
loading. During bootstrapping the project's \c{build/bootstrap.build} file is
read. Then, when (and if) the project is loaded completely, its
\c{build/root.build} file is read followed by the \c{buildfile} (normally from
project root but could also be from a subdirectory).

The \c{bootstrap.build} file is required. Let's see what it would look like
for a typical project using our \c{hello} as an example:

\
project = hello

using version
using config
using test
using install
using dist
\

The first non-comment line in \c{bootstrap.build} should be the assignment of
the project name to the \c{project} variable. After that, a typical
\c{bootstrap.build} file loads a number of build system modules. While most
modules can be loaded during the project load phase, certain modules have to
be loaded early, while bootstrapping (for example, because they define new
operations).

Let's examine briefly the modules loaded by our \c{bootstrap.build}: The
\l{#module-version \c{version} module} helps with managing our project
versioning. With this module we only maintain the version in a single place
(project's \c{manifest} file) and it is made available in various forms
throughout our project (\c{buildfiles}, header files, etc). The \c{version}
module also automates versioning of snapshots between releases.

The \c{manifest} file is what makes our build system project a \i{package}.
It contains all the metadata that a user of a package might need to know:
name, version, dependencies, etc., all in one place. However, even if you
don't plan to package your project, it is a good idea to create a basic
\c{manifest} if only to take advantage of the version management offered by
the \c{version} module. So let's go ahead and add it next to our root
\c{buildfile}:

\
$ tree hello/
hello/
├── build/
│   └── ...
├── ...
├── buildfile
└── manifest

$ cat hello/manifest
: 1
name: hello
version: 0.1.0
summary: hello executable
\

The \c{config} module provides support for persistent configurations. While
project configuration is a large topic that we will discuss in detail later,
in a nutshell \c{build2} support for configuration is an integral part of the
build system with the same mechanisms available to the build system core,
modules, and your projects. However, without \c{config}, the configuration
information is \i{transient}. That is, whatever configuration information was
automatically discovered or that you have supplied on the command line is
discarded after each build system invocation. With the \c{config} module,
however, we can \i{configure} a project to make the configuration
\i{persistent}. We will see an example of this shortly.

Next up are the \c{test}, \c{install}, and \c{dist} modules. As their names
suggest, they provide support for testing, installation and preparation of
distributions. Specifically, the \c{test} modules defines \c{test} operation,
the \c{install} module defines the \c{install} and \c{uninstall} operations,
and the \c{dist} module defines the \c{dist} (meta-)operation. Again, we will
try them in a moment.

Moving on, the \c{root.build} file is optional though most projects will have
it. This is the place where we normally load build system modules that provide
support for the languages/tools that we use as well as establish project-wide
settings. Here is what it could look like for our \c{hello} example:

\
cxx.std = latest

using cxx

hxx{*}: extension = hxx
cxx{*}: extension = cxx
\

As you can see, we've moved the loading of the \c{cxx} modules and setting of
the default file extensions from the root \c{buildfile} in our simple project
to \c{root.build} when using the standard layout. We've also set the
\c{cxx.std} variable to tell the \c{cxx} module to select the latest C++
standard available in any particular C++ compiler we use.

\N|Selecting the C++ standard is a messy issue. If we don't specify the
standard explicitly with \c{cxx.std}, then the default standard in each
compiler will be used, which, currently, can range from C++98 to C++14. So
unless you carefully write your code to work with any standard, this is
probably not a good idea.

Fixing the standard (for example, to \c{c++11}, \c{c++14}, etc) should work
theoretically. In practice, however, compilers add support for new standards
incrementally and many versions, while perfectly usable, are not
feature-complete. As a result, a better practical strategy is to specify the
set of minimum supported compiler versions rather than the C++ standard.

There is also the issue of using libraries that require newer standard in
older code. For example, headers from a library that relies on C++14 features
will not compile when included in a project that is built as C++11.  And, even
if the headers compile (that is, C++14 features are only used in the
implementation), strictly speaking, there is no guarantee that codebases
compiled with different C++ standards are ABI compatible (in fact, some
changes to the C++ language leave the implementations no choice but to break
the ABI).

As result, our recommendation is to set the standard to \c{latest} and specify
the minimum supported compilers/versions. Practically, this should allow you
to include and link any library, regardless of the C++ standard that it uses.|

Let's now take a look at the root \c{buildfile}:

\
./: {*/ -build/}
\

In plain English this \c{buildfile} declares that building this directory
(and, since it's the root of our project, building this entire project) means
building all its subdirectories excluding \c{build/}. Let's now try to
understand how this is actually achieved.

We already know this is a dependency declaration, \c{./} is the target, and
what's after \c{:} are its prerequisites, which seem to be generated with some
kind of a name pattern (the wildcard character in \c{*/} should be the
giveaway). What's unusual about this declaration, however, is the lack of any
target types plus that strange-looking \c{./}.

Let's start with the missing target types. In fact, the above \c{buildfile}
can be rewritten as:

\
dir{.}: dir{* -build}
\

So the trailing slash (always forward, even on Windows) is a special shorthand
notation for \c{dir{\}}. As we will see shortly, it fits naturally with other
uses of directories in \c{buildfiles} (for example, in scopes).

The \c{dir{\}} target type is an \i{alias} (and, in fact, is derived from the
more general \c{alias{\}}). Building it means building all its prerequisites.

\N|If you are familiar with \c{make}, then you can probably see the similarity
with the ubiquitous \c{all} \"alias\" pseudo-target. In \c{build2} we instead
use directory names as more natural aliases for the \"build everything in this
directory\" semantics.

Note also that \c{dir{\}} is purely an alias and doesn't have anything to do
with the filesystem. In particular, it does not create any directories. If you
do want explicit directory creation (which should be rarely needed), use the
\c{fsdir{}} target type instead.|

The \c{./} target is a special \i{default target}. If we run the build system
without specifying the target explicitly, then this target is built by
default. Every \c{buildfile} has the \c{./} target. If we don't declare it
explicitly, then a declaration with the first target in the \c{buildfile} as
its prerequisite is implied. Recall our \c{buildfile} from the simple
\c{hello} project:

\
exe{hello}: cxx{hello}
\

It is equivalent to:

\
./: exe{hello}
exe{hello}: cxx{hello}
\

The last unexplained bit in our root \c{buildfile} is the \c{{*/\ -build/\}}
name pattern. All it does is exclude \c{build/} from the subdirectories to
build. See \l{#name-patterns Name Patterns} for details.

Let's take a look at a slightly more realistic root \c{buildfile}:

\
./: {*/ -build/} doc{README LICENSE} manifest
\

Here we have the customary \c{README} and \c{LICENSE} files as well as the
package \c{manifest}. Listing them as prerequisites achieves two things: they
will be installed if/when our project is installed and, as discussed earlier,
they will be included into the project distribution.

The \c{README} and \c{LICENSE} files use the \c{doc{\}} target type. We could
have used the generic \c{file{\}} but using the more precise \c{doc{\}} makes
sure they are installed into the appropriate documentation directory. The
\c{manifest} file doesn't need an explicit target type since it is a fixed
name (\c{manifest{manifest}} is valid but redundant).

The standard project infrastructure in place, where should we put our source
code? While we could have everything in the root directory of our project,
just like we did in the simple layout, it is recommended to instead place the
source code into a subdirectory named the same as the project. For example:

\
hello/
├── build/
│   └── ...
├── hello/
│   ├── hello.cxx
│   └── buildfile
├── buildfile
└── manifest
\

\N|There are several reasons for this layout: It implements the canonical
inclusion scheme where each header is prefixed with its project name. It also
has a predictable name where users can expect to find our project's source
code. Finally, this layout prevents clutter in the project's root directory
which usually contains various other files.|

The source subdirectory \c{buildfile} is identical to the simple project minus
the parts moved to \c{root.build}:

\
exe{hello}: {hxx cxx}{**}
\

Let's now build our project and see where the build system output ends up
in this new layout:

\
$ cd hello/
$ b
c++ hello/cxx{hello}
ld hello/exe{hello}

$ tree ./
./
├── build/
│   └── ...
├── hello/
│   ├── hello.cxx
│   ├── hello
│   ├── hello.d
│   ├── hello.o
│   ├── hello.o.d
│   └── buildfile
├── buildfile
└── manifest

$ hello/hello
Hello, World!
\

If we don't specify a target to build (as we did above), then \c{build2} will
build the current directory or, more precisely, the default target in the
\c{buildfile} in the current directory. We can also build a directory other
than the current, for example:

\
$ b hello/
\

\N|Note that the trailing slash is required. In fact, \c{hello/} in the above
command line is a target and is equivalent to \c{dir{hello\}}, just like in
the \c{buildfiles}.|

Or we can build a specific target:

\
$ b hello/exe{hello}
\

Naturally, nothing prevents us from building multiple targets or even projects
with the same build system invocation. For example, if we had the \c{libhello}
project next to our \c{hello/}, then we could build both at once:

\
$ ls -1
hello/
libhello/

$ b hello/ libhello/
\

Speaking of libraries, let's see what the standard project structure looks
like for one, using \c{libhello} created by \l{bdep-new(1)} as an example:

\
$ bdep new --no-init -t lib libhello

$ tree libhello/
libhello/
├── build/
│   ├── bootstrap.build
│   ├── root.build
│   └── export.build
├── libhello/
│   ├── hello.hxx
│   ├── hello.cxx
│   ├── export.hxx
│   ├── version.hxx.in
│   └── buildfile
├── tests/
│   └──  ...
├── buildfile
└── manifest
\

The overall layout (\c{build/}, \c{libhello/} source directory) as well as the
contents of the root files (\c{bootstrap.build}, \c{root.build}, root
\c{buildfile}) are exactly the same. There is, however, a new file,
\c{export.build}, in \c{build/}, a new subdirectory, \c{tests/}, and the
contents of the project's source subdirectory, \c{libhello/}, look quite a bit
different. We will examine all of these differences in the coming sections, as
we learn more about the build system.

\N|The standard project structure is not type (executable, library, etc) or
even language specific. In fact, the same project can contain multiple
executables and/or libraries (for example, both \c{hello} and \c{libhello}).
However, if you plan to package your projects, it is a good idea to keep them
as separate build system projects (they can still reside in the same version
control repository, though).

Speaking of terminology, the term \i{project} is unfortunately overloaded to
mean two different things at different levels of software organization. At the
bottom we have \i{build system projects} which, if packaged, become
\i{packages}. And at the top, related packages are often grouped into what is
also commonly referred to as \i{projects}. At this point both usages are
probably too well established to look for alternatives.|

And this completes the conversion of our simple \c{hello} project to the
standard structure. Earlier, when examining \c{bootstrap.build}, we mentioned
that modules loaded in this file usually provide additional operations. So we
still need to discuss what exactly the term \i{build system operation} means
and see how to use operations that are provided by the modules we have loaded.
But before we do that, let's see how we can build our projects \i{out of
source} tree and learn about another cornerstone \c{build2} concept:
\i{scopes}.


\h#intro-dirs-scopes|Directories and Scopes|

The two common requirements places on modern build systems are the ability to
build projects out of the source directory tree (referred to as just \i{out of
source} vs \i{in source}) as well as isolation of \c{buildfiles} from each
other when it comes to target and variable names. In \c{build2} these
mechanisms are closely-related, integral parts of the build system.

\N|This tight integration has advantages, like being always available and
working well with other build system mechanisms, as well disadvantages, like
inability to implement a completely different out of source arrangement and/or
isolation model. In the end, if you find yourself \"fighting\" this aspect of
\c{build2}, it will likely be easier to use a different build system than
subvert it.|

Let's start with an example of an out of source build of our \c{hello}
project. To recap, this is what we have:

\
$ ls -1
hello/

$ tree hello/
hello/
├── build/
│   └── ...
├── hello/
│   └── ...
├── buildfile
└── manifest
\

To start, let's build it in the \c{hello-out/} directory, next to the project:

\
$ b hello/@hello-out/
mkdir fsdir{hello-out/}
mkdir hello-out/fsdir{hello/}
c++ hello/hello/cxx{hello}@hello-out/hello/
ld hello-out/hello/exe{hello}

$ ls -1
hello/
hello-out/

$ tree hello-out/
hello-out/
└── hello/
    ├── hello
    ├── hello.d
    ├── hello.o
    └── hello.o.d
\

This definitely requires some explaining. Let's start from bottom, with the
\c{hello-out/} layout. It is \i{parallel} to the source directory. This
mirrored side-by-side listing (of relevant parts) should illustrate this
clearly:

\
hello/             ~~~                     hello-out/
└── hello/         ~~~            hello/ ──┘
    └── hello.cxx  ~~~  hello.o ──┘
\

In fact, if we copy the contents of \c{hello-out/} over to \c{hello/}, we will
end up with exactly the same result as when we did an in source build. And
this is not accidental: an in source build is just a special case of an out of
source build where the \i{out} directory is the same as \i{src}.

\N|The parallel structure of the out and src directories is a cornerstone
design decision in \c{build2} and is non-negotiable, so to speak. In
particular, out cannot be inside src. And while we can stash the build system
output (objects files, executables, etc) into (potentially different)
subdirectories, this is not recommended. As will be shown later, \c{build2}
offers better mechanisms to achieve the same benefits (like reduced clutter,
ability to run executables) but without the drawbacks (like name clashes).|

Let's now examine how we invoked the build system to achieve this out of
source build. Specifically, if we were building in source, our command line
would have been

\
$ b hello/
\

but for the out of source build, we have

\
$ b hello/@hello-out/
\

In fact, that strange-looking construct, \c{hello/@hello-out/} is just a more
complete target specification that explicitly spells out the target's src and
out directories. Let's add an explicit target type to make it clearer:

\
$ b hello/@hello-out/dir{.}
\

What we have on the right of \c{@} is the target in the out directory and on
the left \- its src directory. In plain English this command line means
\"build me the default target from \c{hello/} in the \c{hello-out/}
directory\".

As an example, if instead we only wanted to build just the \c{hello}
executable out of source, then the invocation would have looked like this:

\
$ b hello/hello/@hello-out/hello/exe{hello}
\

We could also specify out for an in source build, but that's redundant:

\
$ b hello/@hello/
\

There is another example of this complete target specification in the build
diagnostics:

\
c++ hello/hello/cxx{hello}@hello-out/hello/
\

Notice, however, that now the target (\c{cxx{hello\}}) is on the left of
\c{@}, that is, in the src directory. It does, however, make sense if you
think about it - our \c{hello.cxx} is a \i{source file}, it is not built and
it lives in the project's source directory. This is in contrast, for example,
to the \c{exe{hello\}} target which is the output of the build system and goes
to the out directory. So in \c{build2} targets can be either in src or in out
(there can also be \i{out of project} targets, for example, installed files).

The complete target specification can also be used in \c{buildfiles}. We
haven't encountered any so far because targets mentioned without explicit
src/out default to out and, naturally, most of the targets we mention in
\c{buildfiles} are things we want built. One situation where you may encounter
an src target mentioned explicitly is when configuring its installability
(discussed in the next section). For example, if our project includes the
customary \c{INSTALL} file, it probably doesn't make sense to install it.
However, since it is a source file, we have to use the complete target
specification when disabling its installation:

\
doc{INSTALL}@./: install = false
\

Note also that only targets but not prerequisites have this notion of src/out
directories. In a sense, prerequisites are relative to the target they are
prerequisites of and are resolved to targets in a manner that is specific to
their target types. For \c{file{\}}-based prerequisites the corresponding
target in out is first looked up and if found used. Otherwise, an existing file
in src is searched for and if found the corresponding target (now in src) is
used. In particular, this semantics gives preferences to generated code over
static.

\N|More precisely, a prerequisite is relative to the scope (discussed below)
in which the dependency is declared and not to the target that it is a
prerequisite of. But in most practical cases, however, this means the
same thing.|

And this pretty much covers out of source builds. Let's summarize the key
points we have established so far: Every build has two parallel directory
trees, src and out, with the in source build being just a special where they
are the same. Targets in a project can be either in the src or in out
directory though most of the time targets we mention in our \c{buildfiles}
will be in out, which is the default. Prerequsites are relative to targets
they are prerequisites of and \c{file{}}-based prerequisites are first
searched as existing targets in out and then as existing files in src.

Note also that we can have as many out of source builds as we want and we can
place them anywhere we want (but not inside src), say, on a RAM-backed
disk/filesystem. For example, we can build our \c{hello} project with two
different compilers:

\
$ b hello/@hello-gcc/    config.cxx=g++
$ b hello/@hello-clang/  config.cxx=clang++
\

In the next section we will see how to configure these out of source builds so
that we don't have to keep repeating these long command lines.

\N|While technically you can have both an in source and out of source builds
at the same time, this is not recommended. While it may work for simple
projects, as soon as you start using generated source code (which is fairly
common in \c{build2}), it becomes difficult to predict where the compiler will
pick generated headers. There is support for remapping mis-picked headers but
this may not work for older compilers. In other words, while you may have your
cake and eat it too, it might not taste particularly great. Plus, as will be
discussed in the next section, \c{build2} supports \i{forwarded
configurations} which provide most of the benefits of an in source build but
without the drawbacks.|

Let's now turn to \c{buildfile} isolation. It is a common, well-establishes
practice to organize complex software projects in directory hierarchies. One
of the benefits of this organization is isolation: we can use the same, short
file names in different subdirectories. In \c{build2} the project's directory
tree is used as a basis for its \i{scope} hierarchy. In a sense, scopes are
like C++ namespaces that track the project's filesystem structure and use
directories as their names. The following listing illustrates the parallel
directory and scope hierarchies for our \c{hello} project. \N{The \c{build/}
subdirectory is special and does not have a corresponding scope.}

\
hello/                   hello/
│                        {
└── hello/                 hello/
    │                      {
    └── ...                  ...
                           }
                         }
\

Every \c{buildfile} is loaded in its corresponding scope, variables set in a
\c{buildfile} are set in this scope and relative target mentioned in a
\c{buildfile} are relative to this scope's directory. Let's \"load\" the
\c{buildfile} contents from our \c{hello} to the above listing:

\
hello/                   hello/
│                        {
├── buildfile              ./: {*/ -build/}
│
└── hello/                 hello/
    │                      {
    └── buildfile            exe{hello}: {hxx cxx}{**}
                           }
                         }
\

In fact, to be absolutely precise, we should also add the contents of
\c{bootstrap.build} and \c{root.build} to the project's root scope (module
loading is omitted for brevity):

\
hello/                   hello/
│                        {
├── build/
│   ├── bootstrap.build    project = hello
│   │
│   └── root.build         cxx.std = latest
│                          hxx{*}: extension = hxx
│                          cxx{*}: extension = cxx
│
├── buildfile              ./: {*/ -build/}
│
└── hello/                 hello/
    │                      {
    └── buildfile            exe{hello}: {hxx cxx}{**}
                           }
                         }
\

The above scope structure is very similar to what you will see (besides a lot
of other things) if you build with \c{--verbose\ 6}. At this verbosity level
the build system driver dumps the build state before and after matching the
rules. Here is an abbreviated output for our \c{hello} (assuming an in source
build from \c{/tmp/hello}):

\
$ b --verbose 6

/
{
  [target_triplet] build.host = x86_64-linux-gnu
  [string] build.host.class = linux
  [string] build.host.cpu = x86_64
  [string] build.host.system = linux-gnu

  /tmp/hello/
  {
    [dir_path] src_base = /tmp/hello/
    [dir_path] out_root = /tmp/hello/

    [dir_path] src_root = /tmp/hello/
    [dir_path] out_base = /tmp/hello/

    [project_name] project = hello
    [string] project.summary = hello executable
    [string] project.url = https://example.org/hello

    [string] version = 1.2.3
    [uint64] version.major = 1
    [uint64] version.minor = 2
    [uint64] version.patch = 3

    [string] cxx.std = latest

    [string] cxx.id = gcc
    [string] cxx.version = 8.1.0
    [uint64] cxx.version.major = 8
    [uint64] cxx.version.minor = 1
    [uint64] cxx.version.patch = 0

    [target_triplet] cxx.target = x86_64-w64-mingw32
    [string] cxx.target.class = windows
    [string] cxx.target.cpu = x86_64
    [string] cxx.target.system = mingw32

    hxx{*}: [string] extension = hxx
    cxx{*}: [string] extension = cxx

    hello/
    {
      [dir_path] src_base = /tmp/hello/hello/
      [dir_path] out_base = /tmp/hello/hello/

      dir{./}: exe{hello}
      exe{hello.}: cxx{hello.cxx}
    }

    dir{./}: dir{hello/} manifest{manifest}
  }
}
\

This is probably quite a bit more information than what you've expected to see
so let's explain a couple of things. Firstly, it appears there is another
scope outer to our project's root. In fact, \c{build2} extends scoping outside
of projects with the root of the filesystem (denoted by the special \c{/})
being the \i{global scope}. This extension becomes useful when we try to build
multiple unrelated projects or import one project in another. In this model
all projects are part of single scope hierarchy with the global scope at its
root.

The global scope is read-only and contains a number of pre-defined
\i{build-wide} variables such as the build system version, host platform
(shown in the above listing), etc.

Next, inside the global scope, we see our project's root scope
(\c{/tmp/hello/}). Besides the variables that we have set ourselves (like
\c{project}), it also contains a number of variable set by the build system
core (for example, those \c{out_base}, \c{src_root}, etc) as well by build
system modules (for example, \c{project.*} and \c{version.*} variables set by
the \c{version} module and \c{cxx.*} variables set by the \c{cxx} module).

The scope for our project's source directory (\c{hello/}) should look
familiar. We again have a few special variables (\c{out_base}, \c{src_base}).
Notice also that the name patterns in prerequisites have been expanded to
the actual files.

As you can probably guess from their names, the \c{src_*} and \c{out_*}
variables track the association between scopes and src/out directories. They
are maintained automatically by the build system core with the
\c{src/out_base} pair set on each scope within the project and an additional
\c{src/out_root} pair set on the project's root scope (so that we can get the
project's root directories from anywhere in the project). Note that directory
paths in their values are always absolute.

In the above example the corresponding src/out variable pairs have the same
values because we were building in source. As an example, this is what the
association will look for an out of source build:

\
hello/ ~~~       hello-out/                             ~~~ hello-out/
│                {                                          │
│                  src_root = .../hello/                    │
│                  out_root = .../hello-out/                │
│                                                           │
│                  src_base = .../hello/                    │
│                  out_base = .../hello-out/                │
│                                                           │
└── hello/ ~~~     hello/                      ~~~ hello/ ──┘
                   {
                     src_base = .../hello/hello/
                     out_base = .../hello-out/hello/
                   }
                 }
\

Now that we have some scopes and variables to play with, it's good time to
introduce variable expansion. To get the value stored in a variable we use
\c{$} followed by the variable's name. The variable is first looked up in the
current scope (that is, the scope in which the expansion was encountered) and,
if not found, in the outer scopes all the way to the global scope.

\N|To be precise, this is the default \i{variable visibility}. Variables,
however, can have more limited visibilities, such as \i{project}, \i{scope},
\i{target}, or \i{prerequisite}.|

To illustrate the lookup semantics, let's add the following line to each
\c{buildfile} in our \c{hello} project:

\
$ cd hello/ # project root

$ cat buildfile
...
info \"src_base: $src_base\"

$ cat hello/buildfile
...
info \"src_base: $src_base\"
\

And then build it:

\
$ b
buildfile:3:1: info: src_base: /tmp/hello/
hello/buildfile:8:1: info: src_base: /tmp/hello/
\

In this case \c{src_base} is defined in each of the two scopes and we get
their respective values. If, however, we change the above line to print
\c{src_root} instead of \c{src_base} we will get the same value from the
root scope:

\
buildfile:3:1: info: src_root: /tmp/hello/
hello/buildfile:8:1: info: src_root: /tmp/hello/
\

One common place to find \c{src/out_root} expansions is in include search path
options. For example, the source directory \c{buildfile} generated by
l{bdep-new(1)} for an executable project actually looks like this
(\i{poptions} stands for \i{preprocessor options}):

\
exe{hello}: {hxx cxx}{**}

cxx.poptions =+ \"-I$out_root\" \"-I$src_root\"
\

This allows us to include our headers using the project's name as a prefix,
inline with the \l{intro#structure-canonical Canonical Project Structure}
guidelines. For example, if we added the \c{utility.hxx} header to our
\c{hello} project, we would include it like this:

\
#include <iostream>

#include <hello/utility.hxx>

int main ()
{
...
}
\

\N|Besides \c{poptions}, there are also \c{coptions} (compile options),
\c{loptions} (link options) and \c{libs} (extra libraries to link). If you are
familiar with \c{make}, these are roughly equivalent to \c{CPPFLAGS},
\c{CFLAGS}/\c{CXXFLAGS}, \c{LDFLAGS}, and \c{LIBS}.

Specifically, there are three sets of these variables: \c{cc.*} (stands for
\i{C-common}) which apply to all C-like languages as well as \c{c.*} and
\c{cxx.*} which only apply during the C and C++ compilation, respectively.  We
can use these variables in our \c{buildfiles} to adjust the compiler/linker
behavior. For example:

\
if ($cc.class == 'gcc')
{
  cc.coptions  += -fno-strict-aliasing  # C and C++
  cxx.coptions += -fno-exceptions       # only C++
}

if ($c.target.class != 'windows')
  c.libs += -lpthread  # only C
\

Additionally, as we will see in \l{#intro-operations-config Configuration},
there are also the \c{config.cc.*}, \c{config.c.*}, and \c{config.cxx.*} sets
which are used by the users of our projects to provide external configuration.
The initial values of the \c{cc.*}, \c{c.*}, and \c{cxx.*} variables are taken
from the corresponding \c{config.*.*} values.

Finally, as we will learn in \l{#intro-lib Library Exportation}, there are
also the \c{cc.export.*}, \c{c.export.*}, and \c{cxx.export.*} sets that are
used to specify options that should be exported to the users of our library.|


\N|In this section we've only scratched the surface when it comes to
variables. In particular, variables and variable values in \c{build2} are
optionally typed (those \c{[string]}, \c{[uint64]} we've seen in the build
state dump). And in certain contexts the lookup semantics actually starts from
the target, not from the scope (target-specific variables; there are also
prerequisite-specific). These and other variable-related topics will be
discussed in subsequent sections.|

As mentioned above, each \c{buildfile} in a project is loaded into its
corresponding scope. As a result, we rarely need to open scopes explicitly.
In the few cases that we do, we use the following syntax.

\
<directory>/
{
  ...
}
\

If the scope directory is relative, then it is assumed to be relative to the
current scope. As an exercise in understanding, let's reimplement our
\c{hello} project as a single \c{buildfile}. That is, we move the contents of
the source directory \c{buildfile} into the root \c{buildfile}:

\
$ tree hello/
hello/
├── build/
│   └── ...
├── hello/
│   └── hello.cxx
└── buildfile

$ cat hello/buildfile

./: hello/

hello/
{
  ./: exe{hello}
  exe{hello}: {hxx cxx}{**}
}
\

\N|While this single \c{buildfile} setup is not recommended for new projects,
it can be useful for a non-intrusive conversion of existing projects to
\c{build2}. One approach is to place the unmodified original project into a
subdirectory (potentially automating this with a mechanism such as \c{git(1)}
submodules) then adding the \c{build/} directory and the root \c{buildfile}
which opens explicit scope to define the build over the project's
subdirectory.|

Seeing this merged \c{buildfile} may make you wonder what exactly causes the
loading of the source directory \c{buildfile} in our normal setup. In other
words, when we build our \c{hello} from the project root, who and why loads
\c{hello/buildfile}?

Actually, in the earlier days of \c{build2} we had to explicitly load
\c{buildfiles} that define targets we depend on with the \c{include}
directive. In fact, we still can (and have to if we are depending on
targets other than directories). For example:

\
./: hello/

include hello/buildfile
\

We can also omit \c{buildfile} for brevity and have just:

\
include hello/
\

This explicit inclusion, however, quickly becomes tiresome as the number of
directories grows. It also makes using wildcard patterns for subdirectory
prerequisites a lot less appealing.

To resolve this the \c{dir{\}} target type implements an interesting
prerequisite to target resolution semantics: if there is no existing target
with this name, a \c{buildfile} that (presumably) defines this target is
automatically loaded from the corresponding directory. In fact, it goes
a step further and, if the \c{buildfile} does not exist, then assumes
one with the following contents was implied:

\
./: */
\

That is, it simply builds all the subdirectories. This is especially handy
when organizing related tests into subdirectories.

\N|As mentioned above, this automatic inclusion is only triggered if the
target we depend on is \c{dir{\}} and we still have to explicitly include the
necessary \c{buildfiles} for other target. One common example is a project
consisting of a library and an executable that uses it, each residing in a
separate directory next to each other (as noted earlier, not recommended for
projects that are to be packaged). For example:

\
hello/
├── build/
│   └── ...
├── hello/
│   ├── main.cxx
│   └── buildfile
├── libhello/
│   ├── hello.hxx
│   ├── hello.cxx
│   └── buildfile
└── buildfile
\

In this case the executable \c{buildfile} could look along these lines:

\
include ../libhello/ # Include lib{hello}.

exe{hello}: {hxx cxx}{**} lib{hello}
\

Note also that \c{buildfile} inclusion is not the mechanism for accessing
targets from other projects. For that we use \l{#intro-import Target
Importation}.|


\h#intro-operations|Operations|

Modern build systems have to perform operations other than just building:
cleaning the build output, running tests, installing/uninstalling the build
results, preparing source distributions, and so on. And, if the build system has
integrated configuration support, configuring the project would naturally
belong on this list as well.

\N|If you are familiar with \c{make}, you should recognize the parallel with
the common \c{clean} \c{test}, \c{install}, etc., \"operation\"
pseudo-targets.|

In \c{build2} we have the concept of a \i{build system operation} performed on
a target. The two pre-defined operations are \c{update} and \c{clean} with
other operations provided by build system modules.

Operations to perform and targets to perform them on are specified on the
command line. As discussed earlier, \c{update} is the default operation and
\c{./} in the current directory is the default target if no operation and/or
target is specified explicitly. And, similar to targets, we can specify
multiple operations (not necessarily on the same targets) in a single build
system invocation. The list of operations to perform and targets to perform
them on is called a \i{build specification} or \i{buildspec} for short (see
\l{b(1)} for details). Here are a few example:

\
$ cd hello        # Change to project root.

$ b               # Update current directory.
$ b ./            # As above.
$ b update        # As above.
$ b update: ./    # As above.

$ b clean update  # Rebuild.

$ b clean:  hello/             # Clean specific target.
$ b update: hello/exe{hello}   # Update specific target

$ b update: libhello/ tests/   # Update two targets.
\

Let's revisit \c{build/bootstrap.build} from our \c{hello} project:

\
project = hello

using version
using config
using test
using install
using dist
\

Other than \c{version}, all the modules we load define new operations. So
let's examine each of them starting with \c{config}.


\h2#intro-operations-config|Configuration|

As mentioned briefly earlier, the \c{config} module provides support for
persisting configurations by allowing us to \i{configure} our projects. At
first it may feels natural for \c{configure} to be another operation. There
is, however, a conceptual problem: we don't really configure a target. And,
perhaps after some meditation, it should become clear that what we are really
doing is configuring operations on targets. For example, configuring updating
a C++ project might involve detecting and saving information about the C++
compiler while configuring installing it may require specifying the
installation directory.

So \c{configure} is an operation on operation on targets \- a meta-operation.
And so in \c{build2} we have the concept of a \i{build system meta-operation}.
If not specified explicitly (as part of the buildspec), the default is
\c{perform}, which is to simply perform the operation.

Back to \c{config}, this module provides two meta-operations: \c{configure}
which saves the configuration of a project into the \c{build/config.build}
file as well as \c{disfigure} which removes it.

\N|While the common meaning of the word \i{disfigure} is somewhat different to
what we make it mean in this context, we still prefer it over the commonly
suggested \i{deconfigure} for the symmetry of their Latin \i{con-}
(\"together\") and \i{dis-} (\"apart\") prefixes.|

Let's say for the in source build of our \c{hello} project we want to use
\c{Clang} and enable debug information. Without persistence we would have to
repeat this configuration on every build system invocation:

\
$ cd hello  # Change to project root.

$ b config.cxx=clang++ config.cxx.coptions=-d
\

Instead, we can configure our project with this information once and from
then on invoke the build system without any arguments:

\
$ b configure config.cxx=clang++ config.cxx.coptions=-d

$ tree ./
./
├── build/
│   ├── ...
│   └── config.build
└── ...

$ b
$ b clean
$ b
...
\

Let's take a look at \c{config.build}:

\
$ cat build/config.build

config.cxx = clang++
config.cxx.poptions = [null]
config.cxx.coptions = -d
config.cxx.loptions = [null]
config.cxx.libs = [null]
...
\

As you can see, it's just a buildfile with a bunch of variable assignments. In
particular, this means you can tweak your build configuration by modifying
this file with your favorite editor. Or, alternatively, you can adjust the
configuration by reconfigure the project:

\
$ b configure config.cxx=g++

$ cat build/config.build

config.cxx = g++
config.cxx.poptions = [null]
config.cxx.coptions = -d
config.cxx.loptions = [null]
config.cxx.libs = [null]
...
\

Any variable value specified on the command line overrides those specified in
the \c{buildfiles}. As a result, \c{config.cxx} was updated while the value of
\c{config.cxx.coptions} was preserved.

Command line variable overrides are also handy to adjust the configuration for
a single build system invocation. For example, let's say we want to quickly
check that our project builds with optimization but without changing the
configuration:

\
$ b config.cxx.coptions=-O3  # Rebuild with -O3.
$ b                          # Rebuild with -d.
\

We can also configure out of source builds of our projects. In this case,
besides \c{config.build}, \c{configure} also saves the location of the source
directory so that we don't have to repeat that either. Remember, this is how
we used to build our \c{hello} out of source:

\
$ b hello/@hello-gcc/   config.cxx=g++
$ b hello/@hello-clang/ config.cxx=clang++
\

And now we can do:

\
$ b configure: hello/@hello-gcc/   config.cxx=g++
$ b configure: hello/@hello-clang/ config.cxx=clang++

$ hello-clang/
hello-clang/
└── build/
    ├── bootstrap/
    │   └── src-root.build
    └── config.build

$ b hello-gcc/
$ b hello-clang/
$ b hello-gcc/ hello-clang/
\

One major benefit of an in source build is the ability to run executables as
well as examine build/test output (test results, generated source code,
documentation, etc) without leaving the source directory. Unfortunately we
cannot have multiple in source builds and as was discussed earlier, mixing in
and out of source builds is not recommended.

To overcome this limitation \c{build2} has a notion of \i{forwarded
configurations}. As the name suggests, we can configure a project's source
directory to forward to one of its out of source builds. Specifically,
whenever we run the build system from the source directory, it will
automatically build in the corresponded forwarded output
directory. Additionally, it will \i{backlink} (using symlinks or another
suitable mechanism) certain \"interesting\" targets (\c{exe{\}}, \c{doc{\}})
to the source directory for easy access. As an example, let's configure our
\c{hello/} source directory to forward to the \c{hello-gcc/} build:

\
$ b configure: hello/@hello-gcc/,forward

$ cd hello/
$ b
c++ hello/cxx{hello}@../hello-gcc/hello/
ld ../hello-gcc/hello/exe{hello}
ln ../hello-gcc/hello/exe{hello} -> hello/
\

Notice the last line in the above listing: it indicates that \c{exe{hello}}
from the out directory was backlinked in our project's source subdirectory:

\
$ tree ./
./
├── build/
│   ├── bootstrap/
│   │   └── out-root.build
│   └── ...
├── hello/
│   ├── ...
│   └── hello -> ../../hello-gcc/hello/hello*
└── ...

$ ./hello/hello
Hello World!
\

\N|By default only \c{exe{\}} and \c{doc{\}} targets are backlinked. This,
however, can be customized with the \c{backlink} target-specific variable.|


\h2#intro-operations-test|Testing|

The next module we load in \c{bootstrap.build} is \c{test} which defines the
\c{test} operation. As the name suggests, this module provides support for
running tests.

There are two types of tests that we can run with the \c{test} module: simple
tests and \c{testscript}-based.

A simple test is just an executable target with the \c{test} target-specific
variable set to \c{true}. For example:

\
exe{hello}: test = true
\

A simple test is executed once and in its most basic form (typical for unit
testing) doesn't take any inputs nor produce any output, indicating success
via the zero exit status. If we test our \c{hello} project with the above
addition to its \c{buildfile}, then we will see the following output:

\
$ b test
test hello/exe{hello}
Hello, World!
\

While the test passes (since it exited with zero status), we probably don't
want to see that \c{Hello, World!} every time we run it (this can, however, be
quite useful for running examples). More importantly, we don't really test its
functionality and if tomorrow our \c{hello} starts swearing rather than
greeting, the test will still pass.

Besides checking the exit status we can also supply some basic information to
a simple test (more common for integration testing). Specifically, we can pass
command line options (\c{test.options}) and arguments (\c{test.arguments}) as
well as input (\c{test.stdin}, used to supply test's \c{stdin}) and output
(\c{test.stdout}, used to compare test's \c{stdout}).

Let's see how we can use this to fix our \c{hello} test by making sure our
program prints the expected greeting. First we need to add a file that will
contain the expected output, let's call it \c{test.out}:

\
$ ls -1 hello/
hello.cxx
test.out
buildfile

$ cat hello/test.out
Hello, World!
\

Next we arrange for it to be compared to our test's \c{stdout}. Here is the
new \c{hello/buildfile}:

\
exe{hello}: {hxx cxx}{**}
exe{hello}: file{test.out}: test.stdout = true
\

Ok, this looks new. What we have here is a \i{prerequisite-specific variable}
assignment. By setting \c{test.stdout} for the \c{file{test.out\}}
prerequisite of target \c{exe{hello\}} we mark it as expected \c{stdout}
output of \i{this} target (theoretically, we could have marked it as
\c{test.input} for another target). Notice also that we no longer need the
\c{test} target-specific variable. It's unnecessary if one of the other
\c{test.*} variable is specified.

Now, if we run our test, we won't see any output:

\
$ b test
test hello/exe{hello}
\

And if we try to change the greeting in \c{hello.cxx} but not in \c{test.out},
our test will fail printing the \c{diff(1)} comparison of the expected and
actual output:

\
$ b test
c++ hello/cxx{hello}
ld hello/exe{hello}
test hello/exe{hello}
--- test.out
+++ -
@@ -1 +1 @@
-Hello, World!
+Hi, World!
error: test hello/exe{hello} failed
\

Notice another interesting thing: we have modified \c{hello.cxx} to change the
greeting and our test executable was automatically rebuilt before testing.
This happened because the \c{test} operation performs \c{update} as its
\i{pre-operation} on all the targets to be tested.

Let's make our \c{hello} program more flexible by accepting the name to
greet on the command line:

\
#include <iostream>

int main (int argc, char* argv[])
{
  if (argc < 2)
  {
    std::cerr << \"error: missing name\" << std::endl;
    return 1;
  }

  std::cout << \"Hello, \" << argv[1] << '!' << std::endl;
}
\

We can test its successful execution path with a simple test fairly easily:

\
exe{hello}: test.arguments = 'World'
exe{hello}: file{test.out}: test.stdout = true
\

What if we also wanted to test its error handling? Since simple tests are
single-run, this won't be easy. Even if we could overcome this, having
expected output for each test in a separate file will quickly become untidy.
And this is where \c{testscript}-based tests come in. Testscript is a portable
language for running tests. It vaguely resembles Bash and is optimized for
concise test description and fast, parallel execution.

Just to give you an idea (see \l{testscript#intro Testscript Introduction} for
a proper introduction), here is what testing our \c{hello} program with
Testscript would look like:

\
$ ls -1 hello/
hello.cxx
testscript
buildfile

$ cat hello/buildfile

exe{hello}: {hxx cxx}{**} testscript
\

And these are the contents of \c{hello/testscript}:

\
: basics
:
$* 'World' >'Hello, World!'

: missing-name
:
$* 2>>EOE != 0
error: missing name
EOE
\

A couple of key points: The \c{test.out} file is gone with all the test inputs
and expected outputs incorporated into \c{testscript}. To test an executable
with Testscript all we have to is list the corresponding \c{testscript} file
as its prerequisite (and which, being a fixed name, doesn't need an explicit
target type, similar to \c{manifest}).

To see Testscript in action, let's say we've made our program more
user-friendly by falling back to a default name if one wasn't specified:

\
#include <iostream>

int main (int argc, char* argv[])
{
  const char* n (argc > 1 ? argv[1] : \"World\");
  std::cout << \"Hello, \" << n << '!' << std::endl;
}
\

If we forgot to adjust the \c{missing-name} test, then this is what we could
expect to see when running the tests:

\
b test
c++ hello/cxx{hello}
ld hello/exe{hello}
test hello/testscript{testscript} hello/exe{hello}
hello/testscript:7:1: error: hello/hello exit code 0 == 0
  info: stdout: hello/test-hello/missing-name/stdout
\

Testscript-based integration testing is the default setup for executable
(\c{-t\ exe}) projects created by \l{bdep-new(1)}. Here is the recap of the
overall layout:

\
hello/
├── build/
│   └── ...
├── hello/
│   ├── hello.cxx
│   ├── testscript
│   └── buildfile
├── buildfile
└── manifest
\

For libraries (\c{-t\ lib}), however, the integration testing setup is a bit
different. Here are the relevant parts of the layout:

\
libhello/
├── build/
│   └── ...
├── libhello/
│   ├── hello.hxx
│   ├── hello.cxx
│   ├── export.hxx
│   ├── version.hxx.in
│   └── buildfile
├── tests/
│   ├── build/
│   │   ├── bootstrap.build
│   │   └── root.build
│   ├── basics/
│   │   ├── driver.cxx
│   │   └── buildfile
│   └── buildfile
├── buildfile
└── manifest
\

Specifically, there is no \c{testscript} in \c{libhello/}, the project's
source directory. Instead we have the \c{tests/} subdirectory which itself
looks like a project: it contains the \c{build/} subdirectory with all the
familiar files, etc. In fact, \c{tests} is a \i{subproject} of our
\c{libhello} project.

While we will be examining \c{tests} in greater detail later, in a nutshell,
the reason it is a subproject is to be able to test an installed version of
our library. By default, when \c{tests} is built as part of its parent project
(called \c{amalgamation}), the locally built \c{libhello} library will be
automatically imported. However, we can also configure a build of \c{tests}
out of its amalgamation, in which case we can import an installed version of
\c{libhello}. We will learn how to do all that as well as the underlying
concepts (\i{subproject}/\i{amalgamation}, \i{import}, etc) in the coming
sections.

Inside \c{tests/} we have the \c{basics/} subdirectory which contains a simple
test for our library's API. By default it doesn't use Testscript but if you
want to, you can. You can also rename \c{basics/} to something more meaningful
and add more tests next to it. For example, if we were creating an XML parsing
and serialization library, then our \c{tests/} could have the following
layout:

\
tests/
├── build/
│   └── ...
├── parser/
│   └── ...
├── serializer/
│   └── ...
└── buildfile
\

\N|Nothing prevents us from having the \c{tests/} subdirectory for executable
projects. And it can be just a subdirectory or a subproject, the same as for
libraries. Making it a subproject makes sense if your program has a complex
installation, for example, if its execution requires configuration and/or data
files that need to be found, etc. For simple programs, however, testing the
executable before installing is usually sufficient.

For a general discussion of functional/integration and unit testing refer to
the \l{intro#proj-struct-tests Tests} section in the toolchain introduction.|


\h2#intro-operations-install|Installation|

The \c{install} module defines the \c{install} and \c{uninstall} operations.
As the name suggests, this module provides support for project installation.

\N|Project installation in \c{build2} is modeled after UNIX-like operation
systems though the installation directory layout is highly customizable. While
\c{build2} projects can import \c{build2} libraries directly, installation is
often a way to \"export\" them in a form usable by other build systems.|

The root installation directory is specified with the \c{config.install.root}
configuration variable. Let's install our \c{hello} program into
\c{/tmp/install}:

\
$ cd hello/
$ b install config.install.root=/tmp/install/
\

And see what we've got (executables are marked with \c{*}):

\
$ tree /tmp/install/

/tmp/install/
├── bin/
│   └── *hello
└── share/
    └── doc/
        └── hello/
            └── manifest
\

Similar to the \c{test} operation, \c{install} performs \c{update} as a
pre-operation for targets that it installs.

\N|We can also configure our project with the desired \c{config.install.*}
values so we don't have to repeat them on every install/uninstall. For
example:

\
$ b configure config.install.root=/tmp/install/
$ b install
$ b uninstall
\

|

Now the same for \c{libhello} (symbolic link targets are shown with \c{->} and
actual static/shared library names may differ on your operating system):

\
$ rm -r /tmp/install

$ cd libhello/
$ b install config.install.root=/tmp/install/

$ tree /tmp/install/

/tmp/install/
├── include/
│   └── libhello/
│       ├── hello.hxx
│       ├── export.hxx
│       └── version.hxx
├── lib/
│   ├── pkgconfig/
│   │   ├── libhello.shared.pc
│   │   └── libhello.static.pc
│   ├── libhello.a
│   ├── libhello.so -> libhello-0.1.so
│   └── libhello-0.1.so
└── share/
    └── doc/
        └── libhello/
            └── manifest
\

As you can see, the library headers go into the customary \c{include/}
subdirectory while static and shared libraries (and their \c{pkg-config(1)}
files) \- into \c{lib/}. Using this installation we should be able to import
this library from other build systems or even use it in a manual build:

\
$ g++ -I/tmp/install/include -L/tmp/install/lib greet.cxx -lhello
\

If we want to install into a system-wide location like \c{/usr} or
\c{/usr/local}, then we most likely will need to specify the \c{sudo(1)}
program:

\
$ cd hello/
$ b config.install.root=/usr/local/ config.install.sudo=sudo
\

\N|In \c{build2} only actual install/uninstall commands are executed with
\c{sudo(1)}. And while on the topic of sensible implementations, \c{uninstall}
can be generally trusted to work reliably.|

The default installability of a target as well as where it is installed is
determined by its target type. For example, \c{exe{\}} is by default installed
into \c{bin/}, \c{doc{\}} \- into \c{share/doc/<project>/}, and \c{file{\}} is
not installed.

We can, however, override this with the \c{install} target-specific variable.
Its value should be either special \c{false} indicating that the target should
not be installed or the directory to install the target to. As an example,
here is what the root \c{buildfile} from our \c{libhello} project looks like:

\
./: {*/ -build/} manifest

tests/: install = false
\

The first line we have already seen and the purpose of the second line should
now be clear: it makes sure we don't try to install anything in the \c{tests/}
subdirectory.

If the value of the \c{install} variable is not \c{false}, then it is normally
a relative path with the first path component being one of these names:

\
name        default                         override
----        -------                         --------
root                                        config.install.root

data_root   root/                           config.install.data_root
exec_root   root/                           config.install.exec_root

bin         exec_root/bin/                  config.install.bin
sbin        exec_root/sbin/                 config.install.sbin
lib         exec_root/lib/                  config.install.lib
libexec     exec_root/libexec/<project>/    config.install.libexec
pkgconfig   lib/pkgconfig/                  config.install.pkgconfig

data        data_root/share/<project>/      config.install.data
include     data_root/include/              config.install.include

doc         data_root/share/doc/<project>/  config.install.doc
man         data_root/man/                  config.install.man
man<N>      man/man<N>/                     config.install.man<N>
\

Let's see what happens here: The default install directory tree is derived
from the \c{config.install.root} value but the location of each node in this
tree can be overridden by the user that installs our project using the
corresponding \c{config.install.*} variables. In our \c{buildfiles}, in turn,
we use the node names instead of actual directories. As an example, here is a
\c{buildfile} fragment from the source directory of our \c{libhello} project:

\
hxx{*}: install         = include/libhello/
hxx{*}: install.subdirs = true
\

Here we set the installation location for headers to be the \c{libhello/}
subdirectory of the \c{include} installation location. Assuming
\c{config.install.root} is \c{/usr/}, the \c{install} module will perform the
following steps to resolve this relative path to the actual, absolute
installation directory:

\
include/libhello/
data_root/include/libhello/
root/include/libhello/
/usr/include/libhello/
\

In the above example we also see the use of the \c{install.subdirs} variable.
Setting it to \c{true} instructs the \c{install} module to recreate
subdirectories starting from this point in the project's directory hierarchy.
For example, if our \c{libhello/} source directory had the \c{details/}
subdirectory with the \c{utility.hxx} header, then this header would have been
installed as \c{.../include/libhello/details/utility.hxx}.


\h2#intro-operations-dist|Distribution|

The last module that we load in our \c{bootstrap.build} is \c{dist} which
provides support for preparation of distributions by defining the \c{dist}
meta-operation. Similar to \c{configure}, \c{dist} is a meta-operation rather
than an operation because, conceptually, we are preparing a distribution for
performing operations (like \c{update}, \c{test}) on targets rather than
targets themselves.

Preparation of a correct distribution relies on all the necessary project
files (sources, documentation, etc) being listed as prerequisites in the
project's \c{buildfiles}.

\N|You may wonder why not just use the export support offered by version
control systems? The main reason is that in most real-world projects version
control repositories contain a lot more than what needs to be distributed. In
fact, it is not uncommon to host multiple build system projects/packages in a
single repository. As a result, with this approach we seem to inevitably end
up maintaining an exclusion list which feels backwards \- why specify all the
things we don't want in a new list instead of just making sure the existing
list of things that we do want is complete? Also, once we have the complete
list, it can be put to good use by other tools, such as editors, IDEs, etc.|

Preparation of a distribution requires an out of source build. This allows the
\c{dist} module to distinguish between source and output targets. By default,
targets found in src are includes into the distribution while those in out are
excluded. However, we can customize this with the \c{dist} target-specific
variable.

As an example, let's prepare a distribution of our \c{hello} project using the
out of source build configured in \c{hello-out/}. We use \c{config.dist.root}
to specify the directory to place the distribution to:

\
$ b dist: hello-out/ config.dist.root=/tmp/dist

$ ls -1 /tmp/dist
hello-0.1.0/

$ tree /tmp/dist/hello-0.1.0/
/tmp/dist/hello-0.1.0/
├── build/
│   ├── bootstrap.build
│   └── root.build
├── hello/
│   ├── hello.cxx
│   ├── testscript
│   └── buildfile
├── buildfile
└── manifest
\

As we can see, the distribution directory includes the project version (comes
from the \c{version} variable which, in our case, is extracted from
\c{manifest} by the \c{version} module). Inside the distribution directory we
have our project's source files (but, for example, without any \c{.gitignore}
files that we may have had in \c{hello/}.

We can also ask the \c{dist} module to package the distribution directory
into one or more archives and generate their checksum files. For example:

\
$ b dist: hello-out/ \
  config.dist.root=/tmp/dist \
  config.dist.archives=\"tar.gz zip\" \
  config.dist.checksums=sha256

$ ls -1 /tmp/dist
hello-0.1.0/
hello-0.1.0.tar.gz
hello-0.1.0.tar.gz.sha256
hello-0.1.0.zip
hello-0.1.0.zip.sha256
\

\N|We can also configure our project with the desired \c{config.dist.*} values
so we don't have to repeat them every time. For example:

\
$ b configure: hello-out/ config.dist.root=/tmp/dist ...
$ b dist
\

|

Let's now take a look at an example of customizing what gets distributed.
Most of the time you will be using this mechanism to include certain targets
from out. Here is a fragment from the \c{libhello} source directory
\c{buildfile}:

\
hxx{version}: in{version} $src_root/manifest
hxx{version}: dist = true
\

Our library provides the \c{version.hxx} header that the users can include to
examine its version. This header is generated by the \c{version} module from
the \c{version.hxx.in} template. In essence, the \c{version} module takes the
version value from our manifest, splits it into various components (major,
minor, patch, etc) and then preprocesses the \c{in{\}} file substituting these
value (see \l{#module-version \c{version} Module} for details). The end result
is an automatically maintained version header.

One problem with auto-generated headers is that if one does not yet exist,
then the compiler may still find it somewhere else. For example, we may have
an older version of a library installed somewhere where the compiler searches
for headers by default (for example, \c{/usr/local/}). To overcome this
problem it is a good idea to ship pre-generated headers in our distributions.
But since they are output targets, we have to explicitly request this with
\c{dist=true}.


\h#intro-import|Target Importation|

If we need to depend on a target defined in another \c{buildfile} within our
project, then we simply include said \c{buildfile} and reference the target.
For example, if our \c{hello} included both an executable and a library in
separate directories next to each other:

\
hello/
├── build/
│   └── ...
├── hello/
│   ├── ...
│   └── buildfile
└── libhello/
    ├── ...
    └── buildfile
\

Then our executable \c{buildfile} could look like this:

\
include ../libhello/ # Include lib{hello}.

exe{hello}: {hxx cxx}{**} lib{hello}
\

What if instead \c{libhello} is a separate project? The inclusion no longer
works for two reasons: we don't know the path to \c{libhello} (after all, it's
an independent project and can reside anywhere) and we can't assume the path
to the \c{lib{hello\}} target within \c{libhello} (the project directory
layout can change).

To depend on a target from a separate project we use \i{importation} instead
of inclusion. This mechanism is also used to depend on targets that are not
part of any project, for example, installed libraries.

The importing project's side is pretty simple. This is what the above
\c{buildfile} will look like if \c{libhello} is a separate project:

\
import libs = libhello%lib{hello}

exe{hello}: {hxx cxx}{**} $libs
\

The \c{import} directive is a kind of variable assignment that resolves a
\i{project-qualified} relative target (\c{libhello%lib{hello\}} in our case)
to an unqualified absolute target and stores it in the variable (\c{libs} in
our case). We can then expand the variable (\c{$libs} in our case), normally
in the dependency declaration, to get the imported target.

If we needed to import several libraries then we simply repeat the \c{import}
directive, usually accumulating the result in the same variable, for example:

\
import libs  = libformat%lib{format}
import libs += libprint%lib{print}
import libs += libhello%lib{hello}

exe{hello}: {hxx cxx}{**} $libs
\

Let's now try to build our \c{hello} project that uses imported \c{libhello}:

\
$ b hello/
error: unable to import target libhello%lib{hello}
  info: use config.import.libhello command line variable to specify
        its project out_root
\

While that didn't work out well, it does make sense: the build system cannot
know the location of \c{libhello} or which of its builds we want to use.
Though it does helpfully suggests that we use \c{config.import.libhello} to
specify its out directory (\c{out_root}). Let's point it to \c{libhello}
source directory to use an in source build (\c{out_root\ ==\ src_root}):

\
$ b hello/ config.import.libhello=libhello/
c++ libhello/libhello/cxx{hello}
ld libhello/libhello/libs{hello}
c++ hello/hello/cxx{hello}
ld hello/hello/exe{hello}
\

And it works. Naturally, the importation mechanism works the same for out of
source builds and we can persist the \c{config.import.*} variables in the
project's configuration. As an example, let's setup Clang builds of the two
projects out of source:

\
$ b configure: libhello/@libhello-clang/ config.cxx=clang++
$ b configure: hello/@hello-clang/ config.cxx=clang++ \
  config.import.libhello=libhello-clang/

$ b hello-clang/
c++ libhello/libhello/cxx{hello}@libhello-clang/libhello/
ld libhello-clang/libhello/libs{hello}
c++ hello/hello/cxx{hello}@hello-clang/hello/
ld hello-clang/hello/exe{hello}
\

If the corresponding \c{config.import.*} variable is not specified, \c{import}
searches for a project in a couple of other places. First it looks in the list
of subprojects starting from the importing project itself and then continuing
with its outer amalgamations and their subprojects (see \l{#intro-subproj
Subprojects and Amalgamations} for details on this subject).

\N|We've actually seen an example of this search step in action: the \c{tests}
subproject in \c{libhello}. The tests import \c{libhello} which is
automatically found as an amalgamation containing this subproject.|

If the project being imported cannot be located using any of these methods,
then \c{import} falls back to the rule-specific search. That is, a rule that
matches the target may provide support for importing certain prerequisite
types based on rule-specific knowledge. Support for importing installed
libraries by the C++ link rule is a good example of this. Internally, the
\c{cxx} module extracts the compiler library search paths (that is, paths that
would be used to resolve \c{-lfoo}) and then its link rule uses them to search
for installed libraries. This allows us to use the same \c{import} directive
regardless of whether we import a library from a separate build, from a
subproject, or from an installation directory.

\N|Importation of an installed library will work even if it is not a
\c{build2} project. Besides finding the library itself, the link rule will
also try to locate its \c{pkg-config(1)} file and, if present, extract
additional compile/link flags from it. The link rule also produces
\c{pkg-config(1)} files for libraries that it installs.|

Let's now examine the exporting side of the importation mechanism. While a
project doesn't need to do anything special to be found by \c{import}, it does
need to handle locating the exported target (or targets; there could be
several) within the project as well as loading their \c{buildfiles}. This is
the job of an \i{export stub}, the \c{build/export.build} file that you might
have noticed in the \c{libhello} project:

\
libhello
├── build/
│   └── export.build
└── ...
\

Let's take a look inside:

\
$out_root/
{
  include libhello/
}

export $out_root/libhello/$import.target
\

An export stub is a special kind of \c{buildfile} that bridges from the
importing project into exporting. It is loaded in a special temporary scope
out of any project, in a \"no man's land\" so to speak. The following
variables are set on the temporary scope: \c{src_root} and \c{out_root} of the
project being imported as well as \c{import.target} containing the name of
target (without project qualification) being imported.

Typically, an export stub will open the scope of the exporting project, load
the \c{buildfile} that defines the target being exported and finally
\"return\" the absolute target to the importing project using the \c{export}
directive. And this is exactly what the export stub in our \c{libhello} does.

We now have all the pieces of the importation puzzle and you can probably see
how they all fit together. To summarize, when the build system sees an
\c{import} directive, it looks for a project with the specified name. If
found, it creates a temporary scope, sets the \c{src/out_root} variables to
point to the project and \c{import.target} \- to the target name specified in
the \c{import} directive. And then it load the project's export stub in this
scope. Inside the export stub we switch to the project's root scope, load its
\c{buildfile} and then use the \c{export} directive to set the exported
target. Once the export stub is processed, the build system obtains the
exported target and assigns it to the variable specified in the \c{import}
directive.

\N|Our export stub is quite \"loose\" in that it allows importing any target
defined in the project's source subdirectory \c{buildfile}. While we found it
to be a good balance between strictness and flexibility, if you would like to
\"tighten\" your export stubs, you can. For example:

\
if ($import.target == lib{hello})
  export $out_root/libhello/$import.target
\

If no \c{export} directive is executed in an export stub then the build system
assumes the target is not exported by the project and issues appropriate
diagnostics.|


\h#intro-lib|Library Exportation and Versioning|

By now we have examine and explained every line of every \c{buildfile} in our
\c{hello} executable project. There are, however, a few lines remain to be
covered in the source subdirectory \c{buildfile} in \c{libhello}. Here it
is in its entirety:

\
int_libs = # Interface dependencies.
imp_libs = # Implementation dependencies.

lib{hello}: {hxx ixx txx cxx}{** -version} hxx{version} \
  $imp_libs $int_libs

# Include the generated version header into the distribution (so that
# we don't pick up an installed one) and don't remove it when cleaning
# in src (so that clean results in a state identical to distributed).
#
hxx{version}: in{version} $src_root/manifest
hxx{version}: dist  = true
hxx{version}: clean = ($src_root != $out_root)

cxx.poptions =+ \"-I$out_root\" \"-I$src_root\"

obja{*}: cxx.poptions += -DLIBHELLO_STATIC_BUILD
objs{*}: cxx.poptions += -DLIBHELLO_SHARED_BUILD

lib{hello}: cxx.export.poptions = \"-I$out_root\" \"-I$src_root\"

liba{hello}: cxx.export.poptions += -DLIBHELLO_STATIC
libs{hello}: cxx.export.poptions += -DLIBHELLO_SHARED

lib{hello}: cxx.export.libs = $int_libs

# For pre-releases use the complete version to make sure they cannot
# be used in place of another pre-release or the final version.
#
if $version.pre_release
  lib{hello}: bin.lib.version = @\"-$version.project_id\"
else
  lib{hello}: bin.lib.version = @\"-$version.major.$version.minor\"

# Install into the libhello/ subdirectory of, say, /usr/include/
# recreating subdirectories.
#
{hxx ixx txx}{*}: install         = include/libhello/
{hxx ixx txx}{*}: install.subdirs = true
\

Let's start with all those \c{cxx.export.*} variables. It turns out that
merely exporting a library target is not enough for the importers of the
library to be able to use it. They also need to know where to find its
headers, which other libraries to link, etc. This information is carried in a
set of target-specific \c{cxx.export.*} variables that parallel the \c{cxx.*}
set and that together with the library's prerequisites constitute the
\i{library meta-information protocol}. Every time a source file that depends
on a library is compiled or a binary is linked, this information is
automatically extracted by the compile and link rules from the library
dependency chain, recursively. And when the library is installed, this
information is carried over to its \c{pkg-config(1)} file.

\N|Similar to the \c{c.*} and \c{cc.*} sets discussed earlier, there are also
\c{c.export.*} and \c{cc.export.*} sets.|

Here are the parts relevant to the library meta-information protocol in the
above \c{buildfile}:

\
int_libs = # Interface dependencies.
imp_libs = # Implementation dependencies.

lib{hello}: ... $imp_libs $int_libs

lib{hello}: cxx.export.poptions = \"-I$out_root\" \"-I$src_root\"

liba{hello}: cxx.export.poptions += -DLIBHELLO_STATIC
libs{hello}: cxx.export.poptions += -DLIBHELLO_SHARED

lib{hello}: cxx.export.libs = $int_libs
\

As a first step we classify all our library dependencies into \i{interface
dependencies} and \i{implementation dependencies}. A library is an interface
dependency if it is referenced from our interface, for example, by including
(importing) one of its headers (modules) from one of our (public) headers
(modules) or if one of its functions is called from our inline or template
functions.

The preprocessor options (\c{poptions}) of an interface dependency must be
made available to our library's users. The library itself should also be
explicitly linked whenever our library is linked. All this is achieved by
listing the interface dependencies in the \c{cxx.export.libs} variable (the
last line in the above fragment).

\N|More precisely, the interface dependency should be explicitly linked if a
user of our library may end up with a direct call to the dependency in one of
their object files. Not linking such a library is called \i{underlinking}
while linking a library unnecessarily (which can happen because we've included
its header but are not actually calling any of its non-inline/template
functions) is called \i{overlinking}. Unrelinking is an error on some
platforms while overlinking may slow down process startup and/or waste process
memory.

Note also that this only applies to shared libraries. In case of static
libraries, both interface and implementation dependencies are always linked,
recursively.|

To illustrate the distinction, let's say we've reimplemented our \c{libhello}
to use \c{libformat} to formal the greeting and \c{libprint} to print it.
Here is our new header (\c{hello.hxx}):

\
#include <libformat/format.hxx>

namespace hello
{
  void
  say_hello_formatted (std::ostream&, const std::string& hello);

  inline void
  say_hello (std::ostream& o, const std::string& name)
  {
    say_hello_formatted (o, format::format_hello (\"Hello\", name));
  }
}
\

And this is the new source file (\c{hello.cxx}):

\
#include <libprint/print.hxx>

namespace hello
{
  void
  say_hello_formatted (ostream& o, const string& h)
  {
    print::print_hello (o, h);
  }
}
\

In this implementation, \c{libformat} is our interface dependency since we
both include its header in our interface and call it from one of our inline
functions. In contrast, \c{libprint} is only included and used in the source
file and so we can safely treat it as an implementation dependency. The
corresponding \c{import} directives in our \c{buildfile} will then look
like this:

\
import int_libs = libformat%lib{format}
import imp_libs = libprint%lib{print}
\

The remaining three lines in the library meta-information fragment are:

\
lib{hello}: cxx.export.poptions = \"-I$out_root\" \"-I$src_root\"

liba{hello}: cxx.export.poptions += -DLIBHELLO_STATIC
libs{hello}: cxx.export.poptions += -DLIBHELLO_SHARED
\

The first line makes sure the users of our library can locate its headers by
exporting the relevant \c{-I} options. The last two lines define the library
type macros that are relied upon by the \c{export.hxx} header to setup symbol
exporting.

\N|The \c{liba{\}} and \c{libs{\}} target types correspond to the static and
shared libraries, respectively. And \c{lib{\}} is actually a target group that
can contain one, the other, or both as its members.

Specifically, when we build a \c{lib{\}} target, which members will be built
is determined by the \c{config.bin.lib} variable with the \c{static},
\c{shared}, and \c{both} (default) possible values. So to only build a shared
library we can do:

$ b config.bin.lib=shared

When it comes to linking \c{lib{\}} prerequisites, which member is picked is
controlled by the \c{config.bin.{exe,liba,libs}.lib} variables for the
executable, static library, and shared library targets, respectively. Their
valid values are lists of \c{shared} and \c{static} that determine the member
preference. For example, to build both shared and static libraries but to link
executable to static libraries we can do:

$ b config.bin.lib=both config.bin.exe.lib=static

See \l{#module-bin \c{bin} Module} for more information.|

Note also that we don't need to change anything in the above \c{buildfile} if
our library is header-only. In \c{build2} this is handled dynamically based on
the absence of source file prerequisites. In fact, the same library can be
header-only on some platforms or in some configuration and \"source-full\" in
others.

\N|In \c{build2} a header-only library (or a module interface-only library) is
not a different kind of library compared to static/shared libraries but is
rather a binary-less, or \i{binless} for short, library. So, theoretically, it
is possible to have a library that has a binless static and a binary-full
(\i{binfull}) shared variants. Note also that binless libraries can depend on
binfull libraries and are fully supported where the \c{pkg-config(1)}
functionality is concerned.|

Let's now turn to the second subject of this section and the last unexplained
bit in our \c{buildfile}: shared library versioning. Here is the relevant
fragment:

\
if $version.pre_release
  lib{hello}: bin.lib.version = @\"-$version.project_id\"
else
  lib{hello}: bin.lib.version = @\"-$version.major.$version.minor\"
\

Shared library versioning is a murky, platform-specific area. Instead of
trying to come up with a unified versioning scheme that few will comprehend
(similar to \c{autoconf}), \c{build2} provides a platform-independent
versioning scheme as well as the ability to specify platform-specific version
in a native format.

The library version is specified with the \c{bin.lib.version} target-specific
variable. Its value should be a sequence of \c{@}-pairs with the left hand
side (key) being the platform name and the right hand side (value) being the
version. An empty key signifies the platform-independent version (see \c{bin}
module for the exact semantics). For example:

\
lib{hello}: bin.lib.version = @-1.2 linux@3
\

\N{While the interface for platform-specific versions is defined, their
support is not yet implemented by the C/C++ link and install rules.}

A platform-independent version is embedded as a suffix into the library name
(and into its \c{soname}, on relevant platforms) while platform-specific
versions are handled according to the platform. Continuing with the above
example, these would be the resulting shared library names for certain
platforms:

\
libhello.so.3       # Linux
libhello-1.2.dll    # Windows
libhello-1.2.dylib  # Mac OS
\

With this background we can now explain what's going in our \c{buildfile}:

\
if $version.pre_release
  lib{hello}: bin.lib.version = @\"-$version.project_id\"
else
  lib{hello}: bin.lib.version = @\"-$version.major.$version.minor\"
\

We only use platform-independent library versioning. For releases we embed
both major and minor version components assuming that patch releases are
binary compatible. For pre-releases, however we use the complete version to
make sure it cannot be used in place of another pre-release or the final
version (\c{version.project_id} is the project's, as opposed to package's,
shortest \"version id\"; see the \l{#module-version \c{version} Module} for
details).


\h#intro-subproj|Subprojects and Amalgamations|

@@ TODO


\h#intro-lang|Buildfile Language|

By now we should have a good overall sense of what writing \c{buildfiles}
feels like. In this section examines the language in slightly more detail.

Buildfile is primarily a declarative language with support for variables, pure
functions, repetition (\c{for}-loop), and conditional inclusion/exclusion
(\c{if-else}).

\c{Buildfiles} are line-oriented. That is, every construct ends at the end of
the line unless escaped with line continuation (trailing \c{\\}). Some lines
may start a \i{block} if followed by \c{{} on the next line. Such a block ends
with a closing \c{\}} on a separate line. Some types of blocks can nest. For
example:

\
exe{hello}: {hxx cxx}{**} \
  $libs

if ($cxx.target.class == 'windows')
{
  if ($cxx.target.system == 'ming32')
  {
    ...
  }
}
\

A comment starts with \c{#} and everything from this character and until the
end of the line is ignored. A multi-line comment starts with \c{{#\\} on a
separate line and ends with the same character sequence, again on a separate
line. For example:

\
# Single line comment.

info 'Hello, World!' # Trailing comment.

#\
Multi-
line
comment.
#\
\

The three primary Buildfile construct are dependency declaration, directive,
and variable assignment. We've already used all three but let's see another
example:

\
include ../libhello/                  # Directive.

exe{hello}: {hxx cxx}{**} lib{hello}  # Dependency declaration.

cxx.poptions += -DNDEBUG              # Variable assignment (append).
\

There is also the scope opening (we've seen one in \c{export.build}) as well
as target-specific and prerequisite-specific variable assignment blocks. The
latter two are used to assign several entity-specific variables at once, for
example:

\
hxx{version}:
{
  dist  = true
  clean = ($src_root != $out_root)
}

exe{test}: file{test.roundtrip}:
{
  test.stdin  = true
  test.stdout = true
}
\

\N|All prerequisite-specific variables must be assigned at once as part of the
dependency declaration since repeating the same prerequisite again duplicates
the dependency rather than references the already existing one.|

\c{Buildfiles} are processed linearly with directives executed and variables
expanded as they are encountered. However, certain variables, for example,
\c{cxx.poptions} are also expanded by rules during execution in which case
they will \"see\" the final value set in the \c{buildfile}.

\N|Unlike GNU \c{make(1)}, which has deferred (\c{=}) and immediate (\c{:=})
variable assignments, all assignments in \c{build2} are immediate. For
example:

\
x = x
y = $x
x = X
info $y # Prints 'x', not 'X'.
\

|


\h2#intro-lang-expand|Expansion and Quoting|

While we've discussed variable expansion and lookup earlier, to recap, to get
the variable's value we use \c{$} followed by its name. The variable name is
first looked up in the current scope (that is, the scope in which the
expansion was encountered) and, if not found, in the outer scopes,
recursively.

There are two other kinds of expansions: function calls and \i{evaluation
contexts}, or eval context for short. Let's start with the latter since
function calls are built on top of eval contexts.

An eval context is essentially a fragment of a line with additional
interpretations of certain characters to support value comparison, logical
operators, and a few other things. Eval contexts begin with \c{(}, end with
\c{)}, and can nest. Here are a few examples:

\
info ($src_root != $out_root)                 # Prints true or false.
info ($src_root == $out_root ? 'in' : 'out')  # Prints in or out.

macos = ($cxx.target.class == 'macos')  # Assigns true or false.
linux = ($cxx.target.class == 'linux')  # Assigns true or false.

if ($macos || $linux)  # Also eval context.
  ...
\

\N|Below is the eval context grammar that shows supported operators and their
precedence.

\
eval:         '(' (eval-comma | eval-qual)? ')'
eval-comma:   eval-ternary (',' eval-ternary)*
eval-ternary: eval-or ('?' eval-ternary ':' eval-ternary)?
eval-or:      eval-and ('||' eval-and)*
eval-and:     eval-comp ('&&' eval-comp)*
eval-comp:    eval-value (('=='|'!='|'<'|'>'|'<='|'>=') eval-value)*
eval-value:   value-attributes? (<value> | eval | '!' eval-value)
eval-qual:    <name> ':' <name>

value-attributes: '[' <key-value-pairs> ']'
\

Note that \c{?:} (ternary operator) and \c{!} (logical not) are
right-associative. Unlike C++, all the comparison operators have the same
precedence. A qualified name cannot be combined with any other operator
(including ternary) unless enclosed in parentheses. The \c{eval} option in the
\c{eval-value} production shall contain single value only (no commas).|

A function call starts with \c{$} followed by its name and an eval context
listing its arguments. Note that there is no space between the name and
\c{(}. For example:

\
x =
y = Y

info $empty($x)  # true
info $empty($y)  # false

if $regex.match($y, '[A-Z]')
  ...

p = $src_base/foo.txt

info $path.leaf($src_base)              # foo.txt
info $path.directory($src_base)         # $src_base
info $path.base($path.leaf($src_base))  # foo
\

Note that functions in \c{build2} are \i{pure} in a sense that they do not
alter the build state in any way.

Variable and function names follow the C identifier rules. We can also group
variables into namespaces and functions into families by combining multiple
identifier with \c{.}. These rules are used to determine the end of the
variable name in expansions. If, however, a name is being treated longer than
it should, then we can use eval context to explicitly specify its boundaries.
For example:

\
base = foo
name = $(base).txt
\

What is a structure of a variable value? Consider this assignment:

\
x = foo bar
\

The value of \c{x} could be a string, a list of two strings, or something else
entirely. In \c{build2} the fundamental, untyped value is a \i{list of
names}. A value can be typed to something else later but it always starts with
a list of names. So in the above example we have a list of two names, \c{foo}
and \c{bar}, the same as in this example (notice the extra spaces):

\
x = foo    bar
\

\N|The motivation behind going with a list of names instead of a string or a
list of strings is that at its core we are dealing with targets and their
prerequisites and it would be natural to make the representation of their
names (that is, the way we refer to them) the default. Consider the following
two examples; it would be natural for them to mean the same thing:

\
exe{hello}: {hxx cxx}{**}
\

\
prereqs = {hxx cxx}{**}
exe{hello}: $prereqs
\

Note also that the name semantics was carefully tuned to be \i{reversible} to
its syntactic representation for non-name values, such as paths, command line
options, etc., that are commonly found in \c{buildfiles}.|

Names are split into a list at whitespace boundaries with certain other
characters treated as syntax rather than as part of the value. Here are
a few example:

\
x = $y          # expansion
x = (a == b)    # eval context
x = {foo bar}   # name generation
x = [null]      # attributes
x = name@value  # pairs
x = # comments
\

The complete set of syntax character is \c{$(){\}[]@#} plus space and tab.
Additionally, will be \c{*?} treated as wildcards in a name pattern. If
instead we need these characters to appear literally as part of the value,
then we either have to \i{escape} or \i{quote} them.

To escape a special character we prefix it with a backslash (\c{\\}; to
specify a literal backslash double it). For example:

\
x = \$
y = C:\\\\Program\ Files
\

Similar to UNIX shell, \c{build2} supports single (\c{''}) and double
(\c{\"\"}) quoting with roughly the same semantics. Specifically, expansions
(variable, function call, and eval context) and escaping are performed inside
double-quoted strings but not in single-quoted. Note also that quoted strings
can span multiple lines with newlines treated literally (unless escaped in
double-quoted strings). For example:

\
x = \"(a != b)\"  # true
y = '(a != b)'    # (a != b)

x = \"C:\\\\Program Files\"
y = 'C:\Program Files'

t = 'line one
line two
line three'
\

Since quote characters are now also part of the syntax, if you need to specify
them literally in the value, then they will either have to be escaped or
quoted. For example:

\
cxx.poptions += -DOUTPUT='\"debug\"'
cxx.poptions += -DTARGET=\\\"$cxx.target\\\"
\

An expansion can be of two kinds: \i{spliced} or \c{concatenated}. In the
spliced expansion the variable, function, or eval context is separated from
other text with whitespaces. In this case, as the name suggest, the resulting
list of names is spliced into the value. For example:

\
x = 'foo fox'
y = bar $x baz  # Three names: 'bar' 'foo fox' 'baz'.
\

\N|This is an important difference compared to the semantics in UNIX shells
where result of expansion is re-parsed. In particular, this is the reason why
you won't see quoted expansions in \c{buildfiles} as often as in
(well-written) shell scripts.|

In concatenated expansion the variable, function, or eval context are combined
with unseparated text before and/or after the expansion. For example:

\
x = 'foo fox'
y = bar$(x)foz  # Single name: 'barfoo foxbaz'
\

A concatenated expansion is typed unless it is quoted. In typed concatenated
expansion the parts are combined in a type-aware manner while in untyped \-
literally as string. To illustrate the difference, consider this \c{buildfile}
fragment:

\
info $src_root/foo.txt
info \"$src_root/foo.txt\"
\

If we run it on a UNIX-like operating system, we will see two identical
lines, along these lines:

\
/tmp/test/foo.txt
/tmp/test/foo.txt
\

However, if we run it on Windows (which uses backslash as a directory
separator) we will see the output along these lines:

\
C:\test\foo.txt
C:\test/foo.txt
\

The typed concatenation resulted in a native directory separator because
\c{dir_path} (the \c{src_root} type) did the right thing.

Not every typed concatenation is well defined and in certain situations we may
need to force untyped concatenation with quoting. Options specifying header
search paths (\c{-I}) are a typical case, for example:

\
cxx.poptions =+ \"-I$out_root\" \"-I$src_root\"
\

If we were to remove the quotes, we would see the following diagnostics:

\
buildfile:6:20: error: no typed concatenation of <untyped> to dir_path
  info: use quoting to force untyped concatenation
\

- style guide for quoting


\h2#intro-if-else|Conditions (\c{if-else})|

The \c{if} directive can be used to conditionally exclude \c{buildfile}
fragments from being processed. The conditional fragment can be a single
(separate) line or a block and the initial \c{if} can be optionally followed by
a number of \c{elif} directives and a final \c{else} which together form the
\c{if-else} chain. An \c{if-else} block can contain nested \c{if-else}
chains. For example:

\
if ($cxx.target.class == 'linux')
  info 'linux'
elif ($cxx.target.class == 'windows')
{
  if ($cxx.target.system == 'mingw32')
    info 'windows-mingw'
  elif ($cxx.target.system == 'win32-msvc')
    info 'windows-msvc'
  else
    info 'windows-other'
}
else
  info 'other'
\

The \c{if} and \c{elif} directive names must be followed by something that
expands to a single, literal \c{true} or \c{false}. This can be a variable
expansion, a function call, an eval context, or a literal value. For example:

\
if $version.pre_release
  ...

if $regex.match($x, '[A-Z]')
  ...

if ($cxx.target.class == 'linux')
  ...

if false
{
  # disabled fragment
}

x = X
if $x  # Error, must expand to true or false.
  ...
\

There are also \c{if!} and \c{elif!} directives which negate the condition
that follows (note that there is no space before \c{!}). For example:

\
if! $version.pre_release
  ...
elif! $regex.match($x, '[A-Z]')
  ...
\

Note also that there is no notion of variable locality in \c{if-else} blocks
and any value set inside is visible outside. For example:

\
if true
{
  x = X
}

info $x  # Prints 'X'.
\

The \c{if-else} chains should not be used for conditional dependency
declarations since this would violate the expectation that all the project's
source files are listed as prerequisites, irrespective of the configuration.
Instead, use the special \c{include} prerequisite-specific variable to
conditionally include prerequisites into the build. For example:

\
# Incorrect.
#
if ($cxx.target.class == 'linux')
  exe{hello}: cxx{utility-linux}
elif ($cxx.target.class == 'windows')
  exe{hello}: cxx{utility-win32}

# Correct.
#
exe{hello}: cxx{utility-linux}: include = ($cxx.target.class == 'linux')
exe{hello}: cxx{utility-win32}: include = ($cxx.target.class == 'windows')
\


\h2#intro-fir|Repetitions (\c{for})|

The \c{for} directive can be used to repeat the same \c{buildfile} fragment
multiple times, once for each element of a list. The fragment to repeat can be
a single (separate) line or a block which together form the \c{for} loop. A
\c{for} block can contain nested \c{for} loops. For example:

\
for n: foo bar baz
{
  exe{$n}: cxx{$n}
}
\

The \c{for} directive name must be followed by the variable name (called
\i{loop variable}) that on each iteration will be assigned the corresponding
element, \c{:}, and something that expands to a potentially empty list of
values. This can be a variable expansion, a function call, an eval context, or
a literal list as in the example above. Here is a somewhat more realistic
example that splits a space-separated environment variable value into names and
then generates a dependency declaration for each of them:

\
for n: $regex.split($getenv(NAMES), ' +', '')
{
  exe{$n}: cxx{$n}
}
\

Note also that there is no notion of variable locality in \c{for} blocks and
any value set inside us visible outside. At the end of the iteration the loop
variable contains the value of the last element, if any. For example:

\
for x: x X
{
  y = Y
}

info $x  # Prints 'X'.
info $y  # Prints 'Y'.
\


\h1#name-patterns|Name Patterns|

For convenience, in certain contexts, names can be generated with shell-like
wildcard patterns. A name is a \i{name pattern} if its value contains one or
more unquoted wildcard characters or character sequences. For example:

\
./: */                     # All (immediate) subdirectories
exe{hello}: {hxx cxx}{**}  # All C++ header/source files.
pattern = '*.txt'          # Literal '*.txt'.
\

Pattern-based name generation is not performed in certain contexts.
Specifically, it is not performed in target names where it is interpreted
as a pattern for target type/pattern-specific variable assignments. For
example.

\
s = *.txt             # Variable assignment (performed).
./: cxx{*}            # Prerequisite names (performed).
cxx{*}: dist = false  # Target pattern (not performed).
\

In contexts where it is performed, it can be inhibited with quoting, for
example:

\
pat = 'foo*bar'
./: cxx{'foo*bar'}
\

The following characters are wildcards:

\
*  - match any number of characters (including zero)
?  - match any single character
\

If a pattern ends with a directory separator, then it only matches
directories. Otherwise, it only matches files. Matches that start with a dot
(\c{.}) are automatically ignored unless the pattern itself also starts with
this character.

In addition to the above wildcard characters, \c{**} and \c{***} are
recognized as wildcard character sequences. If a pattern contains \c{**}, then
it is matched just like \c{*} but in all the subdirectories, recursively, but
excluding directories that contain the \c{.buildignore} file. The \c{***}
wildcard behaves like \c{**} but also matches the start directory itself. For
example:

\
exe{hello}: cxx{**}  # All C++ source files recursively.
\

A group-enclosed (\c{{\}}) pattern value may be followed by
inclusion/exclusion patterns/matches. A subsequent value is treated as an
inclusion or exclusion if it starts with a literal, unquoted plus (\c{+}) or
minus (\c{-}) sign, respectively. In this case the remaining group values, if
any, must all be inclusions or exclusions. If the second value doesn't start
with a plus or minus, then all the group values are considered independent
with leading pluses and minuses not having any special meaning. For regularity
as well as to allow patterns without wildcards, the first pattern can also
start with the plus sign. For example:

\
exe{hello}: cxx{f* -foo}            # Exclude foo if exists.
exe{hello}: cxx{f* +bar}            # Include bar if exists.
exe{hello}: cxx{f* -fo?}            # Exclude foo and fox if exist.
exe{hello}: cxx{f* +b* -foo -bar}   # Exclude foo and bar if exist.
exe{hello}: cxx{+f* +b* -foo -bar}  # Same as above.
exe{hello}: cxx{+foo}               # Pattern without wildcards.
exe{hello}: cxx{f* b* -z*}          # Names matching three patterns.
\

Inclusions and exclusions are applied in the order specified and only to the
result produced up to that point. The order of names in the result is
unspecified. However, it is guaranteed not to contain duplicates. The first
pattern and the following inclusions/exclusions must be consistent with
regards to the type of filesystem entry they match. That is, they should all
match either files or directories. For example:

\
exe{hello}: cxx{f* -foo +*oo}  # Exclusion has no effect.
exe{hello}: cxx{f* +*oo}       # Ok, no duplicates.
./: {*/ -build}                # Error: exclusion not a directory.
\

As a more realistic example, let's say we want to exclude source files that
reside in the \c{test/} directories (and their subdirectories) anywhere in the
tree. This can be achieved with the following pattern:

\
exe{hello}: cxx{** -***/test/**}
\

Similarly, if we wanted to exclude all source files that have the \c{-test}
suffix:

\
exe{hello}: cxx{** -**-test}
\

In contrast, the following pattern only excludes such files from the top
directory:

\
exe{hello}: cxx{** -*-test}
\

If many inclusions or exclusions need to be specified, then an
inclusion/exclusion group can be used. For example:

\
exe{hello}: cxx{f* -{foo bar}}
exe{hello}: cxx{+{f* b*} -{foo bar}}
\

This is particularly useful if you would like to list the names to include or
exclude in a variable. For example, this is how we can exclude certain files
from compilation but still include them as ordinary file prerequisites (so
that they are still included into the distribution):

\
exc = foo.cxx bar.cxx
exe{hello}: cxx{+{f* b*} -{$exc}} file{$exc}
\

If we want to specify our pattern in a variable, then we have to use the
explicit inclusion syntax, for example:

\
pat = 'f*'
exe{hello}: cxx{+$pat} # Pattern match.
exe{hello}: cxx{$pat}  # Literal 'f*'.

pat = '+f*'
exe{hello}: cxx{$pat}  # Literal '+f*'.

inc = 'f*'  'b*'
exc = 'f*o' 'b*r'
exe{hello}: cxx{+{$inc} -{$exc}}
\

One common situation that calls for exclusions is auto-generated source
code. Let's say we have auto-generated command line parser in \c{options.hxx}
and \c{options.cxx}. Because of the in-tree builds, our name pattern may or
may not find these files. Note, however, that we cannot just include them as
non-pattern prerequisites. We also have to exclude them from the pattern match
since otherwise we may end up with duplicate prerequisites. As a result, this
is how we have to handle this case provided we want to continue using patterns
to find other, non-generated source files:

\
exe{hello}: {hxx cxx}{* -options} {hxx cxx}{options}
\

If the name pattern includes an absolute directory, then the pattern match is
performed in that directory and the generated names include absolute
directories as well. Otherwise, the pattern match is performed in the
\i{pattern base} directory. In buildfiles this is \c{src_base} while on the
command line \- the current working directory. In this case the generated
names are relative to the base directory. For example, assuming we have the
\c{foo.cxx} and \c{b/bar.cxx} source files:

\
exe{hello}: $src_base/cxx{**}  # $src_base/cxx{foo} $src_base/b/cxx{bar}
exe{hello}:           cxx{**}  #           cxx{foo}           b/cxx{bar}
\

Pattern matching as well as inclusion/exclusion logic is target
type-specific. If the name pattern does not contain a type, then the
\c{dir{\}} type is assumed if the pattern ends with a directory separator and
\c{file{\}} otherwise.

For the \c{dir{\}} target type the trailing directory separator is added to
the pattern and all the inclusion/exclusion patterns/matches that do not
already end with one. Then the filesystem search is performed for matching
directories. For example:

\
./: dir{* -build}  # Search for */, exclude build/.
\

For the \c{file{\}} and \c{file{\}}-based target types the default extension
(if any) is added to the pattern and all the inclusion/exclusion
patterns/matches that do not already contain an extension. Then the filesystem
search is performed for matching files.

For example, the \c{cxx{\}} target type obtains the default extension from the
\c{extension} variable. Assuming we have the following line in our
\c{root.build}:

\
cxx{*}: extension = cxx
\

And the following in our \c{buildfile}:

\
exe{hello}: {cxx}{* -foo -bar.cxx}
\

The pattern match will first search for all the files matching the \c{*.cxx}
pattern in \c{src_base} and then exclude \c{foo.cxx} and \c{bar.cxx} from the
result. Note also that target type-specific decorations are removed from the
result. So in the above example if the pattern match produces \c{baz.cxx},
then the prerequisite name is \c{cxx{baz\}}, not \c{cxx{baz.cxx\}}.

If the name generation cannot be performed because the base directory is
unknown, target type is unknown, or the target type is not directory or
file-based, then the name pattern is returned as is (that is, as an ordinary
name). Project-qualified names are never considered to be patterns.


\h1#module-test|\c{test} Module|

The targets to be tested as well as the tests/groups from testscripts to be
run can be narrowed down using the \c{config.test} variable. While this
value is normally specified as a command line override (for example, to
quickly re-run a previously failed test), it can also be persisted in
\c{config.build} in order to create a configuration that will only run a
subset of tests by default. For example:

\
b test config.test=foo/exe{driver} # Only test foo/exe{driver} target.
b test config.test=bar/baz         # Only run bar/baz testscript test.
\

The \c{config.test} variable contains a list of \c{@}-separated pairs with the
left hand side being the target and the right hand side being the testscript
id path. Either can be omitted (along with \c{@}). If the value contains a
target type or ends with a directory separator, then it is treated as a target
name. Otherwise \- an id path. The targets are resolved relative to the root
scope where the \c{config.test} value is set. For example:

\
b test config.test=foo/exe{driver}@bar
\

To specify multiple id paths for the same target we can use the pair
generation syntax:

\
b test config.test=foo/exe{driver}@{bar baz}
\

If no targets are specified (only id paths), then all the targets are tested
(with the testscript tests to be run limited to the specified id paths). If no
id paths are specified (only targets), then all the testscript tests are run
(with the targets to be tested limited to the specified targets). An id path
without a target applies to all the targets being considered.

A directory target without an explicit target type (for example, \c{foo/}) is
treated specially. It enables all the tests at and under its directory. This
special treatment can be inhibited by specifying the target type explicitly
(for example, \c{dir{foo/\}}).


\h1#module-version|\c{version} Module|

A project can use any version format as long as it meets the package version
requirements. The toolchain also provides additional functionality for
managing projects that conform to the \c{build2} \i{standard version}
format. If you are starting a new project that uses \c{build2}, you are
strongly encouraged to use this versioning scheme. It is based on much thought
and, often painful, experience. If you decide not to follow this advice, you
are essentially on your own when version management is concerned.

The standard \c{build2} project version conforms to \l{http://semver.org
Semantic Versioning} and has the following form:

\
<major>.<minor>.<patch>[-<prerel>]
\

For example:

\
1.2.3
1.2.3-a.1
1.2.3-b.2
\

The \c{build2} package version that uses the standard project version will
then have the following form (\i{epoch} is the versioning scheme version
and \i{revision} is the package revision):

\
[+<epoch>-]<major>.<minor>.<patch>[-<prerel>][+<revision>]
\

For example:

\
1.2.3
1.2.3+1
+2-1.2.3-a.1+2
\

The \i{major}, \i{minor}, and \i{patch} should be numeric values between \c{0}
and \c{999} and all three cannot be zero at the same time. For initial
development it is recommended to use \c{0} for \i{major}, start with version
\c{0.1.0}, and change to \c{1.0.0} once things stabilize.

In the context of C and C++ (or other compiled languages), you should
increment \i{patch} when making binary-compatible changes, \i{minor} when
making source-compatible changes, and \i{major} when making breaking changes.
While the binary compatibility must be set in stone, the source compatibility
rules can sometimes be bent. For example, you may decide to make a breaking
change in a rarely used interface as part of a minor release (though this is
probably still a bad idea if your library is widely depended upon). Note also
that in the context of C++ deciding whether a change is binary-compatible is a
non-trivial task. There are resources that list the rules but no automated
tooling yet. If unsure, increment \i{minor}.

If present, the \i{prerel} component signifies a pre-release. Two types of
pre-releases are supported by the standard versioning scheme: \i{final} and
\i{snapshot} (non-pre-release versions are naturally always final). For final
pre-releases the \i{prerel} component has the following form:

\
(a|b).<num>
\

For example:

\
1.2.3-a.1
1.2.3-b.2
\

The letter '\c{a}' signifies an alpha release and '\c{b}' \- beta. The
alpha/beta numbers (\i{num}) should be between 1 and 499.

Note that there is no support for release candidates. Instead, it is
recommended that you use later-stage beta releases for this purpose (and, if
you wish, call them \"release candidates\" in announcements, etc).

What version should be used during development? The common approach is to
increment to the next version and use that until the release. This has one
major drawback: if we publish intermediate snapshots (for example, for
testing) they will all be indistinguishable both between each other and, even
worse, from the final release. One way to remedy this is to increment the
pre-release number before each publication. However, unless automated, this
will be burdensome and error-prone. Also, there is a real possibility of
running out of version numbers if, for example, we do continuous integration
by publishing and testing each commit.

To address this, the standard versioning scheme supports \i{snapshot
pre-releases} with the \i{prerel} component having the following extended
form:

\
(a|b).<num>.<snapsn>[.<snapid>]
\

For example:

\
1.2.3-a.1.20180319215815.26efe301f4a7
\

In essence, a snapshot pre-release is after the previous final release but
before the next (\c{a.1} and, perhaps, \c{a.2} in the above example) and
is uniquely identified by the snapshot sequence number (\i{snapsn}) and
optional snapshot id (\i{snapid}).

The \i{num} component has the same semantics as in the final pre-releases
except that it can be \c{0}. The \i{snapsn} component should be either the
special value '\c{z}' or a numeric, non-zero value that increases for each
subsequent snapshot. It must not be longer than 16 decimal digits. The
\i{snapid} component, if present, should be an alpha-numeric value that
uniquely identifies the snapshot. It is not required for version comparison
(\i{snapsn} should be sufficient) and is included for reference. It must not
be longer than 16 characters.

Where do the snapshot number and id come from? Normally from the version
control system. For example, for \c{git}, \i{snapsn} is the commit date in the
\i{YYYYMMDDhhmmss} form and UTC timezone and \i{snapid} is a 12-character
abbreviated commit id. As discussed below, the \c{build2} \c{version} module
extracts and manages all this information automatically (but the use of
\c{git} commit dates is not without limitations; see below for details).

The special '\c{z}' \i{snapsn} value identifies the \i{latest} or
\i{uncommitted} snapshot. It is chosen to be greater than any other possible
\i{snapsn} value and its use is discussed further below.

As an illustration of this approach, let's examine how versions change
during the lifetime of a project:

\
0.1.0-a.0.z     # development after a.0
0.1.0-a.1       # pre-release
0.1.0-a.1.z     # development after a.1
0.1.0-a.2       # pre-release
0.1.0-a.2.z     # development after a.2
0.1.0-b.1       # pre-release
0.1.0-b.1.z     # development after b.1
0.1.0           # release
0.1.1-b.0.z     # development after b.0 (bugfix)
0.2.0-a.0.z     # development after a.0
0.1.1           # release (bugfix)
1.0.0           # release (jumped straight to 1.0.0)
...
\

As shown in the above example, there is nothing wrong with \"jumping\" to a
further version (for example, from alpha to beta, or from beta to release, or
even from alpha to release). We cannot, however, jump backwards (for example,
from beta back to alpha). As a result, a sensible strategy is to start with
\c{a.0} since it can always be upgraded (but not downgrade) at a later stage.

When it comes to the version control systems, the recommended workflow is as
follows: The change to the final version should be the last commit in the
(pre-)release. It is also a good idea to tag this commit with the project
version. A commit immediately after that should change the version to a
snapshot, \"opening\" the repository for development.

The project version without the snapshot part can be represented as a 64-bit
decimal value comparable as integers (for example, in preprocessor
directives). The integer representation has the following form:

\
AAABBBCCCDDDE

AAA - major
BBB - minor
CCC - patch
DDD - alpha / beta (DDD + 500)
E   - final (0) / snapshot (1)
\

If the \i{DDDE} value is not zero, then it signifies a pre-release. In this
case one is subtracted from the \i{AAABBBCCC} value. An alpha number is stored
in \i{DDD} as is while beta \- incremented by \c{500}. If \i{E} is \c{1}, then
this is a snapshot after \i{DDD}.

For example:

\
             AAABBBCCCDDDE
0.1.0        0000010000000
0.1.2        0000010010000
1.2.3        0010020030000
2.2.0-a.1    0020019990010
3.0.0-b.2    0029999995020
2.2.0-a.1.z  0020019990011
\

A project that uses standard versioning can rely on the \c{build2} \c{version}
module to simplify and automate version managements. The \c{version} module
has two primary functions: eliminate the need to change the version anywhere
except in the project's manifest file and automatically extract and propagate
the snapshot information (serial number and id).

The \c{version} module must be loaded in the project's \c{bootstrap.build}.
While being loaded, it reads the project's manifest and extracts its version
(which must be in the standard form). The version is then parsed and presented
as the following build system variables (which can be used in the buildfiles):

\
[string] version                     # +2-1.2.3-b.4.1234567.deadbeef+3

[string] version.project             # 1.2.3-b.4.1234567.deadbeef
[uint64] version.project_number      # 0010020025041
[string] version.project_id          # 1.2.3-b.4.deadbeef

[bool]   version.stub                # false (true for 0[+<revision>])

[uint64] version.epoch               # 2

[uint64] version.major               # 1
[uint64] version.minor               # 2
[uint64] version.patch               # 3

[bool]   version.alpha               # false
[bool]   version.beta                # true
[bool]   version.pre_release         # true
[string] version.pre_release_string  # b.4
[uint64] version.pre_release_number  # 4

[bool]   version.snapshot            # true
[uint64] version.snapshot_sn         # 1234567
[string] version.snapshot_id         # deadbeef
[string] version.snapshot_string     # 1234567.deadbeef
[bool]   version.snapshot_committed  # true

[uint64] version.revision            # 3
\

As a convenience, the \c{version} module also extract the \c{summary} and
\c{url} manifest values and sets them as the following build system variables
(this additional information is used, for example, when generating the
\c{pkg-config} files):

\
[string] project.summary
[string] project.url
\

If the version is the latest snapshot (that is, it's in the \c{.z} form), then
the \c{version} module extracts the snapshot information from the version
control system used by the project. Currently only \c{git} is supported with
the following semantics.

If the project's source directory (\c{src_root}) is clean (that is, it does
not have any changed or untracked files), then the \c{HEAD} commit date and id
are used as the snapshot number and id, respectively.

Otherwise (that is, the project is between commits), the \c{HEAD} commit date
is incremented by one second and is used as the snapshot number with no id.
While we can work with such uncommitted snapshots locally, we should not
distribute or publish them since they are indistinguishable from each other.

Finally, if the project does not have \c{HEAD} (that is, the project has
no commits yet), the special \c{19700101000000} (UNIX epoch) commit date is
used.

The use of \c{git} commit dates for snapshot ordering has its limitations:
they have one second resolution which means it is possible to create two
commits with the same date (but not the same commit id and thus snapshot
id). We also need all the committers to have a reasonably accurate
clock. Note, however, that in case of a commit date clash/ordering issue, we
still end up with distinct versions (because of the commit id) \- they are
just not ordered correctly. As a result, we feel that the risks are justified
when the only alternative is manual version management (which is always an
option, nevertheless).

When we prepare a distribution of a snapshot, the \c{version} module
automatically adjusts the package name to include the snapshot information as
well as patches the manifest file in the distribution with the snapshot number
and id (that is, replacing \c{.z} in the version value with the actual
snapshot information). The result is a package that is specific to this
commit.

Besides extracting the version information and making it available as
individual components, the \c{version} module also provide rules for
installing the manifest file as well as automatically generating version
headers (or other similar version-based files).

By default the project's \c{manifest} file is installed as documentation, just
like other \c{doc{}} targets (thus replacing the \c{version} file customarily
shipped in the project root directory). The manifest installation rule in the
\c{version} module in addition patches the installed manifest file with the
actual snapshot number and id, just like during the preparation of
distributions.

The version header rule is based on the \l{#module-in \c{in}} module rule and
can be used to preprocesses a template file with version information. While it
is usually used to generate C/C++ version headers (thus the name), it can
really generate any kind of files.

The rule matches a \c{file}-based target that has the corresponding \c{in}
prerequisite and also depends on the project's \c{manifest} file. As an
example, let's assume we want to auto-generate a header called \c{version.hxx}
for our \c{libhello} library. To accomplish this we add the \c{version.hxx.in}
template as well as something along these lines to our \c{buildfile}:

\
lib{hello}: ... hxx{version}

hxx{version}: in{version} $src_root/file{manifest}
hxx{version}: dist = true
\

The header rule is a line-based preprocessor that substitutes fragments
enclosed between (and including) a pair of dollar signs (\c{$}) with \c{$$}
being the escape sequence (see the \l{#module-in \c{in}} module for
details). As an example, let's assume our \c{version.hxx.in} contains the
following lines:

\
#ifndef LIBHELLO_VERSION

#define LIBHELLO_VERSION     $libhello.version.project_number$ULL
#define LIBHELLO_VERSION_STR \"$libhello.version.project$\"

#endif
\

If our \c{libhello} is at version \c{1.2.3}, then the generated
\c{version.hxx} will look like this:

\
#ifndef LIBHELLO_VERSION

#define LIBHELLO_VERSION     10020030000ULL
#define LIBHELLO_VERSION_STR \"1.2.3\"

#endif
\

The first component after the opening \c{$} should be either the name of the
project itself (like \c{libhello} above) or a name of one of its dependencies
as listed in the manifest. If it is the project itself, then the rest can
refer to one of the \c{version.*} variables that we discussed earlier (in
reality it can be any variable visible from the project's root scope).

If the name refers to one of the dependecies (that is, projects listed with
\c{depends:} in the manifest), then the following special substitutions are
recognized:

\
$<name>.version$                           - textual version constraint
$<name>.condition(<VERSION>[,<SNAPSHOT>])$ - numeric satisfaction condition
$<name>.check(<VERSION>[,<SNAPSHOT>])$     - numeric satisfaction check
\

Here \i{VERSION} is the version number macro and the optional \i{SNAPSHOT} is
the snapshot number macro. The snapshot is only required if you plan to
include snapshot information in your dependency constraints.

As an example, let's assume our \c{libhello} depends on \c{libprint} which
is reflected with the following line in our manifest:

\
depends: libprint >= 2.3.4
\

We also assume that \c{libprint} provides its version information in the
\c{libprint/version.hxx} header and uses analogous-named macros. Here
is how we can add a version check to our \c{version.hxx.in}:

\
#ifndef LIBHELLO_VERSION

#define LIBHELLO_VERSION     $libhello.version.project_number$ULL
#define LIBHELLO_VERSION_STR \"$libhello.version.project$\"

#include <libprint/version.hxx>

$libprint.check(LIBPRINT_VERSION)$

#endif
\

After the substitution our \c{version.hxx} header will look like this:

\
#ifndef LIBHELLO_VERSION

#define LIBHELLO_VERSION     10020030000ULL
#define LIBHELLO_VERSION_STR \"1.2.3\"

#include <libprint/version.hxx>

#ifdef LIBPRINT_VERSION
#  if !(LIBPRINT_VERSION >= 20030040000ULL)
#    error incompatible libprint version, libprint >= 2.3.4 is required
#  endif
#endif

#endif
\

The \c{version} and \c{condition} substitutions are the building blocks of the
\c{check} substitution. For example, here is how we can implement a check with
a customized error message:

\
#if !($libprint.condition(LIBPRINT_VERSION)$)
#  error bad libprint, need libprint $libprint.version$
#endif
\

The \c{version} module also treats one dependency in a special way: if you
specify the required version of the build system in your manifest, then the
module will automatically check it for you. For example, if we have the
following line in our manifest:

\
depends: * build2 >= 0.5.0
\

And someone tries to build our project with \c{build2} \c{0.4.0}, then they
will see an error like this:

\
build/bootstrap.build:3:1: error: incompatible build2 version
  info: running 0.4.0
  info: required 0.5.0
\

What version constraints should be use when depending on other project. We
start with a simple case where we depend on a release. Let's say \c{libprint}
\c{2.3.0} added a feature that we need in our \c{libhello}. If \c{libprint}
follows the source/binary compatibility guidelines discussed above, then
any \c{2.X.Y} version should work provided \c{X >= 3}. And this how we can
specify it in the manifest:

\
depends: libprint ^2.3.0
\

Let's say we are now working on \c{libhello} \c{2.0.0} and would like to start
using features from \c{libprint} \c{3.0.0}. However, currently, only
pre-releases of \c{3.0.0} are available. If you would like to add a dependency
on a pre-release (most likely from your own pre-release), then the
recommendation is to only allow a specific version, essentially \"expiring\"
the combination as soon as newer versions become available. For example:

\
version: 2.0.0-b.1
depends: libprint == 3.0.0-b.2
\

Finally, let's assume we are feeling adventerous and would like to test
development snapshots of \c{libprint} (most likey from our own snapshots). In
this case the recommendation is to only allow a snapshot range for a specific
pre-release with the understanding and a warning that no compatibility between
snapshot versions is guaranteed. For example:

\
version: 2.0.0-b.1.z
depends: libprint [3.0.0-b.2.1 3.0.0-b.3)
\

\h1#module-bin|\c{bin} Module|

@@ TODO


\h1#module-cxx|\c{cxx} Module|

This chapter describes the \c{cxx} build system module which provides the C++
compilation and linking support. Most of its functionality, however, is
provided by the \c{cc} module, a common implementation for the C-family
languages.

\h#cxx-modules|C++ Modules Support|

This section describes the build system support for C++ modules.

\h2#cxx-modules-intro|Modules Introduction|

The goal of this section is to provide a practical introduction to C++ Modules
and to establish key concepts and terminology.

A pre-modules C++ program or library consists of one or more \i{translation
units} which are customarily referred to as C++ source files. Translation
units are compiled to \i{object files} which are then linked together to
form a program or library.

Let's also recap the difference between an \i{external name} and a \i{symbol}:
External names refer to language entities, for example classes, functions, and
so on. The \i{external} qualifier means they are visible across translation
units.

Symbols are derived from external names for use inside object files. They are
the cross-referencing mechanism for linking a program from multiple,
separately-compiled translation units. Not all external names end up becoming
symbols and symbols are often \i{decorated} with additional information, for
example, a namespace. We often talk about a symbol having to be satisfied by
linking an object file or a library that provides it. Similarly, duplicate
symbol issues may arise if more than one object file or library provides
the same symbol.

What is a C++ module? It is hard to give a single but intuitive answer to this
question.  So we will try to answer it from three different perspectives: that
of a module consumer, a module producer, and a build system that tries to make
those two play nice. But we can make one thing clear at the outset: modules
are a \i{language-level} not a preprocessor-level mechanism; it is \c{import},
not \c{#import}.

One may also wonder why C++ modules, what are the benefits? Modules offer
isolation, both from preprocessor macros and other modules' symbols. Unlike
headers, modules require explicit exportation of entities that will be visible
to the consumers. In this sense they are a \i{physical design mechanism} that
forces us to think how we structure our code. Modules promise significant
build speedups since importing a module, unlike including a header, should be
essentially free. Modules are also the first step to not needing the
preprocessor in most translation units. Finally, modules have a chance of
bringing to mainstream reliable and easy to setup distributed C++ compilation,
since with modules build systems can make sure compilers on the local and
remote hosts are provided with identical inputs.

To refer to a module we use a \i{module name}, a sequence of dot-separated
identifiers, for example \c{hello.core}. While the specification does not
assign any hierarchical semantics to this sequence, it is customary to refer
to \c{hello.core} as a submodule of \c{hello}. We discuss submodules and
provide the module naming guidelines below.

From a consumer's perspective, a module is a collection of external names,
called \i{module interface}, that become \i{visible} once the module is
imported:

\
import hello.core
\

What exactly does \i{visible} mean? To quote the standard: \i{An
import-declaration makes exported declarations [...] visible to name lookup in
the current translation unit, in the same namespaces and contexts [...]. [
Note: The entities are not redeclared in the translation unit containing the
module import declaration. -- end note ]} One intuitive way to think about
this visibility is \i{as if} there were only a single translation unit for the
entire program that contained all the modules as well as all their
consumers. In such a translation unit all the names would be visible to
everyone in exactly the same way and no entity would be redeclared.

This visibility semantics suggests that modules are not a name scoping
mechanism and are orthogonal to namespaces. Specifically, a module can export
names from any number of namespaces, including the global namespace. While the
module name and its namespace names need not be related, it usually makes
sense to have a parallel naming scheme, as discussed below. Finally, the
\c{import} declaration does not imply any additional visibility for names
declared inside namespaces. Specifically, to access such names we must
continue using the standard mechanisms, such as qualification or using
declaration/directive.  For example:

\
import hello.core;        // Exports hello::say().

say ();                   // Error.
hello::say ();            // Ok.

using namespace hello;
say ();                   // Ok.
\

Note also that from the consumer's perspective a module does not provide
any symbols, only C++ entity names. If we use names from a module, then we
may have to satisfy the corresponding symbols using the usual mechanisms:
link an object file or a library that provides them. In this respect, modules
are similar to headers and as with headers, module's use is not limited to
libraries; they make perfect sense when structuring programs. Furthermore,
a library may also have private or implementation modules that are not
meant to be consumed by the library's users.

The producer perspective on modules is predictably more complex. In
pre-modules C++ we only had one kind of translation unit (or source
file). With modules there are three kinds: \i{module interface unit},
\i{module implementation unit}, and the original kind which we will
call a \i{non-module translation unit}.

From the producer's perspective, a module is a collection of module translation
units: one interface unit and zero or more implementation units. A simple
module may consist of just the interface unit that includes implementations
of all its functions (not necessarily inline). A more complex module may
span multiple implementation units.

A translation unit is a module interface unit if it contains an \i{exporting
module declaration}:

\
export module hello.core;
\

A translation unit is a module implementation unit if it contains a
\i{non-exporting module declaration}:

\
module hello.core;
\

While module interface units may use the same file extension as normal source
files, we recommend that a different extension be used to distinguish them as
such, similar to header files. While the compiler vendors suggest various (and
predictably different) extensions, our recommendation is \c{.mxx} for the
\c{.hxx/.cxx} source file naming and \c{.mpp} for \c{.hpp/.cpp}. And if you
are using some other naming scheme, then perhaps now is a good opportunity to
switch to one of the above. Continuing using the source file extension for
module implementation units appears reasonable and that's what we recommend.

A module declaration (exporting or non-exporting) starts a \i{module purview}
that extends until the end of the module translation unit. Any name declared
in a module's purview \i{belongs} to said module. For example:

\
#include <string>                // Not in purview.

export module hello.core;        // Start of purview.

void
say_hello (const std::string&);  // In purview.
\

A name that belongs to a module is \i{invisible} to the module's consumers
unless it is \i{exported}. A name can be declared exported only in a module
interface unit, only in the module's purview, and there are several syntactic
ways to accomplish this. We can start the declaration with the \c{export}
specifier, for example:

\
export module hello.core;

export enum class volume {quiet, normal, loud};

export void
say_hello (const char*, volume);
\

Alternatively, we can enclose one or more declarations into an \i{exported
group}, for example:

\
export module hello.core;

export
{
  enum class volume {quiet, normal, loud};

  void
  say_hello (const char*, volume);
}
\

Finally, if a namespace definition is declared exported, then every name
in its body is exported, for example:

\
export module hello.core;

export namespace hello
{
  enum class volume {quiet, normal, loud};

  void
  say (const char*, volume);
}

namespace hello
{
  void
  impl (const char*, volume); // Not exported.
}
\

Up until now we've only been talking about names belonding to a module. What
about the corresponding symbols? For exported names, the resulting symbols
would be the same as if those names were declared outside of a module's
purview (or as if no modules were used at all). Non-exported names, on the
other hand, have \i{module linkage}: their symbols can be resolved from this
module's units but not from other translation units. They also cannot clash
with symbols for identical names from other modules (and non-modules). This is
usually achieved by decorating the non-exported symbols with the module name.

This ownership model has an important backwards compatibility implication: a
library built with modules enabled can be linked to a program that still uses
headers. And even the other way around: we can build and use a module for a
library that was built with headers.

What about the preprocessor? Modules do not export preprocessor macros,
only C++ names. A macro defined in the module interface unit cannot affect
the module's consumers. And macros defined by the module's consumers cannot
affect the module interface they are importing. In other words, module
producers and consumers are isolated from each other when the preprocessor
is concerned. For example, consider this module interface:

\
export module hello;

#ifndef SMALL
#define HELLO
export void say_hello (const char*);
#endif
\

And its consumer:

\
// module consumer
//
#define SMALL       // No effect.
import hello;

#ifdef HELLO        // Not defined.
...
#endif
\

This is not to say that the preprocessor cannot be used by either, it just
doesn't \"leak\" through the module interface. One practical implication of
this model is the insignificance of the import order.

If a module imports another module in its purview, the imported module's
names are not made automatically visible to the consumers of the importing
module. This is unlike headers and can be surprising. Consider this module
interface as an example:

\
export module hello;

import std.core;

export void
say_hello (const std::string&);
\

And its consumer:

\
import hello;

int
main ()
{
  say_hello (\"World\");
}
\

This example will result in a compile error and the diagnostics may
confusingly indicate that there is no known conversion from a C string to
\"something\" called \c{std::string}. But with the understanding of the
difference between \c{import} and \c{#include} the reason should be clear:
while the module interface \"sees\" \c{std::string} (because it imported its
module), we (the consumer) do not (since we did not). So the fix is to
explicitly import \c{std.core}:

\
import std.core;
import hello;

int
main ()
{
  say_hello (\"World\");
}
\

A module, however, can choose to re-export a module it imports. In this case,
all the names from the imported module will also be visible to the importing
module's consumers. For example, with this change to the module interface the
first version of our consumer will compile without errors (note that whether
this is a good design choice is debatable, as discussed below):

\
export module hello;

export import std.core;

export void
say_hello (const std::string&);
\

One way to think of a re-export is \i{as if} an import of a module also
\"injects\" all the imports said module re-exports, recursively. That's
essentially how most compilers implement it.

Module re-export is the mechanism for assembling bigger modules out of
submodules. As an example, let's say we had the \c{hello.core},
\c{hello.basic}, and \c{hello.extra} modules. To make life easier for users
that want to import all of them we can create the \c{hello} module that
re-exports the three:

\
export module hello;

export
{
  import hello.core;
  import hello.basic;
  import hello.extra;
}
\

Besides starting a module purview, a non-exporting module declaration in the
implementation unit makes non-internal linkage names declared or made visible
in the \i{interface purview} also visible in the \i{implementation purview}.
In this sense non-exporting module declaration acts as an extended
\c{import}. For example:

\
import hello.impl;          // Not visible (exports impl()).

void
extra_impl ();              // Not visible.

export module hello.extra;  // Start of interface purview.

import hello.core;          // Visible (exports core()).

void
extra ();                   // Visible.

static void
extra2 ();                  // Not visible (internal linkage).
\

And this is the implementation unit:

\
module hello.extra;         // Start of implementation purview.

void
f ()
{
  impl ();        // Error.
  extra_impl ();  // Error.
  core ();        // Ok.
  extra ();       // Ok.
  extra2 ();      // Error.
}
\

In particular, this means that while the relative order of imports is not
significant, the placement of imports in the module interface unit relative
to the module declaration can be.

The final perspective that we consider is that of the build system. From its
point of view the central piece of the module infrastructure is the \i{binary
module interface}: a binary file that is produced by compiling the module
interface unit and that is required when compiling any translation unit that
imports this module as well as the module's implementation units.

Then, in a nutshell, the main functionality of a build system when it comes to
modules support is figuring out the order in which all the translation units
should be compiled and making sure that every compilation process is able to
find the binary module interfaces it needs.

Predictably, the details are more complex. Compiling a module interface unit
produces two outputs: the binary module interface and the object file. The
latter contains object code for non-inline functions, global variables, etc.,
that the interface unit may define. This object file has to be linked when
producing any binary (program or library) that uses this module.

Also, all the compilers currently implement module re-export as a shallow
reference to the re-exported module name which means that their binary
interfaces must be discoverable as well, recursively. In fact, currently, all
the imports are handled like this, though a different implementation is at
least plausible, if unlikely.

While the details vary between compilers, the contents of the binary module
interface can range from a stream of preprocessed tokens to something fairly
close to object code. As a result, binary interfaces can be sensitive to the
compiler options and if the options used to produce the binary interface (for
example, when building a library) are sufficiently different compared to the
ones used when compiling the module consumers, the binary interface may be
unusable. So while a build system should strive to reuse existing binary
interfaces, it should also be prepared to compile its own versions \"on the
side\".

This also suggests that binary module interfaces are not a distribution
mechanism and should probably not be installed. Instead, we should install and
distribute module interface sources and build systems should be prepared to
compile them, again, on the side.


\h2#cxx-modules-build|Building Modules|

Compiler support for C++ Modules is still experimental. As a result, it is
currently only enabled if the C++ standard is set to \c{experimental}. After
loading the \c{cxx} module we can check if modules are enabled using the
\c{cxx.features.modules} boolean variable. This is what the relevant
\c{root.build} fragment could look like for a modularized project:

\
cxx.std = experimental

using cxx

assert $cxx.features.modules 'compiler does not support modules'

mxx{*}: extension = mxx
cxx{*}: extension = cxx
\

To support C++ modules the \c{cxx} module (build system) defines several
additional target types. The \c{mxx{\}} target is a module interface unit.
As you can see from the above \c{root.build} fragment, in this project we
are using the \c{.mxx} extension for our module interface files. While
you can use the same extension as for \c{cxx{\}} (source files), this is
not recommended since some functionality, such as wildcard patterns, will
become unusable.

The \c{bmi{\}} group and its \c{bmie{\}}, \c{bmia{\}}, and \c{bmis{\}} members
are used to represent binary module interfaces targets. We normally do not
need to mention them explicitly in our buildfiles except, perhaps, to specify
additional, module interface-specific compile options. We will see some
examples of this below.

To build a modularized executable or library we simply list the module
interfaces as its prerequisites, just as we do for source files. As an
example, let's build the \c{hello} program that we have started in the
introduction (you can find the complete project in the
\l{https://build2.org/pkg/hello Hello Repository} under
\c{mhello}). Specifically, we assume our project contains the following files:

\
// file: hello.mxx (module interface)

export module hello;

import std.core;

export void
say_hello (const std::string&);
\

\
// file: hello.cxx (module implementation)

module hello;

import std.io;

using namespace std;

void
say_hello (const string& name)
{
  cout << \"Hello, \" << name << '!' << endl;
}
\

\
// file: driver.cxx

import std.core;
import hello;

int
main ()
{
  say_hello (\"World\");
}
\

To build a \c{hello} executable from these files we can write the following
\c{buildfile}:

\
exe{hello}: cxx{driver} {mxx cxx}{hello}
\

Or, if you prefer to use wildcard patterns:

\
exe{hello}: {mxx cxx}{*}
\

Alternatively, we can package the module into a library and then link the
library to the executable:

\
exe{hello}: cxx{driver} lib{hello}
lib{hello}: {mxx cxx}{hello}
\

As you might have surmised from this example, the modules support in
\c{build2} automatically resolves imports to module interface units that are
specified either as direct prerequisites or as prerequisites of library
prerequisites.

To perform this resolution without a significant overhead, the implementation
delays the extraction of the actual module name from module interface units
(since not all available module interfaces are necessarily imported by all the
translation units). Instead, the implementation tries to guess which interface
unit implements each module being imported based on the interface file
path. Or, more precisely, a two-step resolution process is performed: first a
best match between the desired module name and the file path is sought and
then the actual module name is extracted and the correctness of the initial
guess is verified.

The practical implication of this implementation detail is that our module
interface files must embed a portion of a module name, or, more precisely, a
sufficient amount of \"module name tail\" to unambiguously resolve all the
modules used in a project. Note also that this guesswork is only performed for
direct module interface prerequisites; for those that come from libraries the
module names are known and are therefore matched exactly.

As an example, let's assume our \c{hello} project had two modules:
\c{hello.core} and \c{hello.extra}. While we could call our interface files
\c{hello.core.mxx} and \c{hello.extra.mxx}, respectively, this doesn't look
particularly good and may be contrary to the file naming scheme used in our
project. To resolve this issue the match of module names to file names is
made \"fuzzy\": it is case-insensitive, it treats all separators (dots, dashes,
underscores, etc) as equal, and it treats a case change as an imaginary
separator. As a result, the following naming schemes will all match the
\c{hello.core} module name:

\
hello-core.mxx
hello_core.mxx
HelloCore.mxx
hello/core.mxx
\

We also don't have to embed the full module name. In our case, for example, it
would be most natural to call the files \c{core.mxx} and \c{extra.mxx} since
they are already in the project directory called \c{hello/}. This will work
since our module names can still be guessed correctly and unambiguously.

If a guess turns out to be incorrect, the implementation issues diagnostics
and exits with an error before attempting to build anything. To resolve this
situation we can either adjust the interface file names or we can specify the
module name explicitly with the \c{cxx.module_name} variable. The latter
approach can be used with interface file names that have nothing in common
with module names, for example:

\
mxx{foobar}@./: cxx.module_name = hello
\

Note also that standard library modules (\c{std} and \c{std.*}) are treated
specially: they are not fuzzy-matched and they need not be resolvable to
the corresponding \c{mxx{\}} or \c{bmi{\}} in which case it is assumed
they will be resolved in an ad hoc way by the compiler. This means that if
you want to build your own standard library module (for example, because
your compiler doesn't yet ship one; note that this may not be supported
by all compilers), then you have to specify the module name explicitly.
For example:

\
exe{hello}: cxx{driver} {mxx cxx}{hello} mxx{std-core}

mxx{std-core}@./: cxx.module_name = std.core
\

When C++ modules are enabled and available, the build system makes sure the
\c{__cpp_modules} feature test macro is defined. Currently, its value is
\c{201703} for VC and \c{201704} for GCC and Clang but this will most likely
change in the future.

One major difference between the current C++ modules implementation in VC and
the other two compilers is the use of the \c{export module} syntax to identify
the interface units. While both GCC and Clang have adopted this new syntax,
VC is still using the old one without the \c{export} keyword. We can use the
\c{__cpp_modules} macro to provide a portable declaration:

\
#if __cpp_modules >= 201704
export
#endif
module hello;
\

Note, however, that the modules support in \c{build2} provides temporary
\"magic\" that allows us to use the new syntax even with VC (don't ask how).

\h2#cxx-modules-symexport|Module Symbols Exporting|

When building a shared library, some platforms (notably Windows) require that
we explicitly export symbols that must be accessible to the library users.
If you don't need to support such platforms, you can thank your lucky stars
and skip this section.

When using headers, the traditional way of achieving this is via an \"export
macro\" that is used to mark exported APIs, for example:

\
LIBHELLO_EXPORT void
say_hello (const string&);
\

This macro is then appropriately defined (often in a separate \"export
header\") to export symbols when building the shared library and to import
them when building the library's users.

The introduction of modules changes this in a number of ways, at least as
implemented by VC (hopefully other compilers will follow suit). While we
still have to explicitly mark exported symbols in our module interface
unit, there is no need (and, in fact, no way) to do the same when said
module is imported. Instead, the compiler automatically treats all
such explicitly exported symbols (note: symbols, not names) as imported.

One notable aspect of this new model is the locality of the export macro: it
is only defined when compiling the module interface unit and is not visible to
the consumers of the module. This is unlike headers where the macro has to
have a unique per-library name (that \c{LIBHELLO_} prefix) because a header
from one library can be included while building another library.

We can continue using the same export macro and header with modules and, in
fact, that's the recommended approach when maintaining the dual, header/module
arrangement for backwards compatibility (discussed below). However, for
modules-only codebases, we have an opportunity to improve the situation in two
ways: we can use a single, keyword-like macro instead of a library-specific
one and we can make the build system manage it for us thus getting rid of the
export header.

To enable this functionality in \c{build2} we set the
\c{cxx.features.symexport} boolean variable to \c{true} before loading the
\c{cxx} module. For example:

\
cxx.std = experimental

cxx.features.symexport = true

using cxx

...
\

Once enabled, \c{build2} automatically defines the \c{__symexport} macro to
the appropriate value depending on the platform and the type of library being
built. As library authors, all we have to do is use it in appropriate places
in our module interface units, for example:

\
export module hello;

import std.core;

export __symexport void
say_hello (const std::string&);
\

As an aside, you may be wondering why can't a module export automatically mean
a symbol export? While you will normally want to export symbols of all your
module-exported names, you may also need to do so for some non-module-exported
ones. For example:

\
export module foo;

__symexport void
f_impl ();

export __symexport inline void
f ()
{
  f_impl ();
}
\

Furthermore, symbol exporting is a murky area with many limitations and
pitfalls (such as auto-exporting of base classes). As a result, it would not
be unreasonable to expect such an automatic module exporting to only further
muddy the matter.


\h2#cxx-modules-install|Modules Installation|

As discussed in the introduction, binary module interfaces are not a
distribution mechanism and installing module interface sources appears to be
the preferred approach.

Module interface units are by default installed in the same location as
headers (for example, \c{/usr/include}). However, instead of relying on a
header-like search mechanism (\c{-I} paths, etc.), an explicit list of
exported modules is provided for each library in its \c{.pc} (\c{pkg-config})
file.

Specifically, the library's \c{.pc} file contains the \c{cxx_modules} variable
that lists all the exported C++ modules in the \c{<name>=<path>} form with
\c{<name>} being the module's C++ name and \c{<path>} \- the module interface
file's absolute path. For example:

\
Name: libhello
Version: 1.0.0
Cflags:
Libs: -L/usr/lib -lhello

cxx_modules = hello.core=/usr/include/hello/core.mxx hello.extra=/usr/include/hello/extra.mxx
\

Additional module properties are specified with variables in the
\c{cxx_module_<property>.<name>} form, for example:

\
cxx_module_symexport.hello.core = true
cxx_module_preprocessed.hello.core = all
\

Currently, two properties are defined. The \c{symexport} property with the
boolean value signals whether the module uses the \c{__symexport} support
discussed above.

The \c{preprocessed} property indicates the degree of preprocessing the module
unit requires and is used to optimize module compilation. Valid values are
\c{none} (not preprocessed), \c{includes} (no \c{#include} directives in the
source), \c{modules} (as above plus no module declarations depend on the
preprocessor, for example, \c{#ifdef}, etc.), and \c{all} (the source is fully
preprocessed). Note that for \c{all} the source may still contain comments and
line continuations.


\h2#cxx-modules-guidelines|Modules Design Guidelines|

Modules are a physical design mechanism for structuring and organizing our
code. Their explicit exportation semantics combined with the way they are
built make many aspects of creating and consuming modules significantly
different compared to headers. This section provides basic guidelines for
designing modules. We start with the overall considerations such as module
granularity and partitioning into translation units then continue with the
structure of typical module interface and implementation units. The following
section discusses practical approaches to modularizing existing code and
providing dual, header/module interfaces for backwards-compatibility.

Unlike headers, the cost of importing modules should be negligible. As a
result, it may be tempting to create \"mega-modules\", for example, one per
library. After all, this is how the standard library is modularized with its
fairly large \c{std.core} and \c{std.io} modules.

There is, however, a significant drawback to this choice: every time we make a
change, all consumers of such a mega-module will have to be recompiled,
whether the change affects them or not. And the bigger the module the higher
the chance that any given change does not (semantically) affect a large
portion of the module's consumers. Note also that this is not an issue for the
standard library modules since they are not expected to change often.

Another, more subtle, issue with mega-modules (which does affect the standard
library) is the inability to re-export only specific interfaces, as will be
discussed below.

The other extreme in choosing module granularity is a large number of
\"mini-modules\". Their main drawback is the tediousness of importation by the
consumers.

The sensible approach is then to create modules of conceptually-related and
commonly-used entities possibly complemented with aggregate modules for ease
of importation. This also happens to be generally good design.

As an example, let's consider an XML library that provides support for both
parsing and serialization. Since it is common for applications to only use one
of the functionalities, it makes sense to provide the \c{xml.parser} and
\c{xml.serializer} modules. While it is not too tedious to import both, for
convenience we could also provide the \c{xml} module that re-exports the two.

Once we are past selecting an appropriate granularity for our modules, the
next question is how to partition them into translation units. A module can
consist of just the interface unit and, as discussed above, such a unit can
contain anything an implementation unit can, including non-inline function
definitions. Some may then view this as an opportunity to get rid of the
header/source separation and have everything in a single file.

There are a number of drawbacks with this approach: Every time we change
anything in the module interface unit, all its consumers have to be
recompiled. If we keep everything in a single file, then every time we change
the implementation we trigger recompilations that would have been avoided had
the implementation been factored out into a separate unit. Note that a build
system in cooperation with the compiler could theoretically avoid such
unnecessary recompilations: if the compiler produces identical binary
interface files when the module interface is unchanged, then the build system
could detect this and skip recompiling the module's consumers.

A related issue with single-file modules is the reduction in the build
parallelization opportunities. If the implementation is part of the interface
unit, then the build system cannot start compiling the module's consumers
until both the interface and the implementation are compiled. On the other
hand, had the implementation been split into a separate file, the build system
could start compiling the module's consumers (as well as the implementation
unit) as soon as the module interface is compiled.

Another issues with combining the interface with the implementation is the
readability of the interface which could be significantly reduced if littered
with implementation details. We could keep the interface separate by moving
the implementation to the bottom of the interface file but then we might as
well move it into a separate file and avoid the unnecessary recompilations or
parallelization issues.

The sensible guideline is then to have a separate module implementation unit
except perhaps for modules with a simple implementation that is mostly
inline/template. Note that more complex modules may have several
implementation units, however, based on our granularity guideline, those
should be rare.

Once we start writing our first real module the immediate question that
normally comes up is where to put \c{#include} directives and \c{import}
declarations and in what order. To recap, a module unit, both interface and
implementation, is split into two parts: before the module declaration which
obeys the usual or \"old\" translation unit rules and after the module
declaration which is the module purview. Inside the module purview all
non-exported declarations have module linkage which means their symbols are
invisible to any other module (including the global module). With this
understanding, consider the following module interface:

\
export module hello;

#include <string>
\

Do you see the problem? We have included \c{<string>} in the module purview
which means all its names (as well as all the names in any headers it might
include, recursively) are now declared as having the \c{hello} module linkage.
The result of doing this can range from silent code blot to strange-looking
unresolved symbols.

The guideline this leads to should be clear: including a header in the module
purview is almost always a bad idea. There are, however, a few types of
headers that may make sense to include in the module purview. The first are
headers that only define preprocessor macros, for example, configuration or
export headers. There are also cases where we do want the included
declarations to end up in the module purview. The most common example is
inline/template function implementations that have been factored out into
separate files for code organization reasons. As an example, consider the
following module interface that uses an export header (which presumably sets
up symbols exporting macros) as well as an inline file:

\
#include <string>

export module hello;

#include <libhello/export.hxx>

export namespace hello
{
  ...
}

#include <libhello/hello.ixx>
\

A note on inline/template files: in header-based projects we could include
additional headers in those files, for example, if the included declarations
are only needed in the implementation. For the reasons just discussed, this
does not work with modules and we have to move all the includes into the
interface file, before the module purview. On the other hand, with modules, it
is safe to use namespace-level using-directives (for example, \c{using
namespace std;}) in inline/template files (and, with care, even in the
interface file).

What about imports, where should we import other modules? Again, to recap,
unlike a header inclusion, an \c{import} declaration only makes exported names
visible without redeclaring them. As result, in module implementation
units, it doesn't really matter where we place imports, in or out of the
module purview. There are, however, two differences when it comes to module
interface units: only imports in the purview are visible to implementation
units and we can only re-export an imported module from the purview.

The guideline is then for interface units to import in the module purview
unless there is a good reason not to make the import visible to the
implementation units. And for implementation units to always import in the
purview for consistency. For example:

\
#include <cassert>

export module hello;

import std.core;

#include <libhello/export.hxx>

export namespace hello
{
  ...
}

#include <libhello/hello.ixx>
\

By putting all these guidelines together we can then create a module interface
unit template:

\
// Module interface unit.

<header includes>

export module <name>;      // Start of module purview.

<module imports>

<special header includes>  // Configuration, export, etc.

<module interface>

<inline/template includes>
\

As well as the module implementation unit template:

\
// Module implementation unit.

<header includes>

module <name>;             // Start of module purview.

<extra module imports>     // Only additional to interface.

<module implementation>
\

Let's now discuss module naming. Module names are in a separate \"name plane\"
and do not collide with namespace, type, or function names. Also, as mentioned
earlier, the standard does not assign a hierarchical meaning to module names
though it is customary to assume module \c{hello.core} is a submodule of
\c{hello} and importing the latter also imports the former.

It is important to choose good names for public modules (that is, modules
packaged into libraries and used by a wide range of consumers) since changing
them later can be costly. We have more leeway with naming private modules
(that is, the ones used by programs or internal to libraries) though it's
worth coming up with a consistent naming scheme here as well.

The general guideline is to start names of public modules with the library's
namespace name followed by a name describing the module's functionality. In
particular, if a module is dedicated to a single class (or, more generally,
has a single primary entity), then it makes sense to use its name as the
module name's last component.

As a concrete example, consider \c{libbutl} (the \c{build2} utility library):
All its components are in the \c{butl} namespace so all its module names start
with \c{butl.} One of its components is the \c{small_vector} class template
which resides in its own module called \c{butl.small_vector}. Another
component is a collection of string parsing utilities that are grouped into
the \c{butl::string_parser} namespace with the corresponding module called
\c{butl.string_parser}.

When is it a good idea to re-export a module? The two straightforward cases
are when we are building an aggregate module out of submodules, for example,
\c{xml} out of \c{xml.parser} and \c{xml.serializer}, or when one module
extends or supersedes another, for example, as \c{std.core} extends
\c{std.fundamental}. It is also clear that there is no need to re-export a
module that we only use in the implementation. The case when we use a module
in our interface is, however, a lot less clear cut.

But before considering the last case in more detail, let's understand the
issue with re-export. In other words, why not simply re-export any module we
import in our interface? In essence, re-export implicitly injects another
module import anywhere our module is imported. If we re-export \c{std.core}
then consumers of our module will also automatically \"see\" all the names
exported by \c{std.core}. They can then start using names from \c{std} without
explicitly importing \c{std.core} and everything will compile until one day
they no longer need to import our module or we no longer need to import
\c{std.core}. In a sense, re-export becomes part of our interface and it is
generally good design to keep interfaces minimal.

And so, at the outset, the guideline is then to only re-export the minimum
necessary. This, by the way, is the reason why it may make sense to divide
\c{std.core} into submodules such as \c{std.core.string}, \c{std.core.vector},
etc.

Let's now discuss a few concrete examples to get a sense of when re-export
might or might not be appropriate. Unfortunately, there does not seem to be a
hard and fast rule and instead one has to rely on their good sense of design.

To start, let's consider a simple module that uses \c{std::string} in its
interface:

\
export module hello;

import std.core;

export namespace hello
{
  void say (const std::string&);
}
\

Should we re-export \c{std.core} (or, \c{std.core.string}) in this case? Most
likely not. If consumers of our module want to use \c{std::string} in order to
pass an argument to our function, then it is natural to expect them to
explicitly import the necessary module. In a sense, this is analogous to
scoping: nobody expects to be able to use just \c{string} (without \c{std::})
because of \c{using namespace hello;}.

So it seems that a mere usage of a name in an interface does not generally
warrant a re-export. The fact that a consumer may not even use this part of
our interface further supports this conclusion.

Let's now consider a more interesting case (inspired by real events):

\
export module small_vector;

import std.core;

template <typename T, std::size_t N>
export class small_vector: public std::vector<T, ...>
{
  ...
};
\

Here we have the \c{small_vector} container implemented in terms of
\c{std::vector} by providing a custom allocator and with most of the functions
derived as is. Consider now this innocent-looking consumer code:

\
import small_vector;

small_vector<int, 1> a, b;

if (a == b) // Error.
  ...
\

We don't reference \c{std::vector} directly so presumably we shouldn't need to
import its module. However, the comparison won't compile: our \c{small_vector}
implementation re-uses the comparison operators provided by \c{std::vector}
(via implicit to-base conversion) but they aren't visible.

There is a palpable difference between the two cases: the first merely uses
\c{std.core} interface while the second is \i{based on} and, in a sense,
\i{extends} it which feels like a stronger relationship. Re-exporting
\c{std.core} (or, better yet, \c{std.core.vector}, should it become available)
does not seem unreasonable.

Note also that there is no re-export of headers nor header inclusion
visibility in the implementation units. Specifically, in the previous example,
if the standard library is not modularized and we have to use it via headers,
then the consumers of our \c{small_vector} will always have to explicitly
include \c{<vector>}. This suggest that modularizing a codebase that still
consumes substantial components (like the standard library) via headers can
incur some development overhead compared to the old, headers-only approach.


\h2#cxx-modules-existing|Modularizing Existing Code|

The aim of this section is to provide practical guidelines to modularizing
existing codebases as well as supporting the dual, header/module interface for
backwards-compatibility.

Predictably, a well modularized (in the general sense) set of headers makes
conversion to C++ modules easier. Inclusion cycles will be particularly hard
to deal with (C++ modules do not allow circular interface dependencies).
Furthermore, as we will see below, if you plan to provide the dual
header/module interface, then having a one-to-one header to module mapping
will simplify this task. As a result, it may make sense to spend some time
cleaning and re-organizing your headers prior to attempting modularization.

Let's first discuss why the modularization approach illustrated by the
following example does not generally work:

\
export module hello;

export
{
#include \"hello.hxx\"
}
\

There are several issue that usually make this unworkable. Firstly, the header
we are trying to export most likely includes other headers. For example, our
\c{hello.hxx} may include \c{<string>} and we have already discussed why
including it in the module purview, let alone exporting its names, is a bad
idea. Secondly, the included header may declare more names than what should be
exported, for example, some implementation details. In fact, it may declare
names with internal linkage (uncommon for headers but not impossible) which
are illegal to export. Finally, the header may define macros which will no
longer be visible to the consumers.

Sometimes, however, this can be the only approach available (for example, if
trying to non-intrusively modularize a third-party library). It is possible to
work around the first issue by \i{pre-including} outside of the module purview
headers that should not be exported. Here we rely on the fact that the second
inclusion of the same header will be ignored. For example:

\
#include <string> // Pre-include to suppress inclusion below.

export module hello;

export
{
#include \"hello.hxx\"
}
\

Needless to say this approach is very brittle and usually requires that you
place all the inter-related headers into a single module. As a result, its use
is best limited to exploratory modularization and early prototyping.

When starting modularization of a codebase there are two decisions we have to
make at the outset: the level of the C++ modules support we can assume and the
level of backwards compatibility we need to provide.

The two modules support levels we distinguish are just modules and modules
with the modularized standard library. The choice we have to make then is
whether to support the standard library only as headers, only as modules, or
both. Note that some compiler/standard library combinations may not be usable
in some of these modes.

The possible backwards compatibility levels are \i{modules-only} (consumption
via headers is no longer supported), \i{modules-or-headers} (consumption
either via headers or modules), and \i{modules-and-headers} (as the previous
case but with support for consuming a library built with modules via headers
and vice versa).

What kind of situations call for the last level? We may need to continue
offering the library as headers if we have a large number of existing
consumers that cannot possibly be all modularized at once (or even ever). So
the situation we may end up in is a mixture of consumers trying to use the
same build of our library with some of them using modules and some \-
headers. The case where we may want to consume a library built with headers
via modules is not as far fetched as it may seem: the library might have been
built with an older version of the compiler (for example, it was installed
from a distribution's package) while the consumer is being built with a
compiler version that supports modules. Note also that as discussed earlier
the modules ownership semantics supports both kinds of such \"cross-usage\".

Generally, compiler implementations do not support mixing inclusion and
importation of the same entities in the same translation unit. This makes
migration tricky if you plan to use the modularized standard library because
of its pervasive use. There are two plausible strategies to handling this
aspect of migration: If you are planning to consume the standard library
exclusively as modules, then it may make sense to first change your entire
codebase to do that. Simply replace all the standard library header inclusions
with importation of the relevant \c{std.*} modules.

The alternative strategy is to first complete the modularization of our entire
project (as discussed next) while continuing consuming the standard library as
headers. Once this is done, we can normally switch to using the modularized
standard library quite easily. The reason for waiting until the complete
modularization is to eliminate header inclusions between components which
would often result in conflicting styles of the standard library consumption.

Note also that due to the lack of header re-export and include visibility
support discussed earlier, it may make perfect sense to only support the
modularized standard library when modules are enabled even when providing
backwards compatibility with headers. In fact, if all the compiler/standard
library implementations that your project caters to support the modularized
standard library, then there is little sense not to impose such a restriction.

The overall strategy for modularizing our own components is to identify and
modularize inter-dependent sets of headers one at a time starting from the
lower-level components. This way any newly modularized set will only depend on
the already modularized ones. After converting each set we can switch its
consumers to using imports keeping our entire project buildable and usable.

While ideally we would want to be able to modularize just a single component
at a time, this does not seem to work in practice because we will have to
continue consuming some of the components as headers. Since such headers can
only be imported out of the module purview, it becomes hard to reason (both
for us and often the compiler) what is imported/included and where. For
example, it's not uncommon to end up importing the module in its
implementation unit which is not something that all the compilers can handle
gracefully.

Let's now explore how we can provide the various levels of backwards
compatibility discussed above. Here we rely on two feature test macros to
determine the available modules support level: \c{__cpp_modules} (modules are
available) and \c{__cpp_lib_modules} (standard library modules are available,
assumes \c{__cpp_modules} is also defined).

If backwards compatibility is not necessary (the \i{modules-only} level), then
we can use the module interface and implementation unit templates presented
earlier and follow the above guidelines. If we continue consuming the standard
library as headers, then we don't need to change anything in this area. If we
only want to support the modularized standard library, then we simply replace
the standard library header inclusions with the corresponding module
imports. If we want to support both ways, then we can use the following
templates. The module interface unit template:

\
// C includes, if any.

#ifndef __cpp_lib_modules
<std includes>
#endif

// Other includes, if any.

export module <name>;

#ifdef __cpp_lib_modules
<std imports>
#endif

<module interface>
\

The module implementation unit template:

\
// C includes, if any.

#ifndef __cpp_lib_modules
<std includes>

<extra std includes>
#endif

// Other includes, if any.

module <name>;

#ifdef __cpp_lib_modules
<extra std imports>        // Only additional to interface.
#endif

<module implementation>
\

For example:

\
// hello.mxx (module interface)

#ifndef __cpp_lib_modules
#include <string>
#endif

export module hello;

#ifdef __cpp_lib_modules
import std.core;
#endif

export void say_hello (const std::string& name);
\

\
// hello.cxx (module implementation)

#ifndef __cpp_lib_modules
#include <string>

#include <iostream>
#endif

module hello;

#ifdef __cpp_lib_modules
import std.io;
#endif

using namespace std;

void say_hello (const string& n)
{
  cout << \"Hello, \" << n << '!' << endl;
}
\

If we need support for symbol exporting in this setup (that is, we are
building a library and need to support Windows), then we can use the
\c{__symexport} mechanism discussed earlier, for example:

\
// hello.mxx (module interface)

...

export __symexport void say_hello (const std::string& name);
\

The consumer code in the \i{modules-only} setup is straightforward: they
simply import the desired modules.

To support consumption via headers when modules are unavailable (the
\i{modules-or-headers} level) we can use the following setup. Here we also
support the dual header/modules consumption for the standard library (if this
is not required, replace \c{#ifndef __cpp_lib_modules} with \c{#ifndef
__cpp_modules} and remove \c{#ifdef __cpp_lib_modules}). The module interface
unit template:

\
#ifndef __cpp_modules
#pragma once
#endif

// C includes, if any.

#ifndef __cpp_lib_modules
<std includes>
#endif

// Other includes, if any.

#ifdef __cpp_modules
export module <name>;

#ifdef __cpp_lib_modules
<std imports>
#endif
#endif

<module interface>
\

The module implementation unit template:

\
#ifndef __cpp_modules
#include <module interface file>
#endif

// C includes, if any.

#ifndef __cpp_lib_modules
<std includes>

<extra std includes>
#endif

// Other includes, if any

#ifdef __cpp_modules
module <name>;

#ifdef __cpp_lib_modules
<extra std imports>        // Only additional to interface.
#endif
#endif

<module implementation>
\

Notice the need to repeat \c{<std includes>} in the implementation file due to
the lack of include visibility discussed above. This is necessary when modules
are enabled but the standard library is not modularized since in this case the
implementation does not \"see\" any of the headers included in the interface.

Besides these templates we will most likely also need an export header that
appropriately defines a module export macro depending on whether modules are
used or not. This is also the place where we can handle symbol exporting. For
example, here is what it could look like for our \c{libhello} library:

\
// export.hxx (module and symbol export)

#pragma once

#ifdef __cpp_modules
#  define LIBHELLO_MODEXPORT export
#else
#  define LIBHELLO_MODEXPORT
#endif

#if   defined(LIBHELLO_SHARED_BUILD)
#  ifdef _WIN32
#    define LIBHELLO_SYMEXPORT __declspec(dllexport)
#  else
#    define LIBHELLO_SYMEXPORT
#  endif
#elif defined(LIBHELLO_SHARED)
#  ifdef _WIN32
#    define LIBHELLO_SYMEXPORT __declspec(dllimport)
#  else
#    define LIBHELLO_SYMEXPORT
#  endif
#else
#  define LIBHELLO_SYMEXPORT
#endif
\

And this is the module that uses it and provides the dual header/module
support:

\
// hello.mxx (module interface)

#ifndef __cpp_modules
#pragma once
#endif

#ifndef __cpp_lib_modules
#include <string>
#endif

#ifdef __cpp_modules
export module hello;

#ifdef __cpp_lib_modules
import std.core;
#endif
#endif

#include <libhello/export.hxx>

LIBHELLO_MODEXPORT namespace hello
{
  LIBHELLO_SYMEXPORT void say (const std::string& name);
}
\

\
// hello.cxx (module implementation)

#ifndef __cpp_modules
#include <libhello/hello.mxx>
#endif

#ifndef __cpp_lib_modules
#include <string>

#include <iostream>
#endif

#ifdef __cpp_modules
module hello;

#ifdef __cpp_lib_modules
import std.io;
#endif
#endif

using namespace std;

namespace hello
{
  void say (const string& n)
  {
    cout << \"Hello, \" << n << '!' << endl;
  }
}
\

The consumer code in the \i{modules-or-headers} setup has to use either
inclusion or importation depending on the modules support availability, for
example:

\
#ifdef __cpp_modules
import hello;
#else
#include <libhello/hello.mxx>
#endif
\

Predictably, the final backwards compatibility level (\i{modules-and-headers})
is the most onerous to support. Here existing consumers have to continue
working with the modularized version of our library which means we have to
retain all the existing header files. We also cannot assume that just because
modules are available they are used (a consumer may still prefer headers),
which means we cannot rely on (only) the \c{__cpp_modules} and
\c{__cpp_lib_modules} macros to make the decisions.

One way to arrange this is to retain the headers and adjust them according to
the \i{modules-or-headers} template but with one important difference: instead
of using the standard module macros we use our custom ones (and we can also
have unconditional \c{#pragma once}). For example:

\
// hello.hxx (module header)

#pragma once

#ifndef LIBHELLO_LIB_MODULES
#include <string>
#endif

#ifdef LIBHELLO_MODULES
export module hello;

#ifdef LIBHELLO_LIB_MODULES
import std.core;
#endif
#endif

#include <libhello/export.hxx>

LIBHELLO_MODEXPORT namespace hello
{
  LIBHELLO_SYMEXPORT void say (const std::string& name);
}
\

Now if this header is included (for example, by an existing consumer) then
none of the \c{LIBHELLO_*MODULES} macros will be defined and the header will
act as, well, a plain old header. Note that we will also need to make the
equivalent change in the export header.

We also provide the module interface files which appropriately define the two
custom macros and then simply includes the corresponding headers:

\
// hello.mxx (module interface)

#ifdef __cpp_modules
#define LIBHELLO_MODULES
#endif

#ifdef __cpp_lib_modules
#define LIBHELLO_LIB_MODULES
#endif

#include <libhello/hello.hxx>
\

The module implementation unit can remain unchanged. In particular, we
continue including \c{hello.mxx} if modules support is unavailable. However,
if you find the use of different macros in the header and source files
confusing, then instead it can be adjusted as follows (note also that now we
are including \c{hello.hxx}):

\
// hello.cxx (module implementation)

#ifdef __cpp_modules
#define LIBHELLO_MODULES
#endif

#ifdef __cpp_lib_modules
#define LIBHELLO_LIB_MODULES
#endif

#ifndef LIBHELLO_MODULES
#include <libhello/hello.hxx>
#endif

#ifndef LIBHELLO_LIB_MODULES
#include <string>

#include <iostream>
#endif

#ifdef LIBHELLO_MODULES
module hello;

#ifdef LIBHELLO_LIB_MODULES
import std.io;
#endif
#endif

...
\

In this case it may also make sense to factor the \c{LIBHELLO_*MODULES} macro
definitions into a common header.

In the \i{modules-and-headers} setup the existing consumers that would like to
continue using headers don't require any changes. And for those that would
like to use modules if available the arrangement is the same as for the
\i{modules-or-headers} compatibility level.

If our module needs to \"export\" macros then the recommended approach is to
simply provide an additional header that the consumer includes. While it might
be tempting to also wrap the module import into this header, some may prefer
to explicitly import the module and include the header, especially if the
macros may not be needed by all consumers. This way we can also keep the
header macro-only which means it can be included freely, in or out of module
purviews.


\h1#module-in|\c{in} Module|

The \c{in} build system module provides support for \c{.in} (input) file
preprocessing. Specifically, the \c{.in} file can contain a number of
\i{substitutions} \- build system variable names enclosed with the
substitution symbol (\c{$} by default) \- which are replaced with the
corresponding variable values to produce the output file. For example:

\
# build/root.build

using in
\

\
// config.hxx.in

#define TARGET \"$cxx.target$\"
\

\
# buildfile

hxx{config}: in{config}
\

The \c{in} module defines the \c{in{\}} target type and implements the \c{in}
build system rule.

While we can specify the \c{.in} extension explicitly, it is not necessary
because the \c{in{\}} target type implements \i{target-dependent search} by
taking into account the target it is a prerequisite of. In other words, the
following dependency declarations produce the same result:

\
hxx{config}: in{config}
hxx{config}: in{config.hxx}
hxx{config}: in{config.hxx.in}
\

By default the \c{in} rule uses \c{$} as the substitution symbol. This can be
changed using the \c{in.symbol} variable. For example:

\
// data.cxx.in

const char data[] = \"@data@\";
\

\
# buildfile

cxx{data}: in{data}
cxx{data}: in.symbol = '@'
cxx{data}: data = 'Hello, World!'
\

Note that the substitution symbol must be a single character.

The default substitution mode is strict. In this mode every substitution
symbol is expected to start a substitution with unresolved (to a variable
value) names treated as errors. The double substitution symbol (for example,
\c{$$}) serves as an escape sequence.

The substitution mode can be relaxed using the \c{in.substitution} variable.
Its valid values are \c{strict} (default) and \c{lax}. In the lax mode a pair
of substitution symbols is only treated as a substitution if what's between
them looks like a build system variable name (that is, it doesn't contain
spaces, etc). Everything else, including unterminated substitution symbols, is
copied as is. Note also that in this mode the double substitution symbol is
not treated as an escape sequence.

The lax mode is mostly useful when trying to reuse existing \c{.in} files from
other build systems, such as \c{autoconf}. Note, however, that the lax mode is
still stricter than the \c{autoconf}'s semantics which also leaves unresolved
substitutions as is. For example:

\
# buildfile

h{config}: in{config} # config.h.in
h{config}: in.symbol = '@'
h{config}: in.substitution = lax
h{config}: CMAKE_SYSTEM_NAME = $c.target.system
h{config}: CMAKE_SYSTEM_PROCESSOR = $c.target.cpu
\

The \c{in} rule tracks changes to the input file as well as the substituted
variable values and automatically regenerates the output file if any were
detected. Substituted variable values are looked up starting from the
target-specific variables. Typed variable values are converted to string
using the corresponding \c{builtin.string()} function overload before
substitution.

A number of other build system modules, for example, \l{#module-version
\c{version}} and \l{#module-bash \c{bash}}, are based on the \c{in} module and
provide extended functionality. The \c{in} preprocessing rule matches any
\c{file{\}}-based target that has the corresponding \c{in{\}} prerequisite
provided none of the extended rules match.


\h1#module-bash|\c{bash} Module|

The \c{bash} build system module provides modularization support for \c{bash}
scripts. It is based on the \l{#module-in \c{in}} build system module and
extends its preprocessing rule with support for \i{import substitutions} in
the \c{@import\ <module>@} form. During preprocessing, such imports are
replaced with suitable \c{source} builtin calls. For example:

\
# build/root.build

using bash
\

\
# hello/say-hello.bash

function say_hello ()
{
  echo \"Hello, $1!\"
}
\

\
#!/usr/bin/env bash

# hello/hello.in

@import hello/say-hello@

say_hello 'World'
\

\
# hello/buildfile

exe{hello}: in{hello} bash{say-hello}
\

By default the \c{bash} preprocessing rule uses the lax substitution mode and
\c{@} as the substitution symbol but this can be overridden using the standard
\c{in} module mechanisms.

In the above example, \c{say-hello.bash} is a \i{module}. By convention,
\c{bash} modules have the \c{.bash} extension and we use the \c{bash{\}}
target type (defined by the \c{bash} build system module) to refer to them in
buildfiles.

The \c{say-hello.bash} module is \i{imported} by the \c{hello} script with the
\c{@import\ hello/say-hello@} substitution. The \i{import path}
(\c{hello/say-hello} in our case) is a relative path to the module file within
the project. Its first component (\c{hello} in our case) must be the project
base name and the \c{.bash} module extension can be omitted. \N{The constraint
placed on the first component of the import path is required to implement
importation of installed modules, as discussed below.}

During preprocessing, the import substitution will be replaced with a
\c{source} builtin call and the import path resolved to one of the \c{bash{\}}
prerequisites from the script's dependency declaration. The actual module path
used in \c{source} depends on whether the script is preprocessed for
installation. If it's not (development build), then the absolute path to the
module file is used. Otherwise, a path relative to the sourcing script's
directory is derived. This allows installed scripts and their modules to be
moved around.

\N|The derivation of the sourcing script's directory works even if the script
is executed via a symbolic link from another directory. Implementing this,
however, requires \c{readlink(1)} with support for the \c{-f} option. One
notable platform that does not provide such \c{readlink(1)} by default is Mac
OS. The script, however, can provide a suitable implementation as a function.
See the \c{bash} module tests for a sample implementation of such a function.|

By default, \c{bash} modules are installed into a subdirectory of the \c{bin/}
installation directory named as the project base name. For instance, in the
above example, the script will be installed as \c{bin/hello} and the module as
\c{bin/hello/say-hello.bash} with the script sourcing the module relative to
the \c{bin/} directory. Note that currently it is assumed the script and all
its modules are installed into the same \c{bin/} directory.

Naturally, modules can import other modules and modules can be packaged into
\i{module libraries} and imported using the standard build system import
mechanism. For example, we could factor the \c{say-hello.bash} module into a
separate \c{libhello} project:

\
# build/export.build

$out_root/
{
  include libhello/
}

export $src_root/libhello/$import.target
\

\
# libhello/say-hello.bash

function hello_say_hello ()
{
  echo \"Hello, $1!\"
}
\

And then import it in a module of our \c{hello} project:

\
# hello/hello-world.bash.in

@import libhello/say-hello@

function hello_world ()
{
  hello_say_hello 'World'
}
\

\
#!/usr/bin/env bash

# hello/hello.in

@import hello/hello-world@

hello_world
\

\
# hello/buildfile

import mods = libhello%bash{say-hello}

exe{hello}:        in{hello}       bash{hello-world}
bash{hello-world}: in{hello-world} $mods
\

The \c{bash} preprocessing rule also supports importation of installed modules
by searching in the \c{PATH} environment variable.

By convention, \c{bash} module libraries should use the \c{lib} name prefix,
for example, \c{libhello}. If there is also a native library (that is, one
written in C/C++) that provides the same functionality (or the \c{bash}
library is a language binding for said library), then it is customary to add
the \c{.bash} extension to the \c{bash} library name, for example,
\c{libhello.bash}. Note that in this case the project base name is
\c{libhello}.

Modules can be \i{private} or \i{public}. Private modules are implementation
details of a specific project and are not expected to be imported from other
projects. The \c{hello/hello-world.bash.in} module above is an example of a
private module. Public modules are meant to be used by other projects and are
normally packaged into libraries, like the \c{libhello/say-hello.bash} module
above.

Public modules must take care to avoid name clashes. Since \c{bash} does not
have a notion of namespaces, the recommended way is to prefix all module
functions (and global variables, if any) with the library name (without the
\c{lib} prefix), like in the \c{libhello/say-hello.bash} module above.

While using such decorated function names can be unwieldy, it is relatively
easy to create wrappers with shorter names and use those instead. For example:

\
@import libhello/say-hello@

function say_hello () { hello_say_hello \"$@\"; }
\

A module should normally also prevent itself from being sourced multiple
times. The recommended way to achieve this is to begin the module with a
\i{source guard}. For example:

\
# libhello/say-hello.bash

if [ \"$hello_say_hello\" ]; then
  return 0
else
  hello_say_hello=true
fi

function hello_say_hello ()
{
  echo \"Hello, $1!\"
}
\

The \c{bash} preprocessing rule matches \c{exe{\}} targets that have the
corresponding \c{in{\}} and one or more \c{bash{\}} prerequisites as well as
\c{bash{\}} targets that have the corresponding \c{in{\}} prerequisite (if you
need to preprocess a script that does not depend on any modules, you can use
the \c{in} module's rule).
"