// file : doc/manual.cli // license : MIT; see accompanying LICENSE file "\name=build2-package-manager-manual" "\subject=package manager" "\title=Package Manager" // NOTES // // - Maximum 
 line is 70 characters.
//

"
\h0#preface|Preface|

This document describes \c{bpkg}, the \c{build2} package dependency
manager. For the package manager command line interface refer to the
\l{bpkg(1)} man pages.

\h1#package-name|Package Name|

The \c{bpkg} package name can contain ASCII alphabetic characters
(\c{[a-zA-Z]}), digits (\c{[0-9]}), underscores (\c{_}), plus/minus (\c{+-}),
and dots/periods (\c{\c{.}}). The name must be at least two characters long
with the following additional restrictions:

\ol|

\li|It must start with an alphabetic character.|

\li|It must end with an alphabetic, digit, or plus character.|

\li|It must not be any of the following illegal names:

\
build
con prn aux nul
com1 com2 com3 com4 com5 com6 com7 com8 com9
lpt1 lpt2 lpt3 lpt4 lpt5 lpt6 lpt7 lpt8 lpt9
\

||

The use of the plus (\c{+}) character in package names is discouraged.
\N{Pluses are used in URL encoding which makes specifying packages that
contain pluses in URLs cumbersome.}

The use of the dot (\c{.}) character in package names is discouraged except
for distinguishing the implementations of the same functionality for different
languages. \N{For example, \c{libfoo} and \c{libfoo.bash}.}

Package name comparison is case-insensitive but the original case must be
preserved for display, in file names, etc. \N{The reason for case-insensitive
comparison is Windows file names.}

If the package is a library then it is strongly recommended that you start its
package name with the \c{lib} prefix, for example, \c{libfoo}. Some package
repositories may make this a requirement as part of their submission policy.

If a package (normally a library) supports usage of multiple major versions in
the same project, then it is recommended to append the major version number to
the package name starting from version \c{2.0.0}, for example, \c{libfoo}
(before \c{2.0.0}), \c{libfoo2} (\c{2.Y.Z}), \c{libfoo3} (\c{3.Y.Z}), etc.


\h1#package-version|Package Version|

The \c{bpkg} package version format tries to balance the need of accommodating
existing software versions on one hand and providing a reasonably
straightforward comparison semantics on another. For some background on this
problem see \cb{deb-version(1)} and the \l{http://semver.org Semantic
Versioning} specification.

Note also that if you are starting a new project that will use the \c{build2}
toolchain, then it is strongly recommended that you use the \i{standard
versioning} scheme which is a more strictly defined subset of semantic
versioning that allows automation of many version management tasks. See
\l{b#module-version \c{version} Module} for details.

The \c{bpkg} package version has the following form:

\
[+-][-][+][#]
\

The \i{epoch} part should be an integer. It can be used to change to a new
versioning scheme that would be incompatible with the old one. If not
specified, then \i{epoch} defaults to \c{1} except for a stub version (see
below) in which case it defaults to \c{0}. The explicit zero \i{epoch} can be
used if the current versioning scheme (for example, date-based) is known to be
temporary.

The \i{upstream} part is the upstream software version that this package
is based on. It can only contain alpha-numeric characters and \c{.}. The
\c{.} character is used to separate the version into \i{components}.

The \i{prerel} part is the upstream software pre-release marker, for example,
alpha, beta, candidate, etc. Its format is the same as for \i{upstream} except
for two special values: the absent \i{prerel} (for example, \c{1.2.3})
signifies the maximum or final release while the empty \i{prerel} (for
example, \c{1.2.3-}) signifies the minimum or earliest possible
release. \N{The minimum release is intended to be used for version
constraints (for example, \c{libfoo < 1.2.3-}) rather than actual releases.}

The \i{revision} part should be an integer. It is used to version package
releases that are based on the same upstream versions. If not specified, then
\i{revision} defaults to \c{0}.

The \i{iteration} part is an integer. It is used internally by \c{bpkg} to
automatically version modifications to the packaging information
(specifically, to package manifest and lockfile) in \i{external packages} that
have the same upstream version and revision. As a result, the \i{iteration}
cannot not be specified by the user and is only shown in the \c{bpkg} output
(for example, by \c{pkg-status} command) in order to distinguish between
package iterations with otherwise identical versions. Note also that
\i{iteration} is relative to the \c{bpkg} configuration. Or, in other words,
it is an iteration number of a package as observed by a specific
configuration. As a result, two configurations can \"see\" the same package
state as two different iterations.

\N|Package iterations are used to support package development during which
requiring the developer to manually increment the version or revision after
each modification would be impractical. This mechanism is similar to the
automatic commit versioning provided by the \i{standard version} except that
it is limited to the packaging information but works for uncommitted changes.|

Version \c{+0-0-} (least possible version) is reserved and specifying it
explicitly is illegal. \N{Explicitly specifying this version does not make
much sense since \c{libfoo < +0-0-} is always false and \c{libfoo > +0-0-} is
always true. In the implementation this value is used as a special empty
version.}

Version \c{0} (with a potential revision, for example, \c{0+1}, \c{0+2}) is
used to signify a \i{stub package}. A stub is a package that does not contain
source code and can only be \"obtained\" from other sources, for example, a
system package manager. Note that at some point a stub may be converted into a
full-fledged package at which point it will be assigned a \"real\" version.
It is assumed that this version will always be greater than the stub version.

When displaying the package version or when using the version to derive the
file name, the default \i{epoch} value as well as zero \i{revision} and
\i{iteration} values are omitted (even if they were explicitly specified, for
instance, in the package manifest). For example, \c{+1-1.2.3+0} will be used
as \c{libfoo-1.2.3}.

\N|This versioning scheme and the choice of delimiter characters (\c{.-+})
is meant to align with semantic versioning.|

Some examples of versions:

\
0+1
+0-20180112
1.2.3
1.2.3-a1
1.2.3-b2
1.2.3-rc1
1.2.3-alpha1
1.2.3-alpha.1
1.2.3-beta.1
1.2.3+1
+2-1.2.3
+2-1.2.3-alpha.1+3
+2.2.3#1
1.2.3+1#1
+2-1.2.3+1#2
\

The version sorting order is \i{epoch}, \i{upstream}, \i{prerel},
\i{revision}, and finally, \i{iteration}. The \i{upstream} and \i{prerel}
parts are compared from left to right, one component at a time, as described
next.

To compare two components, first the component types are determined.  A
component that only consists of digits is an integer. Otherwise, it is a
string. If both components are integers, then they are compared as
integers. Otherwise, they are compared lexicographically and
case-insensitively. \N{The reason for case-insensitive comparison is Windows
file names.}

A non-existent component is considered 0 if the other component is an integer
and an empty string if the other component is a string.  For example, in
\c{1.2} vs \c{1.2.0}, the third component in the first version is 0 and the
two versions are therefore equal. As a special exception to this rule, an
absent \i{prerel} part is always greater than any non-absent part. \N{And
thus making the final release always older than any pre-release.}

This algorithm gives correct results for most commonly-used versioning
schemes, for example:

\
1.2.3 < 12.2
1.alpha < 1.beta
20151128 < 20151228
2015.11.28 < 2015.12.28
\

One notable versioning scheme where this approach gives an incorrect result is
hex numbers (consider \c{A} vs \c{1A}). The simplest work around is to convert
such numbers to decimal. Alternatively, one can fix the width of the hex
number and pad all the values with leading zeros, for example: \c{00A} vs
\c{01A}.

It is also possible to convert the \i{upstream} and \i{prerel} parts into a
\i{canonical representation} that will produce the correct comparison result
when always compared lexicographically and as a whole. \N{This can be
useful, for example, when storing versions in the database which would
otherwise require a custom collation implementation to obtain the correct sort
order.}

To convert one of these parts to its canonical representation, all its string
components are converted to the lower case while all its integer components
are padded with leading zeros to the fixed length of \c{16} characters, with
all trailing zero-only components removed. Note that this places an
implementation limit on the length of integer components which should be
checked by the implementation when converting to the canonical
representation. \N{The \c{16} characters limit was chosen to still be able
to represent (with some spare) components in the \i{YYYYMMDDhhmmss} form while
not (visually) bloating the database too much.} As a special case, the absent
\i{prerel} part is represented as \c{~}. \N{Since the ASCII code for
\c{~} is greater than any other character that could appear in \i{prerel},
such a string will always be greater than any other representation.} The empty
\i{prerel} part is represented as an empty string.

Note that because it is not possible to perform a reverse conversion without
the possibility of loss (consider \c{01.AA.BB}), the original parts may also
have to be stored, for example, for display, to derive package archive names,
etc.

\N|In quite a few contexts the implementation needs to ignore the \i{revision}
and/or \i{iteration} parts. For example, this is needed to implement the
semantics of newer revisions/iterations of packages replacing their old ones
since we do not keep multiple revisions/iterations of the same upstream
version in the same repository. As a result, in the package object model, we
have a version key as just {\i{epoch}, \i{upstream}, \i{prerel}} but also
store the package revision and iteration so that it can be shown to the user,
etc.|


\h1#package-version-constraint|Package Version Constraint|

The \c{bpkg} package version constraint may follow the package name in certain
contexts, such as the manifest values and \c{bpkg} command line, to restrict
the allowed package version set. It can be specified using comparison
operators, shortcut (to range) operators, or ranges and has the following
form:

\
 =  |  | 
         = ('==' | '>' | '<' | '>=' | '<=') 
           = ('^' | '~') 
              = ('(' | '[')   (')' | ']')
\

The shortcut operators can only be used with \l{b#module-version standard
versions} (a semantic version without the pre-release part is a standard
version). They are equivalent to the following ranges. \N{The \c{X.Y.Z-} version
signifies the earliest pre-release in the \c{X.Y.Z} series; see
\l{#package-version Package Version} for details}.

\
~X.Y.Z  [X.Y.Z  X.Y+1.0-)

^X.Y.Z  [X.Y.Z  X+1.0.0-)  if X >  0
^0.Y.Z  [0.Y.Z  0.Y+1.0-)  if X == 0
\

That is, the tilde (\c{~}) constraint allows upgrades to any further patch
version while the caret (\c{^}) constraint \- also to any further minor
version.

\N|Zero major version component is customarily used during early development
where the minor version effectively becomes major. As a result, the tilde
constraint has special semantics for this case.|

Note that the shortuct operators can only be used with the complete,
three-component versions (\c{X.Y.Z} with the optional pre-release part per the
standard version). Specifically, there is no support for special \c{^X.Y} or
\c{~X} semantics offered by some package manager \- if desired, such
functionality can be easily achieved with ranges. Also, the \c{0.0.Z} version
is not considered special except as having zero major component for the tilde
semantics discussed above.

Note also that pre-releases do not require any special considerations when
used with the shortcut operators. For example, if package \c{libfoo} is
usable starting with the second beta of the \c{2.0.0} release, then our
constraint could be expressed as:

\
libfoo ^2.0.0-b.2
\

\N|Internally, shortcuts and comparisons can be represented as ranges (that
is, \c{[v, v]} for \c{==}, \c{(v, inf)} for \c{>}, etc). However, for display
and serialization such representations should be converted back to simple
operators. While it is possible that the original manifest specified equality
or shortucts as full ranges, it is acceptable to display/serialize them as
simpler operators.|

Instead of a concrete value, the version in the constraint can be specified in
terms of the dependent package's version (that is, the version of the package
placing the constraint) using the special \c{$} value. For example:

\
libfoo == $
\

A constraint that contains \c{$} is called incomplete. This mechanism is
primarily useful when developing related packages that should track each
other's versions exactly or closely.

In comparison operators and ranges the \c{$} value is replaced with the
dependent version ignoring the revision. For shortcut operators, the dependent
version must be a standard version and the following additional processing is
applied depending on whether the version is a release, final pre-release, or a
snapshot pre-release.

\ol|

\li|For a release we set the min version patch to zero. For \c{^} we also set
the minor version to zero, unless the major version is zero (reduces to
\c{~}). The max version is set according to the standard shortcut logic. For
example, \c{~$} is completed as follows:

\
1.2.0 -> [1.2.0 1.3.0-)
1.2.1 -> [1.2.0 1.3.0-)
1.2.2 -> [1.2.0 1.3.0-)
\

And \c{^$} is completed as follows:

\
1.0.0 -> [1.0.0 2.0.0-)
1.1.1 -> [1.0.0 2.0.0-)
\

|

\li|For a final pre-release the key observation is that if the patch
component for \c{~} or minor and patch components for \c{^} are not zero, then
that means there has been a compatible release and we treat this case the same
as release, ignoring the pre-release part. If, however, it/they are zero, then
that means there may yet be no final release and we have to start from the
first alpha. For example, for the \c{~$} case:

\
1.2.0-a.1 -> [1.2.0-a.1 1.3.0-)
1.2.0-b.2 -> [1.2.0-a.1 1.3.0-)
1.2.1-a.1 -> [1.2.0     1.3.0-)
1.2.2-b.2 -> [1.2.0     1.3.0-)
\

And for the \c{^$} case:

\
1.0.0-a.1 -> [1.0.0-a.1 2.0.0-)
1.0.0-b.2 -> [1.0.0-a.1 2.0.0-)
1.0.1-a.1 -> [1.0.0     2.0.0-)
1.1.0-b.2 -> [1.0.0     2.0.0-)
\

|

\li|For a snapshot pre-release we distinguish two cases: a patch snapshot
(the patch component is not zero) and a major/minor snapshot (the patch
component is zero). For the patch snapshot case we assume that it is (most
likely) developed independently of the dependency and we treat it the same as
the final pre-release case. For example, if the dependent version is
\c{1.2.1-a.0.nnn}, the dependency could be \c{1.2.0} or \c{1.2.2} (or
somewhere in-between).

For the major/minor snapshot we assume that all the packages are developed in
the lockstep and have the same \c{X.Y.0} version. In this case we make the
range start from the earliest possible version in this \"snapshot series\" and
end before the final pre-release. For example (in this case \c{~} and \c{^}
are treated the same):

\
1.2.0-a.0.nnn -> [1.2.0-a.0.1 1.2.0-a.1)
2.0.0-b.2.nnn -> [2.0.0-b.2.1 2.0.0-b.3)
\

||


\h1#package-skeleton|Package Build System Skeleton|

There are situations where \c{bpkg} may need to evaluate \c{buildfile}
expressions and fragments before committing to a particular version of the
package and therefore before actually unpacking anything. For example,
\c{bpkg} may need to evaluate a condition in the conditional dependency or it
may need to negotiate a configuration among several dependents of a package
which requires it to know this package's configuration variable types and
default values.

To solve this chicken and egg kind of problem, \c{bpkg} includes a minimal
subset of the build system files along with the package's standard metadata
(name, version, etc) into the repository metadata
(\l{#manifest-package-list-pkg \c{packages.manifest}}). This subset is called
the package build system skeleton, or just package skeleton for short, and
includes the \c{build/bootstrap.build} and \c{build/root.build} files (or
their alternative naming scheme variants) as well as any files that may be
sourced by \c{root.build}.

The inclusion of \c{build/bootstrap.build} and \c{build/root.build} (if
present) as well as any \c{build/config/*.build} (or their alternative naming
scheme variants) is automatic. However, if \c{root.build} sources any files
other than \c{build/config/*.build}, then they must be specified explicitly in
the package manifest using the \l{#manifest-package-build-file \c{build-file}}
value.

Inside these buildfiles the skeleton load can be distinguished from normal
load by examining the \c{build.mode} variable, which is set to \c{skeleton}
during the skeleton load. In particular, this variable must be used to omit
loading of build system modules that are neither built-in nor standard
pre-installed and which are therefore listed as package dependencies. Such
modules are not yet available during the skeleton load. For example:

\
# root.build

using cxx       # Ok, built-in module.
using autoconf  # Ok, standard pre-installed module.

if ($build.mode != 'skeleton')
  using hello
\

The \c{build.mode} variable can also be used to omit parts of \c{root.build}
that are expensive to evaluate and which are only necessary during the actual
build. Here is a realistic example:

\
# root.build

...

using cxx

# Determine the GCC plugin directory. But omit doing it during the
# skeleton load.
#
if ($build.mode != 'skeleton')
{
  if ($cxx.id != 'gcc')
    fail 'this project can only be built with GCC'

  # If plugin support is disabled, then -print-file-name will print
  # the name we have passed (the real plugin directory will always
  # be absolute).
  #
  plugin_dir = [dir_path] \
    $process.run($cxx.path -print-file-name=plugin)

  if (\"$plugin_dir\" == plugin)
    fail \"$recall($cxx.path) does not support plugins\"

  plugin_dir = $normalize($plugin_dir)
}
\


\h1#dep-config-negotiation|Dependency Configuration Negotiation|

In \c{bpkg}, a dependent package may specify a desired configuration for a
dependency package. Because there could be multiple such dependents, \c{bpkg}
needs to come up with a dependency configuration that is acceptable to all of
them. This process is called the dependency configuration negotiation.

The desired dependency configuration is specified as part of the
\l{#manifest-package-depends \c{depends}} manifest value and can be expressed
as either a single \c{require} clause or as a pair of \c{prefer}/\c{accept}
clauses.

The \c{require} clause is essentially a shortcut for specifying the
\c{prefer}/\c{accept} clauses where the \c{accept} condition simply verifies
all the variable values assigned in the \c{prefer} clause. It is, however,
further restricted to the common case of only setting \c{bool} variables and
only to \c{true} to allow additional optimizations during the configuration
negotiation. The remainder of this section only deals with the general
\c{prefer}/\c{accept} semantics.

While the exact format of \c{prefer}/\c{accept} is described as part of the
\l{#manifest-package-depends \c{depends}} manifest value, for this section it
is sufficient to know that the \c{prefer} clause is an arbitrary \c{buildfile}
fragment that is expected to set one or more dependency configuration
variables to the values preferred by this dependent while the \c{accept}
clause is a \c{buildfile} eval context expression that should evaluate to
\c{true} or \c{false} indicating whether the dependency configuration values
it is evaluated on are acceptable to this dependent. For example:

\
libfoo ^1.0.0
{
  # We prefer the cache but can work without it.
  # We need the buffer of at least 4KB.
  #
  prefer
  {
    config.libfoo.cache = true

    config.libfoo.buffer = ($config.libfoo.buffer < 4096 \
                            ? 4096                       \
                            : $config.libfoo.buffer)
  }

  accept ($config.libfoo.buffer >= 4096)
}
\

The configuration negotiation algorithm can be summarized as cooperative
refinement. Specifically, whenever a \c{prefer} clause of a dependent changes
any configuration value, all other dependents' \c{prefer} clauses are
re-evaluated. This process continues until there are no more changes
(success), one of the \c{accept} clauses returned \c{false} (failure), or the
process starts \"yo-yo'ing\" between two or more configurations (failure).

The dependents are expected to cooperate by not overriding \"better\" values
that were set by other dependents. Consider the following two \c{prefer}
clauses:

\
prefer
{
  config.libfoo.buffer = 4096
}

prefer
{
  config.libfoo.buffer = ($config.libfoo.buffer < 4096 \
                          ? 4096                       \
                          : $config.libfoo.buffer)
}
\

The first version is non-cooperative and should only be used if this dependent
requires the buffer to be exactly 4KB. The second version is cooperative: it
will increase the buffer to the minimum required by this dependent but will
respect values above 4KB.

One case where we don't need to worry about this is when setting the
configuration variable to the \"best\" possible value. One common example of
this is setting a \c{bool} configuration to \c{true}.

With a few exceptions discussed below, a dependent must always re-set the
configuration variable, even if to the better value. For example, the
following is an incorrect attempt at the above cooperative \c{prefer} clause:

\
prefer
{
  if ($config.libfoo.buffer < 4096)  # Incorrect.
    config.libfoo.buffer = 4096
}
\

The problem with the above attempt is that the default value could be greater
than 4KB, in which case \c{bpkg} will have no idea that there is a dependent
relying on this configuration value.

Before each \c{prefer} clause re-evaluation, variables that were first set to
their current values by this dependent are reset to their defaults thus
allowing the dependent to change its mind, for instance, in response to other
configuration changes. For example:

\
# While we have no preference about the cache, if enabled/disabled,
# we need a bigger/smaller buffer.
#
prefer
{
  min_buffer = ($config.libfoo.cache ? 8192 : 4096)

  config.libfoo.buffer = ($config.libfoo.buffer < $min_buffer \
                          ? $min_buffer                       \
                          : $config.libfoo.buffer)
}

accept ($config.libfoo.buffer >= ($config.libfoo.cache ? 8192 : 4096))
\

The interesting case to consider in the above example is when
\c{config.libfoo.cache} changes from \c{true} to \c{false}: without the reset
to defaults semantics the \c{prefer} clause would have kept the buffer at 8KB
(since it's greater than the 4KB minimum).

\N|Currently \c{accept} is always evaluated after \c{prefer} and temporary
variables (like \c{min_buffer} in the above example) set in \c{prefer} are
visible in \c{accept}. But it's best not to rely on this in case it changes
in the future. For example, we may try harder to resolve the \"yo-yo'ing\"
case mentioned above by checking if one of the alternating configurations
are acceptable to everyone without re-evaluation.

This is also the reason why we need a separate \c{accept} in the first place.
Plus, it allows for more advanced configuration techniques where we may need
to have an acceptance criteria but no preferences.|

Configuration variables that are set by the dependent in the \c{prefer} clause
are visible in the subsequent clauses as well as in the subsequent \c{depends}
values of this dependent. Configuration variables that are not set, however,
are only visible until the immediately following \c{reflect} clause. For
example, in the above listing, \c{config.libfoo.cache} would still be visible
in the \c{reflect} clause if it were to follow \c{accept} but no further. As a
result, if we need to make decisions based on configuration variables that we
have no preference about, they need to be saved in the \c{reflect} clause. For
example:

\
depends:
\\
libfoo ^1.0.0
{
  # We have no preference about the cache but need to
  # observe its value.
  #
  prefer
  {
  }

  accept (true)

  reflect
  {
    config.hello.libfoo_cache = $config.libfoo.cache
  }
}
\\

depends: libbar ^1.0.0 ? ($config.hello.libfoo_cache)
\

It is possible to determine the origin of the configuration variable value
using the \c{$config.origin()} function. It returns either \c{undefined} if
the variable is undefined (only possible if it has no default value),
\c{default} if the variable has the default value from the \c{config}
directive in \c{root.build}, \c{buildfile} if the value is from a
\c{buildfile}, normally \c{config.build}, or \c{override} if the value is a
command line override (that is, user configuration). For example, this is how
we could use it if we only wanted to change the default value (notice that
it's the variable's name and not its \c{$}-expansion that we pass to
\c{$config.origin()}):

\
prefer
{
  config.libfoo.buffer = (                            \
    $config.origin(config.libfoo.buffer) == 'default' \
    ? 4096                                            \
    : $config.libfoo.buffer)
}
\

The following sub-sections discuss a number of more advanced configuration
techniques that are based on the functionality described in this section.


\h#dep-config-prefer-x-accept-xy|Prefer X but Accept X or Y|

Consider a configuration variable that is a choice between several mutually
exclusive values, for example, user interface backends that could be, say,
\c{cli}, \c{gui}, or \c{none}. In such situations it's common to prefer one
value but being able to work with some subset of them. For example, we could
prefer \c{gui} but were also able to make do with \c{cli} but not with
\c{none}. Here is how we could express such a configuration:

\
libfoo ^1.0.0
{
  # We prefer `gui`, can also work with `cli` but not `none`.
  #
  prefer
  {
    config.libfoo.ui = (                                         \
      $config.origin(config.libfoo.ui) == 'default' ||           \
      ($config.libfoo.ui != 'gui' && $config.libfoo.ui != 'cli') \
      ? 'gui'                                                    \
      : $config.libfoo.ui)
  }

  accept ($config.libfoo.ui == 'gui' || $config.libfoo.ui == 'cli')
}
\

\h#dep-config-use-if-enabled|Use If Enabled|

Sometimes we may want to use a feature if it is enabled by someone else but
not enable it ourselves. For example, the feature might be expensive and our
use of it tangential, but if it's enabled anyway, then we might as well take
advantage of it. Here is how we could express such a configuration:

\
libfoo ^1.0.0
{
  # Use config.libfoo.x only if enabled by someone else.
  #
  prefer
  {
  }

  accept (true)

  reflect
  {
    config.hello.libfoo_x = $config.libfoo.x
  }
}
\

\h#dep-config-disable-default|Disable If Enabled by Default|

Sometimes we may want to disable a feature that is enabled by default provided
that nobody else needs it. For example, the feature might be expensive and we
would prefer to avoid paying the cost if we are the only ones using this
dependency. Here is how we could express such a configuration:

\
libfoo ^1.0.0
{
  prefer
  {
    if ($config.origin(config.libfoo.x) == 'default')
      config.libfoo.x = false
  }

  accept (true)
}
\


\h1#manifests|Manifests|

This chapter describes the general manifest file format as well as the
concrete manifests used by \c{bpkg}.

Currently, three manifests are defined: package manifest, repository manifest,
and signature manifest. The former two manifests can also be combined into a
list of manifests to form the list of available packages and the description
of a repository, respectively.

\h#manifest-format|Manifest Format|

A manifest is a UTF-8 encoded text restricted to the Unicode graphic
characters, tabs (\c{\\t}), carriage returns (\c{\\r}), and line feeds
(\c{\\n}). It contains a list of name-value pairs in the form:

\
: 
\

For example:

\
name: libfoo
version: 1.2.3
\

\N|If a value needs to be able to contain other Unicode codepoints, they
should be escaped in a value-specific manner. For example, the backslash
(\c{\\}) escaping described below can be extended for this purpose.|

The name can contain any characters except \c{:} and whitespaces. Newline
terminates the pair unless escaped with \c{\\} (see below). Leading and
trailing whitespaces before and after name and value are ignored except in the
multi-line mode (see below).

If the first non-whitespace character on the line is \c{#}, then the rest
of the line is treated as a comment and ignored except if the preceding
newline was escaped or in the multi-line mode (see below). For example:

\
# This is a comment.
short: This is #not a comment
long: Also \
#not a comment
\

The first name-value pair in the manifest file should always have an empty
name. The value of this special pair is the manifest format version. The
version value shall use the default (that is, non-multi-line) mode and shall
not use any escape sequences. Currently it should be \c{1}, for example:

\
: 1
name: libfoo
version: 1.2.3
\

Any new name that is added without incrementing the version must be optional
so that it can be safely ignored by older implementations.

The special empty name pair can also be used to separate multiple
manifests. In this case the version may be omitted in the subsequent
manifests, for example:

\
: 1
name: libfoo
version: 1.2.3
:
name: libbar
version: 2.3.4
\

To disable treating of a newline as a name-value pair terminator we can escape
it with \c{\\}. Note that \c{\\} is only treated as an escape sequence when
followed by a newline and both are simply removed from the stream (as opposed
to being replaced with a space). To enter a literal \c{\\} at the end of the
value, use the \c{\\\\} sequence. For example:

\
description: Long text that doesn't fit into one line \
so it is continued on the next line.
\

\
windows-path: C:\foo\bar\\\\
\

Notice that in the final example only the last \c{\\} needs special handling
since it is the only one that is followed by a newline.

One may notice that in this newline escaping scheme a line consisting of just
\c{\\} followed by a newline has no use, except, perhaps, for visual
presentation of, arguably, dubious value. For example, this representation:

\
description: First line. \
\\
Second line.
\

Is semantically equivalent to:

\
description: First line. Second line.
\

As a result, such a sequence is \"overloaded\" to provide more useful
functionality in two ways: Firstly, if \c{:} after the name is followed on the
next line by just \c{\\} and a newline, then it signals the start of the
multi-line mode. In this mode all subsequent newlines and \c{#} are treated as
ordinary characters rather than value terminators or comments until a line
consisting of just \c{\\} and a newline (the multi-line mode terminator). For
example:

\
description:
\\
First paragraph.
#
Second paragraph.
\\
\

Expressed as a C-string, the value in the above example is:

\
\"First paragraph.\n#\nSecond paragraph.\"
\


\N|Originally, the multi-line mode was entered if \c{:} after the name were
immediately followed by \c{\\} and a newline but on the same line. While this
syntax is still recognized for backwards compatibility, it is deprecated and
will be discontinued in the future.|

Note that in the multi-line mode we can still use newline escaping to split
long lines, for example:

\
description:
\\
First paragraph that doesn't fit into one line \
so it is continued on the next line.
Second paragraph.
\\
\

And secondly, in the simple (that is, non-multi-line) mode, the sole \c{\\}
and newline sequence is overloaded to mean a newline. So the previous example
can also be represented like this:

\
description: First paragraph that doesn't fit into one \
line so it is continued on the next line.\
\\
Second paragraph.
\

Note that the multi-line mode can be used to capture a value with leading
and/or trailing whitespaces, for example:

\
description:
\\
  test

\\
\

The C-string representing this value is:

\
\"  test\n\"
\

EOF can be used instead of a newline to terminate both simple and multi-line
values. For example the following representation results in the same value as
in the previous example.

\
description:
\\
  test


\

By convention, names are all in lower case and multi-word names are separated
with \c{-}. Note that names are case-sensitive.

Also by convention, the following name suffixes are used to denote common
types of values:

\
-file
-url
-email
\

For example:

\
description: Inline description
description-file: README
package-url: http://www.example.com
package-email: john@example.com
\

Other common name suffixes (such as -feed) could be added later.

\N|Generally, unless there is a good reason not to, we keep values
lower-case (for example, \c{requires} values such as \c{c++11} or
\c{linux}). An example where we use upper/mixed case would be \c{license}; it
seems unlikely \c{gplv2} would be better than \c{GPLv2}.|

A number of name-value pairs described below allow for the value proper to be
optionally followed by \c{;} and a comment. Such comments serve as additional
documentation for the user and should be one or more full sentences, that is
start with a capital letter and end with a period. Note that unlike
\c{#}-style comments which are ignored, these comments are considered to be
part of the value. For example:

\
email: foo-users@example.com ; Public mailing list.
\

It is recommended that you keep comments short, single-sentence. Note that
non-comment semicolons in such values have to be escaped with a backslash, for
example:

\
url: http://git.example.com/?p=foo\;a=tree
\

The only other recognized escape sequence in such values is \c{\\\\}, which is
replaced with a single backslash. If a backslash is followed by any other
character, then it is treated literally.

If a value with a comment is multi-line, then \c{;} must appear on a separate
line, for example:

\
url:
\\
http://git.example.com/?p=foo;a=tree
;
Git repository tree.
\\
\

In this case, only lines that consist of a sole non-comment semicolon need
escaping, for example:

\
license:
\\
other: strange
\;
license
\\
\

The only other recognized escape sequence in such multi-line values is lines
consisting of two or more backslashes followed by a semicolon.

In the manifest specifications described below optional components are
enclosed in square brackets (\c{[]}). If the name is enclosed in \c{[]} then
the name-value pair is optional, otherwise \- required. For example:

\
name: 
license:  [; ]
[description]: 
\

In the above example \c{name} is required, \c{license} has an optional
component (comment), and \c{description} is optional.

In certain situations (for example, shell scripts) it can be easier to parse
the binary manifest representation. The binary representation does not include
comments and consists of a sequence of name-value pairs in the following form:

\
:\0
\

That is, the name and the value are separated by a colon and each pair
(including the last) is terminated with the \c{NUL} character. Note that there
can be no leading or trailing whitespace characters around the name and any
whitespaces after the colon and before the \c{NUL} terminator are part of the
value. Finally, the manifest format versions are always explicit (that is, not
empty) in binary manifest lists.


\h#manifest-package|Package Manifest|

The package manifest (the \c{manifest} file found in the package's root
directory) describes a \c{bpkg} package. The manifest synopsis is presented
next followed by the detailed description of each value in subsequent
sections.

The subset of the values up to and including \c{license} constitute the
package manifest header. Note that the header is a valid package manifest
since all the other values are optional. There is also no requirement for the
header values to appear first or to be in a specific order. In particular, in
a full package manifest they can be interleaved with non-header values.

\
name: 
version: 
[upstream-version]: 
[type]: 
[language]: 
[project]: 
[priority]:  [; ]
summary: 
license:  [; ]
\

\
[topics]: 
[keywords]: 
[description]: 
[description-file]:  [; ]
[description-type]: 
[package-description]: 
[package-description-file]:  [; ]
[package-description-type]: 
[changes]: 
[changes-file]:  [; ]
[changes-type]: 

[url]:  [; ]
[doc-url]:  [; ]
[src-url]:  [; ]
[package-url]:  [; ]

[email]:  [; ]
[package-email]:  [; ]
[build-email]:  [; ]
[build-warning-email]:  [; ]
[build-error-email]:  [; ]

[depends]: [*]  [; ]
[requires]: [*]  [; ]

[tests]: [*]  []
[examples]: [*]  []
[benchmarks]: [*]  []

[builds]:  [; ]
[build-include]: [/] [; ]
[build-exclude]: [/] [; ]

[*-build-config]:  [; ]

[*-builds]:  [; ]
[*-build-include]: [/] [; ]
[*-build-exclude]: [/] [; ]

[build-file]: 

[bootstrap-build]: 
[root-build]: 
[*-build]: 

[bootstrap-build2]: 
[root-build2]: 
[*-build2]: 

[*-name]:  [...]
[*-version]: 
[*-to-downstream-version]: 
\

\h2#manifest-package-name|\c{name}|

\
name: 
\

The package name. See \l{#package-name Package Name} for the package name
format description. Note that the name case is preserved for display, in file
names, etc.


\h2#manifest-package-version|\c{version}|

\
version: 
[upstream-version]: 
\

The package version. See \l{#package-version Package Version} for the version
format description. Note that the version case is preserved for display, in
file names, etc.

When packaging existing projects, sometimes you may want to deviate from the
upstream versioning scheme because, for example, it may not be representable
as a \c{bpkg} package version or simply be inconvenient to work with. In this
case you would need to come up with an upstream-to-downstream version mapping
and use the \c{upstream-version} value to preserve the original version for
information.


\h2#manifest-package-type-language|\c{type}, \c{language}|

\
[type]: 
[language]: 

 = [,]
 = [=impl]
\

The package type and programming language(s).

The currently recognized package types are \c{exe}, \c{lib}, and \c{other}. If
the type is not specified, then if the package name starts with \c{lib}, then
it is assumed to be \c{lib} and \c{exe} otherwise (see \l{#package-name
Package Name} for details). Other package types may be added in the future and
code that does not recognize a certain package type should treat it as
\c{other}. The type name can be followed by a comma-separated list of
sub-options. Currently, the only recognized sub-option is \c{binless} which
applies to the \c{lib} type indicating a header-only (or equivalent) library.
For example:

\
type: lib,binless
\

The package language must be in the lower case, for example, \c{c}, \c{c++},
\c{rust}, \c{bash}. If the language is not specified, then if the package name
has an extension (as in, for example, \c{libbutl.bash}; see \l{#package-name
Package Name} for details) the extension is assumed to name the package
language. Otherwise, \c{cc} (unspecified \c{c}-common language) is assumed. If
a package uses multiple languages, then multiple \c{language} values must be
specified. The languages which are only used in a library's implementation (as
opposed to also in its interface) should be marked as such. For example, for a
C library with C++ implementation:

\
type: lib
language: c
language: c++=impl
\

\N|If the use of a language, such as C++, also always implies the use of
another language, such as C, then such an implied language need not be
explicitly specified.|


\h2#manifest-package-project|\c{project}|

\
[project]: 
\

The project this package belongs to. The project name has the same
restrictions as the package name (see \l{#package-name Package Name} for
details) and its case is preserved for display, in directory names, etc. If
unspecified, then the project name is assumed to be the same as the package
name.

Projects are used to group related packages together in order to help with
organization and discovery in repositories. For example, packages \c{hello},
\c{libhello}, and \c{libhello2} could all belong to project \c{hello}. By
convention, projects of library packages are named without the \c{lib} prefix.


\h2#manifest-package-|\c{priority}|

\
[priority]:  [; ]

 = security | high | medium | low
\

The release priority (optional). As a guideline, use \c{security} for security
fixes, \c{high} for critical bug fixes, \c{medium} for important bug fixes,
and \c{low} for minor fixes and/or feature releases. If not specified, \c{low}
is assumed.


\h2#manifest-package-summary|\c{summary}|

\
summary: 
\

The short description of the package.


\h2#manifest-package-license|\c{license}|

\
license:  [; ]

 =  [, ]*
  = [:] 
   = other
\

The package license. The default license name scheme is
\l{https://spdx.org/licenses/ SPDX License Expression}. In its simplest form,
it is just an ID of the license under which this package is distributed. An
optional comment normally gives the full name of the license, for example:

\
license: MPL-2.0 ; Mozilla Public License 2.0
\

The following table lists the most commonly used free/open source software
licenses and their SPDX license IDs:

\
MIT                ; MIT License.

BSD-2-Clause       ; BSD 2-Clause \"Simplified\" License
BSD-3-Clause       ; BSD 3-Clause \"New\" or \"Revised\" License
BSD-4-Clause       ; BSD 4-Clause \"Original\" or \"Old\" License

GPL-2.0-only       ; GNU General Public License v2.0 only
GPL-2.0-or-later   ; GNU General Public License v2.0 or later
GPL-3.0-only       ; GNU General Public License v3.0 only
GPL-3.0-or-later   ; GNU General Public License v3.0 or later

LGPL-2.0-only      ; GNU Library General Public License v2 only
LGPL-2.0-or-later  ; GNU Library General Public License v2 or later
LGPL-2.1-only      ; GNU Lesser General Public License v2.1 only
LGPL-2.1-or-later  ; GNU Lesser General Public License v2.1 or later
LGPL-3.0-only      ; GNU Lesser General Public License v3.0 only
LGPL-3.0-or-later  ; GNU Lesser General Public License v3.0 or later

AGPL-3.0-only      ; GNU Affero General Public License v3.0 only
AGPL-3.0-or-later  ; GNU Affero General Public License v3.0 or later

Apache-1.0         ; Apache License 1.0
Apache-1.1         ; Apache License 1.1
Apache-2.0         ; Apache License 2.0

MPL-1.0            ; Mozilla Public License 1.0
MPL-1.1            ; Mozilla Public License 1.1
MPL-2.0            ; Mozilla Public License 2.0

BSL-1.0            ; Boost Software License 1.0

Unlicense          ; The Unlicense (public domain)
\

If the package is licensed under multiple licenses, then an SPDX license
expression can be used to specify this, for example:

\
license: Apache-2.0 OR MIT
license: MIT AND BSD-2-Clause
\

A custom license or extra conditions can be expressed either using the license
reference mechanism of the SPDX license expression or using the \c{other}
scheme (described below). For example:

\
license: LicenseRef-My-MIT-Like; Custom MIT-alike license
license: other: MIT with extra attribution requirements
\

The \c{other} license name scheme can be used to specify licenses that are not
defined by SPDX. The license names in this scheme are free form with
case-insensitive comparison. The following names in this scheme have
predefined meaning:

\
other: public domain    ; Released into the public domain
other: available source ; Not free/open source with public source code
other: proprietary      ; Not free/open source
other: TODO             ; License is not yet decided
\

\N|For new projects \l{https://unlicense.org The Unlicense} disclaimer with
the \c{Unlicense} SPDX ID is recommended over \c{other: public domain}.|

To support combining license names that use different schemes, the \c{license}
manifest value can contain a comma-separated list of license names. This list
has the \i{AND} semantics, that is, the user must comply with all the licenses
listed. To capture alternative licensing options (the \i{OR} semantics),
multiple \c{license} manifest values are used, for example:

\
license: GPL-2.0-only, other: available source
license: other: proprietary
\

For complex licensing situations it is recommended to add comments as an aid
to the user, for example:

\
license: LGPL-2.1-only AND MIT ; If linking with GNU TLS.
license: BSD-3-Clause          ; If linking with OpenSSL.
\

\N|For backwards compatibility with existing packages, the following
(deprecated) scheme-less values on the left are recognized as aliases for the
new values on the right:

\
BSD2               BSD-2-Clause
BSD3               BSD-3-Clause
BSD4               BSD-4-Clause
GPLv2              GPL-2.0-only
GPLv3              GPL-3.0-only
LGPLv2             LGPL-2.0-only
LGPLv2.1           LGPL-2.1-only
LGPLv3             LGPL-3.0-only
AGPLv3             AGPL-3.0-only
ASLv1              Apache-1.0
ASLv1.1            Apache-1.1
ASLv2              Apache-2.0
MPLv2              MPL-2.0

public domain      other: public domain
available source   other: available source
proprietary        other: proprietary
TODO               other: TODO
\

|


\h2#manifest-package-topics|\c{topics}|

\
[topics]: 

 =  [, ]*
\

The package topics (optional). The format is a comma-separated list of up to
five potentially multi-word concepts that describe this package. For example:

\
topics: xml parser, xml serializer
\


\h2#manifest-package-keywords|\c{keywords}|

\
[keywords]: 

 =  [ ]*
\

The package keywords (optional). The format is a space-separated list of up to
five words that describe this package. Note that the package and project names
as well as words from its summary are already considered to be keywords and
need not be repeated in this value.


\h2#manifest-package-description|\c{description}, \c{package-description}|

\
[description]: 
[description-file]:  [; ]
[description-type]: 
[package-description]: 
[package-description-file]:  [; ]
[package-description-type]: 
\

The detailed description of the project (\c{description}) and package
(\c{package-description}). If the package description is not specified, it is
assumed to be the same as the project description. It only makes sense to
specify the \c{package-description} value if the project and package are
maintained separately. A description can be provided either inline as a text
fragment or by referring to a file within a package (for example, \c{README}),
but not both. For \c{package-description-file} the recommended file name is
\c{PACKAGE-README} or \c{README-PACKAGE}.

In the web interface (\c{brep}) the description is displayed according to its
type. Currently, pre-formatted plain text, \l{https://github.github.com/gfm
GitHub-Flavored Markdown}, and \l{https://spec.commonmark.org/current
CommonMark} are supported with the following \c{*-type} values, respectively:

\
text/plain
text/markdown;variant=GFM
text/markdown;variant=CommonMark
\

If just \c{text/markdown} is specified, then the GitHub-Flavored Markdown
(which is a superset of CommonMark) is assumed.

If a description type is not explicitly specified and the description is
specified as \c{*-file}, then an attempt to derive the type from the file
extension is made. Specifically, the \cb{.md} and \cb{.markdown} extensions
are mapped to \c{text/markdown}, the \cb{.txt} and no extension are mapped to
\c{text/plain}, and all other extensions are treated as an unknown type,
similar to unknown \c{*-type} values. And if a description is not specified as
a file, \c{text/plain} is assumed.


\h2#manifest-package-changes|\c{changes}|

\
[changes]: 
[changes-file]:  [; ]
[changes-type]: 
\

The description of changes in the release.

\N|The tricky aspect is what happens if the upstream release stays the
same (and has, say, a \c{NEWS} file to which we point) but we need to make
another package release, for example, to apply a critical patch.|

Multiple \c{changes} values can be present which are all concatenated in the
order specified, that is, the first value is considered to be the most recent
(similar to \c{ChangeLog} and \c{NEWS} files). For example:

\
changes: 1.2.3-2: applied upstream patch for critical bug bar
changes: 1.2.3-1: applied upstream patch for critical bug foo
changes-file: NEWS
\

Or:

\
changes:
\\
1.2.3-2
  - applied upstream patch for critical bug bar
  - regenerated documentation

1.2.3-1
  - applied upstream patch for critical bug foo
\\
changes-file: NEWS
\

In the web interface (\c{brep}) the changes are displayed according to their
type, similar to the package description (see the
\l{#manifest-package-description \c{description}} value for details). If
the changes type is not explicitly specified, then the types deduced for
individual \c{changes} values must all be the same.


\h2#manifest-package-url|\c{url}|

\
[url]:  [; ]
\

The project home page URL.


\h2#manifest-package-doc-url|\c{doc-url}|

\
[doc-url]:  [; ]
\

The project documentation URL.


\h2#manifest-package-src-url|\c{src-url}|

\
[src-url]:  [; ]
\

The project source repository URL.


\h2#manifest-package-package-url|\c{package-url}|

\
[package-url]:  [; ]
\

The package home page URL. If not specified, then assumed to be the same as
\c{url}. It only makes sense to specify this value if the project and
package are maintained separately.


\h2#manifest-package-email|\c{email}|

\
[email]:  [; ]
\

The project email address. For example, a support mailing list.


\h2#manifest-package-package-email|\c{package-email}|

\
[package-email]:  [; ]
\

The package email address. If not specified, then assumed to be the same as
\c{email}. It only makes sense to specify this value if the project and
package are maintained separately.


\h2#manifest-package-build-email|\c{build-email}|

\
[build-email]:  [; ]
\

The build notification email address. It is used to send build result
notifications by automated build bots. If unspecified, then no build result
notifications for this package are sent by email.

\N|For backwards compatibility with existing packages, if it is specified but
empty, then this is the same as unspecified.

|


\h2#manifest-package-warning-email|\c{build-warning-email}|

\
[build-warning-email]:  [; ]
\

The build warning notification email address. Unlike \c{build-email}, only
build warning and error notifications are sent to this email.


\h2#manifest-package-error-email|\c{build-error-email}|

\
[build-error-email]:  [; ]
\

The build error notification email address. Unlike \c{build-email}, only
build error notifications are sent to this email.


\h2#manifest-package-depends|\c{depends}|

\
[depends]: [*]  [; ]
\

Single-line form:

\
 =  [ '|' ]*
  =  ['?' ] []
 =  | \
    '{'  []* '}' []
   =  []
  = '('  ')'
  =  '=' 
\

Multi-line form:

\
 =
  [
  '|'
  ]*

  =
  
  '{'
   [
     'enable' 
   ]

   [
     'require'
     '{'
        
     '}'

   ] | [

     'prefer'
     '{'
        
     '}'

     'accept' 
   ]

   [
     'reflect'
     '{'
        
     '}'
   ]
  '}'

  = '('  ')'
\

The dependency packages. The most common form of a dependency is a package
name followed by the optional version constraint. For example:

\
depends: libhello ^1.0.0
\

See \l{#package-version-constraint Package Version Constraint} for the format
and semantics of the version constraint. Instead of a concrete value, the
version in the constraint can also be specified in terms of the dependent
package's version (that is, its \l{#manifest-package-version \c{version}}
value) using the special \c{$} value. This mechanism is primarily useful when
developing related packages that should track each other's versions exactly or
closely. For example:

\
name: sqlite3
version: 3.18.2
depends: libsqlite3 == $
\

If multiple packages are specified within a single \c{depends} value, they
must be grouped with \c{{\}}. This can be useful if the packages share a
version constraint. The group constraint applies to all the packages in
the group that do not have their own constraint. For example:

\
depends: { libboost-any libboost-log libboost-uuid ~1.77.1 } ~1.77.0
\

If the \c{depends} value starts with \c{*}, then it is a \i{build-time}
dependency. Otherwise it is \i{run-time}. For example:

\
depends: * byacc >= 20210619
\

\N|Most of the build-time dependencies are expected to be tools such as code
generators, so you can think of \c{*} as the executable mark printed by
\c{ls}. An important difference between the two kinds of dependencies is that
in case of cross-compilation a build-time dependency must be built for the
host machine, not the target. Build system modules are also build-time
dependencies.|

Two special build-time dependency names are recognized and checked in an ad
hoc manner: \c{build2} (the \c{build2} build system) and \c{bpkg} (the
\c{build2} package manager). This allows us to specify the minimum required
build system and package manager versions, for example:

\
depends: * build2 >= 0.15.0
depends: * bpkg >= 0.15.0
\

\N|If you are developing or packaging a project that uses features from the
not yet released (staged) version of the \c{build2} toolchain, then you can
use the pre-release version in the constraint. For example:

\
depends: * build2 >= 0.16.0-
depends: * bpkg >= 0.16.0-
\

|

A dependency can be conditional, that is, it is only enabled if a certain
condition is met. For example:

\
depends: libposix-getopt ^1.0.0 ? ($cxx.target.class == 'windows')
\

The condition after \c{?} inside \c{()} is a \c{buildfile} eval context
expression that should evaluate to \c{true} or \c{false}, as if it were
specified in the \c{buildfile} \c{if} directive (see \l{b#intro-lang-expand
Expansion and Quoting} and \l{b#intro-if-else Conditions (\c{if-else})} for
details).

The condition expression is evaluated after loading the package build system
skeleton, that is, after loading its \c{root.build} (see \l{#package-skeleton
Package Build System Skeleton} for details). As a result, variable values set
by build system modules that are loaded in \c{root.build} as well as the
package's configuration (including previously reflected; see below) or
computed values can be referenced in dependency conditions. For example, given
the following \c{root.build}:

\
# root.build

...

using cxx

# MinGW ships POSIX .
#
need_getopt = ($cxx.target.class == 'windows' && \
               $cxx.target.system != 'mingw32')

config [bool] config.hello.regex ?= false
\

We could have the following conditional dependencies:

\
depends: libposix-getopt ^1.0.0 ? ($need_getopt) ; Windows && !MinGW.
depends: libposix-regex  ^1.0.0 ? ($config.hello.regex && \
                                   $cxx.target.class == 'windows')
\

The first \c{depends} value in the above example also shows the use of an
optional comment. It's a good idea to provide it if the condition is not
sufficiently self-explanatory.

A dependency can \"reflect\" configuration variables to the subsequent
\c{depends} values and to the package configuration. This can be used to
signal whether a conditional dependency is enabled or which dependency
alternative was selected (see below). The single-line form of \c{depends} can
only reflect one configuration variable. For example:

\
depends: libposix-regex ^1.0.0       \
  ? ($cxx.target.class == 'windows') \
  config.hello.external_regex=true
\

\
# root.build

...

using cxx

config [bool] config.hello.external_regex ?= false
\

\
# buildfile

libs =

if $config.hello.external_regex
  import libs += libposix-regex%lib{posix-regex}

exe{hello}: ... $libs
\

In the above example, if the \c{hello} package is built for Windows, then the
dependency on \c{libposix-regex} will be enabled and the package will be
configured with \c{config.hello.external_regex=true}. This is used in the
\c{buildfile} to decide whether to import \c{libposix-regex}. While in this
example it would have probably been easier to just duplicate the check for
Windows in the \c{buildfile} (or, better yet, factor this check to
\c{root.build} and share the result via a computed variable between
\c{manifest} and \c{buildfile}), the reflect mechanism is the only way to
communicate the selected dependency alternative (discussed next).

\N|An attempt to set a reflected configuration variable that is overridden by
the user is an error. In a sense, configuration variables that are used to
reflect information should be treated as the package's implementation details
if the package management is involved. If, however, the package is configured
without \c{bpkg}'s involvement, then these variables could reasonably be
provided as user configuration.

If you feel the need to allow a reflected configuration variable to also
potentially be supplied as user configuration, then it's probably a good sign
that you should turn things around: make the variable only user-configurable
and use the enable condition instead of reflect. Alternatively, you could try
to recognize and handle user overrides with the help of the
\c{$config.origin()} function discussed in \l{#dep-config-negotiation
Dependency Configuration Negotiation}.|

While multiple \c{depends} values are used to specify multiple packages with
the \i{AND} semantics, inside \c{depends} we can specify multiple packages (or
groups of packages) with the \i{OR} semantics, called dependency
alternatives. For example:

\
depends: libmysqlclient >= 5.0.3 | libmariadb ^10.2.2
\

When selecting an alternative, \c{bpkg} only considers packages that are
either already present in the build configuration or are selected as
dependencies by other packages, picking the first alternative with a
satisfactory version constraint and an acceptable configuration. As a result,
the order of alternatives expresses a preference. If, however, this does not
yield a suitable alternative, then \c{bpkg} fails asking the user to make the
selection.

For example, if the package with the above dependency is called \c{libhello}
and we build it in a configuration that already has both \c{libmysqlclient}
and \c{libmariadb}, then \c{bpkg} will select \c{libmysqlclient}, provided the
existing version satisfies the version constraint. If, however, there are no
existing packages in the build configuration and we attempt to build just
\c{libhello}, then \c{bpkg} will fail asking the user to pick one of the
alternatives. If we wanted to make \c{bpkg} select \c{libmariadb} we could
run:

\
$ bpkg build libhello ?libmariadb
\

\N|While \c{bpkg}'s refusal to automatically pick an alternative that would
require building a new package may at first seem unfriendly to the user,
practical experience shows that such extra user-friendliness would rarely
justify the potential confusion that it may cause.

Also note that it's not only the user that can pick a certain alternative but
also a dependent package. Continuing with the above example, if we had
\c{hello} that depended on \c{libhello} but only supported MariaDB (or
provided a configuration variable to explicitly select the database), then we
could have the following in its \c{manifest}:

\
depends: libmariadb       ; Select MariaDB in libhello.
depends: libhello ^1.0.0
\

|

Dependency alternatives can be combined with all the other features discussed
above: groups, conditional dependencies, and reflect. As mentioned earlier,
reflect is the only way to communicate the selection to subsequent \c{depends}
values and the package configuration. For example:

\
depends: libmysqlclient >= 5.0.3 config.hello.db='mysql'      | \
         libmariadb ^10.2.2 ? ($cxx.target.class != 'windows')  \
           config.hello.db='mariadb'

depends: libz ^1.2.1100 ? ($config.hello.db == 'mysql')
\

If an alternative is conditional and the condition evaluates to \c{false},
then this alternative is not considered. If all but one alternative are
disabled due to conditions, then this becomes an ordinary dependency. If all
the alternatives are disabled due to conditions, then the entire dependency
is disabled. For example:

\
depends: libmysqlclient >= 5.0.3 ? ($config.hello.db == 'mysql') | \
         libmariadb ^10.2.2      ? ($config.hello.db == 'mariadb')
\

While there is no need to use the dependency alternatives in the above example
(since the alternatives are mutually exclusive), it makes for good
documentation of intent.

Besides as a single line, the \c{depends} value can also be specified in a
multi-line form which, besides potentially better readability, provides
additional functionality. In the multi-line form, each dependency alternative
occupies a separate line and \c{|} can be specified either at the end of
the dependency alternative line or on a separate line. For example:

\
depends:
\\
libmysqlclient >= 5.0.3 ? ($config.hello.db == 'mysql') |
libmariadb ^10.2.2      ? ($config.hello.db == 'mariadb')
\\
\

A dependency alternative can be optionally followed by a block containing a
number of clauses. The \c{enable} clause is the alternative way to specify the
condition for a conditional dependency while the \c{reflect} clause is the
alternative way to specify the reflected configuration variable. The block may
also contain \c{#}-style comments, similar to \c{buildfile}. For example:

\
depends:
\\
libmysqlclient >= 5.0.3
{
  reflect
  {
    config.hello.db = 'mysql'
  }
}
|
libmariadb ^10.2.2
{
  # TODO: MariaDB support on Windows.
  #
  enable ($cxx.target.class != 'windows')

  reflect
  {
    config.hello.db = 'mariadb'
  }
}
\\
\

While the \c{enable} clause is essentially the same as its inline \c{?}
variant, the \c{reflect} clause is an arbitrary \c{buildfile} fragment that
can have more complex logic and assign multiple configuration variables. For
example:

\
libmariadb ^10.2.2
{
  reflect
  {
    if ($cxx.target.class == 'windows')
      config.hello.db = 'mariadb-windows'
    else
      config.hello.db = 'mariadb-posix'
  }
}
\

The multi-line form also allows us to express our preferences and requirements
for the dependency configuration. If all we need is to set one or more
\c{bool} configuration variables to \c{true} (which usually translates to
enabling one or more features), then we can use the \c{require} clause. For
example:

\
libmariadb ^10.2.2
{
  require
  {
    config.libmariadb.cache = true

    if ($cxx.target.class != 'windows')
      config.libmariadb.tls = true
  }
}
\

For more complex dependency configurations instead of \c{require} we can use
the \c{prefer} and \c{accept} clauses. The \c{prefer} clause can set
configuration variables of any type and to any value in order to express the
package's preferred configuration while the \c{accept} condition evaluates
whether any given configuration is acceptable. If used instead of \c{require},
both \c{prefer} and \c{accept} must be present. For example:

\
libmariadb ^10.2.2
{
  # We prefer the cache but can work without it.
  # We need the buffer of at least 4KB.
  #
  prefer
  {
    config.libmariadb.cache = true

    config.libmariadb.buffer = ($config.libmariadb.buffer < 4096 \
                                ? 4096                           \
                                : $config.libmariadb.buffer)
  }

  accept ($config.libmariadb.buffer >= 4096)
}
\

\N|The \c{require} clause is essentially a shortcut for specifying the
\c{prefer}/\c{accept} clauses where the \c{accept} condition simply verifies
all the variable values assigned in the \c{prefer} clause. It is, however,
further restricted to the common case of only setting \c{bool} variables and
only to \c{true} to allow additional optimizations during the configuration
negotiation.|

The \c{require} and \c{prefer} clauses are arbitrary \c{buildfile} fragments
similar to \c{reflect} while the \c{accept} clause is a \c{buildfile} eval
context expression that should evaluate to \c{true} or \c{false}, similar to
\c{enable}.

Given the \c{require} and \c{prefer}/\c{accept} clauses of all the dependents
of a particular dependency, \c{bpkg} tries to negotiate a configuration
acceptable to all of them as described in \l{#dep-config-negotiation
Dependency Configuration Negotiation}.

All the clauses are evaluated in the specified order, that is, \c{enable},
then \c{require} or \c{prefer}/\c{accept}, and finally \c{reflect}, with the
(negotiated, in case of \c{prefer}) configuration values set by preceding
clauses available for examination by the subsequent clauses in this
\c{depends} value as well as in all the subsequent ones. For example:

\
depends:
\\
libmariadb ^10.2.2
{
  prefer
  {
    config.libmariadb.cache = true

    config.libmariadb.buffer = ($config.libmariadb.buffer < 4096 \
                                ? 4096                           \
                                : $config.libmariadb.buffer)
  }

  accept ($config.libmariadb.buffer >= 4096)

  reflect
  {
    config.hello.buffer = $config.libmariadb.buffer
  }
}
\\

depends: liblru ^1.0.0 ? ($config.libmariadb.cache)
\

The above example also highlights the difference between the
\c{require}/\c{prefer} and \c{reflect} clauses that is easy to mix up: in
\c{require}/\c{prefer} we set the dependency's while in \c{reflect} we set the
dependent's configuration variables.


\h2#manifest-package-requires|\c{requires}|

\
[requires]: [*]  [; ]

 =  [ '|' ]*
  =  ['?' []] []
 = [] | \
    '{'  []* '}' []
  =  []
  = '('  ')'
  =  '=' 
\

The package requirements other than other packages. Such requirements are
normally checked in an ad hoc way during package configuration by its
\c{buildfiles} and the primary purpose of capturing them in the manifest is
for documentation.  However, there are some special requirements that are
recognized by the tooling (see below). For example:

\
requires: c++11
requires: linux | windows | macos
requires: libc++ ? ($macos) ; libc++ if using Clang on Mac OS.
\

The format of the \c{requires} value is similar to
\l{#manifest-package-depends \c{depends}} with the following differences. The
requirement name (with or without version constraint) can mean anything (but
must still be a valid package name). Only the \c{enable} and \c{reflect}
clauses are permitted. There is a simplified syntax with either the
requirement or enable condition or both being empty and where the comment
carries all the information (and is thus mandatory). For example:

\
requires: ; X11 libs.
requires: ? ($windows) ; Only 64-bit.
requires: ? ; Only 64-bit if on Windows.
requires: x86_64 ? ; Only if on Windows.
\

Note that \c{requires} can also be used to specify dependencies on system
libraries, that is, the ones not to be packaged. In this case it may make
sense to also specify the version constraint. For example:

\
requires: libx11 >= 1.7.2
\

To assist potential future automated processing, the following pre-defined
requirement names should be used for the common requirements:

\
c++98
c++03
c++11
c++14
c++17
c++20
c++23
\

\
posix
linux
macos
freebsd
openbsd
netbsd
windows
\

\
gcc[_X.Y.Z]  ; For example: gcc_6, gcc_4.9, gcc_5.0.0
clang[_X.Y]  ; For example: clang_6, clang_3.4, clang_3.4.1
msvc[_N.U]   ; For example: msvc_14, msvc_15.3
\

The following pre-defined requirement names are recognized by automated build
bots:

\
bootstrap
host
\

The \c{bootstrap} value should be used to mark build system modules that
require bootstrapping. The \c{host} value should be used to mark packages,
such source code generators, that are normally specified as build-time
dependencies by other packages and therefore should be built in a host
configuration. See the \l{bbot \c{bbot} documentation} for details.


\h2#manifest-package-tests-examples-benchmarks|\c{tests}, \c{examples}, \c{benchmarks}|

\
[tests]: [*]  []
[examples]: [*]  []
[benchmarks]: [*]  []
\

Separate tests, examples, and benchmarks packages. If the value starts with
\c{*}, then the primary package is a \i{build-time} dependency for the
specified package. Otherwise it is \i{run-time}. See the
\l{#manifest-package-depends \c{depends}} value for details on \i{build-time}
dependencies.

These packages are built and tested by automated build bots together with the
primary package (see the \l{bbot \c{bbot} documentation} for details). This,
in particular, implies that these packages must be available from the primary
package's repository or its complement repositories, recursively. The
recommended naming convention for these packages is the primary package name
followed by \c{-tests}, \c{-examples}, or \c{-benchmarks}, respectively. For
example:

\
name: hello
tests : hello-tests
examples: hello-examples
\

See \l{#package-version-constraint Package Version Constraint} for the format
and semantics of the optional version constraint. Instead of a concrete value,
it can also be specified in terms of the primary package's version (see the
\l{#manifest-package-depends \c{depends}} value for details), for example:

\
tests: hello-tests ~$
\

Note that normally the tests, etc., packages themselves do not have an
explicit dependency on the primary package (in a sense, the primary package
has a special dependency on them). They are also not built by automated build
bots separately from their primary package but may have their own build
constraints, for example, to be excluded from building on some platforms where
the primary package is still built, for example:

\
name: hello-tests
builds: -windows
\

\h2#manifest-package-builds|\c{builds}|

\
[builds]: [ ':' ] [] [; ]

 =  [ ]*
 =  [ ]*
 = ('+'|'-'|'&')['!']( | '('  ')')
\

The common package build target configurations. They specify the target
configuration classes the package should or should not be built for by
automated build bots, unless overridden by a package configuration-specific
value (see \l{#manifest-package-build-config \c{*-build-config}} for details).
For example:

\
builds: -windows
\

Build target configurations can belong to multiple classes with their names
and semantics varying between different build bot deployments. However, the
pre-defined \c{none}, \c{default}, \c{all}, \c{host}, and \c{build2} classes
are always provided. If no \c{builds} value is specified in the package
manifest, then the \c{default} class is assumed.

\N|A target configuration class can also derive from another class in which
case configurations that belong to the derived class are treated as also
belonging to the base class (or classes, recursively). See the Build
Configurations page of the build bot deployment for the list of available
target configurations and their classes.|

The \c{builds} value consists of an optional underlying class set
(\c{}) followed by a class set expression (\c{}). The
underlying set is a space-separated list of class names that define the set of
build target configurations to consider. If not specified, then all the
configurations belonging to the \c{default} class are assumed. The class set
expression can then be used to exclude certain configurations from this
initial set.

The class expression is a space-separated list of terms that are evaluated
from left to right. The first character of each term determines whether the
build target configuration that belong to its set are added to (\c{+}),
subtracted from (\c{-}), or intersected with (\c{&}) the current set. If the
second character in the term is \c{!}, then its set of configuration is
inverted against the underlying set. The term itself can be either the class
name or a parenthesized expression. Some examples (based on the
\l{https://ci.cppget.org/?build-configs cppget.org} deployment):

\
builds: none                          ; None.
builds: all                           ; All (suitable for libraries).
builds: all : &host                   ; All host (suitable for tools).
builds: default                       ; All default.
builds: default : &host               ; Default host.
builds: default legacy                ; All default and legacy.
builds: default legacy : &host        ; Default and legacy host.
builds: -windows                      ; Default except Windows.
builds: all : -windows                ; All except Windows.
builds: all : -mobile                 ; All except mobile.
builds: all : &gcc                    ; All with GCC only.
builds: all : &gcc-8+                 ; All with GCC 8 and up only.
builds: all : &gcc -optimized         ; All GCC without optimization.
builds: all : &gcc &( +linux +macos ) ; All GCC on Linux and Mac OS.
\

Notice that the colon and parentheses must be separated with spaces from both
preceding and following terms.

Multiple \c{builds} values are evaluated in the order specified and as if they
were all part of a single expression. Only the first value may specify the
underlying set. The main reason for having multiple values is to provide
individual reasons (as the \c{builds} value comments) for different parts of
the expression. For example:

\
builds: default experimental ; Only modern compilers are supported.
builds: -gcc                 ; GCC is not supported.
builds: -clang               ; Clang is not supported.
\

\
builds: default
builds: -( +macos &gcc)      ; Homebrew GCC is not supported.
\

\N|The \c{builds} value comments are used by the web interface (\c{brep}) to
display the reason for the build target configuration exclusion.|

After evaluating all the \c{builds} values, the final configuration set can be
further fine-tuned using the \l{#manifest-package-include-exclude
\c{build-{include, exclude\}}} patterns.


\h2#manifest-package-include-exclude|\c{build-{include, exclude\}}|

\
[build-include]: [/] [; ]
[build-exclude]: [/] [; ]
\

The common package build inclusions and exclusions. The \c{build-include} and
\c{build-exclude} values further reduce the configuration set produced by
evaluating the \l{#manifest-package-builds \c{builds}} values. The \i{config}
and \i{target} values are filesystem wildcard patterns which are matched
against the build target configuration names and target names (see the \l{bbot
\c{bbot} documentation} for details). In particular, the \c{*} wildcard
matches zero or more characters within the name component while the \c{**}
sequence matches across the components. Plus, wildcard-only pattern components
match absent name components. For example:

\
build-exclude: windows**    # matches windows_10-msvc_15
build-exclude: macos*-gcc** # matches macos_10.13-gcc_8.1-O3
build-exclude: linux-gcc*-* # matches linux-gcc_8.1 and linux-gcc_8.1-O3
\

The exclusion and inclusion patterns are applied in the order specified with
the first match determining whether the package will be built for this
configuration and target. If none of the patterns match (or none we
specified), then the package is built.

As an example, the following value will exclude 32-bit builds for the MSVC
14 compiler:

\
build-exclude: *-msvc_14**/i?86-** ; Linker crash.
\

As another example, the following pair of values will make sure that a package
is only built on Linux:

\
build-include: linux**
build-exclude: ** ; Only supported on Linux.
\

Note that the comment of the matching exclusion is used by the web interface
(\c{brep}) to display the reason for the build target configuration exclusion.


\h2#manifest-package-build-config|\c{*-build-config}|

\
[*-build-config]:  [; ]

 = [[[+|-]:](