aboutsummaryrefslogtreecommitdiff
diff options
context:
space:
mode:
-rw-r--r--doc/manual.cli605
1 files changed, 368 insertions, 237 deletions
diff --git a/doc/manual.cli b/doc/manual.cli
index c565278..aa0f92d 100644
--- a/doc/manual.cli
+++ b/doc/manual.cli
@@ -155,388 +155,519 @@ windows-vc_14-32-debug
linux-gcc_6-cross-arm-eabi
\
-\h#arch-machine-header-manifest|Machine Header Manifest|
+\h#arch-machine-header|Machine Header Manifest|
-\
-SYNOPSIS
+@@ TODO: need ref to general manifest overview in bpkg, or, better yet,
+move it to libbutl and ref to that from both places.
-id: <machine-id>
-name: <machine-name>
+The build machine header manifest contains basic information about a build
+machine on the build host. A list of machine header manifests is sent by
+\c{bbot} agents to controllers. The manifest synopsis is presented next
+followed by the detailed description of each value in subsequent sections.
+
+\
+id: <machine-id>
+name: <machine-name>
summary: <string>
\
-The build machine header manifest contains basic information about a build
-machine on the build host. A list of machine header manifests is sent by
-\c{bbot} agents to controllers.
+For example:
-\dl|
+\
+id: windows_10-msvc_14-1.3
+name: windows_10-msvc_14
+summary: Windows 10 build 1607 with VC 14 update 3
+\
-\li|\n\c{id: <machine-id>}\n
+\h2#arch-machine-header-id|\c{id}|
- The \i{machine-id} uniquely identifies a machine version/revision/build.
- For virtual machines this can be the disk image checksum. For a container
- this can be UUID that is re-generated every time a container filesystem
- is altered.|
+\
+id: <machine-id>
+\
-\li|\n\c{name: <machine-name>}\n
+The uniquely machine version/revision/build identifies. For virtual machines
+this can be the disk image checksum. For a container this can be UUID that is
+re-generated every time a container filesystem is altered.
- The machine name as described above.|
-\li|\n\c{summary: <string>}\n
+\h2#arch-machine-header-name|\c{name}|
- A one-line description of the machine. For example:
+\
+name: <machine-name>
+\
- \
- id: windows_10-msvc_14-1.3
- name: windows_10-msvc_14
- summary: Windows 10 build 1607 with VC 14 update 3
- \
+The machine name.
-||
-\h#arch-machine-manifest|Machine Manifest|
+\h2#arch-machine-header-summary|\c{summary}|
\
-SYNOPSIS
-
-id: <machine-id>
-name: <machine-name>
summary: <string>
-
-type: <machine-type>
-mac: <macaddr>
-options: <machine-options>
\
+The one-line description of the machine.
+
+
+\h#arch-machine|Machine Manifest|
+
The build machine manifest contains the complete description of a build
machine on the build host (see the Build OS documentation for their origin and
-location). The machine manifest starts with the machine manifest header. All
-the header values must appear before any non-header values.
+location). The machine manifest starts with the machine manifest header with
+all the header values appearing before any non-header values. The non-header
+part of manifest synopsis is presented next followed by the detailed
+description of each value in subsequent sections.
-\dl|
+\
+type: kvm|nspawn
+[mac]: <addr>
+[options]: <machine-options>
+\
-\li|\n\c{type: <machine-type>}\n
- The machine type. Valid values are \c{kvm} (QEMU/KVM virtual machine) and
- \c{nspawn} (\c{systemd-nspawn} container).|
+\h2#arch-machine-type|\c{type}|
-\li|\n\c{mac: <macaddr>}\n
+\
+type: kvm|nspawn
+\
- Optional fixed MAC address for the machine in the hexadecimal,
- comma-separated format. For example:
+The machine type. Valid values are \c{kvm} (QEMU/KVM virtual machine) and
+\c{nspawn} (\c{systemd-nspawn} container).
- \
- mac: de:ad:be:ef:de:ad
- \
- If it is not specified, then a random address is generated on the first
- machine bootstrap which is then reused for each build/re-bootstrap. Note
- that it you specify a fixed address, then the machine can only be used by a
- single \c{bbot} agent.|
+\h2#arch-machine-mac|\c{mac}|
+\
+[mac]: <addr>
+\
-\li|\n\c{options: <machine-options>}\n
+The fixed MAC address for the machine. Must be in the hexadecimal,
+comma-separated format. For example:
- Optional list of machine options. The exact semantics is machine
- type-dependent (see below). A single level of quotes (either single or
- double) is removed in each option before being passed on. Options can be
- separated with spaces or newlines.
+\
+mac: de:ad:be:ef:de:ad
+\
- For \c{kvm} machines, if this value is present, then it replaces the
- default network and disk configuration when starting the QEMU/KVM
- hypervisor The options are pre-processed by replacing the question
- mark in \c{ifname=?} and \c{mac=?} strings with the network interface
- and MAC address, respectively.||
+If it is not specified, then a random address is generated on the first
+machine bootstrap which is then reused for each build/re-bootstrap. Note that
+if you specify a fixed address, then the machine can only be used by a single
+\c{bbot} agent.
-\h#arch-task-manifest|Task Manifest|
+\h2#arch-machine-options|\c{options}|
\
-SYNOPSIS
+[options]: <machine-options>
+\
-name: <package-name>
-version: <package-version>
-#location: <package-url>
-repository: <repository-url>
-trust: <repository-fp>
+The list of machine options. The exact semantics is machine type-dependent
+(see below). A single level of quotes (either single or double) is removed in
+each option before being passed on. Options can be separated with spaces or
+newlines.
-machine: <machine-name>
-target: <target-triplet>
-config: <config-vars>
-warning-regex: <warning-regexes>
-\
+For \c{kvm} machines, if this value is present, then it replaces the default
+network and disk configuration when starting the QEMU/KVM hypervisor. The
+options are pre-processed by replacing the question mark in \c{ifname=?} and
+\c{mac=?} strings with the network interface and MAC address, respectively.
+
+
+\h#arch-task|Task Manifest|
The task manifest describes a build task. It consists of two groups of values.
The first group defines the package to build. The second group defines the
-build configuration to use for building the package.
+build configuration to use for building the package. The manifest synopsis is
+presented next followed by the detailed description of each value in
+subsequent sections.
-\dl|
+\
+name: <package-name>
+version: <package-version>
+#location: <package-url>
+repository: <repository-url>
+[trust]: <repository-fp>
-\li|\n\c{name: <package-name>}\n
+machine: <machine-name>
+target: <target-triplet>
+[config]: <config-vars>
+[warning-regex]: <warning-regex>
+\
- Package name to test.|
-\li|\n\c{version: <package-version>}\n
+\h2#arch-task-name|\c{name}|
- Package version to test.|
+\
+name: <package-name>
+\
-\li|\n\c{repository: <repository-url>}\n
+The package name to build.
- The \c{bpkg} repository that contains the package and its dependencies.|
-\li|\n\c{trust: <repository-fp>}\n
+\h2#arch-task-version|\c{version}|
- The SHA256 repository certificate fingerprint to trust (see the \c{bpkg}
- \c{--trust} option for details). This value may be specified multiple times
- to establish the authenticity of multiple certificates. If the special
- \c{yes} value is specified, then all repositories will be trusted without
- authentication (see the \c{bpkg} \c{--trust-yes} option).
+\
+version: <package-version>
+\
- Note that while the controller may return a task with \c{trust} values,
- whether they will be used is up to the agent's configuration. For example,
- some agents may only trust their internally-specified fingerprints to
- prevent the \"man in the middle\" type of attacks.|
+The package version to build.
-\li|\n\c{machine: <machine-name>}\n
- The name of the build machine to use.|
+\h2#arch-task-repository|\c{repository}|
-\li|\n\c{target: <target-triplet>}\n
+\
+repository: <repository-url>
+\
- The target triplet to build for.
+The \c{bpkg} repository that contains the package and its dependencies.
- Compared to the autotools terminology, the \c{machine} value corresponds
- to \c{--build} (the machine we are building on) and \c{target} \- to
- \c{--host} (the machine we are building for). While we use essentially
- the same \i{target triplet} format as autotools for \c{target}, it is
- not flexible enough for \c{machine}.|
-\li|\n\c{config: <config-vars>}\n
+\h2#arch-task-trust|\c{trust}|
+
+\
+[trust]: <repository-fp>
+\
+
+The SHA256 repository certificate fingerprint to trust (see the \c{bpkg}
+\c{--trust} option for details). This value may be specified multiple times to
+establish the authenticity of multiple certificates. If the special \c{yes}
+value is specified, then all repositories will be trusted without
+authentication (see the \c{bpkg} \c{--trust-yes} option).
+
+Note that while the controller may return a task with \c{trust} values,
+whether they will be used is up to the agent's configuration. For example,
+some agents may only trust their internally-specified fingerprints to prevent
+the \"man in the middle\" attacks.
- Additional build system configuration variables.
- A single level of quotes (either single or double) is removed in each
- variable before being passed to \c{bpkg}. For example, the following value:
+\h2#arch-task-machine|\c{machine}|
- \
- config: config.cc.coptions=\"-O3 -stdlib='libc++'\"
- \
+\
+machine: <machine-name>
+\
- Will be passed to \c{bpkg} as the following (single) argument:
+The name of the build machine to use.
- \
- config.cc.coptions=-O3 -stdlib='libc++'
- \
- Variables can be separated with spaces or newlines.|
+\h2#arch-task-target|\c{target}|
-\li|\n\c{warning-regex: <warning-regexes>}\n
+\
+target: <target-triplet>
+\
- Additional regular expressions that should be used to detect warnings in
- the logs. Note that only the first 512 bytes of each log line is considered.
+The target to build for.
- A single level of quotes (either single or double) is removed in each
- expression before being used for search. For example, the following value:
+Compared to the autotools terminology, the \c{machine} value corresponds to
+\c{--build} (the machine we are building on) and \c{target} \- to \c{--host}
+(the machine we are building for). While we use essentially the same \i{target
+triplet} format as autotools for \c{target}, it is not flexible enough for
+\c{machine}.
- \
- warning-regex: \"warning C4\d{3}: \"
- \
- Will be treated as the following (single) regular expression (with a
- trailing space):
+\h2#arch-task-config|\c{config}|
+
+\
+[config]: <config-vars>
+\
+
+The additional build system configuration variables. A single level of quotes
+(either single or double) is removed in each variable before being passed to
+\c{bpkg}. For example, the following value:
+
+\
+config: config.cc.coptions=\"-O3 -stdlib='libc++'\"
+\
+
+Will be passed to \c{bpkg} as the following (single) argument:
+
+\
+config.cc.coptions=-O3 -stdlib='libc++'
+\
+
+Variables can be separated with spaces or newlines.
+
+\h2#arch-task-warning-regex|\c{warning-regex}|
+
+\
+[warning-regex]: <warning-regex>
+\
- \
- warning C4\d{3}:
- \
+Additional regular expressions that should be used to detect warnings in the
+build logs. Note that only the first 512 bytes of each log line is considered.
- Expressions can be separated with spaces or newlines. They will be added to
- the following default list of regular expressions that detect the \c{build2}
- toolchain warnings:
+A single level of quotes (either single or double) is removed in each
+expression before being used for search. For example, the following value:
- \
- ^warning:
- ^.+: warning:
- \
+\
+warning-regex: \"warning C4\d{3}: \"
+\
- Note that this built-in list also covers GCC and Clang warnings (for the
- English locale).||
+Will be treated as the following (single) regular expression (with a trailing
+space):
+\
+warning C4\d{3}:
+\
-\h#arch-result-manifest|Result Manifest|
+Expressions can be separated with spaces or newlines. They will be added to
+the following default list of regular expressions that detect the \c{build2}
+toolchain warnings:
\
-SYNOPSIS
+^warning:
+^.+: warning:
+\
-name: <package-name>
-version: <package-version>
+Note that this built-in list also covers GCC and Clang warnings (for the
+English locale).
-status: <status>
-configure-status: <status>
-update-status: <status>
-test-status: <status>
-install-status: <status>
-test-installed-status: <status>
-uninstall-status: <status>
-configure-log: <text>
-update-log: <text>
-test-log: <text>
-install-log: <text>
-test-installed-log: <text>
-uninstall-log: <text>
+\h#arch-result|Result Manifest|
+
+The result manifest describes a build result. The manifest synopsis is
+presented next followed by the detailed description of each value in
+subsequent sections.
+
\
+name: <package-name>
+version: <package-version>
-The result manifest describes a build result.
+status: <status>
+[configure-status]: <status>
+[update-status]: <status>
+[test-status]: <status>
+[install-status]: <status>
+[test-installed-status]: <status>
+[uninstall-status]: <status>
-\dl|
+[configure-log]: <text>
+[update-log]: <text>
+[test-log]: <text>
+[install-log]: <text>
+[test-installed-log]: <text>
+[uninstall-log]: <text>
+\
-\li|\n\c{name: <package-name>}\n
- Package name from the task manifest.|
+\h2#arch-result-name|\c{name}|
-\li|\n\c{version: <package-version>}\n
+\
+name: <package-name>
+\
+
+The package name from the task manifest.
+
+
+\h2#arch-result-version|\c{version}|
- Package version from the task manifest.|
+\
+version: <package-version>
+\
-\li|\n\c{status: <status>}\n
+The package version from the task manifest.
- An overall (cumulative) build result status. Valid values are:
- \
- success # All operations completed successfully.
- warning # One or more operations completed with warnings.
- error # One or more operations completed with errors.
- abort # One or more operations were aborted.
- abnormal # One or more operations terminated abnormally.
- \
+\h2#arch-result-status|\c{status}|
+
+\
+status: <status>
+\
- The \c{abort} status indicates that the operation has been aborted by
- \c{bbot}, for example, because it was consuming too many resources and/or
- was taking too long. Note that a task can be aborted both by the \c{bbot}
- worker as well as the agent. In the later case the whole machine is shut
- down and no operation-specific status or logs will be included (@@ Maybe
- we should just include 'log:' with commands that start VM, for
- completeness?).
+The overall (cumulative) build result status. Valid values are:
- The \c{abnormal} status indicates that the operation has terminated
- abnormally, for example, due to the package manager or build system crash.
+\
+success # All operations completed successfully.
+warning # One or more operations completed with warnings.
+error # One or more operations completed with errors.
+abort # One or more operations were aborted.
+abnormal # One or more operations terminated abnormally.
+\
- Note that the overall \c{status} value should appear before any
- per-operation \c{*-status} values.|
+The \c{abort} status indicates that the operation has been aborted by
+\c{bbot}, for example, because it was consuming too many resources and/or was
+taking too long. Note that a task can be aborted both by the \c{bbot} worker
+as well as the agent. In the later case the whole machine is shut down and no
+operation-specific status or logs will be included (@@ Maybe we should just
+include 'log:' with commands that start VM, for completeness?).
-\li|\n\c{*-status: <status>}\n
+The \c{abnormal} status indicates that the operation has terminated
+abnormally, for example, due to the package manager or build system crash.
- A per-operation result status. Note that the \c{*-status} values should
- appear in the same order as the corresponding operations were performed
- and for each \c{*-status} there should be a corresponding \c{*-log}.|
+Note that the overall \c{status} value should appear before any per-operation
+\c{*-status} values.
-\li|\n\c{*-log: <text>}\n
- A per-operation result log. Note that the \c{*-log} values should appear
- last and in the same order as the corresponding \c{*-status} values.||
+\h2#arch-result-x-status|\c{*-status}|
+\
+[*-status]: <status>
+\
-\h#arch-task-req-manifest|Task Request Manifest|
+The per-operation result status. Note that the \c{*-status} values should
+appear in the same order as the corresponding operations were performed
+and for each \c{*-status} there should be the corresponding \c{*-log}
+value. Currently supported operation names:
\
-SYNOPSIS
+configure
+update
+test
+install
+test-installed
+uninstall
+\
+
+
+\h2#arch-result-x-log|\c{*-log}|
-agent: <agent-name>
-toolchain-name: <name>
-toolchain-version: <standard-version>
-fingerprint: <agent-fingerprint>
\
+[*-log]: <text>
+\
+
+The per-operation result log. Note that the \c{*-log} values should appear
+last and in the same order as the corresponding \c{*-status} values. For
+the list of supported operation names refer to the \c{*-status} value
+description.
+
+
+\h#arch-task-req|Task Request Manifest|
An agent (or controller acting as an agent) sends a task request to its
controller via HTTP/HTTPS POST method (@@ URL/API endpoint). The task request
starts with the task request manifest followed by a list of machine manifests.
+The task request manifest synopsis is presented next followed by the detailed
+description of each value in subsequent sections.
-\dl|
+\
+agent: <name>
+toolchain-name: <name>
+toolchain-version: <standard-version>
+[fingerprint]: <agent-fingerprint>
+\
-\li|\n\c{agent: <agent-name>}\n
- The name of the agent host (\c{hostname}). This should be unique in a
- particular \c{bbot} deployment.|
+\h2#arch-task-req-agent|\c{agent}|
-\li|\n\c{toolchain-name: <name>}\n
+\
+agent: <name>
+\
- The \c{build2} toolchain name being used by the agent.|
+The name of the agent host (\c{hostname}). The name should be unique in a
+particular \c{bbot} deployment.
-\li|\n\c{toolchain-version: <standard-version>}\n
- The \c{build2} toolchain version being used by the agent.|
+\h2#arch-task-req-toolchain-name|\c{toolchain-name}|
-\li|\n\c{fingerprint: <agent-fingerprint>}\n
+\
+toolchain-name: <name>
+\
- The SHA256 fingerprint of the agent's public key. An agent may be configured
- not to use the public key-based authentication in which case it does not
- include this value. However, the controller may be configured to require
- the authentication in which case it will respond with the
- 401 (unauthorized) HTTP status code.||
+The \c{build2} toolchain name being used by the agent.
-\h#arch-task-res-manifest|Task Response Manifest|
+
+\h2#arch-task-req-toolchain-version|\c{toolchain-version}|
\
-SYNOPSIS
+toolchain-version: <standard-version>
+\
+
+The \c{build2} toolchain version being used by the agent.
+
+
+\h2#arch-task-req-fingerprint|\c{fingerprint}|
-session: <session-id>
-challenge: <text>
-result-url: <url>
\
+[fingerprint]: <agent-fingerprint>
+\
+
+The SHA256 fingerprint of the agent's public key. An agent may be configured
+not to use the public key-based authentication in which case it does not
+include this value. However, the controller may be configured to require the
+authentication in which case it should respond with the 401 (unauthorized)
+HTTP status code.
+
+
+\h#arch-task-res|Task Response Manifest|
A controller sends the task response manifest in response to the task request
initiated by an agent. The response is delivered as a result of the POST
method. The task response starts with the task response manifest optionally
-followed by a task manifest.
+followed by the task manifest. The task response manifest synopsis is
+presented next followed by the detailed description of each value in
+subsequent sections.
-\dl|
+\
+session: <id>
+[challenge]: <text>
+[result-url]: <url>
+\
-\li|\n\c{session: <session-id>}\n
- An identifier assigned to this session by the controller. An empty value
- indicates that the controller has no tasks at this time in which case all
- the following values as well as the task manifest are absent.|
+\h2#arch-task-res-session|\c{session}|
-\li|\n\c{challenge: <string>}\n
+\
+session: <id>
+\
- Random 64-character string (nonce) used to challenge the agent's private
- key. If present, then the agent must sign this string and include the
- signature in the result request.
+The identifier assigned to this session by the controller. An empty value
+indicates that the controller has no tasks at this time in which case all the
+following values as well as the task manifest are absent.
- The signature should be calculated by encrypting the string with the agent's
- private key and then base64-encoding the result.|
-\li|\n\c{result-url: <url>}\n
+\h2#arch-task-res-challenge|\c{challenge}|
- The URL to post the result (upload) request to.||
+\
+[challenge]: <text>
+\
+
+The random, 64 characters long string (nonce) used to challenge the agent's
+private key. If present, then the agent must sign this string and include the
+signature in the result request (see below).
+
+The signature should be calculated by encrypting the string with the agent's
+private key and then \c{base64}-encoding the result.
-\h#arch-result-req-manifest|Result Request Manifest|
+\h2#arch-task-res-result-url|\c{result-url}|
\
-SYNOPSIS
+[result-url]: <url>
+\
+
+The URL to POST (upload) the result request to.
+
+
+\h#arch-result-req|Result Request Manifest|
-session: <session-id>
-challenge: <text>
+On completion of a task an agent (or controller acting as an agent) sends the
+result (upload) request to the controller via the POST method using the URL
+returned in the task response (see above). The result request starts with the
+result request manifest followed by the result manifest. Note that there is no
+result response and only a successful but empty POST result is returned. The
+result request manifest synopsis is presented next followed by the detailed
+description of each value in subsequent sections.
+
+\
+session: <id>
+[challenge]: <text>
\
-On completion of a task an agent (or controller acting as an agent) sends a
-result (upload) request to the controller via HTTP/HTTPS POST method using the
-URL returned in the task response. The result request starts with the
-result request manifest followed by a result manifest. Note that there is no
-result response and only a successful but empty POST result is returned.
-\dl|
+\h2#arch-result-req-session|\c{session}|
+
+\
+session: <session-id>
+\
+
+The session id as returned by the controller in the task response.
-\li|\n\c{session: <session-id>}\n
- The session id as returned by the controller in the task response.|
+\h2#arch-result-req-challenge|\c{challenge}|
-\li|\n\c{challenge: <text>}\n
+\
+[challenge]: <text>
+\
- The answer to the private key challenge as posed by the controller in the
- task response. Must be present only if the challenge value was present in
- the task response.||
+The answer to the private key challenge as posed by the controller in the task
+response. It must be present only if the challenge value was present in the
+task response.
\h#arch-worker|Worker Logic|