Spec CI functionality

2 files changed, 253 insertions, 15 deletions

diff --git a/doc/manual.cli b/doc/manual.cli
index eba170d..f3f1335 100644
--- a/doc/manual.cli
+++ b/doc/manual.cli
@@ -27,7 +27,7 @@ method using the \c{multipart/form-data} content type. The implementation in
 duplicates) expecting the rest of the submission and publishing logic to be
 handled by a separate entity according to the repository policy. Such an
 entity can be notified by \c{brep} about a new submission as an invocation of
-the \i{handler program} (as part of the submission request) or via email. It
+the \i{handler program} (as part of the HTTP request) and/or via email. It
 could also be a separate process that monitors the upload data directory.
 
 The submission request without any parameters is treated as the submission
@@ -98,15 +98,16 @@ directory still exists, then it is handled by \c{brep} depending on the
 handler process exit status and the submission result manifest status value.
 If the process has terminated abnormally or with a non-zero exit status or the
 result manifest status is in the [500-599] range (HTTP server error), then the
-directory is saved for troubleshooting by appending a numeric extension to its
-name. Otherwise, if the status is in the [400-499] range (HTTP client error),
-then the directory is removed. If the directory is left in place by the
-handler or is saved for troubleshooting, then the submission result manifest
-is saved as \c{result.manifest} into this directory, next to the request
-manifest and archive.
+directory is saved for troubleshooting by appending the \c{.fail} extension
+followed by a numeric extension to its name (for example,
+\c{ff5a1a53d318.fail.1}). Otherwise, if the status is in the [400-499] range
+(HTTP client error), then the directory is removed. If the directory is left
+in place by the handler or is saved for troubleshooting, then the submission
+result manifest is saved as \c{result.manifest} into this directory, next to
+the request manifest and archive.
 
 If \c{submit-handler-timeout} is configured and the handler program does not
-exit in the alloted time, then it is killed and its termination is treated as
+exit in the allotted time, then it is killed and its termination is treated as
 abnormal.
 
 If the handler program is not specified, then the following submission result
@@ -124,8 +125,7 @@ reference: <abbrev-checksum>
 \li|Send the submission email.
 
 If \c{submit-email} is configured, send an email to this address containing
-the submission request manifest and, unless implied, the submission result
-manifest.|
+the submission request manifest and the submission result manifest.|
 
 \li|Respond to the client.
 
@@ -161,7 +161,7 @@ corresponding to the custom request parameters.
 \
 archive: <name>
 sha256sum: <sum>
-timestamp: <datetime>
+timestamp: <date-time>
 [simulate]: <outcome>
 [client-ip]: <string>
 [user-agent]: <string>
@@ -185,4 +185,160 @@ message: <string>
 [reference]: <string>
 \
 
+
+\h1#ci|Package CI|
+
+The CI functionality allows submission of package CI requests as well as
+additional, repository-specific information via the HTTP \c{GET} and \c{POST}
+methods. The implementation in \c{brep} only handles reception as well as
+basic parameter verification expecting the rest of the CI logic to be handled
+by a separate entity according to the repository policy. Such an entity can be
+notified by \c{brep} about a new CI request as an invocation of the \i{handler
+program} (as part of the HTTP request) and/or via email. It could also be a
+separate process that monitors the CI data directory.
+
+The CI request without any parameters is treated as the CI form request. If
+\c{ci-form} is configured, then such a form is generated and returned.
+Otherwise, such a request is treated as an invalid CI request (missing
+parameters).
+
+For each CI request \c{brep} performs the following steps.
+
+\ol|
+
+\li|Verify the required \c{repository} and optional \c{package} parameters.
+
+The \c{repository} parameter is the repository URL that contains the packages
+to be tested. If one or more \c{package} parameters are present, then only the
+specified packages are tested. If no \c{package} parameters are specified,
+then all the packages present in the repository (but excluding complement
+repositories) are tested.
+
+Each \c{package} parameter can specify either just the package name, in which
+case all the versions of this package present in the repository will be
+tested, or both the name and version in the \c{<name>/<version>} form (for
+example, \c{libhello/1.2.3}.|
+
+\li|Verify other parameters are valid manifest name/value pairs.
+
+The value can only contain printable ASCII characters as well as tab
+(\c{\\t}), carriage return (\c{\\r}), and line feed (\c{\\n}).|
+
+\li|Generate CI request id and create request directory.
+
+For each CI request a unique id (UUID) is generated and a request subdirectory
+is created in the \c{ci-data} directory with this id as its name.|
+
+\li|Save the CI request manifest into the request directory.
+
+The CI request manifest is saved as \c{request.manifest} into the request
+subdirectory created on the previous step.|
+
+
+\li|Invoke the CI handler program.
+
+If \c{ci-handler} is configured, invoke the handler program passing to it
+additional arguments specified with \c{ci-handler-argument} (if any) followed
+by the absolute path to the CI request directory.
+
+The handler program is expected to write the CI result manifest to \c{stdout}
+and terminate with the zero exit status. A non-zero exit status is treated as
+an internal error. The handler program's \c{stderr} is logged.
+
+Note that the handler program should report temporary server errors (service
+overload, network connectivity loss, etc.) via the CI result manifest status
+values in the [500-599] range (HTTP server error) rather than via a non-zero
+exit status.
+
+The handler program assumes ownership of the CI request directory and can
+move/remove it. If after the handler program terminates the request directory
+still exists, then it is handled by \c{brep} depending on the handler process
+exit status and the CI result manifest status value. If the process has
+terminated abnormally or with a non-zero exit status or the result manifest
+status is in the [500-599] range (HTTP server error), then the directory is
+saved for troubleshooting by appending the \c{.fail} extension to its
+name. Otherwise, if the status is in the [400-499] range (HTTP client error),
+then the directory is removed. If the directory is left in place by the
+handler or is saved for troubleshooting, then the CI result manifest
+is saved as \c{result.manifest} into this directory, next to the request
+manifest.
+
+If \c{ci-handler-timeout} is configured and the handler program does not
+exit in the allotted time, then it is killed and its termination is treated as
+abnormal.
+
+If the handler program is not specified, then the following CI result
+manifest is implied (note that it is not saved):
+
+\
+status: 200
+message: CI request is queued
+reference: <request-id>
+\
+
+|
+
+\li|Send the CI request email.
+
+If \c{ci-email} is configured, send an email to this address containing the CI
+request manifest and the CI result manifest.|
+
+\li|Respond to the client.
+
+Respond to the client with the CI result manifest and its \c{status} value as
+the HTTP status code.|
+
+|
+
+Check violations that are explicitly mentioned above are always reported with
+the CI result manifest. Other errors (for example, internal server errors)
+might be reported with unformatted text, including HTML.
+
+If the CI request contains the \c{simulate} parameter, then the CI service
+simulates the specified outcome of the CI process without actually performing
+any externally visible actions (e.g., testing the package, publishing the
+result, etc). Note that the CI request email (\c{ci-email}) is not sent for
+simulated requests.
+
+Pre-defined simulation outcome values are \c{internal-error-text},
+\c{internal-error-html}, and \c{success}. The simulation outcome is included
+into the CI request manifest and the handler program must at least handle
+\c{success} but may recognize additional outcomes.
+
+
+\h#ci-request-manifest|CI Request Manifest|
+
+The CI request manifest starts with the below values and in that order
+optionally followed by additional values in the unspecified order
+corresponding to the custom request parameters.
+
+\
+id: <request-id>
+repository: <url>
+[package]: <name>[/<version>]
+timestamp: <date-time>
+[simulate]: <outcome>
+[client-ip]: <string>
+[user-agent]: <string>
+\
+
+The \c{package} value can be repeated multiple time. The \c{timestamp} value
+is in the ISO-8601 \c{<YYYY>-<MM>-<DD>T<hh>:<mm>:<ss>Z} form (always
+UTC). Note also that \c{client-ip} can be IPv4 or IPv6.
+
+
+\h#ci-result-manifest|CI Result Manifest|
+
+The CI result manifest starts with the below values and in that order
+optionally followed by additional values if returned by the handler program.
+If the CI request is successful, then the \c{reference} value must be present
+and contain a string that can be used to identify this request (for example,
+the CI request id).
+
+\
+status: <http-code>
+message: <string>
+[reference]: <string>
+\
+
 "
diff --git a/mod/options.cli b/mod/options.cli
index 59aceb6..33ed9a1 100644
--- a/mod/options.cli
+++ b/mod/options.cli
@@ -444,7 +444,7 @@ namespace brep
       {
         "<path>",
         "The handler program to be executed on package submission. The handler
-         is executed as part of the submission request and is passed additional
+         is executed as part of the HTTP request and is passed additional
          arguments that can be specified with \cb{submit-handler-argument}
          followed by the absolute path to the submission directory. See
          \l{brep The \cb{build2} Repository Interface Manual} for more
@@ -462,9 +462,69 @@ namespace brep
       size_t submit-handler-timeout
       {
         "<seconds>",
-        "The handler program timeout in seconds. If specified and the handler
-         does not exit in the alloted time, then it is killed and its
-         termination is treated as abnormal."
+        "The submission handler program timeout in seconds. If specified and
+         the handler does not exit in the allotted time, then it is killed and
+         its termination is treated as abnormal."
+      }
+    };
+
+    class ci: page, handler
+    {
+      dir_path ci-data
+      {
+        "<dir>",
+        "The directory to save CI request data to. If unspecified, the
+         package CI functionality will be disabled. See \l{brep The
+         \cb{build2} Repository Interface Manual} for more information on
+         package CI.
+
+         Note that the directory path must be absolute and the directory
+         itself must exist and have read, write, and execute permissions
+         granted to the user that runs the web server."
+      }
+
+      path ci-form
+      {
+        "<file>",
+        "The package CI form fragment. If specified, then its contents are
+         treated as an XHTML5 fragment that is inserted into the <body>
+         element of the CI page. If unspecified, then no CI page will be
+         displayed. Note that the file path must be absolute."
+      }
+
+      string ci-email
+      {
+        "<email>",
+        "The package CI email. If specified, the CI request and result
+         manifests will be sent to this address. See \l{brep The \cb{build2}
+         Repository Interface Manual} for more information."
+      }
+
+      path ci-handler
+      {
+        "<path>",
+        "The handler program to be executed on CI request. The handler is
+         executed as part of the HTTP request and is passed additional
+         arguments that can be specified with \cb{ci-handler-argument}
+         followed by the absolute path to the CI request directory. See
+         \l{brep The \cb{build2} Repository Interface Manual} for more
+         information. Note that the program path must be absolute."
+      }
+
+      strings ci-handler-argument
+      {
+        "<arg>",
+        "Additional arguments to be passed to the CI handler program (see
+         \cb{ci-handler} for details). Repeat this option to specify multiple
+         arguments."
+      }
+
+      size_t ci-handler-timeout
+      {
+        "<seconds>",
+        "The CI handler program timeout in seconds. If specified and the
+         handler does not exit in the allotted time, then it is killed and
+         its termination is treated as abnormal."
       }
     };
 
@@ -655,5 +715,27 @@ namespace brep
       //
       string simulate;
     };
+
+    // Parameters, except simulate, must either be all present (actual CI
+    // request) or absent (CI form request).
+    //
+    // Note also that besides these parameters there can be others. We don't
+    // recognize their semantics and just save them to the CI request
+    // manifest.
+    //
+    class ci
+    {
+      // Package repository URL.
+      //
+      url repository;
+
+      // Package names/versions.
+      //
+      strings package;
+
+      // Submission simulation outcome.
+      //
+      string simulate;
+    };
   }
 }