diff options
Diffstat (limited to 'INSTALL')
| -rw-r--r-- | INSTALL | 228 |
1 files changed, 187 insertions, 41 deletions
@@ -1,7 +1,8 @@ -This guide shows how to install and configure brep on a "deployment" machine as -opposed to a "development" one (see INSTALL-DEV for the latter). Here we assume -you are using a systemd-based distribution. If not, then you will need to -replace systemctl commands with the equivalent init.d ones. +This guide describes how to install and configure brep on a "deployment" +machine as opposed to a "development" one (see INSTALL-DEV for the +latter). Here we assume you are using a systemd-based distribution. If not, +then you will need to replace systemctl commands with the equivalent init.d +ones. The below instructions include steps for setting up brep as the build2 build bot controller, package submission, and CI request services. All these @@ -233,6 +234,61 @@ $ psql -d brep_build -c 'SELECT DISTINCT name FROM build_package' $ cp install/share/brep/etc/brep-module.conf config/ $ edit config/brep-module.conf # Adjust default values if required. +See the following sub-sections for details on configuring various optional +brep functionality. + +Once the brep module configuration is ready, the next step is to enable +it in the Apache2 configuration file. Here we assume you have setup an +appropriate Apache2 virtual server. Open the corresponding Apache2 .conf +file and add the contents of brep/etc/brep-apache2.conf into the +<VirtualHost> section. + +The output content types of the brep module are application/xhtml+xml, +text/manifest and text/plain. If you would like to make sure they get +compressed (along with linked CSS), also add the following lines: + + # Compress brep output (xhtml+xml) and CSS. + # + AddOutputFilterByType DEFLATE application/xhtml+xml + AddOutputFilterByType DEFLATE text/manifest + AddOutputFilterByType DEFLATE text/plain + AddOutputFilterByType DEFLATE text/css + +Then restart Apache2: + +$ sudo systemctl restart apache2 + +To verify, visit the repository root. To troubleshoot, see Apache logs. + +Now that Apache2 loads the brep module which requires PostgreSQL, it is a good +idea to make the Apache2 service depend on PostgreSQL so that they are started +in proper order. Here is how we can do it with systemd (with newer versions +you can use 'systemctl edit' instead of mkdir and cat): + +# mkdir -p /etc/systemd/system/apache2.service.d/ +# cat >/etc/systemd/system/apache2.service.d/postgresql.conf +[Unit] +Requires=postgresql.service +After=postgresql.service +^D + +# mkdir -p /etc/systemd/system/postgresql.service.d/ +# cat >/etc/systemd/system/postgresql.service.d/apache2.conf +[Unit] +Wants=apache2.service +^D + +# systemctl daemon-reload +# systemctl cat apache2 # Verify override is listed. +# systemctl cat postgresql # Verify override is listed. +# systemctl stop postgresql +# systemctl status apache2 # Verify stopped. +# systemctl start postgresql +# systemctl status apache2 # Verify started. + + +6.1 Enabling build bot controller functionality + To enable the build2 build bot controller functionality you will need to set the build-config option in brep-module.conf. To also enable the build artifacts upload functionality you will need to specify the upload-data @@ -250,6 +306,9 @@ $ setfacl -m g:www-data:rwx /home/brep/bindist-data For sample upload handler implementations see brep/handler/upload/. + +6.2 Enabling package submission functionality + To enable the package submission functionality you will need to specify the submit-data and submit-temp directories in brep-module.conf. Note that these directories must exist and have read, write, and execute permissions granted @@ -272,6 +331,9 @@ $ edit config/submit.xhtml # Add custom form fields, adjust CSS style, etc. For sample submission handler implementations see brep/handler/submit/. + +6.3 Enabling CI request functionality + To enable the CI request functionality you will need to specify the ci-data directory in brep-module.conf. Note that this directory must exist and have read, write, and execute permissions granted to the www-data user. This, for @@ -291,52 +353,119 @@ $ edit config/ci.xhtml # Add custom form fields, adjust CSS style, etc. For sample CI request handler implementations see brep/handler/ci/. -Here we assume you have setup an appropriate Apache2 virtual server. Open the -corresponding Apache2 .conf file and add the contents of -brep/etc/brep-apache2.conf into the <VirtualHost> section. -The output content types of the brep module are application/xhtml+xml, -text/manifest and text/plain. If you would like to make sure they get -compressed (along with linked CSS), also add the following lines: +6.4 Enabling GitHub CI integration - # Compress brep output (xhtml+xml) and CSS. - # - AddOutputFilterByType DEFLATE application/xhtml+xml - AddOutputFilterByType DEFLATE text/manifest - AddOutputFilterByType DEFLATE text/plain - AddOutputFilterByType DEFLATE text/css +6.4.1 Background -Restart Apache2: +The GitHub CI integration has one user-configurable setting: +warning=<success|failure> (whether or not to fail on warnings). -$ sudo systemctl restart apache2 +In order not to have to support repository configuration files, a deployment +will consist of two registered GitHub Apps with the same webhook URL (i.e., +the same brep instance) but different query parameters: one with +warning=success and the other with warning=failure. The App id is passed (as a +query parameter) so that we know which private key to use (the key cannot be +shared between Apps). -To verify, visit the repository root. To troubleshoot, see Apache logs. +We will call the warning=success App the "Default App" and the warning=failure +App the "Werror App". -Now that Apache2 loads the brep module which requires PostgreSQL, it is a good -idea to make the Apache2 service depend on PostgreSQL so that they are started -in proper order. Here is how we can do it with systemd (with newer versions -you can use 'systemctl edit' instead of mkdir and cat): +6.4.2 Create the GitHub Apps -# mkdir -p /etc/systemd/system/apache2.service.d/ -# cat >/etc/systemd/system/apache2.service.d/postgresql.conf -[Unit] -Requires=postgresql.service -After=postgresql.service -^D +To create a GitHub App under the <org> organization, visit +https://github.com/organizations/<org>/settings/apps (Settings -> Developer +settings -> GitHub Apps). Then click on New GitHub App. -# mkdir -p /etc/systemd/system/postgresql.service.d/ -# cat >/etc/systemd/system/postgresql.service.d/apache2.conf -[Unit] -Wants=apache2.service -^D +App names (note: 34 character limit): -# systemctl daemon-reload -# systemctl cat apache2 # Verify override is listed. -# systemctl cat postgresql # Verify override is listed. -# systemctl stop postgresql -# systemctl status apache2 # Verify stopped. -# systemctl start postgresql -# systemctl status apache2 # Verify started. + Default App: "<org> CI" + Werror App: "<org> CI - warnings as errors" + +App description: + + Default App: "Trigger <org> CI on branch push and pull request." + Werror App: "Trigger <org> CI on branch push and pull request. Warnings are + treated as errors". + +App homepage: + + https://ci.<org>.org/ + +Skip the "Identifying and authorizing users" and "Post installation" sections. + +Leave webhooks active. + +Webhook URL: + + Default App: https://ci.<org>.org/?ci-github&app-id=XXX&warning=success + Werror App: https://ci.<org>.org/?ci-github&app-id=XXX&warning=failure + +Note that the App id only becomes available once the App has been registered +so we update it later in both URLs. + +Webhook secret: Use the same random 64-character string for both Apps. + + echo `tr -dc -- A-Za-z0-9 </dev/urandom | head -c 64` + +Note that GitHub says only that the secret should be "a random string with +high entropy." However lots of sources say 32 bytes should be secure enough +for HMAC-SHA256, while other sources recommend 64 bytes for maximal security +at an insignificant performance cost. (Keys longer than 64 bytes are hashed to +match the internal block size and are therefore not recommended.) + +Repository permissions: + - Checks: RW + - Contents: RO (for Push events) + - Metadata (mandatory): RO + - Pull requests: RO + +Subscribed events: + - Check suite + - Pull request + - Push + +Note that GitHub Apps with write access to the "Checks" permission are +automatically subscribed to check_suite(requested|rerequested) and check_run +events so no need to subscribe explicitly. However in order to receive +check_suite(completed) events, which we need, one does have to subscribe to +Check suite. + +Select "Any account" under "Where can this GitHub App be installed?". + +Click "Create GitHub App". + +When the page reloads (should be the General tab), note the App id and replace +the XXX in the webhook URL with it. + +Still in the General tab, scroll to Private keys and generate a private key. +The file will be downloaded by the browser. + +@@ TODO Logo +@@ TODO Create Marketplace listing + +6.4.3 Configure brep + +Assume the following configuration values: + +- Webhook secret: abcdefg +- Default App id: 12345 +- Werror App id: 67890 + +In brep-module.conf: + +Set the webhook secret from the GitHub App settings: + + ci-github-app-webhook-secret abcdefg + +Associate each GitHub App id with the App's private key: + + ci-github-app-id-private-key 12345=path/to/default-app-private-key.pem + ci-github-app-id-private-key 67890=path/to/werror-app-private-key.pem + +Now brep should be ready to handle the webhook event requests triggered by +branch pushes and pull requests in repositories into which one of these Apps +has been installed. 7. Optimize CSS @@ -407,6 +536,23 @@ PATH=/usr/local/bin:/bin:/usr/bin Note that here we assume that bpkg (which is executed by brep-load) is in one of the PATH's directories (usually /usr/local/bin). +To additionally load the package metadata (reviews, etc) to the database, the +brep-load-with-metadata wrapper script can be used instead of brep-load +directly. In this case, the package git repository that contains the owners/ +metadata directory should be pre-cloned (read-only and shallowly) as follows: + +$ git init public-metdata +$ cd public-metdata +$ git remote add origin <repository-url> +$ git config core.sparsecheckout true +$ echo "owners/" > .git/info/sparse-checkout +$ git pull --depth=1 origin master + +And the above crontab brep-load entry needs to be replaced with the following +version: + +$HOME/install/bin/brep-load-with-metadata --timeout 60 /home/brep/public-metdata $HOME/config/loadtab + 8.b Setup Periodic Loader, Cleaner, and Monitor Execution with systemd |