aboutsummaryrefslogtreecommitdiff
path: root/INSTALL
diff options
context:
space:
mode:
authorKaren Arutyunov <karen@codesynthesis.com>2020-03-18 22:17:49 +0300
committerKaren Arutyunov <karen@codesynthesis.com>2020-03-27 17:28:44 +0300
commit35359f038f571dc46de3d14af72a2bc911fb0a24 (patch)
treede3e89d678e78b9efc4d395274fd7ccc68f4a213 /INSTALL
parent8ad672cc7211952716ffe1fbf76c179b4f1149e3 (diff)
Implement brep-monitor
Diffstat (limited to 'INSTALL')
-rw-r--r--INSTALL72
1 files changed, 46 insertions, 26 deletions
diff --git a/INSTALL b/INSTALL
index 59cc081..00f7975 100644
--- a/INSTALL
+++ b/INSTALL
@@ -11,8 +11,9 @@ can be omitted.
1. Create 'brep' User
This user will be used to run the brep package database loader, build database
-cleaner, and the database schemes migration utility. We will also use its home
-directory to build and install the brep module, store its configuration, etc.
+cleaner, monitor, and database schemas migration utility. We will also use its
+home directory to build and install the brep module, store its configuration,
+etc.
Note: if the deployment machine employs SELinux, then this approach may
require additional configuration steps (not shown) in order to allow Apache2
@@ -194,7 +195,7 @@ CREATE EXTENSION citext;
Exit psql (^D)
-5. Create Database Schemes and Load Repositories
+5. Create Database Schemas and Load Repositories
$ mkdir config
$ edit config/loadtab # Loader configuration, see brep-load(1).
@@ -275,7 +276,7 @@ can also find this fragment in install/share/brep/etc/brep-apache2.conf):
#
brep-email admin@example.org
- # Repository host. It specifies the scheme and the host address (but
+ # Repository host. It specifies the schema and the host address (but
# not the root path; see brep-root below) that will be used whenever
# brep needs to construct an absolute URL to one of its locations (for
# example, a link to a build log that is being send via email).
@@ -384,15 +385,15 @@ $ cd install/share/brep/www/
$ for i in *.scss; do sassc -s compressed $i `basename -s .scss $i`.css; done
-8. Setup Periodic Loader and Cleaner Execution
+8. Setup Periodic Loader, Cleaner, and Monitor Execution
Initially this guide suggested using systemd user session support to run the
-loader and the cleaner. However, the current state of user sessions has one
-major drawback: they are not started/attached-to when logging in with su -l
-(see Debian bug #813789 for details). This limitation makes them unusable in
-our setup. If you still would like to use systemd to run the loader and the
-cleaner, then you can set it up as a system-wide service which runs the
-utilities as the brep user/group. Otherwise, a cron job is a natural choice.
+loader, cleaner, and monitor. However, the current state of user sessions has
+one major drawback: they are not started/attached-to when logging in with su
+-l (see Debian bug #813789 for details). This limitation makes them unusable
+in our setup. If you still would like to use systemd to run the utilities,
+then you can set it up as a system-wide service which runs them as the brep
+user/group. Otherwise, a cron job is a natural choice.
Note that the builds cleaner execution is optional and is only required if the
build2 build bot functionality is enabled (see the build bot documentation for
@@ -402,29 +403,37 @@ parts in the subsequent subsections.
If the CI request functionality is enabled you most likely will want to
additionally setup the tenants cleanup.
+The monitor execution is also optional and currently only makes sense if the
+build2 build bot functionality is enabled. Note that you may need to replace
+the public toolchain name argument in the monitor utility command with a real
+list of toolchain names (and optionally versions) used in the brep build
+infrastructure.
-8.a Setup Periodic Loader and Cleaner Execution with cron
-The following crontab entries will execute the loader every five minutes
-and the tenants and builds cleaners once a day at midnight:
+8.a Setup Periodic Loader, Cleaner, and Monitor Execution with cron
+
+The following crontab entries will execute the loader every five minutes, the
+tenants and builds cleaners once a day at midnight, and the monitor every hour
+(all shifted by a few minutes in order not to clash with other jobs):
$ crontab -l
MAILTO=<brep-admin-email>
PATH=/usr/local/bin:/bin:/usr/bin
*/5 * * * * $HOME/install/bin/brep-load $HOME/config/loadtab
-0 0 * * * $HOME/install/bin/brep-clean tenants 240
-0 0 * * * $HOME/install/bin/brep-clean builds $HOME/config/buildtab
+1 0 * * * $HOME/install/bin/brep-clean tenants 240
+2 0 * * * $HOME/install/bin/brep-clean builds $HOME/config/buildtab
+3 * * * * $HOME/install/bin/brep-monitor --report-timeout 86400 --clean $HOME/config/brep-module.conf public
^D
Note that here we assume that bpkg (which is executed by brep-load) is in one
of the PATH's directories (usually /usr/local/bin).
-8.b Setup Periodic Loader and Cleaner Execution with systemd
+8.b Setup Periodic Loader, Cleaner, and Monitor Execution with systemd
In this version we will use the systemd user session to periodically run the
-loader and the cleaner as the brep user. If your installation doesn't use
-systemd, then a cron job would be a natural alternative (see above).
+loader, cleaner, and monitor as the brep user. If your installation doesn't
+use systemd, then a cron job would be a natural alternative (see above).
As the first step, make sure systemd user sessions support is working for the
brep user:
@@ -443,6 +452,7 @@ $ sudo loginctl enable-linger brep
$ mkdir -p .config/systemd/user
$ cp install/share/brep/etc/systemd/brep-load.* .config/systemd/user/
$ cp install/share/brep/etc/systemd/brep-clean.* .config/systemd/user/
+$ cp install/share/brep/etc/systemd/brep-monitor.* .config/systemd/user/
Start the service to make sure there are no issues:
@@ -452,16 +462,21 @@ $ journalctl
$ systemctl --user start brep-clean.service
$ journalctl
+$ systemctl --user start brep-monitor.service
+$ journalctl
+
Start the timers and monitor them to make sure they fire:
$ systemctl --user start brep-load.timer
$ systemctl --user start brep-clean.timer
+$ systemctl --user start brep-monitor.timer
$ journalctl -f
If everything looks good, enable the timer to be started at boot time:
$ systemctl --user enable brep-load.timer
$ systemctl --user enable brep-clean.timer
+$ systemctl --user enable brep-monitor.timer
9. Upgrade Procedure
@@ -483,18 +498,20 @@ $ cd brep
$ bpkg fetch
$ bpkg build brep
-If you are using a systemd-based setup, then stop and disable the loader and
-the cleaner:
+If you are using a systemd-based setup, then stop and disable the loader,
+cleaner, and monitor:
$ systemctl --user disable --now brep-load.timer
$ systemctl --user disable --now brep-clean.timer
+$ systemctl --user disable --now brep-monitor.timer
$ systemctl --user stop brep-load.service
$ systemctl --user stop brep-clean.service
+$ systemctl --user stop brep-monitor.service
If you are using a cron-based setup, then it is not worth it commenting out the
-job entries. If the new version of the loader or the cleaner gets executed
-before or during the migration, then it will fail and you will get an email
-with the diagnostics. Other than that, it should be harmless.
+job entries. If the new version of the brep utilities gets executed before or
+during the migration, then it will fail and you will get an email with the
+diagnostics. Other than that, it should be harmless.
Stop apache:
@@ -510,7 +527,7 @@ Review brep-module.conf changes that may need to be merged:
$ diff -u install/share/brep/etc/brep-module.conf config/brep-module.conf
-Migrate database schemes:
+Migrate database schemas:
$ install/bin/brep-migrate package
$ install/bin/brep-migrate build
@@ -521,17 +538,20 @@ is not possible), then one way to do it would be:
$ psql -d brep_package -c 'DROP OWNED BY brep'
$ psql -d brep_build -c 'DROP OWNED BY brep'
-If using systemd, then start and enable the loader and the cleaner:
+If using systemd, then start and enable the loader, cleaner, and monitor:
$ systemctl --user start brep-load.service
$ systemctl --user status brep-load.service
$ systemctl --user start brep-clean.service
$ systemctl --user status brep-clean.service
+$ systemctl --user start brep-monitor.service
+$ systemctl --user status brep-monitor.service
If everything looks good, enable periodic execution:
$ systemctl --user enable --now brep-load.timer
$ systemctl --user enable --now brep-clean.timer
+$ systemctl --user enable --now brep-monitor.timer
If using cron, then simply wait for the next run.