element. +# +# search-title Packages + + +# Web page logo. It is displayed in the page header aligned to the left edge. +# The value is treated as an XHTML5 fragment. +# +# logo "" + + +# Web page menu. Each entry is displayed in the page header in the order +# specified and aligned to the right edge. A link target that starts with '/' +# or contains ':' is used as is. Otherwise, it is prefixed with the repository +# web interface root. +# +menu Packages= +# menu Builds=?builds +# menu Configs=?build-configs +menu Submit=?submit +# menu CI=?ci +menu About=?about + + +# Number of packages per page. +# +# search-page-entries 20 + + +# Number of pages in navigation (pager). +# +# search-pages 5 + + +# Number of package description characters to display in brief pages. +# +# package-description 500 + + +# Number of package changes characters to display in brief pages. +# +# package-changes 5000 + + +# The package database connection configuration. By default, brep will try to +# connect to the local instance of PostgreSQL with the operating system- +# default mechanism (Unix-domain socket, etc) and use operating system +# (login) user name and the database called 'brep_package'. If the role name +# is not empty then the login user will be switched (with SET ROLE) to this +# user prior to executing any statements. If not specified, then 'brep' is +# used. See brep(1) for details. +# +# package-db-user +# package-db-role brep +# package-db-password +# package-db-name brep_package +# package-db-host +# package-db-port + + +# The maximum number of concurrent package database connections per web server +# process. If 0, then no limitation is applied. +# +# package-db-max-connections 5 + + +# The maximum number of times to retry package database transactions in the +# face of recoverable failures (deadlock, loss of connection, etc). +# +# package-db-retry 10 + + +# Build configuration file. If not specified (default), then the package +# building functionality will be disabled. If specified, then the build +# database must be configured (see next). Note: must be an absolute path. +# +# build-config + + +# Number of build configurations per page. +# +# build-config-page-entries 20 + + +# Number of pages in navigation (pager). +# +# build-config-pages 5 + + +# Directory containing build bot agent public keys. If specified, then brep +# will perform agent authentication and will reject build results from +# unauthenticated ones. If not specified, then build results are accepted from +# all agents (which will be a security risk if the brep instance is publicly +# accessible). +# +# The directory is expected to contain one PEM-encoded public key per file with +# the .pem extension. All other files and subdirectories are ignored. The brep +# instance needs to be restarted after adding new key files for the changes to +# take effect. +# +# build-bot-agent-keys + + +# Number of builds per page. +# +# build-page-entries 20 + + +# Number of pages in navigation (pager). +# +# build-pages 5 + + +# Time to wait before considering a package for a forced rebuild. Must be +# specified in seconds. Default is 10 minutes. +# +# build-forced-rebuild-timeout 600 + + +# Time to wait before considering a package for a normal rebuild. Must be +# specified in seconds. Default is 24 hours. +# +# build-normal-rebuild-timeout 86400 + + +# Alternative package rebuild timeout to use instead of the normal rebuild +# timeout (see the build-normal-rebuild-timeout option for details) during +# the specified time interval. Must be specified in seconds. Default is the +# time interval length. +# +# The alternative rebuild timeout can be used to "pull" the rebuild window to +# the specified time of day, for example, to optimize load and/or power +# consumption of the build infrastructure (off-work hours, solar, off-peak +# electricity tariffs, etc). A shorter than the time interval rebuild timeout +# can also be used to force continuous rebuilds, for example, to shake out +# flaky tests. Note also that if the alternative rebuild timeout is greater +# than the normal rebuild timeout, then this will result in slower rebuilds +# during the alternative time interval. In this case, if the build +# infrastructure is monitored for delayed package builds, then the alternative +# rebuild timeout should only be made slightly greater than the normal timeout +# (see brep-monitor(1) for details). +# +# The time interval boundaries must be specified as times of day (in the local +# timezone) in the <hours>:<minutes> form. If the stop time is less than the +# start time then the interval extends through midnight. The start and stop +# times must both be either specified or absent. If unspecified, then no +# alternative rebuild timeout will be used. +# +# build-alt-rebuild-timeout +# build-alt-rebuild-start +# build-alt-rebuild-stop + + +# The maximum size of the build task request manifest accepted. Note that the +# HTTP POST request body is cached to retry database transactions in the face +# of recoverable failures (deadlock, loss of connection, etc). Default is +# 100K. +# +# build-task-request-max-size 102400 + + +# Time to wait before considering the expected task result lost. Must be +# specified in seconds. Default is 3 hours. +# +# build-result-timeout 10800 + + +# The maximum size of the build result manifest accepted. Note that the HTTP +# POST request body is cached to retry database transactions in the face of +# recoverable failures (deadlock, loss of connection, etc). Default is 10M. +# +# build-result-request-max-size 10485760 + + +# The build database connection configuration. By default, brep will try to +# connect to the local instance of PostgreSQL with the operating system-default +# mechanism (Unix-domain socket, etc) and use operating system (login) user +# name and the database called 'brep_build'. If the role name is not empty +# then the login user will be switched (with SET ROLE) to this user prior +# to executing any statements. If not specified, then 'brep' is used. See +# brep(1) for details. +# +# build-db-user +# build-db-role brep +# build-db-password +# build-db-name brep_build +# build-db-host +# build-db-port + + +# The maximum number of concurrent build database connections per web server +# process. If 0, then no limitation is applied. +# +# build-db-max-connections 5 + + +# The maximum number of times to retry build database transactions in the +# face of recoverable failures (deadlock, loss of connection, etc). +# +# build-db-retry 10 + + +# The openssl program to be used for crypto operations. You can also specify +# additional options that should be passed to the openssl program with +# openssl-option. If the openssl program is not explicitly specified, then brep +# will use openssl by default. +# +# openssl openssl + + +# Additional option to be passed to the openssl program (see openssl for +# details). Repeat this option to specify multiple openssl options. +# +# openssl-option + + +# Environment variable to be set (<name>=<value>) or unset (just <name>) for +# the openssl program (see openssl for details). Repeat this option to specify +# multiple openssl variables. Note that unspecified variables are inherited +# from the web server process. +# +# You need to at least set the RANDFILE environment variable to change the +# default location of the openssl program seed file and maybe also the +# OPENSSL_CONF variable if you would like to use a custom openssl configuration +# file. +# +# openssl-envvar RANDFILE=/home/brep/www-data-openssl.rnd +# openssl-envvar OPENSSL_CONF=/home/brep/www-data-openssl.cnf +# +# To create www-data-openssl.rnd with suitable permissions, run (as user brep): +# +# $ touch www-data-openssl.rnd +# $ setfacl -b -m g:www-data:rw www-data-openssl.rnd +# + + +# The directory to save final submission data to. If unspecified, the package +# submission functionality will be disabled. If specified, then submit-temp +# must be specified as well. +# +# Note that the directory path must be absolute and the directory itself must +# exist and have read, write, and execute permissions granted to the user that +# runs the web server. +# +submit-data /home/brep/submit-data + + +# The directory to save temporary submission data to. Must be specified if the +# package submission functionality is enabled. +# +# Note that this directory must be on the same filesystem and satisfy the same +# requirements as submit-data. It is also the user's responsibility to clean +# it up after an unclean web server shutdown. +# +submit-temp /home/brep/submit-temp + + +# The maximum size of the submission data accepted. Note that currently the +# entire submission request is read into memory. Default is 10M. +# +# 100M. +# +submit-max-size 104857600 + + +# The package submission form fragment. If specified, then its contents are +# treated as an XHTML5 fragment that is inserted into the <body> element of +# the submission page. If unspecified, then no submission page will be +# displayed. Note that the file path must be absolute. +# +submit-form /home/brep/install/share/brep/www/submit.xhtml + + +# The package submission email. If specified, the submission request and +# result manifests will be sent to this address. +# +# submit-email + + +# The handler program to be executed on package submission. The handler is +# executed as part of the submission request and is passed additional +# arguments that can be specified with submit-handler-argument followed by +# the absolute path to the submission directory. Note that the program path +# must be absolute. +# +submit-handler /home/brep/install/bin/brep-submit-pub + + +# Additional arguments to be passed to the submission handler program (see +# submit-handler for details). Repeat this option to specify multiple +# arguments. +# +submit-handler-argument --user +submit-handler-argument brep +submit-handler-argument --result-url +submit-handler-argument http://unknown +submit-handler-argument /home/brep/install/bin/brep-load +submit-handler-argument --db-name=brep_submit_package +submit-handler-argument /var/brep/bpkg/pkg + + +# The handler program timeout in seconds. If specified and the handler does +# not exit in the alloted time, then it is killed and its termination is +# treated as abnormal. +# +submit-handler-timeout 120 + + +# The directory to save CI request data to. If unspecified, the package CI +# functionality will be disabled. +# +# Note that the directory path must be absolute and the directory itself must +# exist and have read, write, and execute permissions granted to the user that +# runs the web server. +# +# ci-data + + +# The package CI form fragment. If specified, then its contents are treated as +# an XHTML5 fragment that is inserted into the <body> element of the CI page. +# If unspecified, then no CI page will be displayed. Note that the file path +# must be absolute. +# +# ci-form + + +# The package CI email. If specified, the CI request and result manifests will +# be sent to this address. +# +# ci-email + + +# The handler program to be executed on CI request. The handler is executed as +# part of the HTTP request and is passed additional arguments that can be +# specified with ci-handler-argument followed by the absolute path to the CI +# request directory. Note that the program path must be absolute. +# +# ci-handler + + +# Additional arguments to be passed to the CI handler program (see ci-handler +# for details). Repeat this option to specify multiple arguments. +# +# ci-handler-argument + + +# The CI handler program timeout in seconds. If specified and the handler does +# not exit in the allotted time, then it is killed and its termination is +# treated as abnormal. +# +# ci-handler-timeout + + +# The default view to display for the global repository root. The value is one +# of the supported services (packages, builds, submit, ci, etc). Default is +# packages. +# +# root-global-view packages + + +# The default view to display for the tenant repository root. The value is one +# of the supported services (packages, builds, submit, ci, etc). Default is +# packages. +# +# root-tenant-view packages + + +# Name to call the tenant values on web pages. If not specified, then 'tenant' +# is used. +# +# tenant-name tenant + + +# Trace verbosity. Disabled by default. +# +# verbosity 0 diff --git a/etc/private/install/brep-startup b/etc/private/install/brep-startup new file mode 100755 index 0000000..80cb255 --- /dev/null +++ b/etc/private/install/brep-startup @@ -0,0 +1,67 @@ +#! /usr/bin/env bash + +# file : etc/private/install/brep-startup +# license : MIT; see accompanying LICENSE file + +# (Re-)initialize the brep private instance, normally on the machine startup. +# +# Specifically: +# +# - Create the pkg repository and symlink to it, unless already exists. +# +# - Migrate the brep databases as a sanity check. +# +# - Adjust the brep module configuration file using the current host name/IP. +# +# - Generate the loadtab using the current host name/IP and run the loader. +# +trap "{ exit 1; }" ERR + +# Create the pkg repository, if required. +# +d=/var/brep/bpkg + +if [ ! -L "$d/pkg" ]; then + rd="$(date "+pkg-%Y%m%d-%H%M%S-%N")" + + mkdir -p "$d/$rd/1" + ln -s "$rd" "$d/pkg" +fi + +r="$d/pkg/1" + +if [ ! -f "$r/repositories.manifest" ]; then + echo ": 1" >"$r/repositories.manifest" +fi + +if [ ! -f "$r/packages.manifest" ]; then + bpkg rep-create -q "$r" +fi + +# Migrate the databases. +# +"$HOME/install/bin/brep-migrate" package +"$HOME/install/bin/brep-migrate" build +"$HOME/install/bin/brep-migrate" -n brep_submit_package package + +# Deduce the machine host name. +# +h="$(hostname -f)" +if [ "$h" == "localhost" ]; then + h="$(hostname -I | sed 's/ *$//')" # Strip the potential trailing space(s). +fi + +# Adjust the submission result URL host name in the brep module configuration +# file. +# +sed --in-place -re \ +"\$!N;s%^\s*(submit-handler-argument\s+--result-url\s*\\n)\ +\s*(submit-handler-argument\s+https?://)[^/]+(.*)\$%\1\2$h\3%;P;D" \ +"$HOME/config/brep-module.conf" + +# (Re-)generate the loadtab file and reload the repository. +# +f="$HOME/config/loadtab" + +echo "http://$h/1 private cache:$r" >"$f" +"$HOME/install/bin/brep-load" "$f" diff --git a/etc/private/install/brep-startup.service b/etc/private/install/brep-startup.service new file mode 100644 index 0000000..a3dc546 --- /dev/null +++ b/etc/private/install/brep-startup.service @@ -0,0 +1,17 @@ +[Unit] +Description=brep instance initialization service + +Wants=network-online.target +After=network-online.target + +Requires=postgresql.service +After=postgresql.service + +[Service] +Type=oneshot +User=brep +Group=brep +ExecStart=/home/brep/bin/brep-startup + +[Install] +WantedBy=default.target diff --git a/etc/private/install/vm-gen-service b/etc/private/install/vm-gen-service new file mode 100755 index 0000000..c4938c2 --- /dev/null +++ b/etc/private/install/vm-gen-service @@ -0,0 +1,204 @@ +#! /usr/bin/env bash + +# Generate systemd .service file for QEMU/KVM virtual machines. +# +# Normally the machines are run from a dedicated user account with its home +# directory containing all the relevant files (management scripts, images, +# configurations, and sockets). However, this can be overriden with the +# following options +# +# --home <dir> +# The virtual machines "home" directory. If unspecified, the user's home +# directory is assumed. +# +# --bin <dir> +# The virtual machines management scripts directory. If unspecified, +# <home>/bin is assumed. If specified as relative, then assumed relative +# to <home>. +# +# --etc <dir> +# The virtual machines configuration files directory. If unspecified, +# <home>/etc is assumed. If specified as relative, then assumed relative +# to <home>. +# +# --var <dir> +# The virtual machines image files directory. If unspecified, <home>/var is +# assumed. If specified as relative, then assumed relative to <home>. +# +# --run <dir> +# The virtual machines sockets directory. If unspecified, <home>/run is +# assumed. If specified as relative, then assumed relative to <home>. +# +# If <user> is unspecified, the current user is assumed. If <group> is +# unspecified, the user's primary group is assumed. +# +usage="usage: $0 [<options>] [<user>] [<group>]" + +owd="$(pwd)" +trap "{ cd '$owd'; exit 1; }" ERR +set -o errtrace # Trap in functions. + +function info () { echo "$*" 1>&2; } +function error () { info "$*"; exit 1; } + +home= +bin= +etc= +var= +run= + +while [ "$#" -gt 0 ]; do + case "$1" in + --home) + shift + home="$1" + shift + ;; + --bin) + shift + bin="$1" + shift + ;; + --etc) + shift + etc="$1" + shift + ;; + --var) + shift + var="$1" + shift + ;; + --run) + shift + run="$1" + shift + ;; + *) + break + ;; + esac +done + +user="$1" + +if [ -z "$user" ]; then + user="$(id -un)" +fi + +group="$2" + +if [ -z "$group" ]; then + group="$(id -un "$user")" +fi + +if [ -z "$home" ]; then + home="$(eval echo ~$user)" +fi + +function complete_dir () # <default> <home> <dir> +{ + local r + if [ -z "$3" ]; then + r="$2/$1" + elif [ "${3:0:1}" != "/" ]; then + r="$2/$3" + else + r="$3" + fi + echo "$(realpath --no-symlinks --canonicalize-missing "$r")" +} + +bin="$(complete_dir bin "$home" "$bin")" +etc="$(complete_dir etc "$home" "$etc")" +var="$(complete_dir var "$home" "$var")" +run="$(complete_dir run "$home" "$run")" + +name="vm-$user" +file="$name@.service" + +# Thinks that must be \-escaped: +# +# - $ (including in comments) +# - \ (e.g., in line continuations) +# +cat <<EOF >"$file" +# $file -- QEMU/KVM machine service template for systemd +# +# user: $user +# group: $group +# bin: $bin +# etc: $etc +# var: $var +# run: $run +# +# To install: +# +# sudo cp $file /etc/systemd/system/ +# sudo chmod 644 /etc/systemd/system/$file +# +# cp ... $var/<machine>.img +# nano $etc/<machine>.conf # Specify RAM, CPU, TAP, MAC, etc. +# +# sudo systemctl start $name@<machine> +# sudo systemctl status $name@<machine> +# login-machine $run/<machine>-con.sock +# sudo systemctl stop $name@<machine> +# +# sudo systemctl enable $name@<machine> + +[Unit] +Description=QEMU/KVM virtual machine %I + +Wants=network-online.target +#After=network-online.target +After=multi-user.target + +[Service] +User=$user +Group=$user +UMask=0007 +WorkingDirectory=~ + +Environment=CPU=1 +Environment=RAM=2G + +# These MUST be specific in EnvironmentFile! +# +#Environment=TAP= +#Environment=MAC= + +# Note that using variable expansion in EnvironmentFile does not work (at +# least not with systemd 229). +# +EnvironmentFile=$etc/%i.conf + +# Note that the first word of ExecStart cannot contain variable expansions. +# +ExecStart=$bin/vm-start \\ + --cpu \${CPU} \\ + --ram \${RAM} \\ + --tap \${TAP} \\ + --mac \${MAC} \\ + --monitor $run/%i-mon.sock \\ + --console $run/%i-con.sock \\ + $var/%i.img + +ExecStop=$bin/vm-stop $run/%i-mon.sock + +# Make systemd wait for ExecStop completion. +# +KillMode=none +TimeoutStopSec=60 + +[Install] +WantedBy=multi-user.target +EOF + +info "generated $file for" +info " user: $user" +info " group: $group" +info " bin: $bin" +info " etc: $etc" +info " var: $var" +info " run: $run" -- cgit v1.1

From f42eb41a5164780ac8bf5934d0fa6278a6ace6f0 Mon Sep 17 00:00:00 2001 From: Karen Arutyunov Date: Thu, 3 Sep 2020 11:44:20 +0200 Subject: Initial support for private brep instance setup --- etc/private/install/brep-apache2.conf | 99 +++++++ etc/private/install/brep-install | 433 +++++++++++++++++++++++++++++++ etc/private/install/brep-load.service | 11 + etc/private/install/brep-load.timer | 33 +++ etc/private/install/brep-logrotate | 20 ++ etc/private/install/brep-module.conf | 387 +++++++++++++++++++++++++++ etc/private/install/brep-startup | 67 +++++ etc/private/install/brep-startup.service | 17 ++ etc/private/install/vm-gen-service | 204 +++++++++++++++ 9 files changed, 1271 insertions(+) create mode 100644 etc/private/install/brep-apache2.conf create mode 100755 etc/private/install/brep-install create mode 100644 etc/private/install/brep-load.service create mode 100644 etc/private/install/brep-load.timer create mode 100644 etc/private/install/brep-logrotate create mode 100644 etc/private/install/brep-module.conf create mode 100755 etc/private/install/brep-startup create mode 100644 etc/private/install/brep-startup.service create mode 100755 etc/private/install/vm-gen-service (limited to 'etc/private/install') diff --git a/etc/private/install/brep-apache2.conf b/etc/private/install/brep-apache2.conf new file mode 100644 index 0000000..99186d1 --- /dev/null +++ b/etc/private/install/brep-apache2.conf @@ -0,0 +1,99 @@ +# Keep in the main server configuration context. This way the directive will +# be in effect during module initialization and request handling. +# +# Note that initialization log messages are written to the main server log +# file (/var/log/apache2/error.log), and request handling messages to the +# virtual server log file (/var/www/brep/log/error.log). +# +LogLevel brep:info + + + #ServerName + #ServerAdmin + + #DocumentRoot /var/www/brep/public + #Options +Indexes + + AddOutputFilterByType DEFLATE application/xhtml+xml + AddOutputFilterByType DEFLATE text/manifest + AddOutputFilterByType DEFLATE text/plain + AddOutputFilterByType DEFLATE text/css + + Alias "/1" "/var/brep/bpkg/pkg/1" + + ErrorLog /var/www/brep/log/error.log + CustomLog /var/www/brep/log/access.log combined + + # brep configuration + # + + # Load the brep module. + # + + LoadModule brep_module /home/brep/install/libexec/brep/mod_brep.so + + + # Repository email. This email is used for the From: header in emails send + # by brep (for example, build failure notifications). + # + #brep-email + + # Repository host. It specifies the scheme and the host address (but not the + # root path; see brep-root below) that will be used whenever brep needs to + # construct an absolute URL to one of its locations (for example, a link to + # a build log that is being send via email). + # + #brep-host http:// + + # Repository root. This is the part of the URL between the host name and the + # start of the repository. For example, root value /pkg means the repository + # URL is http://example.org/pkg/. Specify / to use the web server root + # (e.g., http://example.org/). If using a different repository root, don't + # forget to also change Location and Alias directives below. + # + brep-root / + + + SetHandler brep + + + DirectoryIndex disabled + DirectorySlash Off + + + + # Brep module configuration. If you prefer, you can paste the contents of + # this file here. However, you will need to prefix every option with + # 'brep-'. + # + brep-conf /home/brep/config/brep-module.conf + + # Static brep content (CSS files). + # + + Error "mod_alias is not enabled" + + + # Note: trailing slashes are important! + # + Alias /@/ /home/brep/install/share/brep/www/ + + + Require all granted + + + # brep config override (must come after). + # + + SetHandler none + + DirectoryIndex enabled + DirectorySlash On + + + + + Options Indexes FollowSymLinks + AllowOverride None + Require all granted + diff --git a/etc/private/install/brep-install b/etc/private/install/brep-install new file mode 100755 index 0000000..0c6c2e6 --- /dev/null +++ b/etc/private/install/brep-install @@ -0,0 +1,433 @@ +#! /usr/bin/env bash + +# file : etc/private/install/brep-install +# license : MIT; see accompanying LICENSE file + +# Setup HTTP-only brep instance with unsigned package submission support via +# direct repository publishing (brep/handler/submit/submit-pub). +# +# NOTE: this setup should only be used in private/trusted environments. +# +# Unless the --setup option is specified, create the 'brep' group and user and +# re-run itself with the --setup option under this user. In the setup mode +# install and configure the brep instance, automating the instructions from +# the INSTALL file, including: +# +# - Build the build2 toolchain (installing it to /usr/local/) and brep +# (installing it to ~brep/install/). +# +# - Install PostgreSQL and create brep users/databases. +# +# - Installing Apache2 and configure HTTP server with the brep module. +# +# Note that the script is written for use on Debian-based distributions so you +# will need to adjust it to match other distributions or operating systems. +# +# Options: +# +# --mount +# +# Mount the virtio-9p device with id 'state' as the /var/brep directory. +# This directory is expected to either contain the pkg repository or be +# empty, in which case an empty repository will be automatically +# initialized. If this option is unspecified, the directory will be created +# in the local filesystem. +# +# --brep-user +# +# User and group ids to use when creating the 'brep' group and user. If +# unspecified, 63700 is used. +# +# --setup +# +# Install and configure the brep instance, assuming that the 'brep' user +# already exists and this script is executed as this user. +# +usage="Usage: $0 []" + +# build2 toolchain repository certificate fingerprint. Note: this is a +# repository the toolchain installation script downloads the build2 packages +# from. +# +toolchain_repo_cert_fp="86:BA:D4:DE:2C:87:1A:EE:38:C7:F1:64:7F:65:77:02:15:79:F3:C4:83:C0:AB:5A:EA:F4:F7:8C:1D:63:30:C6" +#toolchain_repo_cert_fp="37:CE:2C:A5:1D:CF:93:81:D7:07:46:AD:66:B3:C3:90:83:B8:96:9E:34:F0:E7:B3:A2:B0:6C:EF:66:A4:BE:65" + +# brep package repository URL and certificate fingerprint. +# +#brep_repo_url="https://pkg.cppget.org/1/alpha" +#brep_repo_cert_fp="86:BA:D4:DE:2C:87:1A:EE:38:C7:F1:64:7F:65:77:02:15:79:F3:C4:83:C0:AB:5A:EA:F4:F7:8C:1D:63:30:C6" +brep_repo_url="https://stage.build2.org/1" +brep_repo_cert_fp="37:CE:2C:A5:1D:CF:93:81:D7:07:46:AD:66:B3:C3:90:83:B8:96:9E:34:F0:E7:B3:A2:B0:6C:EF:66:A4:BE:65" + +owd=`pwd` +trap "{ exit 1; }" ERR +trap "{ cd $owd; }" EXIT +set -o errtrace # Trap in functions. + +function info () { echo "$*" 1>&2; } +function error () { info "error: $*"; exit 1; } + +# Trace a command line, quoting empty arguments as well as those that contain +# spaces. +# +function trace () # ... +{ + local s="+" + while [ "$#" -gt 0 ]; do + if [ -z "$1" -o -z "${1##* *}" ]; then + s="$s \"$1\"" + else + s="$s $1" + fi + + shift + done + + info "$s" +} + +# Trace and run a command. +# +run () # ... +{ + trace "$@" + "$@" +} + +# The chosen fixed id for the 'brep' user. Note: must match the id of the +# 'brep' user on the host. +# +# Note that Linux assigns the [0 99] range for the statically allocated system +# users and [100 499] -- for dynamic allocations by administrators and post- +# install scripts. Debian, in turn, assigns the [100 999] range for the +# dynamically allocated system users and [60000 64999] -- for statically +# allocated on demand "obscure package users". +# +brep_id=63700 # Update the README file on change. + +# Parse the command line options and, while at it, compose the options array +# for potential re-execution as the 'brep' user. +# +mount= +setup= +ops=() + +while [ "$#" -gt 0 ]; do + case "$1" in + --mount) + mount=true + ops+=("$1") + shift + ;; + --brep-user) + shift + brep_id="$1" + shift + ;; + --setup) + setup=true + shift + ;; + *) + break # The end of options is encountered. + ;; + esac +done + +if [ "$#" -ne 0 ]; then + error "$usage" +fi + +scr_exe="$(realpath "${BASH_SOURCE[0]}")" +scr_dir="$(dirname "$scr_exe")" + +# Unless we are not in the setup mode, non-interactively add the 'brep' +# user/group and re-execute the script in the setup mode as this user. +# +if [ ! "$setup" ]; then + run sudo addgroup --gid "$brep_id" brep + + run sudo adduser --uid "$brep_id" --gid "$brep_id" --disabled-password \ + --gecos "" brep + + run sudo tee -a /etc/sudoers.d/brep >/dev/null </dev/null </dev/null +run psql -d brep_build -c "$q" >/dev/null +run psql -d brep_submit_package -c "$q" >/dev/null + +run sudo sudo -u www-data psql -d brep_package -c "$q" >/dev/null +run sudo sudo -u www-data psql -d brep_build -c "$q" >/dev/null + +# Setup the connection between the databases. +# +run sudo sudo -u postgres psql -d brep_build </dev/null +run psql -d brep_build -c 'SELECT package_name FROM build' >/dev/null +run psql -d brep_build -c 'SELECT DISTINCT name FROM build_package' >/dev/null +run psql -d brep_submit_package -c 'SELECT canonical_name, summary FROM repository' >/dev/null + +# Setup executing the brep-startup script on boot. +# +run sudo cp "$scr_dir/brep-startup.service" /etc/systemd/system/ + +run sudo systemctl start brep-startup.service # Make sure there are no issues. +run sudo systemctl enable brep-startup.service + +# Prepare directories for the package submission service. +# +run mkdir submit-data +run mkdir submit-temp +run setfacl -m g:www-data:rwx submit-data +run setfacl -m g:www-data:rwx submit-temp + +# Make the Apache2 user owned directories fully accessible by the 'brep' user +# (which the submit-pub submission handler will run as). +# +run setfacl -dm g:brep:rwx submit-data +run setfacl -dm g:brep:rwx submit-temp + +# Add the Apache2 user to sudoers, so the submission handler can re-execute +# itself as the 'brep' user. +# +run sudo tee -a /etc/sudoers.d/www-data >/dev/null </dev/null + +# Make sure the Apache2 service depends on PostgreSQL and +# brep-startup.service, so that they are started in proper order. +# +run sudo mkdir -p /etc/systemd/system/apache2.service.d/ +run sudo tee /etc/systemd/system/apache2.service.d/postgresql.conf >/dev/null </dev/null </dev/null </dev/null || ec="$?" + +if [ "$ec" -ne 3 ]; then + error "exit code 3 (unit is not active) is expected instead of $ec" +fi + +# Verify that Apache2 is started after PostgreSQL is started. +# +run sudo systemctl start postgresql + +run sleep 3 +run sudo systemctl status apache2 >/dev/null + +# Setup periodic loader execution. +# +run sudo cp "$scr_dir/brep-load.service" /etc/systemd/system/ +run sudo cp "$scr_dir/brep-load.timer" /etc/systemd/system/ + +run sudo systemctl start brep-load.service # Make sure there are no issues. + +run sudo systemctl start brep-load.timer +run sudo systemctl status brep-load.timer >/dev/null +run sudo systemctl enable brep-load.timer +run sudo systemctl status brep-load.timer >/dev/null diff --git a/etc/private/install/brep-load.service b/etc/private/install/brep-load.service new file mode 100644 index 0000000..272a124 --- /dev/null +++ b/etc/private/install/brep-load.service @@ -0,0 +1,11 @@ +[Unit] +Description=brep repository loader service + +[Service] +Type=oneshot +User=brep +Group=brep +ExecStart=/home/brep/install/bin/brep-load /home/brep/config/loadtab + +[Install] +WantedBy=default.target diff --git a/etc/private/install/brep-load.timer b/etc/private/install/brep-load.timer new file mode 100644 index 0000000..1bf78c4 --- /dev/null +++ b/etc/private/install/brep-load.timer @@ -0,0 +1,33 @@ +[Unit] +Description=brep repository loader timer +RefuseManualStart=no +RefuseManualStop=no + +# Note that due to brep-startup service's oneshot type, this unit won't be +# started until the brep-startup process exits successfully. +# +# Also note that if brep-startup fails and is restarted manually, similar to +# services, the timer is not started automatically. Instead, it has to be +# started manually with `systemctl start brep-load.timer`. +# +Requires=brep-startup.service +After=brep-startup.service + +[Timer] +Unit=brep-load.service + +# Don't keep track of the timer across reboots. +# +Persistent=false + +# Start the timer for the first time. +# +OnBootSec=1 + +# Then wait 4-5 seconds until the next run. +# +OnUnitInactiveSec=4 +AccuracySec=1 + +[Install] +WantedBy=timers.target diff --git a/etc/private/install/brep-logrotate b/etc/private/install/brep-logrotate new file mode 100644 index 0000000..67c7c90 --- /dev/null +++ b/etc/private/install/brep-logrotate @@ -0,0 +1,20 @@ +/var/www/brep/log/*.log { + weekly + missingok + rotate 4 + compress + delaycompress + notifempty + create 640 root adm + sharedscripts + postrotate + if /etc/init.d/apache2 status > /dev/null ; then \ + /etc/init.d/apache2 reload > /dev/null; \ + fi; + endscript + prerotate + if [ -d /etc/logrotate.d/httpd-prerotate ]; then \ + run-parts /etc/logrotate.d/httpd-prerotate; \ + fi; \ + endscript +} diff --git a/etc/private/install/brep-module.conf b/etc/private/install/brep-module.conf new file mode 100644 index 0000000..0bff58d --- /dev/null +++ b/etc/private/install/brep-module.conf @@ -0,0 +1,387 @@ +# Configuration file for the brep module (note: this is not an apache2 .conf +# file but it can be converted to one by prefixing all the options with +# brep-). See brep(1) for detailed description of each configuration option. +# Commented out options indicate their default values. +# +# Besides being parsed by the brep module, this file may also be parsed by +# brep utilities that are normally only interested in the subset of the +# options. To simplify skipping of unrecognized, this file must always have an +# option name and its value on the same line. +# + +# Package search page title. It is placed inside XHTML5 element. +# +# search-title Packages + + +# Web page logo. It is displayed in the page header aligned to the left edge. +# The value is treated as an XHTML5 fragment. +# +# logo "" + + +# Web page menu. Each entry is displayed in the page header in the order +# specified and aligned to the right edge. A link target that starts with '/' +# or contains ':' is used as is. Otherwise, it is prefixed with the repository +# web interface root. +# +menu Packages= +# menu Builds=?builds +# menu Configs=?build-configs +menu Submit=?submit +# menu CI=?ci +menu About=?about + + +# Number of packages per page. +# +# search-page-entries 20 + + +# Number of pages in navigation (pager). +# +# search-pages 5 + + +# Number of package description characters to display in brief pages. +# +# package-description 500 + + +# Number of package changes characters to display in brief pages. +# +# package-changes 5000 + + +# The package database connection configuration. By default, brep will try to +# connect to the local instance of PostgreSQL with the operating system- +# default mechanism (Unix-domain socket, etc) and use operating system +# (login) user name and the database called 'brep_package'. If the role name +# is not empty then the login user will be switched (with SET ROLE) to this +# user prior to executing any statements. If not specified, then 'brep' is +# used. See brep(1) for details. +# +# package-db-user +# package-db-role brep +# package-db-password +# package-db-name brep_package +# package-db-host +# package-db-port + + +# The maximum number of concurrent package database connections per web server +# process. If 0, then no limitation is applied. +# +# package-db-max-connections 5 + + +# The maximum number of times to retry package database transactions in the +# face of recoverable failures (deadlock, loss of connection, etc). +# +# package-db-retry 10 + + +# Build configuration file. If not specified (default), then the package +# building functionality will be disabled. If specified, then the build +# database must be configured (see next). Note: must be an absolute path. +# +# build-config + + +# Number of build configurations per page. +# +# build-config-page-entries 20 + + +# Number of pages in navigation (pager). +# +# build-config-pages 5 + + +# Directory containing build bot agent public keys. If specified, then brep +# will perform agent authentication and will reject build results from +# unauthenticated ones. If not specified, then build results are accepted from +# all agents (which will be a security risk if the brep instance is publicly +# accessible). +# +# The directory is expected to contain one PEM-encoded public key per file with +# the .pem extension. All other files and subdirectories are ignored. The brep +# instance needs to be restarted after adding new key files for the changes to +# take effect. +# +# build-bot-agent-keys + + +# Number of builds per page. +# +# build-page-entries 20 + + +# Number of pages in navigation (pager). +# +# build-pages 5 + + +# Time to wait before considering a package for a forced rebuild. Must be +# specified in seconds. Default is 10 minutes. +# +# build-forced-rebuild-timeout 600 + + +# Time to wait before considering a package for a normal rebuild. Must be +# specified in seconds. Default is 24 hours. +# +# build-normal-rebuild-timeout 86400 + + +# Alternative package rebuild timeout to use instead of the normal rebuild +# timeout (see the build-normal-rebuild-timeout option for details) during +# the specified time interval. Must be specified in seconds. Default is the +# time interval length. +# +# The alternative rebuild timeout can be used to "pull" the rebuild window to +# the specified time of day, for example, to optimize load and/or power +# consumption of the build infrastructure (off-work hours, solar, off-peak +# electricity tariffs, etc). A shorter than the time interval rebuild timeout +# can also be used to force continuous rebuilds, for example, to shake out +# flaky tests. Note also that if the alternative rebuild timeout is greater +# than the normal rebuild timeout, then this will result in slower rebuilds +# during the alternative time interval. In this case, if the build +# infrastructure is monitored for delayed package builds, then the alternative +# rebuild timeout should only be made slightly greater than the normal timeout +# (see brep-monitor(1) for details). +# +# The time interval boundaries must be specified as times of day (in the local +# timezone) in the <hours>:<minutes> form. If the stop time is less than the +# start time then the interval extends through midnight. The start and stop +# times must both be either specified or absent. If unspecified, then no +# alternative rebuild timeout will be used. +# +# build-alt-rebuild-timeout +# build-alt-rebuild-start +# build-alt-rebuild-stop + + +# The maximum size of the build task request manifest accepted. Note that the +# HTTP POST request body is cached to retry database transactions in the face +# of recoverable failures (deadlock, loss of connection, etc). Default is +# 100K. +# +# build-task-request-max-size 102400 + + +# Time to wait before considering the expected task result lost. Must be +# specified in seconds. Default is 3 hours. +# +# build-result-timeout 10800 + + +# The maximum size of the build result manifest accepted. Note that the HTTP +# POST request body is cached to retry database transactions in the face of +# recoverable failures (deadlock, loss of connection, etc). Default is 10M. +# +# build-result-request-max-size 10485760 + + +# The build database connection configuration. By default, brep will try to +# connect to the local instance of PostgreSQL with the operating system-default +# mechanism (Unix-domain socket, etc) and use operating system (login) user +# name and the database called 'brep_build'. If the role name is not empty +# then the login user will be switched (with SET ROLE) to this user prior +# to executing any statements. If not specified, then 'brep' is used. See +# brep(1) for details. +# +# build-db-user +# build-db-role brep +# build-db-password +# build-db-name brep_build +# build-db-host +# build-db-port + + +# The maximum number of concurrent build database connections per web server +# process. If 0, then no limitation is applied. +# +# build-db-max-connections 5 + + +# The maximum number of times to retry build database transactions in the +# face of recoverable failures (deadlock, loss of connection, etc). +# +# build-db-retry 10 + + +# The openssl program to be used for crypto operations. You can also specify +# additional options that should be passed to the openssl program with +# openssl-option. If the openssl program is not explicitly specified, then brep +# will use openssl by default. +# +# openssl openssl + + +# Additional option to be passed to the openssl program (see openssl for +# details). Repeat this option to specify multiple openssl options. +# +# openssl-option + + +# Environment variable to be set (<name>=<value>) or unset (just <name>) for +# the openssl program (see openssl for details). Repeat this option to specify +# multiple openssl variables. Note that unspecified variables are inherited +# from the web server process. +# +# You need to at least set the RANDFILE environment variable to change the +# default location of the openssl program seed file and maybe also the +# OPENSSL_CONF variable if you would like to use a custom openssl configuration +# file. +# +# openssl-envvar RANDFILE=/home/brep/www-data-openssl.rnd +# openssl-envvar OPENSSL_CONF=/home/brep/www-data-openssl.cnf +# +# To create www-data-openssl.rnd with suitable permissions, run (as user brep): +# +# $ touch www-data-openssl.rnd +# $ setfacl -b -m g:www-data:rw www-data-openssl.rnd +# + + +# The directory to save final submission data to. If unspecified, the package +# submission functionality will be disabled. If specified, then submit-temp +# must be specified as well. +# +# Note that the directory path must be absolute and the directory itself must +# exist and have read, write, and execute permissions granted to the user that +# runs the web server. +# +submit-data /home/brep/submit-data + + +# The directory to save temporary submission data to. Must be specified if the +# package submission functionality is enabled. +# +# Note that this directory must be on the same filesystem and satisfy the same +# requirements as submit-data. It is also the user's responsibility to clean +# it up after an unclean web server shutdown. +# +submit-temp /home/brep/submit-temp + + +# The maximum size of the submission data accepted. Note that currently the +# entire submission request is read into memory. Default is 10M. +# +# 100M. +# +submit-max-size 104857600 + + +# The package submission form fragment. If specified, then its contents are +# treated as an XHTML5 fragment that is inserted into the <body> element of +# the submission page. If unspecified, then no submission page will be +# displayed. Note that the file path must be absolute. +# +submit-form /home/brep/install/share/brep/www/submit.xhtml + + +# The package submission email. If specified, the submission request and +# result manifests will be sent to this address. +# +# submit-email + + +# The handler program to be executed on package submission. The handler is +# executed as part of the submission request and is passed additional +# arguments that can be specified with submit-handler-argument followed by +# the absolute path to the submission directory. Note that the program path +# must be absolute. +# +submit-handler /home/brep/install/bin/brep-submit-pub + + +# Additional arguments to be passed to the submission handler program (see +# submit-handler for details). Repeat this option to specify multiple +# arguments. +# +submit-handler-argument --user +submit-handler-argument brep +submit-handler-argument --result-url +submit-handler-argument http://unknown +submit-handler-argument /home/brep/install/bin/brep-load +submit-handler-argument --db-name=brep_submit_package +submit-handler-argument /var/brep/bpkg/pkg + + +# The handler program timeout in seconds. If specified and the handler does +# not exit in the alloted time, then it is killed and its termination is +# treated as abnormal. +# +submit-handler-timeout 120 + + +# The directory to save CI request data to. If unspecified, the package CI +# functionality will be disabled. +# +# Note that the directory path must be absolute and the directory itself must +# exist and have read, write, and execute permissions granted to the user that +# runs the web server. +# +# ci-data + + +# The package CI form fragment. If specified, then its contents are treated as +# an XHTML5 fragment that is inserted into the <body> element of the CI page. +# If unspecified, then no CI page will be displayed. Note that the file path +# must be absolute. +# +# ci-form + + +# The package CI email. If specified, the CI request and result manifests will +# be sent to this address. +# +# ci-email + + +# The handler program to be executed on CI request. The handler is executed as +# part of the HTTP request and is passed additional arguments that can be +# specified with ci-handler-argument followed by the absolute path to the CI +# request directory. Note that the program path must be absolute. +# +# ci-handler + + +# Additional arguments to be passed to the CI handler program (see ci-handler +# for details). Repeat this option to specify multiple arguments. +# +# ci-handler-argument + + +# The CI handler program timeout in seconds. If specified and the handler does +# not exit in the allotted time, then it is killed and its termination is +# treated as abnormal. +# +# ci-handler-timeout + + +# The default view to display for the global repository root. The value is one +# of the supported services (packages, builds, submit, ci, etc). Default is +# packages. +# +# root-global-view packages + + +# The default view to display for the tenant repository root. The value is one +# of the supported services (packages, builds, submit, ci, etc). Default is +# packages. +# +# root-tenant-view packages + + +# Name to call the tenant values on web pages. If not specified, then 'tenant' +# is used. +# +# tenant-name tenant + + +# Trace verbosity. Disabled by default. +# +# verbosity 0 diff --git a/etc/private/install/brep-startup b/etc/private/install/brep-startup new file mode 100755 index 0000000..80cb255 --- /dev/null +++ b/etc/private/install/brep-startup @@ -0,0 +1,67 @@ +#! /usr/bin/env bash + +# file : etc/private/install/brep-startup +# license : MIT; see accompanying LICENSE file + +# (Re-)initialize the brep private instance, normally on the machine startup. +# +# Specifically: +# +# - Create the pkg repository and symlink to it, unless already exists. +# +# - Migrate the brep databases as a sanity check. +# +# - Adjust the brep module configuration file using the current host name/IP. +# +# - Generate the loadtab using the current host name/IP and run the loader. +# +trap "{ exit 1; }" ERR + +# Create the pkg repository, if required. +# +d=/var/brep/bpkg + +if [ ! -L "$d/pkg" ]; then + rd="$(date "+pkg-%Y%m%d-%H%M%S-%N")" + + mkdir -p "$d/$rd/1" + ln -s "$rd" "$d/pkg" +fi + +r="$d/pkg/1" + +if [ ! -f "$r/repositories.manifest" ]; then + echo ": 1" >"$r/repositories.manifest" +fi + +if [ ! -f "$r/packages.manifest" ]; then + bpkg rep-create -q "$r" +fi + +# Migrate the databases. +# +"$HOME/install/bin/brep-migrate" package +"$HOME/install/bin/brep-migrate" build +"$HOME/install/bin/brep-migrate" -n brep_submit_package package + +# Deduce the machine host name. +# +h="$(hostname -f)" +if [ "$h" == "localhost" ]; then + h="$(hostname -I | sed 's/ *$//')" # Strip the potential trailing space(s). +fi + +# Adjust the submission result URL host name in the brep module configuration +# file. +# +sed --in-place -re \ +"\$!N;s%^\s*(submit-handler-argument\s+--result-url\s*\\n)\ +\s*(submit-handler-argument\s+https?://)[^/]+(.*)\$%\1\2$h\3%;P;D" \ +"$HOME/config/brep-module.conf" + +# (Re-)generate the loadtab file and reload the repository. +# +f="$HOME/config/loadtab" + +echo "http://$h/1 private cache:$r" >"$f" +"$HOME/install/bin/brep-load" "$f" diff --git a/etc/private/install/brep-startup.service b/etc/private/install/brep-startup.service new file mode 100644 index 0000000..a3dc546 --- /dev/null +++ b/etc/private/install/brep-startup.service @@ -0,0 +1,17 @@ +[Unit] +Description=brep instance initialization service + +Wants=network-online.target +After=network-online.target + +Requires=postgresql.service +After=postgresql.service + +[Service] +Type=oneshot +User=brep +Group=brep +ExecStart=/home/brep/bin/brep-startup + +[Install] +WantedBy=default.target diff --git a/etc/private/install/vm-gen-service b/etc/private/install/vm-gen-service new file mode 100755 index 0000000..c4938c2 --- /dev/null +++ b/etc/private/install/vm-gen-service @@ -0,0 +1,204 @@ +#! /usr/bin/env bash + +# Generate systemd .service file for QEMU/KVM virtual machines. +# +# Normally the machines are run from a dedicated user account with its home +# directory containing all the relevant files (management scripts, images, +# configurations, and sockets). However, this can be overriden with the +# following options +# +# --home <dir> +# The virtual machines "home" directory. If unspecified, the user's home +# directory is assumed. +# +# --bin <dir> +# The virtual machines management scripts directory. If unspecified, +# <home>/bin is assumed. If specified as relative, then assumed relative +# to <home>. +# +# --etc <dir> +# The virtual machines configuration files directory. If unspecified, +# <home>/etc is assumed. If specified as relative, then assumed relative +# to <home>. +# +# --var <dir> +# The virtual machines image files directory. If unspecified, <home>/var is +# assumed. If specified as relative, then assumed relative to <home>. +# +# --run <dir> +# The virtual machines sockets directory. If unspecified, <home>/run is +# assumed. If specified as relative, then assumed relative to <home>. +# +# If <user> is unspecified, the current user is assumed. If <group> is +# unspecified, the user's primary group is assumed. +# +usage="usage: $0 [<options>] [<user>] [<group>]" + +owd="$(pwd)" +trap "{ cd '$owd'; exit 1; }" ERR +set -o errtrace # Trap in functions. + +function info () { echo "$*" 1>&2; } +function error () { info "$*"; exit 1; } + +home= +bin= +etc= +var= +run= + +while [ "$#" -gt 0 ]; do + case "$1" in + --home) + shift + home="$1" + shift + ;; + --bin) + shift + bin="$1" + shift + ;; + --etc) + shift + etc="$1" + shift + ;; + --var) + shift + var="$1" + shift + ;; + --run) + shift + run="$1" + shift + ;; + *) + break + ;; + esac +done + +user="$1" + +if [ -z "$user" ]; then + user="$(id -un)" +fi + +group="$2" + +if [ -z "$group" ]; then + group="$(id -un "$user")" +fi + +if [ -z "$home" ]; then + home="$(eval echo ~$user)" +fi + +function complete_dir () # <default> <home> <dir> +{ + local r + if [ -z "$3" ]; then + r="$2/$1" + elif [ "${3:0:1}" != "/" ]; then + r="$2/$3" + else + r="$3" + fi + echo "$(realpath --no-symlinks --canonicalize-missing "$r")" +} + +bin="$(complete_dir bin "$home" "$bin")" +etc="$(complete_dir etc "$home" "$etc")" +var="$(complete_dir var "$home" "$var")" +run="$(complete_dir run "$home" "$run")" + +name="vm-$user" +file="$name@.service" + +# Thinks that must be \-escaped: +# +# - $ (including in comments) +# - \ (e.g., in line continuations) +# +cat <<EOF >"$file" +# $file -- QEMU/KVM machine service template for systemd +# +# user: $user +# group: $group +# bin: $bin +# etc: $etc +# var: $var +# run: $run +# +# To install: +# +# sudo cp $file /etc/systemd/system/ +# sudo chmod 644 /etc/systemd/system/$file +# +# cp ... $var/<machine>.img +# nano $etc/<machine>.conf # Specify RAM, CPU, TAP, MAC, etc. +# +# sudo systemctl start $name@<machine> +# sudo systemctl status $name@<machine> +# login-machine $run/<machine>-con.sock +# sudo systemctl stop $name@<machine> +# +# sudo systemctl enable $name@<machine> + +[Unit] +Description=QEMU/KVM virtual machine %I + +Wants=network-online.target +#After=network-online.target +After=multi-user.target + +[Service] +User=$user +Group=$user +UMask=0007 +WorkingDirectory=~ + +Environment=CPU=1 +Environment=RAM=2G + +# These MUST be specific in EnvironmentFile! +# +#Environment=TAP= +#Environment=MAC= + +# Note that using variable expansion in EnvironmentFile does not work (at +# least not with systemd 229). +# +EnvironmentFile=$etc/%i.conf + +# Note that the first word of ExecStart cannot contain variable expansions. +# +ExecStart=$bin/vm-start \\ + --cpu \${CPU} \\ + --ram \${RAM} \\ + --tap \${TAP} \\ + --mac \${MAC} \\ + --monitor $run/%i-mon.sock \\ + --console $run/%i-con.sock \\ + $var/%i.img + +ExecStop=$bin/vm-stop $run/%i-mon.sock + +# Make systemd wait for ExecStop completion. +# +KillMode=none +TimeoutStopSec=60 + +[Install] +WantedBy=multi-user.target +EOF + +info "generated $file for" +info " user: $user" +info " group: $group" +info " bin: $bin" +info " etc: $etc" +info " var: $var" +info " run: $run" -- cgit v1.1