diff options
| author | Boris Kolpackov <boris@codesynthesis.com> | 2017-03-25 17:16:13 +0200 |
|---|---|---|
| committer | Boris Kolpackov <boris@codesynthesis.com> | 2017-03-25 17:16:13 +0200 |
| commit | 6aead8d49c6aa020ef6630d25f30df9df1ace5d2 (patch) | |
| tree | 48cad6fbfa19e0155c1d90ee452ec56c360ef8d9 | |
| parent | 864fc1c2e0597606ec8e2e04ae3150144aa3ed39 (diff) | |
Implement buildid monitoring and automatic reboot
| -rw-r--r-- | .gitignore | 1 | ||||
| -rwxr-xr-x | bootstrap | 22 | ||||
| -rwxr-xr-x | buildos | 45 | ||||
| -rw-r--r-- | doc/manual.cli | 30 | ||||
| -rwxr-xr-x | init | 2 |
5 files changed, 81 insertions, 19 deletions
@@ -1,3 +1,4 @@ *.cpio.gz +buildos-buildid buildos-initrd buildos-image @@ -366,6 +366,15 @@ fi # if [ "$stage" -le "5" ]; then + # Generate buildid and store it in /etc/os-release and in buildos-buildid. + # These are used by the monitor to detect when it's time to reboot. + # + # Note that /etc/os-release is a symlink to /usr/lib/os-release. + # + buildid="$(uuidgen)" + + sudo tee -a "$root/etc/os-release" >/dev/null <<<"BUILD_ID=\"$buildid\"" + # Install init and buildos monitor. # sudo install -m 755 ./init "$root/" @@ -379,6 +388,7 @@ if [ "$stage" -le "5" ]; then cd "$root" sudo cpio -o -H newc <<EOF | \ gzip -9 > "$owd/buildos-init.cpio.gz" +usr/lib/os-release init usr/sbin/buildos usr/lib/systemd/system/buildos.service @@ -391,19 +401,19 @@ EOF # Copy the kernel image next to the initramfs for convenience. # cp "$root/vmlinuz" buildos-image + echo "$buildid" >buildos-buildid subvol_snapshot -r "$root" "$root-5" fi # Test. # -# To test PXE boot, replace -kernel/-initrd with '-boot n'. -# - if [ ! -e /tmp/buildos-disk ]; then qemu-img create -f raw /tmp/buildos-disk 100M fi +# To test PXE boot, replace -kernel/-initrd/-append with '-boot n'. +# sudo kvm \ -m 8G \ -device "e1000,netdev=net0,mac=$macaddr" \ @@ -411,5 +421,7 @@ sudo kvm \ -device "virtio-scsi-pci,id=scsi" \ -device "scsi-hd,drive=disk1" \ -drive "if=none,id=disk1,file=/tmp/buildos-disk,format=raw" \ - -kernel buildos-image -initrd buildos-initrd \ - -append "buildos.smtp_relay=build2.org buildos.admin_email=admin@build2.org" + -boot n + +# -kernel buildos-image -initrd buildos-initrd \ +# -append "buildos.smtp_relay=build2.org buildos.admin_email=admin@build2.org" @@ -63,14 +63,53 @@ for v in "${cmdline[@]}"; do fi done +hname="$(hostname)" + +# Get the build id. +# +buildid="$(sed -n -re 's/^BUILD_ID="(.+)"$/\1/p' /etc/os-release)" + function email () # <subject> < <body> { - (echo -e "Subject: $1\n"; cat -) | sendmail -i "$admin_email" + (echo -e "Subject: [$hname] $1\n"; cat -) | sendmail -i "$admin_email" +} + +function restart () +{ + sendmail -q # Flush mail queue. + sleep 10 # Give any remaining mail chance to go through. + sudo systemctl reboot } -email "starting buildos monitor on $(hostname)" <<<"" +if [ -z "$buildid_url" ]; then + info "no buildos.buildid_url specified, not monitoring for new os builds" +fi + +email "starting buildos monitor" <<EOF +buildid: $buildid +buildid_url: $buildid_url +EOF while true; do info "monitoring..." - sleep 2 + + sleep 5 + + if [ -n "$buildid_url" ]; then + # Fetch the current id. While normally it will be a TFTP URL, it could also + # be HTTP(S) so we configure sensible behavior for that. + # + if id="$(curl -f -L -s -S "$buildid_url")"; then + if [ "$id" != "$buildid" ]; then + email "rebooting because of new os build" <<EOF +old_buildid: $buildid +new_buildid: $id +EOF + info "new os build ($id), rebooting..." + restart + fi + else + info "unable to fetch $buildid_url, will try again" + fi + fi done diff --git a/doc/manual.cli b/doc/manual.cli index 961627e..863841c 100644 --- a/doc/manual.cli +++ b/doc/manual.cli @@ -34,9 +34,17 @@ mode} and receive \i{build tasks} from their respective agents. \h1#boot|Booting| \c{buildos} is normally booted from the network using PXE but can also be -booted locally from initrd directly. +booted locally from the kernel image and initrd directly. -\h2#boot-net|Network| +\h2#boot-reboot|Reboot| + +Build OS can detect when the OS build has been updated and automatically +reboot the build host. This is achieved by polling the URL specified +with the \c{buildos.buildid_url} kernel command line parameter. It should +point to the \c{buildos-buildid} file that comes along the kernel image +and initrd. See \l{#boot-net Network Boot} for the usage example. + +\h2#boot-net|Network Boot| Here we assume that you have already established your PXE setup using PXELINUX. That is, you have configured a TFTP server that hosts the @@ -47,18 +55,20 @@ To setup PXE boot of \c{buildos}, perform the following steps: \ol| -\li|Copy the kernel image and initrd to the TFTP server. For example: +\li|Copy the Build OS \c{-image}, \c{-initrd}, and \c{-buildid} files to the +TFTP server. For example: \ # mkdir -p /var/lib/tftpboot/buildos -# cp buildos-image buildos-initrd /var/lib/tftpboot/buildos/ +# cp buildos-image buildos-initrd buildos-buildid \ + /var/lib/tftpboot/buildos/ \ | -\li|Assuming the host has MAC address \c{de:ad:be:ef:b8:da}, create a +\li|Assuming the host MAC address is \c{de:ad:be:ef:b8:da}, create a host-specific configuration file (or use \c{default} as the last path - component for a configuration that applies to all the hosts): + component for a configuration that applies to all hosts): \ # cat <<EOF >/var/lib/tftpboot/pxelinux.cfg/01-de-ad-be-ef-b8-da @@ -70,11 +80,13 @@ label buildos menu label buildos kernel /buildos/buildos-image initrd /buildos/buildos-initrd - append buildos.smtp_relay=example.org buildos.admin_email=admin@example.org + append buildos.smtp_relay=example.org buildos.admin_email=admin@example.org buildos.buildid_url=tftp://<tftp-host>/buildos/buildos-buildid EOF \ -| +Where \c{<tftp-host>} in \c{buildid_url} is the address of the TFTP server +(the same address as returned by the DHCP server to PXE clients). Note that +all the parameters in \c{append} must be specified on a single line.| \li|You can test the setup using QEMU/KVM, for example: @@ -88,7 +100,7 @@ $ sudo kvm \ || -\h2#boot-local|Local| +\h2#boot-local|Local Boot| During testing it is often useful to boot \c{buildos} directly from the kernel image and initrd files. As an example, here is how this can be done @@ -334,8 +334,6 @@ if [ -z "$machines" ]; then error fi -/bin/bash - # Hand off to systemd. But first arrange to keep console output (which # becomes tty1). # |