From b026efa58533e7ce1f04bf9fbb8eb5e7da19db65 Mon Sep 17 00:00:00 2001 From: Boris Kolpackov Date: Fri, 4 Sep 2020 13:03:41 +0200 Subject: Add private install README --- etc/private/README | 270 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 270 insertions(+) create mode 100644 etc/private/README (limited to 'etc/private') diff --git a/etc/private/README b/etc/private/README new file mode 100644 index 0000000..22255ae --- /dev/null +++ b/etc/private/README @@ -0,0 +1,270 @@ +This directory contains a virtual machine (VM) with a build2 repository web +interface (brep) installation for a private package repository plus a number +of scripts and configuration files for running this machine as a systemd +service. + +A brep installation consists of a web server (Apache2), a database server +(PostgreSQL), and a number of auxiliary processes (repository loader, +submission handler, etc). While all this can be installed and configured +manually (as described in brep/INSTALL), this VM makes it possible to quickly +setup a private repository. + +Note that the configuration offered by this VM is only suitable for a +private/trusted environment, normally either for personal use (host-local) +or for use within an organization's private network. Specifically: + + - The repository is accessed via HTTP. + + - The repository is not signed. + + - Submitted packages are published directly and without ownership + authentication. + + - The VM does not perform auto-update (since it does not assume the presence + of an Internet connection) and therefore may not have the latest security + updates. + +The below setup instructions are for host machines that run systemd-based +Linux distributions. Note, however, that it should be possible to use other +distributions or operating systems provided they are capable of running +QEMU/KVM virtual machines. The following utilities are expected to be +available on the host machine: + + - systemd >= 229 (systemd --version) + - bash >= 4.3 (base --version) + - qemu >= 2.5.0 (qemu-system-x86_64 --version) + - screen, socat (screen --version, socan -V) + +Consult your distribution's package manager if any of these utilities are +missing. + +The host machine is also expected to have KVM virtualization support as +well as at least 1G or RAM (2G recommended) and at least 5G of disk space +(4G for VM image and the rest for package storage) that can be dedicated +to the VM. + +Note: commands that start with the `#` prompt must be executed as root. + + +1. Create the brep user and group. + +In this setup, the VM image, scripts, etc., as well as the repository packages +are all kept in the home directory of the special brep user. In particular, +the packages are stored on the host machine (as opposed to inside the VM +image) and are shared with the VM (using the virtio-9p filesystem). As a +result, if necessary, you can manipulate the package repository from the host +machine (but see Step 6 below for the rules). This setup also makes it easier +to upgrade VM images by simply replacing the old one with a new. + +However, to make this arrangement work reliably, the brep user/group IDs on +the host machine must match those inside the VM. As a result, we create the +brep user with specific IDs: + +# addgroup --gid 63700 brep +# adduser --uid 63700 --gid 63700 --gecos '' --disabled-password brep + +Additionally, if your distribution requires users that are allowed to use KVM +to belong to a special group (normally kvm), then add the brep user to this +group: + +# usermod -G kvm brep + +If unsure whether this is required, skip this step and come back to it if you +get 'KVM: permission denied' error on Step 4. + + +2. Download and unpack the VM archive into the brep user's home directory. + +# su - brep +$ pwd +/home/brep + +$ curl -fO https://download.build2.org/X.Y.Z/linux-debian-N-brep-X.Y.Z.tar.xz +$ tar xf --strip-components=1 linux-debian-N-brep-X.Y.Z.tar.xz +$ ls +bin/ etc/ vm/ vm-brep@.service + + +3. Configure networking for the VM. + +This setup expects the VM to use bridged networking with a persistent tap +interface. This allows for a wide variety of configurations ranging between +host-local (private bridge without routing), subnet (private bridge with +routing/NAT), and local network (public bridge over host's Ethernet +adapter). In particular, the last configuration would make the repository +accessible from other machines on the same local network. + +The exact steps on how to setup bridged networking and create a persistent tap +interface depend the network manager used so consult your distribution's +documentation for details. The guide found in etc/systemd-networkd/README +shows how to setup the local network configuration mentioned above using the +systemd-networkd network manager available on most systemd-based +distributions. + +In the rest of this guide we assume that tap interface called tap0 is +appropriately configured and is owned by user/group brep. + + +4. Generate a MAC address and start the VM for testing. + +The recommended way to obtain a MAC address for the VM is to generate it +based on the host's Ethernet adapter address (see inside vm-gen-macaddress +for details): + +$ ~/bin/vm-gen-macaddress xx:yy:yy:yy:yy:yy 0 + +Where xx:yy:yy:yy:yy:yy is the MAC address of the host's Ethernet adapter that +can be viewed with the following command: + +# ip link show + +The address printed by vm-gen-macaddress will be in the 02:yy:yy:yy:yy:yy +form. + +If you are using a local network configuration, then now is a good time to +assign the VM its IP address and hostname. If you need to submit a request to +your network administrator, then the following text could serve as a template: + +"I would like to run a VM on the machine that needs to have its own IP + address and domain name (configured via DHCP. It will have fixed MAC address + (which was derived from 's physical Ethernet address; but you are + welcome to assign a different MAC address if desired). I would like this + machine to have the name if possible. + + This will be a QEMU/KVM virtual machine running as a systemd service. It will + use bridged networking with a tap interface." + +Where: + + host machine name, for example, myserver.lan (run hostname -f) + the generated mac address (02:yy:yy:yy:yy:yy) + VM machine name, for example, mybrep.lan + +Note that the VM is configured to receive its hostname from DHCP. Failed that, +the repository URL will use the IP address. + +Next, create the package repository directory and start the VM for testing +(replace 02:yy:yy:yy:yy:yy with the actual MAC address): + +$ mkdir -p state/bpkg +$ ~/bin/vm-start --stdio --tap tap0 --mac 02:yy:yy:yy:yy:yy vm/brep.img + +After booting, you will be presented with a login. Login as root with password +123 (VM command prompts are shown indented with two spaces). Then verify IP +address, hostname, and the network functionality: + + # ip addr show + # hostname -f + # ping example.org + +If everything appears correct, visit the repository web page with a browser +(for example, http://mybrep.lan). Check the About page to see the repository +URL. + +Try to submit a package (for example, from your development machine): + +$ bdep new hello +$ cd hello +$ git add . && git commit -m test +$ bdep init -C @test cc +$ bdep publish --control=none --repository http://mybrep.lan --force=snapshot + +Visit the repository web page and confirm the package is there. Then try to +build the submitted package from the repository: + +$ bpkg create -d test +$ bpkg build -d test hello@http://mybrep.lan/1 + +If everything is working fine, shut the VM down: + + # shutdown -h now + + +5. Setup the VM to run as a systemd service. + +To start the VM as a systemd service on host boot, perform the following +steps. + +First, create the VM configuration file (replace 02:yy:yy:yy:yy:yy with the +actual MAC address):: + +$ cat <vm/brep.conf +RAM=2G +CPU=1 +TAP=tap0 +MAC=02:yy:yy:yy:yy:yy +EOF + +Then configure the systemd service: + +# cp ~brep/vm-brep@.service /etc/systemd/system/ +# chmod 644 /etc/systemd/system/vm-brep@.service +# systemctl status vm-brep@brep +# systemctl start vm-brep@brep +# systemctl status vm-brep@brep + +If the VM fails to start, study the log for a possible cause: + +# journalctl -u vm-brep@brep + +If the VM has started successfully, perform the same verifications as on step +4 above. + +To login to the VM running as a systemd service (for example, to verify IP +and hostname) use the vm-login script (which uses screen to connect to the +VM's console): + +$ ~/bin/vm-login ~/brep-con.sock + +To close the login, press Ctrl-a k (or Ctrl-a a k if already running inside +screen). + +If everything functions correctly, verify the VM can be stopped: + +# systemctl stop vm-brep@brep +# systemctl status vm-brep@brep + +Finally, if desired, enable the VM service to start on boot: + +# systemctl enable vm-brep@brep + +After this you may also want to reboot the host machine and confirm the VM is +started on boot. + + +6. Managing the repository state. + +While you can submit packages to the repository using bdep-publish(1), you can +also add them manually. Also, packages can only be removed manually. + +The repository packages and metadata are stored in the ~brep/state/bpkg/pkg/ +directory. If you need to make any modifications in this directory, there are +two rules that you must follow: + + 1. You must stop the VM before making any modifications. + + 2. You must make any modification only as user brep. + +After performing the modifications, remove the 1/packages.manifest file to +trigger the repository metadata regeneration on the next VM startup. + +Putting it all together, the steps could look like this: + +# systemctl stop vm-brep@brep +# su - brep +$ cd state/bpkg/pkg +$ ... +$ rm 1/packages.manifest +$ exit +# systemctl start vm-brep@brep + +Note also that it's easy to break the repository with manual modifications. +For example, you may add a package that has an unmet dependency or remove a +package that still has some dependents. In this case, the repository web +interface will be unavailable. As a result, you may want to login into the VM +and verify the repository regeneration was successful: + +$ ~/bin/vm-login ~/brep-con.sock + + # systemctl status brep-startup + # journalctl -u brep-startup -- cgit v1.1