path: root/etc/private/README

This directory contains a virtual machine (VM) with a build2 repository web
interface (brep) installed and configured for a private package repository.
It also includes a number of scripts and configuration files for running
this VM as a systemd service.

A brep installation consists of a web server (Apache2), a database server
(PostgreSQL), and a number of auxiliary processes (repository loader,
submission handler, etc). While all this can be installed and configured
manually (as described in brep/INSTALL), this VM has everything pre-installed
and pre-configured which makes it possible to quickly get a private
repository up and running.

Note that the configuration offered by this VM is only suitable for a
private/trusted environment, normally either for personal use (host-local)
or for use within an organization's private network. Specifically:

  - The repository is accessed via HTTP.

  - The repository is not signed.

  - Submitted packages are published directly and without ownership
    authentication.

  - The VM does not auto-update (since it does not assume the presence of an
    Internet connection) and therefore may not have the latest security
    patches.

The below setup instructions are for host machines that run systemd-based
Linux distributions. Note, however, that it should be possible to use other
distributions or operating systems provided they are capable of running
QEMU/KVM virtual machines. The following utilities are expected to be
available on the host machine:

  - systemd >= 229    (systemd --version)
  - bash    >= 4.3    (bash --version)
  - qemu    >= 2.5.0  (qemu-system-x86_64 --version)
  - screen, socat     (screen --version, socat -V)

Consult your distribution's package manager if any of these utilities are
missing.

The host machine is also expected to have KVM virtualization support as
well as at least 1G or RAM (2G recommended) and at least 5G of disk space
(4G for VM image and the rest for package storage) that can be dedicated
to the VM.

Commands shown in this guide use several prompts with the following meaning:

  # -- must be executed as root on the host machine
  $ -- must be execute as user brep on the host machine
  > -- can be executed for testing on any other machine with build2 installed


1. Create the brep user and group
---------------------------------

In this setup, the VM image, scripts, etc., as well as the repository packages
are all kept in the home directory of the special user brep. In particular,
the packages are stored on the host machine (as opposed to inside the VM
image) and are shared with the VM (using the virtio-9p filesystem). As a
result, if necessary, you can manipulate the package repository from the host
machine (but see Step 6 below for the rules). This setup also makes it easier
to upgrade VM images by simply replacing the old image with a new (see Step 7
below for details).

However, to make this arrangement work reliably, the brep user/group IDs on
the host machine must match those inside the VM. As a result, we create the
brep user/group with specific IDs:

# groupadd --gid 63700 brep
# useradd --uid 63700 --gid 63700 --create-home brep
# usermod --lock brep # disable password login (if desired)

Additionally, if your distribution requires users that are allowed to use KVM
to belong to a special group (normally kvm), then add the brep user to this
group:

# usermod -G kvm brep

If unsure whether this is required, skip this step and come back to it if you
get the 'KVM: permission denied' error on Step 4.


2. Download and unpack the VM archive into the brep user's home directory
-------------------------------------------------------------------------

# su - brep
$ pwd
/home/brep

$ curl -fO https://download.build2.org/X.Y.Z/linux-debian-N-brep-X.Y.Z.tar.xz
$ sha256sum -b linux-debian-N-brep-X.Y.Z.tar.xz

Verify the checksum matches the one from https://build2.org/download.xhtml

$ tar -xf linux-debian-N-brep-X.Y.Z.tar.xz --strip-components=1
$ ls
bin/ etc/ vm/ vm-brep@.service README NEWS


3. Configure networking for the VM
----------------------------------

This setup expects the VM to use bridged networking with a persistent tap
interface. This allows for a wide variety of configurations ranging between
host-local (private bridge without routing), subnet (private bridge with
routing/NAT), and local area network (public bridge over host's Ethernet
adapter). In particular, the last configuration would make the repository
accessible from other machines on the same local network.

The exact steps on how to setup bridged networking and create a persistent tap
interface depend the network manager used thus consult your distribution's
documentation for details. The guide found in etc/systemd-networkd/README
shows how to setup the local area network configuration mentioned above using
the systemd-networkd network manager available on most systemd-based
distributions.

In the rest of this guide we assume that tap interface called tap0 is
appropriately configured and is owned by user/group brep.


4. Generate a MAC address and start the VM for testing
------------------------------------------------------

The recommended way to obtain a MAC address for the VM is to generate it based
on the address of the host's Ethernet adapter (see inside vm-gen-macaddress
for details):

$ ~/bin/vm-gen-macaddress xx:yy:yy:yy:yy:yy 0

Where xx:yy:yy:yy:yy:yy is the MAC address of the host's Ethernet adapter
which can can be viewed with the following command:

# ip link show

The address printed by vm-gen-macaddress will be in the 02:yy:yy:yy:yy:yy
form.

If you are using a local network configuration, then now is a good time to
assign the VM its IP address and hostname. If you need to submit a request to
your network administrator, then the following text could serve as a template:

"I would like to run a VM on the <host> machine that needs to have its own IP
 address and domain name (configured via DHCP). It will have fixed MAC address
 <mac> (which was derived from <host>'s physical Ethernet address; but you are
 welcome to assign a different MAC address if required). The DHCP client ID is
 the same as the MAC address. I would like this machine to have the <vm> name
 if possible.

 FYI, this is a QEMU/KVM virtual machine running as a systemd service. It
 will use bridged networking with a tap interface."

Where:

  <host>  host machine's name, for example, myserver.lan (run hostname -f)
  <mac>   the generated mac address (02:yy:yy:yy:yy:yy)
  <vm>    VM machine's name, for example, mybrep.lan

Note that the VM is configured to receive its hostname from DHCP server (the
DHCP protocol option 12, "Host Name"). Failed that, the repository URL will
use the IP address.

Next, create the package repository directory and start the VM for testing
(replace 02:yy:yy:yy:yy:yy with the actual MAC address):

$ mkdir -p state/bpkg
$ ~/bin/vm-start --stdio --tap tap0 --mac 02:yy:yy:yy:yy:yy vm/brep.img

After booting, you will be presented with a login. Login as root with password
123 (VM command prompts are shown indented with two spaces). Then verify IP
address, hostname, and the network functionality:

  # ip addr show
  # hostname -f
  # ping example.org

If everything appears correct, visit the repository web page with a browser
(for example, http://mybrep.lan). Check the About page to verify the
repository URL matches the hostname or IP address.

Try to submit a package (for example, from your development machine):

> bdep new hello
> cd hello
> git add . && git commit -m test
> bdep init -C @test cc
> bdep publish --control=none --repository http://mybrep.lan --force=snapshot

Visit the repository web page and confirm the package is there. Then try to
consume the submitted package from the repository:

> bpkg create -d test
> bpkg build -d test hello@http://mybrep.lan/1

If everything is working fine, shut the VM down:

  # shutdown -h now


5. Setup the VM to run as a systemd service
-------------------------------------------

To start the VM as a systemd service on host boot, perform the following
steps.

First, create the VM configuration file (replace 02:yy:yy:yy:yy:yy with the
actual MAC address)::

$ cat <<EOF >vm/brep.conf
RAM=2G
CPU=1
TAP=tap0
MAC=02:yy:yy:yy:yy:yy
EOF

Then configure the systemd service:

# cp ~brep/vm-brep@.service /etc/systemd/system/
# chmod 644 /etc/systemd/system/vm-brep@.service
# systemctl status vm-brep@brep
# systemctl start vm-brep@brep
# systemctl status vm-brep@brep

If the VM fails to start, study the logs for a possible cause:

# journalctl -u vm-brep@brep

If the VM has started successfully, perform the same verifications as on Step
4 above.

To login to the VM running as a systemd service (for example, to verify IP and
hostname) use the vm-login script (which uses screen(1) to connect to the VM's
console):

$ ~/bin/vm-login ~/brep-con.sock

Note that the screen may be blank (due to this being a serial console). In
this case, press Enter to see the login. To close the login, press 'Ctrl-a k'
(or 'Ctrl-a a k' if already running inside screen).

If everything functions correctly, verify the VM can be stopped:

# systemctl stop vm-brep@brep
# systemctl status vm-brep@brep

Finally, if desired, enable the VM service to start on boot:

# systemctl enable vm-brep@brep

After this you may also want to reboot the host machine and confirm the VM is
started on boot.


6. Manage the repository state
------------------------------

While you can submit packages to the repository using bdep-publish(1), they
can also be added them manually. Also, currently, packages can only be removed
manually.

The repository packages and metadata are stored in the ~brep/state/bpkg/pkg/
directory. If you need to make any modifications in this directory, there are
two rules that you must follow:

  1. You must stop the VM before making any modifications.

  2. You must make any modification only as user brep.

After performing the modifications, remove the 1/packages.manifest file to
trigger the repository metadata regeneration on the next VM startup. You can
also customize the repositories.manifest file in the same way. For example,
you could add cppget.org as a prerequisite repository for your private
repository.

Putting it all together, the steps could look like this:

# systemctl stop vm-brep@brep
# su - brep
$ cd state/bpkg/pkg/1
$ <make your changes here>
$ rm packages.manifest
$ exit
# systemctl start vm-brep@brep

Note also that it's easy to break the repository with manual modifications.
For example, you may add a package that has an unmet dependency or remove a
package that still has some dependents. In this case, the brep service inside
the VM will fail to start and the repository web interface will be
unavailable. In this case, you can login into the VM to investigate:

$ ~/bin/vm-login ~/brep-con.sock

  # systemctl status brep-startup
  # journalctl -u brep-startup


7. Upgrade the VM
-----------------

To upgrade to the new version of the VM, first download and unpack the new
VM archive similar to Step 2:

$ curl -fO https://download.build2.org/X.Y.Z/linux-debian-N-brep-X.Y.Z.tar.xz
$ sha256sum -b linux-debian-N-brep-X.Y.Z.tar.xz

Verify the checksum matches the one from https://build2.org/download.xhtml

$ tar -xf linux-debian-N-brep-X.Y.Z.tar.xz

Next read the linux-debian-N-brep-X.Y.Z/NEWS file for changes and potential
backwards compatibility issues. Unless instructed otherwise by the NEWS file,
the upgrade procedure is as follows:

# systemctl stop vm-brep@brep

$ cd
$ mkdir bak
$ mv -t bak/ bin etc vm vm-brep@.service README NEWS
$ mv -t ./ linux-debian-N-brep-X.Y.Z/*
$ cp bak/vm/brep.conf vm/
$ rm state/bpkg/pkg/1/packages.manifest

# cp ~brep/vm-brep@.service /etc/systemd/system/
# chmod 644 /etc/systemd/system/vm-brep@.service
# systemctl daemon-reload
# systemctl start vm-brep@brep
# systemctl status vm-brep@brep

If the VM has started successfully, perform the same verifications as on Step
4 above. If everything is functioning correctly, you can remove the backup
files:

$ rm -r bak

If there are any issues, investigate as on Step 6.