aboutsummaryrefslogtreecommitdiff
path: root/doc/manual.cli
blob: 91dc68bb627c3aa47d21d2fcdd75acc28b02a3b0 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
// file      : doc/manual.cli
// copyright : Copyright (c) 2014-2019 Code Synthesis Ltd
// license   : MIT; see accompanying LICENSE file

"\name=build2-buildos-manual"
"\subject=Build OS"
"\title=Operating System"

// NOTES
//
// - Maximum <pre> line is 70 characters.
//

"
\h0#preface|Preface|

This document describes Build OS, the \c{build2} build operating system.

\h1#intro|Introduction|

Build OS is a Debian GNU/Linux-based in-memory network-booted operating system
specialized for autonomous building of software using the \c{build2}
toolchain. It's primary purpose is to run the \c{build2} build bot (\c{bbot}),
build slave (\c{bslave}), or both.

A machine that run an instance of Build OS is called a \i{build host}. A
build host runs the \c{bbot} and/or \c{bslave} in the \i{agent mode}. The
actual building is performed in the virtual machines and/or containers. For
\c{bbot} it is normally one-shot virtual machines and for \c{bslave} it is
normally containers but can also be long-running virtual machines. Inside
virtual machines/containers, \c{bbot} and \c{bslave} run in the \i{worker
mode} and receive \i{build tasks} from their respective agents.

\h1#arch|Architecture|

Build OS root filesystem (\c{rootfs}) resides entirely in RAM with all changes
(such as installation of the \c{build2} toolchain) discarded on the next
reboot. A small amount of persistent (but not precious) state is stored in
\c{/state} (see \l{#config-storage-state State}). A minimum of 4GB of RAM
is required for Build OS itself (that is, excluding any virtual machines
and containers).

After booting the kernel, the Build OS execution starts with a custom \c{init}
script which performs low-level configuration and setup and then hands off the
initialization to \c{systemd}. At the end of \c{systemd} startup the Build OS
monitor script (\c{buildos}) is started as a \c{systemd} service. On startup
this script bootstraps the \c{build2} toolchain, builds the \c{bbot} package,
and starts it (as another \c{systemd} service) in the agent mode. After that
the monitor starts monitoring for OS and toolchain changes. If either is
detected, the host is rebooted, which will trigger both booting the latest
OS and building of the latest \c{build2} toolchain and \c{bbot}.

@@ TODO: init steps.

The monitor service (and \c{bbot} that it starts) are executed as the
\c{build} user/group. The home directory of this user is \c{/build/}. It has
the following subdirectories:

\ul|

\li|\n\c{/build/tftp/}\n

  A size-limited \c{tmpfs} filesystem that is used to communicate with build
  machines as well as for build host log access.

  This directory is read-accessible via the TFTP server running on the default
  port. A \c{bbot} agent also makes the \c{bootstrap/} and \c{build/}
  sub-directories inside this directory temporarily write-accessible to build
  machines by running custom instances of the TFTP server on other ports.|

\li|\n\c{/build/toolchains/}\n

  Contains \c{build2} toolchain installations after bootstrap. Each version
  is installed into a subdirectory named as the toolchain name. See
  \l{#config-toolchains Toolchains} for details.|

\li|\n\c{/build/bots/}\n

  Contains \c{bbot} installations. Each version is installed into a
  subdirectory named as the toolchain name. See \l{#config-toolchains
  Toolchains} for details.|

\li|\n\c{/build/machines/}\n

  Contains virtual machines and containers. See \l{#config-storage-machines
  Machines} for details.||


\h1#boot|Booting|

Build OS is normally booted over the network using PXE but can also be booted
locally from the kernel image and initrd directly.

\h#boot-reboot|Reboot|

Build OS can detect when the OS or toolchain have been updated and
automatically reboot the build host. This is achieved by polling the URLs
specified with the \c{buildos.buildid_url} and \c{buildos.toolchain_url}
kernel command line parameters.

The \c{buildos.buildid_url} value should point to the \c{buildos-buildid} file
that comes along the kernel image and initrd. The \c{buildos.toolchain_url}
value is the location of the toolchain checksums file as described in
\l{#config-toolchains Toolchains}. See \l{#boot-net Network Boot} for the usage
example.

\h#boot-net|Network Boot|

Here we assume that you have already established your PXE setup using
PXELINUX. That is, you have configured a TFTP server that hosts the
\c{pxelinux} initial bootstrap program (NBP) and configured a DHCP
server to direct PXE client to this server/NBP.

To setup PXE boot of Build OS, perform the following steps:

\ol|

\li|Copy the Build OS \c{-image}, \c{-initrd}, and \c{-buildid} files to the
TFTP server. For example:

\
# mkdir -p /var/lib/tftpboot/buildos
# cp buildos-image buildos-initrd buildos-buildid \
  /var/lib/tftpboot/buildos/
\

|

\li|Assuming the host MAC address is \c{de:ad:be:ef:b8:da}, create a
    host-specific configuration file (or use \c{default} as the last path
    component for a configuration that applies to all hosts):

\
# cat <<EOF >/var/lib/tftpboot/pxelinux.cfg/01-de-ad-be-ef-b8-da
default buildos
prompt 1
timeout 50

label buildos
  menu label Build OS
  kernel /buildos/buildos-image
  initrd /buildos/buildos-initrd
  append buildos.smtp_relay=example.org buildos.admin_email=admin@example.org buildos.buildid_url=tftp://<os-host>/buildos/buildos-buildid buildos.toolchain_url=https://<toolchain-host>/toolchain.sha256 buildos.toolchain_trust=<repo-fp>
EOF
\

Where \c{<os-host>} is the address of the TFTP server (the same address as
returned by the DHCP server to PXE clients), \c{<toolchain-host>} is the host
that serves the toolchain archives, and \c{<repo-fp>} is the toolchain
repository certificate fingerprint to trust. Note that all the parameters in
\c{append} must be specified on a single line.|

\li|You can test the setup using QEMU/KVM, for example:

\
$ sudo kvm \
  -m 8G \
  -netdev tap,id=net0,script=./qemu-ifup \
  -device e1000,netdev=net0,mac=de:ad:be:ef:b8:da \
  -boot n
\

||

\h#boot-local|Local Boot|

During testing it is often useful to boot Build OS directly from the kernel
image and initrd files. As an example, here is how this can be done using
QEMU/KVM:

\
sudo kvm \
  -m 8G \
  -netdev tap,id=net0,script=./qemu-ifup \
  -device e1000,netdev=net0,mac=de:ad:be:ef:b8:da \
  -kernel buildos-image -initrd buildos-initrd
\

\h1#config|Configuration|

\h#config-cpu-ram|CPU and RAM|

A Build OS instances divides available CPUs and RAM (minus reserved, see
below) into \i{slices} that are then \i{committed} to each instance of each
toolchain. If you don't expect your builds to utilize these resources at the
same time, then it may make sense to overcommit them to improve utilization.
The respective overcommit values can be specified as ratios with the
\c{buildos.cpu_overcommit} and \c{buildos.ram_overcommit} kernel command
line parameters. For example, given the following CPU overcommit:

\
buildos.cpu_overcommit=3/2
\

A Build OS machine with 8 CPUs (hardware threads) and three toolchains will
assign 4 CPUs (\c{8 * 3/2 / 3}) to each slice.

It is also possible to reserve a number of CPUs and an amount of RAM to
Build OS with the \c{buildos.cpu_reserved} and \c{buildos.ram_reserved}
(in GB) kernel command line parameters. If unspecified, 4GB of RAM is
reserved by default.


\h#config-storage|Storage|

Build OS configures storage based on the labels assigned to disks and
partitions (collectively refered to as disks from now on). Build OS requires
storage for state as well as virtual machines and containers.

\h2#config-storage-state|State|

Build OS stores a small amount of state on a disk labeled \c{buildos.state}
(mounted as \c{/state}). This includes random number generator state, SSH
server host keys, and so on. While this state is persistent, it is not
precious.

The stored state is fairly small (hundreds of megabytes) and is not
performance-critical. While one can create a small state partition on the same
physical disk as used for machines (see below), having it on a separate disk
makes it easier to move machine disks around. Based on these requirements, a
small USB flash drive or flash card is a good option.

While any suitable filesystem can be used, \c{ext4} is a good choice, with
journaling disabled if used on a flash drive/card. For example:

\
mkfs.ext4 -L buildos.state -O ^has_journal /dev/sdX
\

\h2#config-storage-machines|Machines|

For virtual machine and container storage we can use a single disk, in which
case it can be labeled just \c{buildos.machines}. If we would like to use
multiple disks, then they should be labeled \c{buildos.machines.<volume>}. In
both cases the disks must be formatted as \c{btrfs}.

In a single disk configuration, the disk is mounted as
\c{/build/machines/default/} (in other words, as the default valume called
\c{default}). In a multi-disk configuration, each disk is mounted as
\c{/build/machines/<volume>/}.

If no disks are found for required storage, then the boot process is
interrupted with a shell prompt where you can format and/or lable a suitable
disk. You can also view the storage configuration on a booted Build OS
instance by examining \c{/etc/fstab}.

As an example, let's consider the first boot of a clean machine that has an
SSD disk as \c{/dev/sda} and which we would like to use for virtual machine
storage. We would also like to over-provision this SSD by 10% to (potentially)
prolong its life and increase performance (you may want to skip this step if
you are using a datacenter-grade SSD that would normally already be generously
over-provisioned).

On the first boot we will be presented with a shell prompt which we use to
over-provision the disk:

\
# fdisk -l /dev/sda            # Query disk information.
# hdparm -N /dev/sda           # Query disk/host protection area sizes.
# hdparm -Np<COUNT> /dev/sda   # COUNT = sector count * 0.9
# hdparm -N /dev/sda           # Verify disk/host protection area sizes.
# ^D                           # Exit shell and reboot.
\

Note that this may not always work, depending on the disk controller used. An
alternative approach is to use the \c{mkfs.btrfs --byte-count} option when
formatting the disk to leave some disk space untouched and unused.

After the reboot we will be presented with a shell prompt again where we
confirm over-provisioning, format the disk as \c{btrfs}, and label it as
\c{buildos.machines}:

\
# fdisk -l /dev/sda            # Confirm disk size decreased by 10%.
# mkfs.btrfs -L buildos.machines -m single /dev/sda
# ^D                           # Exit shell and reboot.
\

\h#config-net|Network|

Network is configured via DHCP. Initially, all Ethernet interfaces that have
carrier are tried in (some) order and the first interface that is successfully
configured via DHCP is used.

Hostname is configured from the DHCP information. Failed that, a name is
generated based on the MAC address, in the form \c{build-xxxxxxxxxx}.
@@ Maybe also kernel cmdline?

Based on the discovery of the Ethernet interface, two bridge interfaces are
configured: \c{br0} is a public bridge that includes the Ethernet interface
and is configured via DHCP. \c{br1} is a private interface with NAT to \c{br0}
with \c{dnsmasq} configured as a DHCP on this interface.

Normally, \c{br0} is used for \c{bslave} virtual machines/container (since
they may need to be accessed directly) and \c{br1} \- for \c{bbot} virtual
machines. You can view the bridge configuration on a booted Build OS instance
by examining \c{/etc/network/interfaces}.

@@ TODO: private network parameters.

\h#config-email|Email|

A Build OS instance sends various notifications (including all messages to
\c{root}) to the admin email address. The admin email is specified with
the \c{buildos.admin_email} kernel command line parameter.

In order to deliver mail, the \c{postfix} MTA is configured to forward to a
relay. The relay host is specified with the \c{buildos.smtp_relay} kernel
command line parameter.

Note that no authentication of any kind is configured for relaying. This means
that the relay host should accept emails from build hosts either because of
their network location (for example, because they are on your organization's
local network and you are using your organization's relay) or because the
relay host accepts emails send to the admin address from anyone (which is
normally the case if the relay is the final destination for the admin
address, for example, \c{example.org} and \c{admin@example.org}).


\h#config-ssh|SSH|

Build OS runs an OpenSSH server with password authentication and \c{root}
login disabled. As a result, the only way to login remotely is as user
\c{build} using public key authentication. To add a public key into the
\c{build}'s \c{authorized_keys} file we can use the \c{buildos.ssh_key}
kernel command line parameter. For example (note the quotes):

\
buildos.ssh_key=\"ssh-rsa AAA...OA0DB user@host\"
\

\h#config-toolchains|Toolchains|

The first step performed by the Build OS monitor is to bootstrap the
\c{build2} toolchain. The location of the toolchain packages is specified with
the \c{buildos.toolchain_url} kernel command line parameter. This URL should
point to the \i{toolchain checksums file}. You will also normally need to pass
the \c{buildos.toolchain_trust} parameter which is the toolchain repository
certificate fingerprint that the monitor should trust. Note also that the
bootstrap process (both on the build host and inside build machines) uses the
default toolchain repository location embedded into the build scripts in the
\c{build2-toolchain} package.

It is also possible to use multiple toolchains on a single Build OS instance.
In this case a toolchain name can be appended after \c{buildos.toolchain_*},
for example, \c{buildos.toolchain_url.<name>} (values without the toolchain
name use the toolchain name \c{default}). The toolchain name may not contain
\c{-}.

Each toolchain may also execute multiple \c{bbot} agent instances. The number
of instances is specified with the \c{buildos.instances[.<name>]} parameter.

All \c{bbot} agent instances of a toolchain are executed with the same nice
value which can be specified with the \c{buildos.nice[.<name>]} parameter. It
should be between -20 (highest priority) and 19 (lowest priority) with 0
being the default. See \cb{sched(7)} for details.

The bridge interface to be used for machine networking can be specified with
the \c{buildos.bridge[.<name>]} parameter. Valid values are \c{br0} (public
bridge to the physical interface) and \c{br1} (private/NAT'ed bridge to
\c{br0}). If unspecified, \c{br1} is used by default.

In the checksums file blank lines and lines that start with \c{#} are ignored.
If the first line is the special \c{disabled} value, then this toolchain is
ignored. Otherwise, each line in the checksums file is the output of the
\c{shaNNNsum(1)} utility, that is, the SHANNN sum following by space, an
asterisk (\c{*}, which signals the binary mode), and the relative file path.
The extension of the checksums file should be \c{.shaNNN} and the first line
should be for the \c{build2-toolchain} \c{tar} archive itself (used to derive
the toolchain version). For example:

\
# toolchain.sha256
ae89[...]87a4 *0.4.0/build2-toolchain-0.4.0.tar.xz
058d[...]c962 *0.4.0/build2-baseutils-0.4.0-x86_64-windows.zip
e723[...]c305 *0.4.0/build2-mingw-0.4.0-x86_64-windows.tar.xz
\

Based on the checksums file the monitor downloads each file into
\c{/build/tftp/toolchains/<name>/} (the file path is taken as relative to
\c{toolchain_url}), verifies their checksums, and creates \i{predictable name}
symlinks (names without the version). It also creates the \c{version} which
contains the toolchain version and the \c{trust} file which contains the
value of the \c{buildos.toolchain_trust} parameter or the special \c{\"no\"}
value if none were specified.

Continuing with the above example, the contents of
\c{/build/tftp/toolchains/default/} would be:

\
version
trust

build2-toolchain-0.4.0.tar.xz
build2-baseutils-0.4.0-x86_64-windows.zip
build2-mingw-0.4.0-x86_64-windows.tar.xz

build2-toolchain-tar.xz             -> build2-toolchain-0.4.0.tar.xz
build2-baseutils-x86_64-windows.zip -> build2-baseutils-0.4.0-x86_64-windows.zip
build2-mingw-x86_64-windows.tar.xz  -> build2-mingw-0.4.0-x86_64-windows.tar.xz
\

While the monitor itself only needs the \c{build2-toolchain} package, build
machine toolchain bootstrap may require additional packages (which will be
accessed via TFTP using predictable names).

\h#config-controllers|Controllers|

For each toolchain the \c{bbot} agent polls one or more controllers for build
tasks to perform. The controller URLs are configured with the
the \c{buildos.controller_url[.<name>]} kernel command line parameter (where
\c{<name>} is optional toolchain name). To specify multiple controllers,
repeat this parameter.

Additionally, we can use the \c{buildos.controller_trust[.<name>]} kernel
command line parameter to specify SHA256 repository certificate fingerprints
to trust (see the \c{trust} build task manifest value for details). To specify
multiple fingerprints, repeat this parameter.


\h1#machines|Build Machines|

At the top level, a machine storage volume (see \l{#config-storage-machines
Machines}) contains machine directories, for example:

\
/build/machines/default/
├── linux-gcc_6/
└── windows-msvc_14/
\

The layout inside a machine directory is as follows, where \c{<name>} is
the machine name and \c{<toolchain>} is the toolchain name:

\
<name>/
├── <name>-1   -> <name>-1.1
├── <name>-1.0/
├── <name>-1.1/
├── <name>-<toolchain>/
└── <name>-<toolchain>-<xxx>/
\

The \c{<name>-<P>.<R>} entries are read-only \c{btrfs} subvolumes that contain
the initial (that is, \i{pre-bootstrap}) machine images. The numeric \c{<P>}
part indicates the \i{bootstrap protocol version}. The numeric \c{<R>} part
indicates the machine revision.

The \c{<name>-<P>} entry is a symbolic link to \c{<name>-<P>.<N>} that is
currently in effect.

The \c{<name>-<toolchain>} entry is the bootstrapped machine image for
\c{<toolchain>}. It is created by cloning \c{<name>-<P>} with a bootstrap
protocol version that matches this toolchain's \c{bbot} and then bootstrapping
the \c{build2} toolchain inside.

The \c{<name>-<toolchain>-<xxx>} entries are the temporary snapshots of
\c{<name>-<toolchain>} created by \c{bbot} for building packages.

A machine can be added, upgraded, or removed on a live Build OS instance.
This needs to be done in a particular order to avoid inconsistencies and race
conditions.

\h#machines-add|Adding a Machine|

Let's assume you have a read-only \c{btrfs} \c{linux-gcc_6-1.0} subvolume on a
development host (we will call it \c{devel}) that contains the initial version
of our virtual machine. We would like to add it to the build host (running
Build OS, we will call it \c{build}) into the default machine volume
(\c{/build/machines/default/}). To achieve this in an atomic way we perform
the following steps:

\
# Create the machine directory.
#
build$ mkdir /build/machines/default/linux-gcc_6

# Send the machine subvolume to build host.
#
devel$ sudo btrfs send linux-gcc_6-1.0 | \
  ssh build@build sudo btrfs receive /build/machines/default/linux-gcc_6/

build$ cd /build/machines/default/linux-gcc_6

# Make user build the owner of the machine subvolume.
#
build$ sudo btrfs property set -ts linux-gcc_6-1.0 ro false
build$ sudo chown build:build linux-gcc_6-1.0 linux-gcc_6-1.0/*
build$ btrfs property set -ts linux-gcc_6-1.0 ro true

# Make the subvolume the current machine.
#
build$ ln -s linux-gcc_6-1.0 linux-gcc_6-1
\

\N|The \c{upload-machine} helper script implements this sequence of steps.|


\h#machines-upgade|Upgrading a Machine|

Continuing with the example started in the previous section, let's assume we
have created \c{linux-gcc_6-1.1} as a snapshot of \c{linux-gcc_6-1.0} and have
made some modification to the virtual machine (all on the development
host). We now would like to switch to this new revision of our machine on the
build host. To achieve this in an atomic way we perform the following steps:

\
# Send the new machine subvolume to build host incrementally.
#
devel$ sudo btrfs send -p linux-gcc_6-1.0 linux-gcc_6-1.1 | \
  ssh build@build sudo btrfs receive /build/machines/default/linux-gcc_6/

build$ cd /build/machines/default/linux-gcc_6

# Make user build the owner of the new machine subvolume.
#
build$ sudo btrfs property set -ts linux-gcc_6-1.1 ro false
build$ sudo chown build:build linux-gcc_6-1.1 linux-gcc_6-1.1/*
build$ btrfs property set -ts linux-gcc_6-1.1 ro true

# Switch the current machine atomically.
#
build$ ln -s linux-gcc_6-1.1 new-linux-gcc_6-1
build$ mv -T new-linux-gcc_6-1 linux-gcc_6-1

# Remove the old machine subvolume (optional).
#
build$ btrfs property set -ts linux-gcc_6-1.0 ro false
build$ btrfs subvolume delete linux-gcc_6-1.0
\

\N|The \c{upload-machine} helper script implements this sequence of steps.|

\h#machines-remove|Remove a Machine|

Continuing with the example started in the previous section, let's assume we
are no longer interested in the \c{linux-gcc_6} machine and would like to
remove it. This operation is complicated by the possibility of \c{bbot}
instances currently building with this machine.

\
build$ cd /build/machines/default/linux-gcc_6

# Remove the current machine symlink.
#
build$ rm linux-gcc_6-1

# Wait for all the linux-gcc_6-<toolchain>-<xxx> subvolumes
# to disappear.
#
build$ for d in linux-gcc_6-*-*/; do           \
  while [ -d $d ]; do \
    echo \"waiting for $d\" && \
    sleep 10; \
  done; \
done

# Remove the initial and bootstrapped machine subvolume(s).
#
build$ for d in linux-gcc_6-*/; do     \
  btrfs property set -ts $d ro false && \
  btrfs subvolume delete $d; \
done

# Remove the machine directory.
#
build$ cd ..
build$ rmdir /build/machines/default/linux-gcc_6
\

\N|The \c{remove-machine} helper script implements this sequence of steps.|

Note also that on reboot the Build OS monitor examines and cleans up
machine directories of any stray subvolumes. As a result, an alternative
approach would be to remove the current machine symlink and reboot the
build host.

\h#machines-login|Logging into a Machine|

A running QEMU/KVM machine (that is, one being bootstrapped or used for
building) can be accessed with a VNC client. Clients based on \c{gtk-vnc},
such as \c{vinagre}, are known to work reasonably well. For example:

\
ssh -f -L 5901:127.0.0.1:5901 build@build sleep 1 && vinagre 127.0.0.1:5901
\

If the machine has been suspended, it can be resumed using the following
command:

\
echo cont | ssh build@build socat - UNIX-CONNECT:/tmp/monitor-<toolchain>-<instance>
\

Other useful QEMU monitor commands are \c{system_powerdown} and
\c{system_reset}.

"