From 757787725cffc24f6c4e353fefa63bf255007457 Mon Sep 17 00:00:00 2001 From: Boris Kolpackov Date: Sun, 24 Feb 2019 15:44:35 +0200 Subject: Updates for 0.9.0 release --- doc/.gitignore | 2 + doc/bash-style.cli | 4 +- doc/cli.sh | 5 + doc/release.cli | 724 ++++++++++++++++++++++++++++++++++++++++++++++++ git/modules | 9 +- publish | 90 ------ release-bugfix-only.txt | 79 ------ release.txt | 307 -------------------- rep-test | 179 ------------ review | 2 +- stage | 125 ++++++--- stage-pkg | 17 +- upgrade | 131 +++++++++ version | 19 +- 14 files changed, 981 insertions(+), 712 deletions(-) create mode 100644 doc/release.cli delete mode 100755 publish delete mode 100644 release-bugfix-only.txt delete mode 100644 release.txt delete mode 100755 rep-test create mode 100755 upgrade diff --git a/doc/.gitignore b/doc/.gitignore index a5873dc..36afaee 100644 --- a/doc/.gitignore +++ b/doc/.gitignore @@ -1 +1,3 @@ build2-bash-style.xhtml +build2-release.xhtml + diff --git a/doc/bash-style.cli b/doc/bash-style.cli index 9e90e8a..d22cc23 100644 --- a/doc/bash-style.cli +++ b/doc/bash-style.cli @@ -9,7 +9,7 @@ // - Maximum 
 line is 70 characters.
 //
 
-"\h1|Contents|"
+"\h1|Table of Contents|"
 "\$TOC$"
 
 "
@@ -27,7 +27,7 @@ You just need to be clear on why you are doing it.
 See also \l{https://google.github.io/styleguide/shell.xml Google's Bash Style
 Guide} as well as \l{https://github.com/progrium/bashstyle Let's do Bash
 right!}; we agree with quite a few (but not all) items in there. In particular,
-it provides a lot more rationale compared to this guide.
+the former provides a lot more rationale compared to this guide.
 
 \h1#style|Style|
 
diff --git a/doc/cli.sh b/doc/cli.sh
index 44d311a..e737ee7 100755
--- a/doc/cli.sh
+++ b/doc/cli.sh
@@ -22,3 +22,8 @@ cli --generate-html --html-suffix .xhtml \
 --html-prologue-file doc-prologue.xhtml \
 --html-epilogue-file doc-epilogue.xhtml \
 --output-prefix build2- bash-style.cli
+
+cli --generate-html --html-suffix .xhtml \
+--html-prologue-file doc-prologue.xhtml \
+--html-epilogue-file doc-epilogue.xhtml \
+--output-prefix build2- release.cli
diff --git a/doc/release.cli b/doc/release.cli
new file mode 100644
index 0000000..6639371
--- /dev/null
+++ b/doc/release.cli
@@ -0,0 +1,724 @@
+// file      : doc/release.cli
+// copyright : Copyright (c) 2014-2018 Code Synthesis Ltd
+// license   : MIT; see accompanying LICENSE file
+
+"\title=Release Process"
+
+// NOTES
+//
+// - Maximum 
 line is 70 characters.
+//
+
+"\h1|Table of Contents|"
+"\$TOC$"
+
+"
+\h0#process|Process|
+
+Review the state and services list (currently on paper) for any new additions.
+Consider how/when they are updated/tested during the release process.
+
+
+\h1#stage|Stage|
+
+The staging repository is completely independent which means it must contain
+all the \c{build2} dependencies (plus a few extra packages for testing; see
+the \c{etc/stage} script for the complete list). This in turn means that if
+any of these dependencies are in the unreleased state, then they should go
+through the applicable steps in this section (e.g., updating of \c{NEWS}, etc)
+and then be queued and published (effectively released) as part of the
+\c{build2} release. Generally, however, we should strive to not unnecessarily
+bundle the release of dependencies with the release of \c{build2} to keep
+the process as streamlined as possible.
+
+\N|When unbundling the release of a dependency we need to remove its
+distribution from \c{etc/stage} and add the pre-distributed packages
+(for example, from \c{public}) to \c{staging/repository/1/}.|
+
+\b{Pre-Conditions:}
+
+\ul|
+
+\li|Current \c{stage} build is clean.||
+
+\h#copy|Update copyright if new year|
+
+  Use the \c{git/copyright} script. See the script header for instructions.
+
+\h#etc|Update \c{etc/git/modules}|
+
+  Review for any new modules. Remove \c{etc/} and \c{private/} from
+  \c{modules} to reduce noise during stat.
+
+\h#review|Review \c{@@}|
+
+  Review \c{@@} notes:
+
+  \
+  ./review.sh | less -R
+  \
+
+  At least look for \c{@@\ TMP}
+
+  \
+  ./review.sh | grep TMP
+  \
+
+\h#review-db|Review database schema changes|
+
+  Review database schema changelog differences in \c{bpkg}, \c{bdep}, and
+  \c{brep} compared to the previous release (tag) for any schema/data
+  migration that may be required (if something is missing, then it would also
+  have to be tested).
+
+\h#news|Update \c{NEWS} files|
+
+  Update in all projects (including dependencies) that will have a release
+  version.
+
+  See \c{etc/stage} for the complete list.
+
+
+\h#packaging|Release \c{packaging/} dependencies|
+
+  Release dependencies that have the snapshot version as described in
+  \c{private/build2-packaging.txt}.
+
+  \N|Maybe this should be done during queuing? Why do we release (but not
+     publish) these now and other dependencies later?|
+
+
+\h#dependencies|Finalize all other dependencies|
+
+  Make sure all other unreleased dependencies listed in \c{etc/stage} are
+  ready to be released. Effectively, the only remaining step should be to
+  change the version.
+
+  Do this in the dependency order and finish each off with:
+
+  \
+  git pull && bdep sync -fura && bdep test -ar
+  \
+
+
+\h#upgrade|Upgrade dependencies|
+
+  Upgrade all dependencies in the \c{build2} toolchain build:
+
+  \
+  etc/upgrade
+  \
+
+
+\h#hello|Update \c{hello/} projects|
+
+  These projects should closely track the output of \c{bdep-new(1)}. The
+  recommended procedure is to generate a new project with the same name and
+  then review/resolve the differences with \c{diff\ -ru}.
+
+  Every change in these projects must be accompanied by a revision increment
+  and a re-tag. This should be managed with \c{bdep-release(1)}:
+
+  \
+  # make changes
+  git add .
+  bdep release --revision --show-push
+  # review commit
+  git push ...
+  \
+
+  Once done, run the \c{intro} scripts and review any changes in the output
+  (this information will be helpful on the next step):
+
+  \
+  cd etc
+
+  ./intro2-tldr 2>&1 | tee intro2-tldr.out
+  diff -u intro2-tldr.orig intro2-tldr.out  # Or use gitk.
+  mv intro2-tldr.out intro2-tldr.orig
+
+  ./intro2-tour 2>&1 | tee intro2-tour.out
+  diff -u intro2-tour.orig intro2-tour.out  # Or use gitk.
+  mv intro2-tour.out intro2-tour.orig
+  \
+
+\h#doc|Review documentation|
+
+  Review the following documentation for (1) sample output changes and (2)
+  still being relevant/making sense.
+
+  \N|Ideally this should be done during development but it's easy to forget.
+     Also, check if there is any new documentation that has been added that is
+     not on the below list.|
+
+  \ul|
+
+    \li|Install guide: 1 & 2.|
+
+    \li|Toolchain introduction: 1 & 2 (use \c{intro} script output).|
+
+    \li|Introduction in the build system manual: 1 (uses \c{bdep-new(1)}
+        output).|
+
+    \li|Testscript manual: 1.||
+
+
+\h#submod|Update all submodules|
+
+  Make sure all working trees are clean. Review \c{sub_modules} in
+  \c{etc/git/modules} for any missing new modules, then:
+
+  \N|\c{etc/} and \c{private/} need to be committed manually.|
+
+  \
+  ./modup.sh
+  ./commit.sh # If changes.
+  ./push.sh
+
+  cd build2-toolchain
+
+  git submodule update --remote --checkout
+  git submodule foreach git submodule update --init --recursive
+
+  git status  # There should only be 'new commits'; see README-GIT
+
+  git commit -a -m \"Update submodules\"
+  git push
+  \
+
+\h#stage-machines|Update \c{stage} \c{buildtab}s and build machines|
+
+  Review \c{stage} \c{buildtab} for any configurations to drop (for example,
+  an intermediate version of a compiler).
+
+  Based on these changes update \c{stage} CI \c{buildtab}, which is a subset
+  of the \c{stage} configurations (and is a base for the \c{queue}/\c{public}
+  configurations).
+
+  Review deployed machines against the updated \c{stage} \c{buildtab} and
+  remove those that are no longer used:
+
+  \
+  cd private/buildos/
+
+  ./ls-machines -c stage -c devel
+
+  ~/work/buildos/remove-machine  
+  \
+
+  Also review deployed machines against the latest available versions and
+  upgrade those that are not the latest:
+
+  \
+  cd private/buildos/
+
+  ./ls-machines -l \"/btrfs/$(whoami)/machines/default/\"
+  ./ls-machines -c stage -c devel
+
+  ~/work/build2/buildos/upload-machine  .../new-ver .../old-ver
+  \
+
+
+\h#restage|Restage|
+
+  Confirm \c{stage} and \c{devel} \c{buildos} images, hardware-specific
+  configurations are the same.
+
+  Review \c{staging/0/} and \c{staging/repository/1/} for anything stray.
+
+  Update all submodules in \c{build2-toolchain}:
+
+  \
+  git submodule update --remote --checkout
+  \
+
+  Restage with \c{baseutils}/\c{mingw} regeneration:
+
+  \
+  etc/stage -b
+  \
+
+  Upgrade \c{brep} on \c{stage} and sync latest \c{buildtab}s.
+
+  Verify \c{stage} build is clean, nothing is unbuilt.
+
+
+\h#install-stage|Test install scripts|
+
+  Test \l{https://stage.build2.org/0/ \c{stage} install scripts}, including
+  upgrading, as described in \c{private/install/testing.txt}.
+
+
+\h1#queue|Queue|
+
+\b{Pre-Conditions:}
+
+\ul|
+
+\li|Final \c{stage} build is clean.|
+
+\li|Build with the \c{queue} toolchain is inactive.||
+
+
+\h#queue-repo|Review \c{queue} repository|
+
+  Pull, review, and clean the \c{queue} \c{git} repository for any stale
+  packages or anything (ownership, etc) that may conflict with packages being
+  released.
+
+  Also clean toolchain distribution:
+
+  \
+  rm -rf cppget.org/queue/0/*
+  \
+
+\h#version-release|Change to release version|
+
+  Change to the release version in all the packages being released and tag
+  (see \c{etc/stage} for the list). Use \c{bdep-release(1)} unless a custom
+  versioning/tagging scripts are used:
+
+  \
+  bdep release --no-open --show-push [--alpha|--beta]
+  # review commit
+  git push ...
+  \
+
+  Do this in the dependency order and finish each off with:
+
+  \
+  bdep sync -fura && bdep update
+  \
+
+  For the \c{build2} packages (commit but don't push after each step if
+  required):
+
+  \ul|
+
+    \li|Close schema versions in \c{bpkg}, \c{bdep}, and \c{brep}.|
+
+    \li|Change \c{BUILD2_STAGE} in \c{build2/build2/config.hxx.in} to \c{false}.|
+
+    \li|Change version by updating (including with new modules) and then
+        executing:
+
+       \
+       etc/version
+       ./commit.sh
+       git -C build2-toolchain commit --amend # \"Change version to X.Y.Z\"
+       \
+
+    |
+
+    \li|Tag by executing \c{tag.sh\ }.|
+
+    \li|Regenerate documentation in each package.|
+
+    \li|Upgrade all dependencies in configure-only mode by executing
+    \c{etc/upgrade\ -c}.|
+
+    \li|Trigger regeneration of version files (might require several runs
+        to \"close off\"):
+
+        \
+        BDEP_SYNC=0 b --match-only ~/work/build2/builds/gcc7-asan/
+        \
+
+        If using GCC prior to 8 then might also need explicit version target
+        upgrades:
+
+        \
+        BDEP_SYNC=0 b ~/work/build2/builds/gcc7-asan/.../hxx{version}
+        \
+
+    |
+
+    \li|Regenerate ODB in relevant packages passing upgraded configuration
+        path explicitly (\c{bdep} is not runnable):
+
+       \
+       ./odb.sh ~/work/build2/builds/gcc7-asan/
+       \
+
+    |
+
+    \li|Finish upgrading all dependencies by executing in the upgraded
+    configuration:
+
+       \
+       BDEP_SYNC=0 b ~/work/build2/builds/gcc7-asan/
+       \
+
+    ||
+
+  Verify key tests pass (in particular, the \c{bdep} tests will now be running
+  against \c{public} services):
+
+  \
+  b test: build2/ bpkg/ bdep/
+  \
+
+  \N|We could have queued after this step before preparing
+     \c{build2-toolchain}. However, splitting this seems to only increase
+     complexity without any major benefits (it's not hard to update submodules
+     and regenerated \c{build2-toolchain} if there are any issues with
+     packages). Plus we can start testing install scripts, etc.|
+
+  Next push the above changes and update all submodules in
+  \c{build2-toolchain}:
+
+  \
+  ./push.sh
+
+  cd build2-toolchain
+
+  git submodule update --remote --checkout
+
+  git status  # There should only be 'new commits'; see README-GIT
+
+  git commit -a -m \"Update submodules\"
+  \
+
+  As well as (commit after each step if required):
+
+  \ul|
+
+    \li|Regenerate documentation in each package inside as well as in
+        \c{build2-toolchain} itself.|
+
+    \li|Update ODB by copying relevant files from the previous step (trust
+        me, this is the easy way).|
+
+    \li|Change \c{BUILD2_REPO} in \c{build2-toolchain} build scripts to
+    \c{queue}.||
+
+  Finally, push the changes:
+
+  \
+  git push
+  \
+
+\h#queuing|Queue|
+
+   Prepare packages and the toolchain distribution:
+
+   \
+   etc/stage -q -b
+   \
+
+   Sort non-alpha packages from \c{cppget.org/queue/1/alpha/} into appropriate
+   sections (we could probably automate this similar to \c{bdep-release(1)}).
+
+   Note also that we assume all the packages already have the corresponding
+   ownership information either in \c{queue} or \c{public}. However, if any
+   new packages were added, then that will have to be added as well.
+
+   Commit and push queue repository:
+
+   \
+   cd cppget.org/queue/
+   git add .
+   git ci -m \"Queue build2 toolchain X.Y.Z\"
+   git push
+   \
+
+\h#build-public|Verify queued packages build with \c{public}|
+
+   This makes sure that the new version can be built with the old toolchain.
+
+   While all the packages should build (except perhaps \c{libhello} which,
+   being based on the latest \c{bdep-new(1)} output, may use new features),
+   some tests may fail. In particular, we will be running \c{bpkg} tests using
+   a previous version of the build system and \c{bdep} tests also using a
+   previous version of \c{bpkg}. None of that is or will be supported. So
+   for now we manually review all the failed builds for anything suspicious
+   and in the future we may skip tests if the tool versions mismatch.
+
+
+\h#install-queue|Test install scripts|
+
+  Test \l{https://download.build2.org/queue/ \c{queue} install scripts}. Here
+  we just smoke-test each script on its \"primary\" platform and make sure
+  \c{queue} URLs/repositories are used.
+
+
+\h#upgrade-brep-bpub|Upgrade \c{brep} and \c{bpub} on \c{cppget.org}|
+
+  Upgrade \c{brep} and \c{bpub} using the queued toolchain according
+  to \c{private/brep-bpub-upgrade.txt}.
+
+  After upgrade, drop all package builds for \c{public} and \c{queue} and
+  verify there are no regressions. This makes sure that the \c{brep} services
+  in the new version are usable with the old toolchain. If there are issues,
+  they have to be fixed by publishing/queuing revisions.
+
+  \N|The \c{bdep} tests that exercise CI and publishing services do it in the
+  simulation mode. To properly test that these services are compatible with
+  the old version of the toolchain we would need to CI/publish a real test
+  package manually.|
+
+
+\h#start-queue|Start \c{queue} builds|
+
+  Update \c{queue} \c{buildtab} based on the \c{stage} CI \c{buildtab}
+  (normally just a copy sans the sanitized toolchain configurations).
+
+  Adjust \c{stage} and \c{devel} build host configurations to enable the
+  \c{queue} toolchain. Shift most instances from \c{stage} to \c{queue}
+  in the hardware class-specific configurations. Regenerate affected
+  configurations and reboot build hosts:
+
+  \
+  cd private/buildos/
+
+  ./gen-config stage 
+  ./gen-config devel 
+
+  ./po-hosts -r -c stage -c devel
+  \
+
+  Verify both \c{queue} and \c{public} builds with the queued toolchain,
+  fixing any breakages with revisions, moving to legacy, and/or queuing
+  replacements. We can only proceed further once we have a \"resolution\"
+  for every (newly) broken package.
+
+
+\h#stop-queue|Stop \c{queue} builds|
+
+  Adjust \c{stage} and \c{devel} build host configurations to disable the
+  \c{queue} toolchain (comment out). Regenerate affected configurations and
+  reboot build hosts as on the previous step.
+
+\h1#public|Public|
+
+\b{Pre-Conditions:}
+
+\ul|
+
+\li|A \"resolution\" is queued or published for every (newly) broken package.||
+
+\h#public-machines|Update \c{public} \c{buildtab}s and build machines|
+
+  Poweroff the old set of \c{public} build hosts:
+
+  \
+  ./po-hosts -c public
+  \
+
+  Update \c{public} \c{buildtab}s based on the \c{queue} \c{buildtab}
+  (normally just a copy).
+
+  Adjust build host configurations and add/remove new/old machines.
+
+  Replace the \c{public} \c{buildos} image on \c{build-cache} with the
+  one for \c{stage}.
+
+  Comment out the \c{public} toolchain in the build host configuration
+  (effectively making it a no-toolchain configuration) and power on the new
+  set of \c{public} build hosts.
+
+  Review deployed machines against the updated \c{public} \c{buildtab} and
+  remove those that are no longer used:
+
+  \
+  cd private/buildos/
+
+  ./ls-machines -c public
+
+  ~/work/buildos/remove-machine  
+  \
+
+  Also review deployed machines against the latest available versions and
+  upgrade those that are not the latest:
+
+  \
+  cd private/buildos/
+
+  ./ls-machines -l \"/btrfs/$(whoami)/machines/default/\"
+  ./ls-machines -c public
+
+  ~/work/build2/buildos/upload-machine  .../new-ver .../old-ver
+  \
+
+  Uncomment the \c{public} toolchain in the build host configuration. The
+  only remaining step is to reboot (not yet):
+
+  \
+  ./po-hosts -r -c public
+  \
+
+\h#pub-dist|Publish distribution|
+
+  Change \c{BUILD2_REPO} in \c{build2-toolchain} build scripts to \c{public}
+  and publish the distribution (this also cleans/disables the \c{queue}
+  toolchain):
+
+  \
+  etc/stage -p
+  \
+
+\h#pub-pkg|Publish packages|
+
+  Move packages (manually for now) from the \c{queue} to \c{public} \c{git}
+  repository, including ownership information. Move old/replaced/FTB
+  packages either to legacy or delete.
+
+  Commit everything (see the commit log for procedure @@ this needs
+  automation) and push (which should trigger auto-publish). Note: commit and
+  push both \c{queue} and \c{public} \c{git} repositories.
+
+  Note that once published, the existing install instructions/download
+  links are no longer usable, so do not linger.
+
+\h#start-public|Start \c{public} builds|
+
+  Reboot \c{public} build hosts. They should next bootstrap and proceed
+  with building all packages.
+
+\h#install-public|Test install scripts|
+
+  Test \l{https://download.build2.org/ \c{public} install scripts}. Here
+  we just smoke-test each script on its \"primary\" platform and make sure
+  \c{public} URLs/repositories are used.
+
+\h#web|Update web|
+
+  \ul|
+
+  \li|Write release notes, use placeholder for announcement URL (or guess).
+      Add link from the \c{doc.cli}. Add blog entry.|
+
+  \li|Add any new FAQ entries or any other updates (main page, etc). Any new
+      documentation to link from the Doc page?|
+
+  \li|Update the Download page.
+
+      \
+      cat `ls -1 cppget.org/public/0/X.Y.Z/*.sha256`
+      \
+
+  |
+
+  \li|Update the Install page.
+
+      \
+      cat cppget.org/public/0/toolchain.sha256
+      \
+
+  |
+
+  \li|Regenerate documentation in \c{private/} for all hosts:
+
+      \
+      cd private/
+
+      cd /www/
+      ./cli.sh
+      \
+
+  |
+
+  \li|Test locally, publish, and test remote:
+
+      \
+      cd private/
+      ./publish --dry-run
+      ./publish
+      \
+
+  ||
+
+
+\h#ann|Announce|
+
+  \ul|
+
+  \li|Write and send the announcement to the mailing lists.
+
+      \
+      cat `ls -1 cppget.org/public/0/X.Y.Z/*.sha256`
+      \
+
+      Add \c{reply-to:} header for \c{users@} announcement.|
+
+  \li|Patch (or verify) the announcement URL in release notes, re-publish.|
+
+  \li|Announce on reddit, slack, and other relevant places.||
+
+\h#tag|Commit, tag, and push the rest.|
+
+   Tag \c{build2-toolchain}:
+
+   \
+   cd build2-toolchain
+   git tag -a vX.Y.Z -m \"Tag version X.Y.Z\"
+   git push --follow-tags
+   \
+
+   Add \c{etc/} and \c{private/} back to \c{modules} in \c{etc/git/modules}
+   (essentially reverse \l{#etc Update \c{etc/git/modules}}).
+
+   Finalize changes, commit, tag, and push \c{style/}, \c{etc/}, and
+   \c{private/}.
+
+   Snapshot \c{buildos} subvolumes:
+
+   \
+   btrfs subvolume snapshot buildos-3 buildos-3-X.Y.Z
+   btrfs subvolume snapshot buildos-6 buildos-6-X.Y.Z
+   \
+
+\h1#reopen|Reopen|
+
+\h#version-snapshot|Change to snapshot version|
+
+  Change to the snapshot version in all the released packages. Use
+  \c{bdep-release(1)} unless a custom versioning script is used:
+
+  \
+  bdep release --open --show-push [--open-*]
+  # review commit
+  git push ...
+  \
+
+  Essentially, the same steps as in \l{#version-release Change to
+  release version} (but no tagging).
+
+\h#stage-machines-reopen|Update \c{stage} \c{buildtab}s and build machines|
+
+  Essentially, the same steps as in \l{#public-machines Update \c{public}
+  \c{buildtab}s and build machines} but for stage. Some differences:
+
+  Clean \c{buildtab}s (both \c{stage} and CI) by removing no longer relevant
+  configurations, moving some to \c{legacy}, etc.
+
+  More generally, this is the time to do sweeping changes such as renaming
+  machines/configurations, adjusting classes, etc. This is also the time to
+  rebalance machines across available hosts.
+
+\h#restage-reopen|Restage|
+
+  Make symlinks for the new version in \c{private/baseutils} (for both
+  \c{baseutils} and \c{mingw}; the idea is that we will start with those and
+  maybe upgrade later).
+
+  Then cleanup and restage:
+
+  \
+  rm -r staging/0/*
+  rm -r staging/repository/1/*/
+
+  etc/stage -b
+  \
+
+\h#start-stage|Start \c{stage} builds|
+
+  Reboot \c{stage} build hosts. They should next bootstrap and proceed with
+  building all the staged packages. Make sure all the builds of the new
+  development snapshot are successful and there is nothing unbuilt.
+
+\h#commit-reopen|Commit and push \c{etc/} and \c{private/}.|
+
+  Commit and push changes to \c{etc/} and \c{private/}.
+
+"
diff --git a/git/modules b/git/modules
index a793829..d138b98 100644
--- a/git/modules
+++ b/git/modules
@@ -23,10 +23,10 @@ modules="$modules buildos"
 modules="$modules msvc-linux"
 modules="$modules openssl-agent"
 
-#modules="$modules etc"
-#modules="$modules private"
+modules="$modules etc"
+modules="$modules private"
 
-# We don't tag git/, etc/, and private/.
+# We don't tag git/, and etc/, private/, build2-toolchain/ are tagged manually.
 #
 tag_modules="     \
 libbutl           \
@@ -42,13 +42,12 @@ bbot              \
 libstd-modules    \
 msvc-linux        \
 openssl-agent     \
-build2-toolchain  \
 buildos"
 
 # Submodule update in build2-toolchain has to be done manually.
-# As well as in private.
 #
 sub_modules="     \
+etc               \
 private           \
 libbutl           \
 libbutl.bash      \
diff --git a/publish b/publish
deleted file mode 100755
index e9f658f..0000000
--- a/publish
+++ /dev/null
@@ -1,90 +0,0 @@
-#! /usr/bin/env bash
-
-# Publish build2 to build2.org/cppget.org (and brep.cppget.org).
-#
-# The distribution is taken from cppget.org/0/ and packages from
-# cppget.org/repository/.
-#
-# -d
-#    Only publish the distribution.
-#
-# -p
-#    Only publish the packages.
-#
-usage="$0 [options] -- []"
-
-owd=`pwd`
-trap "{ cd $owd; exit 1; }" ERR
-set -o errtrace # Trap in functions.
-
-function info () { echo "$*" 1>&2; }
-function error () { info "$*"; exit 1; }
-
-pkg_only=
-dist_only=
-
-while [ $# -gt 0 ]; do
-  case $1 in
-    -d)
-      dist_only=true
-      shift
-      ;;
-    -p)
-      pkg_only=true
-      shift
-      ;;
-    --)
-      shift
-      break
-      ;;
-    *)
-      error "unexpected $1"
-      ;;
-  esac
-done
-
-if [ -z "$pkg_only" ]; then
-  v="$(sed -n -re 's/^version: ([^.]+\.[^.]+\.[^-]+(-[ab]\.[^.+]+)?).*$/\1/p' build2-toolchain/manifest)"
-
-  if [ -z "$v" ]; then
-    error "unable to extract version build2-toolchain/manifest"
-  fi
-fi
-
-
-function sync ()
-{
-  if [ -z "$pkg_only" ]; then
-    info "build2.org:"
-
-    rsync -v -rlpt -c --copy-unsafe-links --prune-empty-dirs --delete-after \
-"${@}" "cppget.org/0/$v/" "build2.org:/var/www/download.build2.org/public/$v/"
-
-    rsync -v -rlpt -c --copy-unsafe-links --prune-empty-dirs --delete-after \
-"${@}" "cppget.org/0/toolchain.sha256" "build2.org:/var/www/download.build2.org/public/"
-  fi
-
-  if [ -z "$dist_only" ]; then
-    info "cppget.org:"
-    etc/rep-publish cppget.org/repository/1/ cppget.org:/var/bpkg/1/ "${@}"
-
-    info "brep.cppget.org:"
-    etc/rep-publish cppget.org/repository/1/ lists.cppget.org:/var/bpkg/1/ \
-"${@}"
-  fi
-}
-
-sync --dry-run "${@}"
-
-r=
-while [ -z "$r" ]; do
-  read -r -p "Continue? [yes/no]: " r
-
-  case "$r" in
-    yes) ;;
-    no) exit 0 ;;
-    *) r= ;;
-  esac
-done
-
-sync --progress "${@}"
diff --git a/release-bugfix-only.txt b/release-bugfix-only.txt
deleted file mode 100644
index f8bb098..0000000
--- a/release-bugfix-only.txt
+++ /dev/null
@@ -1,79 +0,0 @@
-- Create X.Y branch if first bugfix release
-
-  git branch X.Y X.Y.0
-  git checkout X.Y
-
-- See if makes sense to cherry-pick other fixes/changes
-
-- Update NEWS if anything major, commit
-
-- Update version in manifest, cli.sh, commit and push:
-
-  git ci -a -m "Bump version to X.Y.N"
-  git push
-
-- Regenerate odb.sh, cli.sh
-
-- If need to rebuild build2-toolchain:
-
-  - Create branch as above, then:
-
-    git submodule update --checkout
-
-  - Add the branch value for each entry in .gitmodules:
-
-    branch = X.Y
-
-  - Then checkout and commit updated submodules:
-
-    git submodule update --remote 
-
-  - Update version in manifest as above
-
-  - Regenerate odb.sh (may need to checkout old version or copy old),
-    cli.sh as above.
-
-  - Stage and diff with previous version for sanity check.
-
-- Dist to queue:
-
-  etc/stage-pkg -q -d -c  
-
-  If need to dist build2-toolchain, then rename X.Y.N-1 to X.Y.N in
-  private/baseutils, then:
-
-  etc/stage -b -p
-
-  Verify build2-toolchain works by building locally.
-
-- @@ TODO: Test queue.
-
-- Move package from queue to the appropriate repository, normally
-  replacing the old package.
-
-- Regenerate the repository and publish (remove -p if also publishing
-  build2-toolchain):
-
-    cd cppget.org
-    git -C repository add .
-    git -C repository status
-    ./update
-
-    cd ..
-    etc/publish -p
-
-- Update download links/checksums if changed build2-toolchain.
-
-- Tag the bugfix release:
-
-  git tag -a X.Y.Z -m "Tag version X.Y.Z"
-
-  Also in build2-toolchain if applicable.
-
-- Commit cppget.org/repository/ (see history for procedure)
-
-- Commit private if changed build2-toolchain.
-
-- Write and send announcements, remember to include checksum.
-
-- Switch back to master and regenerate cli.sh/odb.sh
diff --git a/release.txt b/release.txt
deleted file mode 100644
index 350bbaf..0000000
--- a/release.txt
+++ /dev/null
@@ -1,307 +0,0 @@
-@@ Need to sync hello/* projects with latest bdep-new output (and re-tag)
-
-@@ No upgrade testing
-
-- Add new bot machines/configurations/options from stage to queue to public
-
-  @@ See paper in Build OS filder.
-
-  @@ Might have to be done at release if things are incompatible. Maybe
-     better/cleaner to always do this at release.
-
-- Replace 0.7.0 and 0.7 in this document with the new version.
-
-- Remove etc/ and private/ from modules in etc/git/modules to reduce
-  noise during stat. Also, review for any new modules.
-
-- Review '@@' items [private/ excluded, update with new modules, at least
-  look for @@ TMP; also check the review script for any new modules]:
-
-  etc/review | less -R
-
-- Identify packages that will be released (see etc/stage). They should all be
-  staged and built successfully, including submodule-updated build2-toolchain
-  and baseutils/mingw. Review stageing/0/ and staging/1/ for anything stray.
-
-- Update and test local builds:
-
-  b test: build2/ bpkg/ bdep/ brep/ bbot/ libbutl/ libbpkg/ libbbot/
-  b msvc-linux/
-
-- Stage hello projects and test:
-
-  cd hello
-  ./foreach-git pull
-  ./foreach-git stat
-  ./stage
-
-  cd ..
-  etc/intro -s `pwd`/hello/repository/1/
-  etc/intro -s https://hello.stage.build2.org/1
-
-  - Make sure all the bot builds are successful (drop build db).
-
-- Test bootstrap build2-toolchain using scripts/batch files (bot builds
-  are using makefile). Need to test build.sh, build-*.bat. Now test via
-  automated install files: see private/install/testing.txt for details.
-
-- Update NEWS files in all project that will have non-pre-release version.
-
-- Review documentation for (1) sample output changes and (2) still being
-  relevant/making sense:
-
-  - Testscript manual (for 1, see hello/hello-testscript).
-
-  - Install guide for (for 1 & 2).
-
-  - Introduction for (for 1 & 2). Use intro script to get output:
-
-    ./intro2-tldr 2>&1 | tee intro2-tldr.out
-    ./intro2-tour 2>&1 | tee intro2-tour.out
-
-- Change to final versions for all packages being released, some by hand,
-  some with version scripts (see etc/stage for list + msvc-linux).
-
-    - Close schema versions. Review schema changlog difference from previous
-      release (tag) for any data migration that may be required (would also
-      need to test this).
-
-    - Push and update all submodules in buil2-toolchain
-
-    - Regenerate all odb & docs (in both packages and inside build2-toolchain)
-
-    - Bootstrap then update and test local
-
-      cd build2
-      b clean
-      git cout build2/version.hxx build2/b-options.?xx
-      ./bootstrap.sh g++-6
-
-      cd ..
-      b-boot '{clean update}(libbutl/ build2/)'
-
-      b clean: bpkg/ bdep/ brep/ bbot/ libbpkg/ libbbot/ msvc-linux/
-
-      Then the local test step above.
-
-    - Cleanup staging repo:
-
-      rm -r staging/0/*
-      rm -r staging/repository/1/*/
-
-    - Restage with -b:
-
-      etc/stage -b
-
-    - Check /0/ and /1/ for anything stray
-
-    - restage hello (libstd-modules)
-
-- Upgrade brep on cppget.org (queue and main repo) using stage
-
-- Queue packages:
-
-  - Change BUILD2_REPO in build2-toolchain build scripts to queue
-
-  - Change build.version.stage in build2/build2/context.cxx to false.
-
-  - Cleanup queue repo:
-
-    rm -rf cppget.org/0/*
-    rm -rf cppget.org/repository/1/queue/*/
-
-  - Queue
-
-    etc/stage -q -b
-
-  - check /0/ and /1/ for anything stray
-
-    @@ Add extra packages (cli, libcutl, mysql/mariadb).
-
-  - Make sure bot builds are successful
-
-- Upgrade brep on build2.org (hello) using queue
-
-- Publish and test hello
-
-    cd hello
-    ./foreach-git stat
-    ./stage -p
-
-    cd ..
-    etc/intro -s https://build2.org/pkg/1/hello
-
-  - Drop build database (package versions do not change)
-
-  - Make sure bot builds are successful
-
-- Publish and test toolchain
-
-  - Change BUILD2_REPO in build2-toolchain build script to public
-
-  - Regenerate build2-toolchain package in cppget.org/0/
-
-    etc/stage -p
-
-    @@ Remove *-queue install scripts.
-
-  - Move packages (manually for now):
-
-    - old/replaced/FTB packages either to legacy or delete
-
-    - from queue to appropriate repositories (don't override existing)
-
-  - Regenerate the repository
-
-    cd cppget.org
-    git -C repository add .
-    git -C repository status  # Review changes.
-    ./update
-
-  - Disable the queue toolchain on bbots (moved packages from queue/),
-    delete distribution
-
-    echo disabled | ssh build2.org tee >/dev/null \
-      /var/www/download.build2.org/public/queue/toolchain.sha256
-
-    ssh build2.org rm -r /var/www/download.build2.org/public/queue/*/
-
-  - Publish everything to build2.org/cppget.org
-
-    etc/publish
-
-  - Make sure bot builds are successful, drop hello build database again.
-
-    @@ First test automated install scripts.
-
-- Release
-
-  - Update submodules (style/) in private/.
-
-  - Write release notes, use placeholder for announcement URL (or guess).
-    Add link from the doc page.
-
-  - Any new FAQ entries or any other updates (main page, etc)? Any new
-    documentation to link from the doc page?
-
-  - Update download page.
-
-    cat `ls -1 *.sha256`
-
-  - Update install page.
-
-    cat toolchain.sha256
-
-  - Regenerate documentation (./cli.sh) for all hosts.
-
-  - Test locally, publish, test production
-
-    cd private/
-    ./publish --dry-run
-    ./publish
-
-  - Write and send the announcement to mailing lists.
-
-    cat `ls -1 *.sha256`
-
-    - Remember upgrade instructions.
-
-    - Add reply-to: header for users@ announcement.
-
-  - Patch/verify the announcement URL in release notes, re-publish.
-
-  - Announce on reddit and other places (see doc/ann/).
-
-  - Commit and tag private/ and style/
-
-    git tag -a 0.7.0 -m "Tag version 0.7.0" && git push --follow-tags
-
-- Tag & Commit hello
-
-  cd hello
-  ./foreach-git stat
-
-  - commit hello/repository/ (see commit history for procedure)
-
-    git tag -a build2-0.7.0 -m "Tag for build2 version 0.7.0" && git push --follow-tags
-
-
-- Tag & Commit
-
-  - commit & push cppget.org/repository/ (see commit history for procedure;
-    commit big chunks first, then reorder with rebase).
-
-  - Tag and push all released packages, some by hand, some with tag scripts
-    (see etc/stage for list but also packages added ad hoc with etc/stage-pkg).
-
-    @@ Use vX.Y.Z tag for packages that are bpkg/bdep-ready (i.e., have
-       suitable repositories.manifest if required).
-
-    cd 
-
-    # X.Y.Z
-    v="$(sed -n -re 's/version: ([^ ]+)/\1/p' manifest)"; echo $v; read; \
-      git tag -a $v -m "Tag version $v" && git push --follow-tags
-
-    # vX.Y.Z
-    v="$(sed -n -re 's/version: ([^ ]+)/\1/p' manifest)"; echo v$v; read; \
-      git tag -a v$v -m "Tag version $v" && git push --follow-tags
-
-    For build2 projects, review tag_modules in etc/git/modules, then:
-
-    ./tag.sh 0.7.0
-    ./push.sh
-
-- Increment versions and open master for business
-
-  - Change to next development versions for all released packages, some by
-    hand, some with version scripts (see etc/stage for list + msvc-linux).
-
-    cd 
-
-    ~ edit manifest
-
-    v="$(sed -n -re 's/version: ([^ ]+)/\1/p' manifest)"; echo $v; read; \
-      git ci -a -m "Bump version to $v, master is open for business" && git push
-
-  - In build2-toolchain, update all submodules (including libodb*, etc),
-    change BUILD2_REPO to staging.
-
-  - Regenerate odb & cli docs everywhere (packages and build2-toolchain).
-
-    New sequence:
-
-    ~ rebuild ODB compiler
-
-    @@ bdep sync -fura
-    @@ b hxx{version} # first lib*, then projects, then odb.sh
-
-  - Update/test local (see early steps of this list).
-
-  - Make symlinks for new version in baseutils (for both baseutils and mingw,
-    the idea is that we will start with those and maybe upgrade during
-    development).
-
-  - Restage
-
-    rm -r staging/0/*
-    rm -r staging/repository/1/*/
-
-    etc/stage -b
-
-    - Make sure bbot builds of the new development snapshot are successful.
-
-- Commit etc/
-
-  - Restore changes to etc/git/modules
-
-  - Change all '+ ' back to '- '.
-
-  - Commit and tag
-
-    git tag -a 0.7.0 -m "Tag version 0.7.0" && git push --follow-tags
-
-- Snapshot buildos subvolumes as buildos-0.7.0
-
-  btrfs subvolume snapshot buildos-3 buildos-3-0.7.0
-  btrfs subvolume snapshot buildos-6 buildos-6-0.7.0
diff --git a/rep-test b/rep-test
deleted file mode 100755
index 09541b6..0000000
--- a/rep-test
+++ /dev/null
@@ -1,179 +0,0 @@
-#! /usr/bin/env bash
-
-# Test repository.
-#
-# Usage: update [options] 
-#
-# First, the script determines the list of repositories/sections. If 
-# contains the repositories.manifest file, then it is the only repository to
-# be tested. Otherwise, every first-level subdirectory of  that doesn't
-# start with '.' and contains the repositories.manifest file is to be tested.
-#
-# Then, it makes sure that every package in every repository can be built
-# in a clean configuration.
-#
-# -n
-#    Only test new packages. For this to work,  should be (part of) a
-#    git repository. Untracked (and changed) files are considered new.
-#
-# -e 
-#    Exclude the specified sub-directory. Currently only one directory can
-#    be excluded.
-#
-# -t 
-#    Specify the build2 toolchain install/stage directory. The script will
-#    use /bin/b and /bin/bpkd instead of just b and
-#    bpkg.
-#
-# -c