// file : doc/release.cli // license : MIT; see accompanying LICENSE file "\title=Release Process" // NOTES // // - Maximum
line is 70 characters. // "\h1|Table of Contents|" "\$TOC$" " \h0#process|Process| Review the state and services list (currently on paper) for any new additions. Consider how/when they are updated/tested during the release process. @@ We currently have an issue in that \c{queue} builds \c{public} using \c{public} \c{buildtabs} (since it's querying \c{public} brep) which means existing packages are not tested with new build configurations. But maybe that's correct, conceptually. \h1#stage|Stage| The staging repository is completely independent which means it must contain all the \c{build2} dependencies (plus a few extra packages for testing; see the \c{etc/stage} script for the complete list). This in turn means that if any of these dependencies are in the unreleased state, then they should go through the applicable steps in this section (e.g., updating of \c{NEWS}, etc) and then be queued and published (effectively released) as part of the \c{build2} release. Generally, however, we should strive to not unnecessarily bundle the release of dependencies with the release of \c{build2} to keep the process as streamlined as possible. In fact, we now have \c{queue.stage} (see \c{etc/stage-queue}) which is the place for such \"extra packages for testing\". \N|When unbundling the release of a dependency we need to remove its distribution from \c{etc/stage} and add the pre-distributed packages (for example, from \c{public}) to \c{staging/repository/1/}.| \b{Pre-Conditions:} \ul| \li|Current \c{stage} build is clean.|| \h#copy|Update copyright if new year| Use the \c{git/copyright} script. See the script header for instructions. Note that most of (all?) our project have been converted to the automatic copyright update notification via a pre-commit hook. So perhaps this step can be removed. \h#install-times|Update install script times.| See \c{private/install/build2-times.txt} for instructions. \h#etc|Update \c{etc/git/modules}| Review for any new modules. Remove \c{etc/} and \c{private/} from \c{modules} to reduce noise during stat. \h#review|Review \c{@@}| Review \c{etc/review} for new modules. Then review \c{@@} notes: \ etc/review | less -R \ At least look for \c{@@\ TMP} \ etc/review | grep TMP \ \h#review-db|Review database schema changes| Review database schema changelog differences in \c{bpkg}, \c{bdep}, and \c{brep} compared to the previous release (tag) for any schema/data migration that may be required (if something is missing, then it would also have to be tested). \h#news|Update \c{NEWS} files| Update in all projects (including dependencies) that will have a release version. See \c{etc/stage} for the complete list. \h#packaging|Release \c{packaging/} dependencies| Release dependencies that have the snapshot version as described in \l{https://github.com/build2-packaging/README#version-and-release-management Version and Release Management}. Also, consider upgrading the \c{curl/ca-certificates-curl}. \N|Maybe this should be done during queuing? Why do we release (but not publish) these now and other dependencies later? Maybe so that we can stage them one more time?| \h#dependencies|Finalize all other dependencies| Make sure all other unreleased dependencies listed in \c{etc/stage} are ready to be released (\c{NEWS}, etc). Effectively, the only remaining step should be to change the version. Do this in the dependency order and finish each off with: \ git pull bdep sync -fura && bdep test -ar \ For some it may make sense to go straight to release; see \l{#version-release Change to release version}. \h#upgrade-dep|Upgrade dependencies| Upgrade all dependencies in the \c{build2} toolchain build: \ etc/upgrade \ \h#hello|Update \c{hello/} projects| These projects should closely track the output of \c{bdep-new(1)}. The recommended procedure is to generate a new project with the same name and then review/resolve the differences with \c{diff\ -ru}. Every change in these projects must be accompanied by a revision increment and a re-tag. This should be managed with \c{bdep-release(1)}: \ # make changes git add . bdep release --revision --show-push # review commit git push ... \ Once done, run the \c{intro} scripts and review any changes in the output (this information will be helpful on the next step): \ cd etc ./intro2-tldr 2>&1 | tee intro2-tldr.out diff -u intro2-tldr.orig intro2-tldr.out # Or use gitk. mv intro2-tldr.out intro2-tldr.orig ./intro2-tour 2>&1 | tee intro2-tour.out diff -u intro2-tour.orig intro2-tour.out # Or use gitk. mv intro2-tour.out intro2-tour.orig \ \h#doc|Review documentation| Review the following documentation for (1) sample output changes and (2) still being relevant/making sense. Also (3) check the \c{NEWS} files for anything new worth mentioning. \N|Ideally this should be done during development but it's easy to forget. Also, check if there is any new documentation that has been added that is not on the below list.| \ul| \li|Install guide: 1 & 2 (also review \c{build2-toolchain} commit log).| \li|Toolchain introduction: 1, 2 & 3 (use \c{intro} script output for 2).| \li|Introduction in the build system manual: 1 (uses \c{bdep-new(1)} output).| \li|Testscript manual: 1.|| \h#submod|Update all submodules| Make sure all working trees are clean. Review \c{sub_modules} in \c{etc/git/modules} for any missing new modules, then: \N|\c{etc/} and \c{private/} need to be committed manually.| \ ./modup.sh ./commit.sh # If changes. ./push.sh cd build2-toolchain git submodule update --remote --checkout git submodule foreach git submodule update --init --recursive git status # There should only be 'new commits'; see README-GIT git commit -a -m \"Update submodules\" git push \ \h#stage-machines|Update \c{stage} \c{buildtab}s and build machines| Note: normally, we try to do all of this as part of normal development (e.g., when adding new machines, etc). Review \c{stage} \c{buildtab} for any configurations to drop (for example, an intermediate version of a compiler), classes to adjust (\c{legacy}, \c{default}, \c{latest}, etc). Based on these changes update \c{stage} CI \c{buildtab}, which is a subset of the \c{stage} configurations (and is a base for the \c{queue}/\c{public} configurations). Review deployed machines against the updated \c{stage} \c{buildtab} and remove those that are no longer used: \ cd private/buildos/ grep '.*' .../brep-buildtab | cut -d' ' -f1 | sort -u ./ls-machines -c stage -c devel ~/work/buildos/remove-machine\ Also review deployed machines against the latest available versions and upgrade those that are not the latest: \ cd private/buildos/ ./ls-machines -l \"/btrfs/$(whoami)/machines/default/\" | sort ./ls-machines -c stage -c devel ~/work/build2/buildos/upload-machine .../new-ver .../old-ver \ \h#restage|Restage| Confirm \c{stage} and \c{devel} \c{buildos} images, hardware-specific configurations are the same. Review \c{staging/0/} and \c{staging/repository/1/} for anything stray. If no upgrade is possible from the previous version, uncomment errors in install scripts (and add a note to restore after the release). Enable extra packages for testing in \c{etc/stage} script. Restage and upgrade \c{brep} by performing the following steps: \ol| \li|Disable building of all repositories on \c{stage} by adding the \c{buildable:no} field at the end of each line in \c{loadtab}.| \li|Restage with \c{baseutils}/\c{mingw} regeneration: \ etc/stage -b \ | \li|While build machines are bootstrapping, upgrade \c{brep} on \c{stage}, sync latest \c{buildtab}s but do not restart the web server.| \li|Once all build machines have bootstrapped, enabling build of all repositories and restart the web server. To check bootstrap progress: \ ./ls-machines -b stage -c stage -c devel \ || Verify \c{stage} build is clean, nothing is unbuilt. \h#test-extra|Perform extra testing| CI \c{libauto-symexport}. Test \c{cxx20-modules-examples} (see \c{README} in each example). \h#install-stage|Test install scripts| Test \l{https://stage.build2.org/0/ \c{stage} install scripts}, including upgrading, as described in \c{private/install/testing.txt}. Perform necessary upgrades, if any, and test on ad hoc test machines (\c{test-*}). Also test \c{bootstrap-mingw.bat} and \c{bootstrap.sh} (preferably on something less mainstream like FreeBSD) since not exercised as part of install. \h1#queue|Queue| \b{Pre-Conditions:} \ul| \li|Final \c{stage} build is clean.| \li|Build with the \c{queue} toolchain is inactive.|| \h#queue-repo|Review \c{queue} repository| Pull, review, and clean the \c{queue} \c{git} repository for any stale packages or anything (ownership, etc) that may conflict with packages being released. Also clean toolchain distribution: \ rm -rf cppget.org/queue/0/* \ \h#version-release|Change to release version| Change to the release version in all the packages being released and tag (see \c{etc/stage} for the list). Use \c{bdep-release(1)} unless a custom versioning/tagging scripts are used: \ git pull bdep release --no-open --show-push [--alpha|--beta] # review commit, run update (in case anything comitted is pre-generated) git push ... \ Do this in the dependency order and finish each off with: \ bdep sync -fura && bdep update \ For the \c{build2} packages (commit but don't push after each step if required): \ul| \li|Close schema versions in \c{bpkg}, \c{bdep}, and \c{brep}.| \li|Change \c{LIBBUILD2_STAGE} in \c{build2/libbuild2/config.hxx.in} to \c{false}.| \li|If necessary, update minimum \c{build2} and \c{bpkg} version requirements in projects generated by \c{bdep-new}. \b{This must be done if created projects use new features.} \N|Why shouldn't we always do this for simplicity? Maybe because then we cannot run tests using \c{public} services? Also if we change this in toolchain packages, the below upgrade steps will break since there is no continuity. Perhaps do it in two stages: first change version to final, upgrade, then change toolchain dependencies to final, upgrade again? But then nobody involed in development will be able to pull and upgrade. Maybe KISS and keep it pre-release.|| \li|Change version by updating \c{etc/version} (including with new modules and/or new dependencies, but keep pre-release in minimum toolchain version requirements) and then executing: \ etc/version ./commit.sh git -C build2-toolchain commit --amend # \"Change version to X.Y.Z\" \ Note that \c{libbuild2-*} modules (e.g., \c{libbuild2-hello}) are independently versioned but may still need to update minimum toolchain version requirements (see below).| \li|Tag by executing \c{tag.sh\ }.| \li|Release all standard pre-installed build system modules. Update minimum toolchain version requirements. \ bdep release --no-open --show-push \ | \li|Regenerate documentation in each package (including standard pre-installed build system modules, use \c{BDEP_SYNC=0}).| \li|Upgrade all dependencies in configure-only mode by executing \c{etc/upgrade\ -c}. Avoid this (see above): If the \c{build2}/\c{bpkg} requirements in the manifests have been bumped to the version being released, then first bootstrap the build system and update \c{bpkg}/\c{bdep} (might have to hack their generated \c{version.hxx} to disable constraint checking; also if you forget \c{BDEP_SYNC=0} it will most likely hose the build configuration). \ BDEP_SYNC=0 b-boot build2/build2/ BDEP_SYNC=0 b bpkg/ bdep/ \ | \li|Trigger regeneration of version files (might require several runs to \"close off\"): \ BDEP_SYNC=0 b --match-only ~/work/build2/builds/gcc7-asan/ \ If using GCC prior to 8 then might also need explicit version target upgrades: \ BDEP_SYNC=0 b ~/work/build2/builds/gcc7-asan/.../hxx{version} \ | \li|Regenerate ODB in relevant packages passing upgraded configuration path explicitly (\c{bdep} is not runnable): \ ./odb.sh ~/work/build2/builds/gcc7-asan/ \ | \li|Finish upgrading all dependencies by executing in the upgraded configuration: \ BDEP_SYNC=0 b ~/work/build2/builds/gcc7-asan/ b build2/ bpkg/ bdep/ # Update standard pre-installed build system modules. \ | \li|Update other \c{libbuild2-*} modules if required.|| Verify key tests pass (in particular, the \c{bdep} tests will now be running against \c{public} services): \ b test: build2/ bpkg/ bdep/ b test: bpkg/ config.bpkg.test.remote=true b test: libbuild2-hello/libbuild2-hello-tests/ # Test standard pre-installed build system modules. \ \N|We could have queued after this step before preparing \c{build2-toolchain}. However, splitting this seems to only increase complexity without any major benefits (it's not hard to update submodules and regenerated \c{build2-toolchain} if there are any issues with packages). Plus we can start testing install scripts, etc.| Next push the above changes and update all submodules in \c{build2-toolchain}: \ ./push.sh # Push (with tags) standard pre-installed build system modules. cd build2-toolchain git submodule update --remote --checkout git status # There should only be 'new commits'; see README-GIT git commit -a -m \"Update submodules\" \ As well as (commit after each step if required): \ul| \li|Regenerate documentation in each package inside as well as in \c{build2-toolchain} itself.| \li|Update ODB by copying relevant files from the previous step (trust me, this is the easy way for now). Make sure all \c{*-odb.*} are copied!| \li|Change \c{build2_repo} in \c{build2-toolchain} \c{buildfile} to \c{queue}.|| Finally, push the changes: \ git push \ \h#queuing|Queue| Prepare packages and the toolchain distribution (disable extra packages first if any were enabled in the \c{etc/stage} script): \ etc/stage -q -b \ Sort non-alpha packages from \c{cppget.org/queue/1/alpha/} into appropriate sections (we could probably automate this similar to \c{bdep-release(1)}). Also check if any of them are already in \c{public}. Note also that we assume all the packages already have the corresponding ownership information either in \c{queue} or \c{public}. However, if any new packages were added, then that will have to be added as well. Commit and push queue repository: \ cd cppget.org/queue/ git add . git ci -m \"Queue build2 toolchain X.Y.Z\" git push \ If queued package manifests contain new values, then the \c{bpkg-rep-publish} script will fail to create repository due to unknown manifest values. To resolve this we temporarily add (to \c{crontab}) \c{--ignore-unknown} and make a note to restore. \h#build-public|Verify queued packages build with \c{public}| This makes sure that the new version can be built with the old toolchain. While all the packages should build (except perhaps \c{libhello} which, being based on the latest \c{bdep-new(1)} output, may use new features), some tests may fail. In particular, we will be running \c{bpkg} tests using a previous version of the build system and \c{bdep} tests also using a previous version of \c{bpkg}. None of that is or will be supported. So for now we manually review all the failed builds for anything suspicious and in the future we may skip tests if the tool versions mismatch. \h#install-queue|Test install scripts| Test \l{https://download.build2.org/queue/ \c{queue} install scripts}. Here we just smoke-test each script on its \"primary\" platform and make sure \c{queue} URLs/repositories are used. \h#upgrade-brep-bpub|Upgrade \c{brep} and \c{bpub} on \c{cppget.org}| Upgrade \c{brep} and \c{bpub} using the queued toolchain according to \c{private/brep-bpub-upgrade.txt}. After upgrade, drop all package builds for \c{public} and \c{queue} and verify there are no regressions. This makes sure that the \c{brep} services in the new version are usable with the old toolchain. If there are issues, they have to be fixed by publishing/queuing revisions. \N|The \c{bdep} tests that exercise CI and publishing services do it in the simulation mode. To properly test that these services are compatible with the old version of the toolchain we would need to CI/publish a real test package manually.| \h#start-queue|Start \c{queue} builds| Update \c{queue} \c{buildtab} based on the \c{stage} CI \c{buildtab} (normally just a copy sans the sanitized toolchain configurations). \N|Note that the \c{queue} \c{buildtab} is shared between \c{public} and \c{queue} builds. As a result, after this update, \c{public} build hosts may not have some of the new (or renamed) build machines. This also means that if the \c{buildtab} contains anything new (options, etc) that are incompatible with \c{public}, then they should only be enabled later, when upgrading \c{public} \c{buildtab} (make a note if that's the case).| Adjust \c{stage} and \c{devel} build host configurations (both \c{*-config} and hardware classes) to enable the \c{queue} toolchain. Shift most instances from \c{stage} to \c{queue} in the hardware class-specific configurations. Regenerate affected configurations and reboot build hosts: \ cd private/buildos/ ./regen ./po-hosts -r -c stage -c devel \ Verify both \c{queue} and \c{public} builds with the queued toolchain, fixing any breakages with revisions, moving to legacy, and/or queuing replacements. We can only proceed further once we have a \"resolution\" for every (newly) broken package. \h#stop-queue|Stop \c{queue} builds| Adjust \c{stage} and \c{devel} build host configurations to disable the \c{queue} toolchain (comment out). Regenerate affected configurations and reboot build hosts as on the previous step (or poweroff until stage reopening). \h1#public|Public| \b{Pre-Conditions:} \ul| \li|A \"resolution\" is queued or published for every (newly) broken package.|| \h#public-machines|Update \c{public} \c{buildtab}s and build machines| Poweroff the old set of \c{public} build hosts: \ ./po-hosts -c public \ Update \c{public} \c{buildtab}s based on the \c{queue} \c{buildtab} (normally just a copy). Adjust build host configurations (hardware classes, etc) and add/remove new/old build hosts. Replace the \c{public} \c{buildos} image on \c{build-cache} with the one for \c{stage}. Comment out the \c{public} toolchain in the build host configuration (effectively making it a no-toolchain configuration), regenerate, and power on the new set of \c{public} build hosts. Review deployed machines against the updated \c{public} \c{buildtab} and remove those that are no longer used: \ cd private/buildos/ grep '.*' .../brep-buildtab | cut -d' ' -f1 | sort -u ./ls-machines -c public ~/work/build2/buildos/remove-machine \ Then move now legacy machines to the \"legacy\" build host: \ grep 'legacy' .../brep-buildtab | cut -d' ' -f1 | sort -u \ Also review deployed machines against the latest available versions and upgrade those that are not the latest: \ cd private/buildos/ ./ls-machines -l \"/btrfs/$(whoami)/machines/default/\" | sort ./ls-machines -c public ~/work/build2/buildos/upload-machine .../new-ver .../old-ver \ Finally, add any new machines: \ grep -v 'legacy' .../brep-buildtab | cut -d' ' -f1 | sort -u \ Uncomment the \c{public} toolchain in the build host configuration and regenerate. The only remaining step is to reboot (not yet): \ ./po-hosts -r -c public \ \h#pub-dist|Publish distribution| Change \c{BUILD2_REPO} in \c{build2-toolchain} build scripts to \c{public}, commit, and publish the distribution (this also cleans/disables the \c{queue} toolchain): \ etc/stage -p \ \h#pub-pkg|Publish packages| @@ Need to add --min-bpkg-version 0.13.0 Move packages (manually for now) from the \c{queue} to \c{public} \c{git} repository, including ownership information. Move old/replaced/FTB packages either to legacy or delete. Commit everything (see the commit log for procedure @@ this needs automation) and push (which should trigger auto-publish). Note: commit and push both \c{queue} and \c{public} \c{git} repositories. Note that once published, the existing install instructions/download links are no longer usable, so do not linger (in fact, may make sense to update Download and Install pages before publishing packages and only sync them immediately after). \h#start-public|Start \c{public} builds| Reboot \c{public} build hosts. They should next bootstrap and proceed with building all packages. Once all the machines have bootstrapped, submit a test CI task. \h#install-public|Test install scripts| Test \l{https://download.build2.org/ \c{public} install scripts}. Here we just smoke-test each script on its \"primary\" platform and make sure \c{public} URLs/repositories are used. \h#web|Update web| \ul| \li|Write release notes, use placeholder for announcement URL (or guess). Add link from the \c{doc.cli}. Add blog entry.| \li|Add any new FAQ entries or any other updates (main page, etc). Any new documentation to link from the Doc page?| \li|Update the Download page. \ cat `ls -1 cppget.org/public/0/X.Y.Z/*.sha256` \ | \li|Update the Install page. \ cat cppget.org/public/0/toolchain.sha256 \ | \li|Regenerate documentation in \c{private/} for all hosts: \ cd private/ cd /www/ ./cli.sh \ | \li|Test locally, publish, and test remote: \ cd private/ ./publish --dry-run ./publish \ || \h#ann|Announce| \ul| \li|Write and send the announcement to the mailing lists. \ cat `ls -1 cppget.org/public/0/X.Y.Z/*.sha256` \ Add \c{reply-to:} header for \c{users@} announcement.| \li|Patch (or verify) the announcement URL in release notes, re-publish.| \li|Announce on reddit, slack, and other relevant places.| \li|Add links to reddit discussions to release notes.|| \h#tag|Commit, tag, and push the rest.| Tag \c{build2-toolchain}: \ cd build2-toolchain git tag -a vX.Y.Z -m \"Tag version X.Y.Z\" git push --follow-tags \ Add \c{etc/} and \c{private/} back to \c{modules} in \c{etc/git/modules} (essentially reverse \l{#etc Update \c{etc/git/modules}}). Finalize changes, commit, tag, and push \c{style/}, \c{etc/}, and \c{private/}. Snapshot \c{buildos} subvolumes: \ btrfs subvolume snapshot buildos-3 buildos-3-X.Y.Z btrfs subvolume snapshot buildos-6 buildos-6-X.Y.Z \ \h1#reopen|Reopen| \h#version-snapshot|Change to snapshot version| Change to the snapshot version in all the released packages. Use \c{bdep-release(1)} unless a custom versioning script is used: \ bdep release --open --show-push [--open-*] # review commit git push ... \ Essentially, the same steps as in \l{#version-release Change to release version} (but no tagging). Revert changes to install scripts if upgrade was disabled. \h#stage-machines-reopen|Update \c{stage} \c{buildtab}s and build machines| Essentially, the same steps as in \l{#public-machines Update \c{public} \c{buildtab}s and build machines} but for stage. Some differences: Clean \c{buildtab}s (both \c{stage} and CI) by removing no longer relevant configurations, moving some to \c{legacy}, etc. More generally, this is the time to do sweeping changes such as renaming machines/configurations, adjusting classes, etc. This is also the time to rebalance machines across available hosts. \h#restage-reopen|Restage| Make symlinks for the new version in \c{private/baseutils} (for both \c{baseutils} and \c{mingw}; the idea is that we will start with those and maybe upgrade later). Then cleanup and restage: \ rm -r staging/0/* rm -r staging/repository/1/*/ etc/stage -b \ \h#start-stage|Start \c{stage} builds| Reboot \c{stage} build hosts. They should next bootstrap and proceed with building all the staged packages. Make sure all the builds of the new development snapshot are successful and there is nothing unbuilt. Upgrade \c{brep} on \c{stage}. \h#commit-reopen|Commit and push \c{etc/} and \c{private/}.| Commit and push changes to \c{etc/} and \c{private/}. \h1#upgrade|Upgrade| \h#upgrade-baseutils|Upgrade \c{baseutils}| Check for abnormalities in new package sizes. Restage: \ rm -r staging/0/* etc/stage -b \ Remember to update the GCC version in the stage \c{buildtab}s for the configuration that uses \c{mingw} package. \h#upgrade-db-migration|Cleanup database migrations| Drop old database migration code. Keep migration entries for two previous versions for \c{bpkg} and \c{bdep} and for one previous version for \c{brep}. \h#upgrade-buildos|Upgrade \c{buildos}| See \c{bootstrap.txt} for the procedure. Start with the \c{devel} configuration and upgrade \c{stage} after testing. \h#upgrade-build-machines|Upgrade build machines| Check every major compiler and OS if a new version is available. \h#upgrade-packages|Upgrade third-party packages| Consider upgrading to new upstream versions in \c{packaging/}. \h#upgrade-projects|Upgrade own projects| Adjust own projects to use any newly available features. \h#upgrade-bdep-new|Upgrade \c{bdep-new}| Adjust \c{bdep-new}-generated projects to use any newly available features. \h1#re-review|Re-review \c{@@}| Review \c{@@} notes (in case some should be fixed after the release): \ etc/review | less -R \ At least look for \c{@@\ TMP} \ etc/review | grep TMP \ \h1#plan|Plan| Plan the next release and update the project roadmap. "