Further work on packaging guide

1 files changed, 292 insertions, 11 deletions

diff --git a/doc/packaging.cli b/doc/packaging.cli
index 8a55296..c5d1863 100644
--- a/doc/packaging.cli
+++ b/doc/packaging.cli
@@ -1104,11 +1104,11 @@ plan is as follows:
 
 \li|Make a smoke test for the library.|
 
-\li|Replace the smoke tests with upstream tests.|
+\li|Replace the smoke test with upstream tests.|
 
 \li|Tweak root \c{buildfile} and \c{manifest}.|
 
-\li|Test the result using the CI service.|
+\li|Test the result using the CI service. @@ Actually doing it as ready.|
 
 |
 
@@ -2371,6 +2371,7 @@ subdirectory which contains the generated test and which is what we will be
 turning into a smoke test. The subproject root \c{buildfile} rarely needs
 changing.
 
+
 \h2#core-test-smoke-build-wide|Review project-wide build system files in \c{tests/build/}|
 
 Review the generated \c{bootstrap.build} and \c{root.build} (there will be no
@@ -2445,7 +2446,7 @@ exe{driver}: {hxx cxx}{**} $libs
 \
 
 
-\h2#core-test-smoke-localy|Test locally|
+\h2#core-test-smoke-locally|Test locally|
 
 With the smoke test ready, we can finally do some end-to-end testing of our
 library build. We will start with doing some local testing to catch basic
@@ -2471,7 +2472,7 @@ initialized} your library in several:
 $ bdep test -a
 \
 
-\h2#core-test-smoke-localy-install|Test locally: installation|
+\h2#core-test-smoke-locally-install|Test locally: installation|
 
 Once this works, let's test the installed version of the library. In
 particular, this makes sure that the public headers are installed in a way
@@ -2514,7 +2515,7 @@ Once done testing the installed case, let's clean things up:
 $ rm -r /tmp/install /tmp/libfoo-tests-out
 \
 
-\h2#core-test-smoke-localy-dist|Test locally: distribution|
+\h2#core-test-smoke-locally-dist|Test locally: distribution|
 
 Another special case worth testing is the preparation of the source
 distribution (see \l{b#intro-operations-dist Distributing} for
@@ -2581,6 +2582,7 @@ $ cd foo/ # Change to the package repository root.
 $ git add .
 $ git status
 $ git commit -m \"Add smoke test\"
+$ git push
 
 $ bdep ci
 \
@@ -2613,17 +2615,298 @@ that it will be fixed in the next upstream version. Note that in this case you
 should not exclude the failing build from CI.|
 
 
+\h#core-test-upstream|Replace smoke test with upstream tests|
+
+With the smoke test working we can now proceed with replacing it with the
+upstream tests.
+
+
+\h2#core-test-upstream-understand|Understand how upstream tests work|
+
+While there are some commonalities in how C/C++ libraries are normally built,
+when it comes to tests there is unfortunately little common ground in how they
+are arranged, built, and executed. As a result, the first step in dealing with
+upstream tests is to study the existing build system and try to understand how
+they work. To get you started, below are some of the questions you would
+likely need answered before you can proceed:
+
+\ul|
+
+\li|\b{Are upstream tests unit tests or integration tests?}
+
+While the distinction is often fuzzy, for our purposes the key differentiator
+between unit and integration tests is which API they use: integration tests
+only use the library's public API while unit tests need access to the
+implementation details.
+
+Normally (but not always), unit tests will reside next to the library source
+code since they need access to more than just the library binary (individual
+object files, utility libraries, etc). While integration tests are normally
+(but again not always) placed into a seperate subdirectory, usually called
+\c{tests} or \c{test}.
+
+If the library has unit tests, then refer to \l{b#intro-unit-test Implementing
+Unit Testing} for background on how to hanle them in \c{build2}.
+
+If the library has integration tests, then use them to to replace (or
+complement) the smoke test.
+
+If the library has unit tests but no integration tests, then it's recommended
+to keep the smoke test since that's the only way the library will be tested
+via its public API.|
+
+
+\li|\b{Do upstream tests use an external testing framework?}
+
+Oftentimes a C++ library will use an external testing framework to implement
+tests. Popular choices include \l{https://cppget.org/catch2 \c{catch2}},
+\l{https://cppget.org/gtest \c{gtest}}, \l{https://cppget.org/doctest
+\c{doctest}}, and \l{https://cppget.org/libboost-test \c{libboost-test}}.
+
+If a library uses such an external testing framework, then it is recommended
+to factor tests into a separate package in order to avoid making the library
+package depend on the testing framework (which is only required during
+testing). See
+\l{https://github.com/build2/HOWTO/blob/master/entries/handle-tests-with-extra-dependencies.md
+How do I handle tests that have extra dependencies?} for details.
+
+\N|Sometimes you will find that upstream bundles the source code of the
+testing framework with their tests. This is especially common with
+\c{catch2}. If that's the case, it is strongly recommended that you
+\"unbundle\" it by making it a proper external dependency.||
+
+
+\li|\b{Are upstream tests in a single or multiple executables?}
+
+It's not unusual for libraries to have a single test executable that runs all
+the test cases. This is especially common if a C++ testing framework is used.
+In this case it is natural to replace the contents of the smoke test with the
+upstream source code, potentially renaming the test subdirectory (\c{basics/})
+to better match upstream naming.
+
+If upstream has multiple test executables, then they could all be in single
+test subdirectory (potentially reusing some common bits) or spread over
+multiple subdirectories. In both cases it's a good idea to follow the upstream
+structure unless you have good reasons to deviate. In the former case (all
+executables in the same subdirectory), you can re-purpose the smoke test
+subdirectory. In the latter case (each executable in a separate subdirectory)
+you can make copies of the smoke test subdirectory.|
+
+
+\li|\b{Are upstream tests well behaved?}
+
+Unfortunately it's not uncommon for upstream tests not to behave well, such as
+write diagnostics to \c{stdout} instead of \c{stderr}, create temporary files
+without cleaning them up, or assume presence of input files in the current
+working directory. For details on how to deal with such situations see
+\l{https://github.com/build2/HOWTO/blob/master/entries/sanitize-test-execution.md
+How do I sanitize the execution of my tests?}||
+
+
+\h2#core-test-upstream-convert|Convert smoke test to upstream tests|
+
+Once you have a good grasp of how upstream tests work, convert or replace the
+smoke test with the upstream tests. If upstream has multiple test executables,
+you may want to deal with one test at a time, making sure that it passes
+before moving to the next one.
+
+It's normally a good idea to use the smoke test \c{buildfile} as a starting
+point for upstream tests. To recap, the smoke test \c{buildfile} for our
+\c{libfoo} example ended up looking like this:
+
+\
+import libs = libfoo%lib{foo}
+
+exe{driver}: {hxx cxx}{**} $libs
+\
+
+At a minimum you will most likely need to change the name of the executable to
+match upstream. If you need to build multiple executables in the same
+directory, then it's probably best to get rid of the name pattern for the
+source files and specify the prerequisite names explicitly, for example:
+
+\
+import libs = libfoo%lib{foo}
+
+./: exe{test1}: cxx{test1} $libs
+./: exe{test2}: cxx{test2} $libs
+\
+
+If you have a large number of such test executables, then a \c{for}-loop might
+be a more scalable option:
+
+\
+import libs = libfoo%lib{foo}
+
+for src: cxx{test*}
+  ./: exe{$name($src)}: $src $libs
+\
+
+
+\h2#core-test-upstream-locally|Test locally|
+
+With the upstream tests ready, we re-do the same end-to-end testing as we did
+with the smoke test:
+
+\l{#core-test-smoke-locally Test locally}\n
+\l{#core-test-smoke-locally-install Test locally: installation}\n
+\l{#core-test-smoke-locally-dist Test locally: distribution}\n
+
+
+\h2#core-test-upstream-ci|Commit and test with CI|
+
+With local testing complete, we commit our changes and submit a remote CI
+job. This step is similar to what \l{#core-test-smoke-ci we did for the smoke
+test} but this time we are using the upstream tests:
+
+\
+$ cd foo/ # Change to the package repository root.
+$ git add .
+$ git status
+$ git commit -m \"Add upstream tests\"
+$ git push
+
+$ bdep ci
+\
+
+
+\h#core-examples-banchmarks|Add upstream examples, benchmarks, if any|
+
+If the upstream project provides examples and/or benchmarks and you wish to
+add them to the \c{build2} build (which is not strictly necessary for the
+\c{build2} package to be usable), then now is a good time to do that.
+
+As was mentioned in \l{#core-package-review Review and test auto-genetated
+\c{buildfile} templates}, the recommended approach is to copy the \c{tests/}
+subproject (potentially from the commit history before the smoke test was
+replaced with the upstream tests) and use that as a starting point for
+examples and/or benchmarks. Just do not forgeting to add the corresponding
+entry in the root \c{buildfile}.
+
+Once that is done, follow the same steps as in \l{#core-test-upstream Replace
+smoke test with upstream tests} to add upstream examples/benchmarks and test
+the result.
+
+
+\h#core-root|Adjust root \c{buildfile} and \c{manifest}|
+
+The last few files that we need to review and potentially adjust are
+the root \c{buildfile} and package \c{manifest}.
+
+
+\h2#core-root-buildfile|Adjust root \c{buildfile}|
+
+The main function of the root \c{buildfile} is to pull all the subdirectories
+that need building plus list targets that are usually found in the root
+directory of a project, typically \c{README.md}, \c{LICENSE}, etc. This is
+what the generated root \c{buildfile} looks like for our \c{libfoo} project
+assuming we have symlinked \c{README.md} and \c{LICENSE} from upstream on the
+\l{#core-package-create Create final package} step:
+
+@@ PACKAGE-README.md?
+
+\
+./: {*/ -build/} doc{README.md} legal{LICENSE} manifest
+
+# Don't install tests.
+#
+tests/: install = false
+\
+
+If the upstream project provides any other documentation (change log, news,
+etc) or legal files (list of authorts, code of conduct, etc), then you may
+want to symlink and list them as the \c{doc{\}} and \c{legal{\}}
+prerequisites, respectively.
+
+\N|One file you don't need listing is \c{INSTALL} (or equivalent) which
+normally contains the installation instructions for the upstream build
+system. In the \c{build2} package the \c{PACKAGE-README.md} file serves this
+purpose.|
+
+
+\h2#core-root-buildfile-doc|Adjust root \c{buildfile}: other subdirectories|
+
+If the upstream project has other subdirectories that makes sense to include
+into the \c{build2} package, then now is good time to take care of that.  The
+most common such case will be extra documentation (besides the root
+\c{README}), typically in a subdirectory called \c{doc/}, \c{docs/}, or
+\c{documentation/}.
+
+The typical procedure for handling such subdirectories will be to symlink the
+relevant files (or the entire subdirectory) and then list the files as
+prerequisites. For this last step, there are two options: we can list the
+files directly in the root \c{buildfile} or we can create a seperate
+\c{buildfile} in the subdirectory.
+
+Let's examine both approaches using our \c{libfoo} as an example. Assume that
+upstream \c{libfoo} contains the \c{docs/} subdirectory with additional
+\c{*.md} files that document its API. It would make sense to include them into
+the \c{build2} package.
+
+Listing the subdirectory files directly in the root \c{buildfile} works best
+for simple case, where you have a bunch of static files that don't require any
+customizations, such as to their installation location. In this case we can
+symlink the entire \c{docs/} subdirectory:
+
+\
+$ cd libfoo/ # Change to the package root.
+$ ln -s ../upstream/docs ./
+\
+
+The adjustment to the root \c{buildfile} are pretty straightforward: we
+exclude the \c{docs/} subdirectory (since it has no \c{buildfile}) and list
+the \c{*.md} files as prerequisites using the \c{doc{\}} target type (which,
+in particular, makes sure they are installed into the appropriate location):
+
+\
+./: {*/ -build/ -docs/} \
+    doc{README.md} docs/doc{*.md} \
+    legal{LICENSE} manifest
+\
+
+The alternative approach (create a seperate \c{buildfile}) is a good choice if
+things are more complicated then that. Let's say we need to adjust the
+installation location of the files in \c{docs/} because there is another
+\c{README.md} that would conflict with the root one when installed into the
+same location. This time we cannot symlink the top-level \c{docs/}
+subdirectory (because we need to place a \c{buildfile} there). The two options
+here is to either symlink the individual files or introduce another
+subdirectory level inside \c{docs/} (which is the same approach as discussed
+in \l{#dont-main-target-root-buildfile Don't build your main targets in root
+\c{buldfile}}). Let's illustrate both sub-cases.
+
+Symlinking individual files works best when you don't expect the set of
+files to change often. For example, if \c{docs/} contains a man page and
+its HTML rendering, then it's unlikely this set will change often. On the
+other hand, if \c{docs/} contains a manual split into an \c{.md} file per
+chapter, then there is good chance this set of files will fluctuate between
+releases.
+
+
+@@ Commit.
+
+
+@@ Any other upstream files besides source? Maybe do Doc here?
+@@ We could just list them in the root buildfile... Could do any
+@@ extra files like this.
+
+
+\h2#core-root-manifest|Adjust \c{manifest}|
+
+@@ changes-file
+
+@@ Maybe we should start with pre-release of upstream version? If want
+   strict versioning?
+
+@@ Maybe re-CI at the ends?
 
-@@ Next section: convert smoke test to upstream tests.
-@@ Next section: adjust root buildfile and manifest.
 @@ Next section: release and publish (see README in build2-packaging/).
+@@ GH issue #?? has some notes.
 
 ========
 
 @@ Add example of propagating config.libfoo.debug to macro on build options?
 
-@@ Upstream tests: link to HOWTO on how to sanitize.
-
 @@ Note on library metadata where talk about configuration. Also about
    autoconf.
 
@@ -2631,8 +2914,6 @@ should not exclude the failing build from CI.|
 
 @@ Squash commits?
 
-@@ Any other upstream files besides source? Doc?
-
 @@ The 'Don't write buildfiles by hand entry' is now mostly duplicate/redundant.
 
 ======================================================================