diff options
Diffstat (limited to 'doc/testscript.cli')
| -rw-r--r-- | doc/testscript.cli | 495 |
1 files changed, 432 insertions, 63 deletions
diff --git a/doc/testscript.cli b/doc/testscript.cli index 50f975d..c539903 100644 --- a/doc/testscript.cli +++ b/doc/testscript.cli @@ -622,15 +622,27 @@ By convention, the testscript file should be called either \c{testscript} if you only have one or have the \c{.testscript} extension, for example, \c{basics.testscript}. The \c{test} module registers the \c{testscript{\}} target type to be used for testscript files. We don't have to use explicit -target type for the \c{testscript} file. +target type for the \c{testscript} file. For example: + +\ +exe{hello}: testscript{basics advanced} +\ A testscript prerequisite can be specified for any target. For example, if -our directory contains a bunch of shell scripts that we want to test together, +our directory contains a bunch of executables that we want to test together, then it makes sense to specify the testscript prerequisite for the directory target: \ -./: testscript{basics} +./: testscript +\ + +Similarly, the same testscript can be used to test multiple targets. For +example: + +\ +exe{hello}: testscript{basics advanced} +exe{hello-lite}: testscript{basics} \ During variable lookup if a variable is not found in one of the testscript @@ -741,6 +753,68 @@ Note also that these \c{test.*} variables only establish a convention. You could also put everything into, say \c{test.arguments}, and it will still work as expected. +\N|The \c{test.redirects}, \c{test.cleanups}, and \c{$*} variables are of the +special \c{cmdline} type, see \l{#lexical Lexical Structure} for details.| + +The special \c{test.*} variables make it fairly easy to arrange the testing of +a single executable. What if we need to run multiple executables from a single +testscript file? For example, we may have a pair of executables, such as +\c{reader} and \c{writer}, that must be tested together. Or we may have a +number of test executables that all require a common setup, for example, +cryptographic key generation, which we would like not to repeating for each +test. While it is possible to achieve this with target-specific variables +similar to \c{test}, things will be less automatic. In particular, there +will be no automatic translation of target names to paths and we will have +to do it manually. For example: + +\ +# buildfile + +./: exe{reader}: cxx{reader} ... +./: exe{writer}: cxx{writer} ... + +./: testscript +{ + reader = exe{reader} + writer = exe{writer} +} +\ + +\ +# testscript + +# Translate targets to paths. +# +reader = $path($reader) +writer = $path($writer) + +: pipe +: +$writer | $reader + +: file +: +$writer output; +$reader output +\ + +\N|Strictly speaking, for local executables, there is no need to pass the +target names from \c{buildfile} to \c{testscript} and instead we could just +list them literally in \c{testscript}. In particular, this could be an +attractive approach if we have a large number of such executables. For +example: + +\ +# testscript + +$path(exe{test1}) : test1 +$path(exe{test2}) : test2 +$path(exe{test3}) : test3 +... +\ + +| + Another pre-defined variable is \c{test.target}. It is used to specify the test target platform when cross-testing (for example, when running Windows test on Linux under Wine). Normally, you would set it in your @@ -964,6 +1038,9 @@ the teardown commands are executed sequentially and in the order specified. Again, if any of them fail, the group execution is terminated and the group is considered to have failed. +\N|Currently, the only way to run several executables serially is to place +them into a single compound test. See \l{#syntax-test Test} for details.| + As an example, consider the following version of \c{basics.testscript}: \ @@ -1285,62 +1362,54 @@ here-document single-quoted here_line_single here-document double-quoted here_line_double expansions \ -Finally, unquoted expansions in command lines (test, setup, and teardown) are -re-lexed in the \c{command_expansion} mode in order to recognize command line -syntax tokens (redirects, pipes, etc). To illustrate why this re-lexing is -necessary, consider the following example of a \"canned\" command line: +Finally, unquoted expansions in command lines (test, setup, and teardown) of +the special \c{cmdline} type are re-lexed in the \c{command_expansion} mode in +order to recognize command line syntax tokens (redirects, pipes, etc). To +illustrate this mechanism, consider the following example of a \"canned\" +command line: \ -x = echo >- -$x foo +cmd = [cmdline] echo >- +$cmd foo \ -The test command line token sequence will be \c{$}, \c{x}, \c{foo}. After the -expansion we have \c{echo}, \c{>-}, \c{foo}, however, the second element -(\c{>-}) is not (yet) recognized as a redirect. To recognize it we re-lex -the result of the expansion. +The test command line token sequence will be \c{$}, \c{cmd}, \c{foo}. After +the expansion we have \c{echo}, \c{>-}, \c{foo}, however, the second element +(\c{>-}) is not (yet) recognized as a redirect. To recognize it, the result of +the expansion is re-lexed. Note that besides the few command line syntax characters, re-lexing will also \"consume\" quotes and escapes, for example: \ -args = \"'foo'\" # 'foo' -echo $args # echo foo +cmd = [cmdline] echo \"'foo'\" # echo 'foo' +$cmd # echo foo \ To preserve quotes in this context we need to escape them: \ -args = \"\\\\'foo\\\\'\" # \'foo\' -echo $args # echo 'foo' -\ - -Alternatively, for a single value, we could quote the expansion (in order -to suppress re-lexing; note, however, that quoting will also inhibit -word-splitting): - -\ -arg = \"'foo'\" # 'foo' -echo \"$arg\" # echo 'foo' +cmd = [cmdline] echo \"\\\\'foo\\\\'\" # echo \'foo\' +$cmd # echo 'foo' \ To minimize unhelpful consumption of escape sequences (for example, in Windows paths), re-lexing only performs the \i{effective escaping} for the \c{'\"\\} characters. All other escape sequences are passed through uninterpreted. Note -that this means there is no way to escape command line syntax characters. The -recommendation is to use quoting except for passing literal quotes, for -example: +that this means there is no way to escape command line syntax characters in +canned commands. The recommendation is to use quoting except for passing +literal quotes, for example: \ -args = \'&foo\' # '&foo' -echo $args # echo &foo +cmd = [cmdline] echo \'&foo\' # echo '&foo' +$cmd # echo &foo \ To make sure that a string is passed as is through both expansions use the \i{doubled single-quoting} idiom, for example: \ -filter = sed -e \''s/foo (bar|baz)/$&/'\' +filter = [cmdline] sed -e \''s/foo (bar|baz)/$&/'\' $* <<EOI | $filter >>EOO ... EOI @@ -1423,6 +1492,7 @@ while potentially spanning several physical lines. The \c{-line} suffix here signifies a \i{logical line}, for example, a command line plus its here-document fragments. + \h#syntax-grammar|Grammar| The complete grammar of the Testscript language is presented next with the @@ -1479,33 +1549,58 @@ test: +(variable-line|command-like) variable-like: - variable-line|variable-if + variable-line|variable-flow variable-line: <variable-name> ('='|'+='|'=+') value-attributes? <value> ';'? value-attributes: '[' <key-value-pairs> ']' +variable-flow: + variable-if|variable-for|variable-while + variable-if: ('if'|'if!') command-line - variable-if-body + variable-flow-body *variable-elif ?variable-else - 'end' + 'end' ';'? variable-elif: ('elif'|'elif!') command-line - variable-if-body + variable-flow-body variable-else: 'else' - variable-if-body + variable-flow-body -variable-if-body: +variable-flow-body: *variable-like +variable-for: + variable-for-args|variable-for-stream + +variable-for-args: + 'for' <variable-name> element-attributes? ':' \ + value-attributes? <value> + variable-flow-body + 'end' ';'? + +element-attributes: value-attributes + +variable-for-stream: + (command-pipe '|')? \ + 'for' (<opt>|stdin)* <variable-name> element-attributes? (stdin)* + variable-flow-body + 'end' ';'? + +variable-while: + 'while' command-line + variable-flow-body + 'end' ';'? + command-like: - command-line|command-if + command-line|command-flow command-line: command-expr (';'|(':' <text>))? *here-document @@ -1518,24 +1613,47 @@ command: <path>(' '+(<arg>|redirect|cleanup))* command-exit? command-exit: ('=='|'!=') <exit-status> +command-flow: + command-if|command-for|command-while + command-if: ('if'|'if!') command-line - command-if-body + command-flow-body *command-elif ?command-else 'end' (';'|(':' <text>))? command-elif: ('elif'|'elif!') command-line - command-if-body + command-flow-body command-else: 'else' - command-if-body + command-flow-body -command-if-body: +command-flow-body: *(variable-line|command-like) +command-for: + command-for-args|command-for-stream + +command-for-args: + 'for' <variable-name> element-attributes? ':' \ + value-attributes? <value> + command-flow-body + 'end' (';'|(':' <text>))? + +command-for-stream: + (command-pipe '|')? \ + 'for' (<opt>|stdin)* <variable-name> element-attributes? (stdin)* + command-flow-body + 'end' (';'|(':' <text>))? + +command-while: + 'while' command-line + command-flow-body + 'end' (';'|(':' <text>))? + redirect: stdin|stdout|stderr stdin: '0'?(in-redirect) @@ -1568,6 +1686,12 @@ description: +(':' <text>) \ +Note that the only purpose of having separate (from the command flow control +constructs) variable-only flow control constructs is to remove the error-prone +requirement of having to specify \c{+} and \c{-} prefixes in group +setup/teardown. + + \h#syntax-script|Script| \ @@ -1578,6 +1702,7 @@ script: A testscript file is an implicit group scope (see \l{#model Model and Execution} for details). + \h#syntax-scope|Scope| \ @@ -1627,6 +1752,7 @@ the scopes in an \c{if-else} chain are alternative implementations of the same group/test (thus the single description). If at least one of them is a group scope, then all the others are treated as groups as well. + \h#syntax-directive|Directive| \ @@ -1640,7 +1766,7 @@ variable is assigned. You can, however, use variables assigned in the buildfile. For example: \ -include common-$(cxx.target.class).testscript +.include common-$(cxx.target.class).testscript \ \h2#syntax-directive-include|Include| @@ -1659,6 +1785,7 @@ this scope should not be included again. The implementation is not required to handle links when determining if two paths are to the same file. Relative paths are assumed to be relative to the including testscript file. + \h#syntax-setup-teardown|Setup and Teardown| \ @@ -1672,11 +1799,12 @@ setup-line: '+' command-like tdown-line: '-' command-like \ -Note that variable assignments (including \c{variable-if}) do not use the +Note that variable assignments (including \c{variable-flow}) do not use the \c{'+'} and \c{'-'} prefixes. A standalone (not part of a test) variable assignment is automatically treated as a setup if no tests have yet been encountered in this scope and as a teardown otherwise. + \h#syntax-test|Test| \ @@ -1695,11 +1823,16 @@ cat <'verbose = true' >=$conf; test1 $conf \ +\N|As discussed in \l{#model Model and Execution}, tests are executed in +parallel. Currently, the only way to run several executables serially is to +place them into a single compound test.| + + \h#syntax-variable|Variable| \ variable-like: - variable-line|variable-if + variable-line|variable-flow variable-line: <variable-name> ('='|'+='|'=+') value-attributes? <value> ';'? @@ -1718,25 +1851,26 @@ echo $args # foo bar fox baz The value can only be followed by \c{;} inside a test to signal the test continuation. + \h#syntax-variable-if|Variable-If| \ variable-if: ('if'|'if!') command-line - variable-if-body + variable-flow-body *variable-elif ?variable-else - 'end' + 'end' ';'? variable-elif: ('elif'|'elif!') command-line - variable-if-body + variable-flow-body variable-else: 'else' - variable-if-body + variable-flow-body -variable-if-body: +variable-flow-body: *variable-like \ @@ -1760,15 +1894,90 @@ with a ternary operator is often more concise: slash = ($cxx.target.class == 'windows' ? \\\\ : /) \ -Note also that the only purpose of having a separate (from \c{command-if}) -variable-only if-block is to remove the error-prone requirement of having to -specify \c{+} and \c{-} prefixes in group setup/teardown. + +\h#syntax-variable-for|Variable-For| + +\ +variable-for: + variable-for-args|variable-for-stream + +variable-for-args: + 'for' <variable-name> element-attributes? ':' \ + value-attributes? <value> + variable-flow-body + 'end' ';'? + +variable-for-stream: + (command-pipe '|')? \ + 'for' (<opt>|stdin)* <variable-name> element-attributes? (stdin)* + variable-flow-body + 'end' ';'? + +variable-flow-body: + *variable-like +\ + +A group of variables can be set in a loop while iterating over elements of a +list. The iteration semantics is the same as in \c{command-for}. For example: + +\ +uvalues = +for v: $values + uvalues += $string.ucase($v) +end +\ + +Another example: + +\ +uvalues = +cat values.txt | for -n v + uvalues += $string.ucase($v) +end +\ + +Or using the \c{stdin} redirect: + +\ +uvalues = +for -n v <=values.txt + uvalues += $string.ucase($v) +end +\ + + +\h#syntax-variable-while|Variable-While| + +\ +variable-while: + 'while' command-line + variable-flow-body + 'end' ';'? + +variable-flow-body: + *variable-like +\ + +A group of variables can be set in a loop while the condition evaluates to +\c{true}. The condition \c{command-line} semantics is the same as in +\c{scope-if}. For example: + +\ +uvalues = +i = [uint64] 0 +n = $size($values) +while ($i != $n) + uvalues += $string.ucase($values[$i]) + i += 1 +end +\ + \h#syntax-command|Command| \ command-like: - command-line|command-if + command-line|command-flow command-line: command-expr (';'|(':' <text>))? *here-document @@ -1783,7 +1992,7 @@ command-exit: ('=='|'!=') <exit-status> \ A command line is a command expression. If it appears directly (as opposed to -inside \c{command-if}) in a test, then it can be followed by \c{;} to signal +inside \c{command-flow}) in a test, then it can be followed by \c{;} to signal the test continuation or by \c{:} and the trailing description. A command expression can combine several command pipes with logical AND and OR @@ -1808,25 +2017,26 @@ to succeed (0 exit code). The logical result of executing a command is therefore a boolean value which is used in the higher-level constructs (pipe and expression). + \h#syntax-command-if|Command-If| \ command-if: ('if'|'if!') command-line - command-if-body + command-flow-body *command-elif ?command-else 'end' (';'|(':' <text>))? command-elif: ('elif'|'elif!') command-line - command-if-body + command-flow-body command-else: 'else' - command-if-body + command-flow-body -command-if-body: +command-flow-body: *(variable-line|command-like) \ @@ -1846,6 +2056,108 @@ end; test1 $foo \ + +\h#syntax-command-for|Command-For| + +\ +command-for: + command-for-args|command-for-stream + +command-for-args: + 'for' <variable-name> element-attributes? ':' \ + value-attributes? <value> + command-flow-body + 'end' (';'|(':' <text>))? + +command-for-stream: + (command-pipe '|')? \ + 'for' (<opt>|stdin)* <variable-name> element-attributes? (stdin)* + command-flow-body + 'end' (';'|(':' <text>))? + +command-flow-body: + *(variable-line|command-like) +\ + +A group of commands can be executed in a loop while iterating over elements of +a list and setting the specified variable (called \i{loop variable}) to the +corresponding element on each iteration. At the end of the iteration the loop +variable contains the value of the last element, if any. Note that in a +compound test, commands inside \c{command-for} must not end with +\c{;}. Rather, \c{;} may follow \c{end}. + +The \c{for} loop has two forms: In the first form the list is specified as +arguments. Similar to the \c{for} loop in the Buildfile language, it can +contain variable expansions, function calls, evaluation contexts, and/or +literal values. For example: + +\ +for v: $values + test1 $v +end; +test2 +\ + +In the second form the list is read from the \c{stdin} input. The input data +is split into elements either at whitespaces (default) or newlines, which can +be controlled with the \c{-n|--newline} and \c{-w|--whitespace} options. +Overall, this form supports the same set of options as the \l{#builtins-set +\c{set}} pseudo-builtin. For example: + +\ +cat values.txt | for -n v + test1 $v +end +\ + +Or using the \c{stdin} redirect: + +\ +for -n v <=values.txt + test1 $v +end +\ + +Both forms can include value attributes enclosed in \c{[]} to be applied to +each element, again similar to the \l{#builtins-set \c{set}} pseudo-builtin. + + +\h#syntax-command-while|Command-While| + +\ +command-while: + 'while' command-line + command-flow-body + 'end' (';'|(':' <text>))? + +command-flow-body: + *(variable-line|command-like) +\ + +A group of commands can be executed in a loop while a condition evaluates to +\c{true}. The condition \c{command-line} semantics is the same as in +\c{scope-if}. Note that in a compound test, commands inside \c{command-while} +must not end with \c{;}. Rather, \c{;} may follow \c{end}. For example: + +\ +i = [uint64] 0; +n = $size($values); +while ($i != $n) + test1 ($values[$i]) + i += 1 +end; +test2 +\ + +Another example: + +\ +while test -f $file + test1 $file +end +\ + + \h#syntax-redirect|Redirect| \ @@ -1974,6 +2286,7 @@ Similar to the input redirects, an output here-document redirect must be specified literally on the command line. See \l{#syntax-here-document Here Document} for details. + \h#syntax-here-document|Here-Document| \ @@ -2464,6 +2777,11 @@ env - --unset=FOO -- $* Terminate the command if it fails to complete within the specified number of seconds. See also the \l{#builtins-timeout \c{timeout}} builtin.| +\li|\n\c{-s|--timeout-success} + + Assume the command terminated due to the timeout specified with the + \c{-t|--timeout} option to have succeeded.| + \li|\n\c{-c|--cwd <dir>} Change the command's working directory.| @@ -2536,6 +2854,56 @@ false Do nothing and terminate normally with the 1 exit code (indicating failure). +\h#builtins-find|\c{find}| + +\ +find <start-path>... [<expression>] +\ + +Search for filesystem entries in a filesystem hierarchy. Traverse filesystem +hierarchies from each \i{start-path} specified on the command line, evaluate +for each filesystem entry the boolean \i{expression} consisting of the +options-like arguments called \i{primaries}, and print the filesystem entry +path if it evaluates to \c{true}, one path per line. The primaries are +combined into the expression with an implicit logical AND operator. The empty +expression always evaluates to \c{true}. + +Note that the implementation deviates from POSIX in a number of ways. It only +supports a small subset of primaries and doesn't support compound expressions, +negations, logical OR and (explicit) AND operators, and the \c{-type} primary +values other than \c{f}, \c{d}, and \c{l}. It, however, supports the +\c{-mindepth} and \c{-maxdepth} primaries which are not specified by POSIX but +are supported by the major \c{find} utility implementations. + +The following primaries are supported: + +\dl| + +\li|\n\c{-name <pattern>} + + Evaluates to \c{true} if a filesystem entry base name matches the specified + wildcard pattern.| + +\li|\n\c{-type <type>} + + Evaluates to \c{true} if a filesystem entry type matches the specified type: + \c{f} for a regular file, \c{d} for a directory, and \c{l} for a symbolic + link.| + +\li|\n\c{-mindepth <depth>} + + Evaluates to \c{true} if a filesystem entry directory level is not less than + the specified depth. The level of the \i{start-path} entries specified on + the command line is 0.| + +\li|\n\c{-maxdepth <depth>} + + Evaluates to \c{true} if a filesystem entry directory level is not greater + than the specified depth. The level of the \i{start-path} entries specified + on the command line is 0. Note that the implementation is smart enough not + to traverse a directory when the maximum depth is reached.|| + + \h#builtins-ln|\c{ln}| \ @@ -2772,6 +3140,7 @@ are supported. \U - Convert next characters until \E to the upper case. \L - Convert next characters until \E to the lower case. + \n - Newline. \\\\ - Literal backslash. \ @@ -2782,7 +3151,7 @@ are supported. \h#builtins-set|\c{set}| \ -set [-e] [-n|-w] [<attr>] <var> +set [-e] [-n|-w] <var> [<attr>] \ Set variable from the \c{stdin} input. @@ -2819,7 +3188,7 @@ If the \i{attr} argument is specified, then it must contain a list of value attributes enclosed in \c{[]}, for example: \ -sed -e 's/foo/bar/' input | set [string] x +sed -e 's/foo/bar/' input | set x [string] \ Note that this is also the only way to set a variable with a computed name, @@ -2827,7 +3196,7 @@ for example: \ foo = FOO -set [null] $foo <- +set $foo [null] <- \ \dl| |