1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
|
- Test description language (testscript) [feature]
Requirements
============
* Run multiple tests
* Multiple testscript files (to group related test)
* Specify exit status (optional, expect 0 by default).
* Run pre/post test actions (setups/teardowns, e.g., database schema creation).
* Test input (optional)
* Test stdout output (to compare to, optional)
* Test stderr output (to compare to, optional)
* Test description
* Create (and automatically clean up) input/output files
? Variable expansion, looked up on target in buildfile
* Ability to pass arguments from build files (how for multiple scripts?)
Could use $0, $1, $*.
? if/else, loops (we will have ternary operator as part of eval context)
* Multi-line comments (to disable a chunk of test; #\)
* Ability to suppress output (/dev/null)
* Ability not to clean up output (if failed)
Questions
=========
* What if the testcript attached at the buildfile level? An executable? What
if the test is not for an executable. I guess we always have the directory
target.
We could reuse the test variable to specify (list of) scripts:
exe{hello}: test = test1 test2
* Do we want a notion of multi-command test (e.g., with pre/post commands)?
Maybe {}? Will need it not to clean up output files too early. Also var
scope.
* What if we need to define some global variables for the entire script?
* Ability to include testscripts (for grouping/organization purposes). Also
include common definitions?
* Ability to redefine $0 (e.g., to src_base to run scripts)?
Notes
=====
* If expected exist status is not 0, then probably makes sense to suppress
stderr output if there is no comparison.
Syntax
======
The goal is to be as concise as possible at least for the common cases.
$0 World # hello World, 0 exit expected
$0 "John Doe" # hello "John Doe"
$0 == 1 # hello, 1 exit expected
$0 != 0 # hello, non-0 exit expected
$0 World >>EOO
Hello, World!
EOO
$0 World >"Hello, World!"
$0 2>>EOE
Usage: $0 <name>
EOE
$0 --trace <<EOI >>EOO 2>>EOE # The data should appear in the order specified.
World
EOI
Hello, World!
EOO
13 characters written
EOE
~ Built-in cat command with auto-cleanup? To create files for tests, etc.
~ Built-in sed(-like) utility? Note that GCC 4.8 has broken <regex> support,
only usable from 4.9. Could be a problem.
~ We should probably allow assignment of variables and since we will also
lookup those defined in the buildfile, makes sense to use the same semantics
(and perhaps even the parser, like we do for command line). Also value
types.
So in a sense for each test we will have an "inner" variable scope where
we lookup first.
~ Could support assigning output of a program to a variable:
foo = `sed ...`
The semantics should probably be "as if you had the output literally",
without any of the extra shell splitting nonsense, just the standard
buildfile logic.
Probably also support piping.
~ Output file redirects should be relative to out_base. Input? Could actually
track created out files and use that if exists, and src otherwise. What if
the test create a file without a redirect? Will need to "register" it
somehow for auto-cleanup. Maybe &<file>? Also for directories &<dir>/.
~ Will there be a way to combine stderr and stdout, could be useful sometimes
(e.g., order of output relative to diagnostics). 2>& syntax?
~ Do we support command piping?
~ What if we actually do want the output to go to a file, e.g., for a second
command to act on it? Maybe keep shell syntax since familiar? Or reverse it
(> - inline, >> - multiline, >>> to file). Simplest thing is the shortest, I
like it.
~ $0 is target path (works out in case of a directory, but will be out).
~ #-comment (also for mid-line)
~ quoting & escaping (need to think and specify). Would be nice to avoid shell
madness. Perhaps only support escaping only inside quoted strings).
~ If we support variable expansion inside "here doc", what if we need to
specify literal '$'? Perhaps treat heredoc as a kind of string? Maybe only
do effective escaping? Otherwise will have to escape backslashes...
~ Will probably need to support both $foo and $(foo) syntax. Will need to
replicate var name lexing mode form build2. Perhaps use it to avoid
duplication?
~ line continuation with \
~ == !=
~ Specify named variables instead/in addition to arguments
...: test = test1 test2
...: test.test1.var1 = ...
~ Perhaps the default variable type should be strings, not names? Thought
we reverse most things pretty well.
~ CWD should probably be running tests in out_base (so if they create some
files).
~ Clean up is tricky: sometimes we may want to keep it, but then how to clean
it up later (the test might fail if there is junk present). Maybe create
temp directory in out_base and run there? Will also help with parallel runs.
|
