Usage

The first parameter specifies which assignment is to be processed. The second and third together specify which test suite is to be exercised and what type of output should be generated. The fourth parameter specifies which students' submissions are to be used. The fifth parameter is optional, and specifies a run_id for this particular rst run.

Other configuration options are available via rstconfig.

Description

Short for "run student tests", rst is used to run a particular test suite on one or more students' submissions, resulting in either a number of potential output files: a file suitable for student feedback (autotesting, handmarking, etc.), a file summarizing the autotesting results for each student in a format meant to be parsed easily in an automated fashion, and a file summarizing the marks for all students tested with the same run_id.

Essentially, this script loops through the specified set of students and runs one or two script files for each student. These script files must be created before invoking rst. The first of these script files, which in some circumstances may be omitted, is called runTests and is intended to do whatever is required in order to compile and execute the student's submission, resulting in output files corresponding to each test case. The second script, which must always be supplied, is called computeMarks and is responsible for generating the final results. If a runTests was used, computeMarks is usually responsible for checking the results obtained from the student's program against the "correct" results and assigning a grade.

Since runTests and computeMarks can encapsulate essentially any action, the system is extremely flexible and accordingly the above description may not apply to a particular application of the system. The important point is to understand the various directories and files related to the system and what they are all used for.

Note that rst will request that the subdirectory answers exist in the suite directory. You don't have to have any files in this directory (although you'll have automatically generated ones if you run rsta); you just need that directory to exist.

Explanation of Parameters

The assign parameter specifies which assignment is to be processed. By default, student submissions in directory /u/csXXX/handin/assign will be used.

The second parameter in version 1, or the [-t] option in version 2, specifies the type of feedback output that is to be generated. This can be any combination of p (postscript), f (pdf), or t (text); it can also just be c (complete). The default is "pft" so all output types will be generated. For backward compatibility, "m" is also maintained as a synonym for "t". In the event of a complete run, not only are postscript files and text files both generated, but all temporary files generated are stored in rst's results directory (as specified in the decription of the run_id below), so it should primarily be used for testing on a small subset of students.

suite specifies the name of the specific suite to be run. Data and executables for the test suite, including the runTests and computeMarks scripts, is found in /u/csXXX/marking/assign/suite/. For backward compatibility, it is possible to name the suite directory test.suite and to make the suite used dependent on the type of test run, although it is usually not necessary to create such a dependency. This can be done by appending the appropriate test run letter after test in the directory name; ie, /u/csXXX/marking/assign/testm.suite/ for a mail test run.

The 'studentglob' parameter in version 1, or the [-s] option in version 2, specifies which students' submissions are to be processed. The quote marks are needed whenever studentglob is anything other than a single UW userid. For running the whole class, studentglob should be just a single asterisk (*). For running subsets of the class, Unix "glob" patterns can be used. These are the same patterns as used with ls or other utilities to specify sets of files. For example, "a*" would indicate all students whose UW userids start with the letter "a". This is useful for running a test suite on a small group of students for testing purposes before taking the time to run it on the whole class. Additionally, studentglob can be more than one word, so an arbitrary set of students can be specified by simply listing all the relevant UW userids. This is most likely to be useful for dealing with a small set of students who for whatever reason need to be handled specially.

The final parameter is optional, and specifies the run_id that will be used for this run of the testing. The results from a particular run are placed in the directory /u/csXXX/marking/assign/suite.run_id/. If the run_id parameter is omitted, a ten-digit timestamp will be constructed from the current time down to the second. In a given test run, any student already processed under that run_id is skipped. This may be useful for continuing test runs that were interrupted. Note that the student who was being processed when the test run was interrupted may be skipped even though they were not actually fully processed. When rerunning rst with a run_id that has been previously used, you will need to first delete the directory in both marking directory and the tmp directory. The run_id specified by this parameter can in fact be any (reasonable) string, not just a ten-digit number like the automatically generated ones.

Related programs

As discussed above, rst runs up to two scripts, runTests and computeMarks, on each student's submission. These are the keys to writing an rst test suite; see their respective pages for more details.

When running on a large group of student submissions, it is often convenient to use distrst, which distributes the run of rst across a number of servers, greatly reducing the time required to complete a test run. If solutions need to be generated, rsta needs to be run first.

After completion of an rst run, the program mailOut or printOut can be used to mail text results to students or print postscript for handmarking by TAs, respectively.

There is also a publictest mechanism available for allowing students to request simple tests and receive the results by e-mail. The students in the course can request this via pub_test_request or an appropriate web front-end. The course account then has to have suites named pt for the test account runners launched via public_test_launch to be able to find them.