autograde

autograde is a toolbox for testing Jupyter notebooks. Autograde executed of notebooks with consecutive unit testing of the final notebook state and allows refining results (e.g. grading plots by hand) in the audit mode. Autograde summarizes results in human and machine-readable formats. https://github.com/cssh-rwth/autograde

Demo

We have included autograde to this envirnment by adding a line jupyter-autograde==0.2.13 to requirements.txt file. Let's evaluate how it automates grading using the example files provided by autograde in the demo/ subdirectory.

Grade

To grade demo/notebook.ipynb we run autograde test against test scenarios specified in demo/test.py using context files stored in demo/context

autograde test demo/test.py demo/notebook.ipynb --target /grading --context demo/context

• demo/test.py contains test cases we want to apply
• demo/notebook.ipynb is the a notebook to be tested (here you may also specify a directory to be recursively searched for notebooks)
• The optional flag --target tells autograde where to store results, /tmp in our case, and the current working directory by default.
• The optional flag --context specifies a directory that is mounted into the sandbox and may contain arbitrary files or subdirectories. This is useful when the notebook expects some external files to be present such as data sets.

This command generates a compressed archive named results_[Lastname1,Lastname2,...]_XXXXXXXX.zip (in this case results_[Bar,Foo]_598b9a96.zip) with the following contents:

• artifacts/: directory with all files that where created or modified by the tested notebook as well as rendered matplotlib plots.
• code.py: code extracted from the notebook including stdout/stderr as comments
• notebook.ipynb: an identical copy of the tested notebook
• restults.json: test results

Generate Reports

Let's generate a human readable HTML report for grading results using autograde report, the command below generates a report inside the results archive grading/results_\[Bar\,Foo\]_598b9a96.zip.

autograde report grading/

Let's uncompress the results

pushd grading/
unzip results_\[Bar\,Foo\]_598b9a96.zip
popd

Let's summarise results and generate inside subfolder grading/ the follwoing 2 files:

• summary.csv with aggregated results
• summary.html with human readable summary report

autograde summary grading/