.gitattributes

+grading/results_[Bar,Foo]_598b9a96.zip filter=lfs diff=lfs merge=lfs -text

Dockerfile

 # For finding latest versions of the base image see
 # https://github.com/SwissDataScienceCenter/renkulab-docker
-ARG RENKU_BASE_IMAGE=renku/renkulab-py:3.9-0.10.1
+ARG RENKU_BASE_IMAGE=renku/renkulab-vnc:0.10.1
 FROM ${RENKU_BASE_IMAGE}
 
 # Uncomment and adapt if code is to be included in the image
@@ -47,4 +47,4 @@ RUN if [ -n "$RENKU_VERSION" ] ; then \
         fi \
     fi
 
-########################################################
\ No newline at end of file
+########################################################

README.md

-# Jupyter-autograde
+# autograde
 
-## Introduction
+`autograde` is a toolbox for testing Jupyter notebooks. Autograde executes notebooks with consecutive unit testing of the final notebook state and allows refining results (e.g. grading plots by hand) in the audit mode. Autograde summarizes results in human (html) and machine-readable (JSON) formats.
+https://github.com/cssh-rwth/autograde
 
-This is a Renku project - basically a git repository with some
-bells and whistles. You'll find we have already created some
-useful things like `data` and `notebooks` directories and
-a `Dockerfile`.
+## Demo
 
-## Working with the project
+We have included `autograde` to this envirnment by adding a line `jupyter-autograde==0.2.13` to `requirements.txt` file.
+Let's evaluate how it automates grading using the example files provided by `autograde` in the `demo/` subdirectory. 
 
-The simplest way to start your project is right from the Renku
-platform - just click on the `Environments` tab and start a new session.
-This will start an interactive environment right in your browser.
+### Grade
+To grade `demo/notebook.ipynb` we run `autograde test` against test scenarios specified in `demo/test.py` using context files stored in `demo/context`
+``` bash
+autograde test demo/test.py demo/notebook.ipynb --target /grading --context demo/context
+```
 
-To work with the project anywhere outside the Renku platform,
-click the `Settings` tab where you will find the
-git repo URLs - use `git` to clone the project on whichever machine you want.
+- `demo/test.py` contains test cases we want to apply
+- `demo/notebook.ipynb` is the a notebook to be tested (here you may also specify a directory to be recursively searched for notebooks)
+- The optional flag `--target` tells autograde where to store results, /tmp in our case, and the current working directory by default.
+- The optional flag `--context` specifies a directory that is mounted into the sandbox and may contain arbitrary files or subdirectories. This is useful when the notebook expects some external files to be present such as data sets.
 
-### Changing interactive environment dependencies
+This command generates a compressed archive named `results_[Lastname1,Lastname2,...]_XXXXXXXX.zip` (in this case `results_[Bar,Foo]_598b9a96.zip`) with the following contents:
 
-Initially we install a very minimal set of packages to keep the images small.
-However, you can add python and conda packages in `requirements.txt` and
-`environment.yml` to your heart's content. If you need more fine-grained
-control over your environment, please see [the documentation](https://renku.readthedocs.io/en/latest/user/advanced_interfaces.html#dockerfile-modifications).
+- `artifacts/`: directory with all files that where created or modified by the tested notebook as well as rendered matplotlib plots.
+- `code.py`: code extracted from the notebook including stdout/stderr as comments
+- `notebook.ipynb`: an identical copy of the tested notebook
+- `restults.json`: test results
 
-## Project configuration
+### Inspect and edit automatic grades
+Let's use `autograde audit` to inspect and edit the grades generated by automatic testing located in subfolder `grading/`.
+```bash
+autograde audit grading/
+```
+  
+To open an interactive audit interface triggered by this function we first need to switch to virtual desktop by clicking the VNC icon:  
+![image](https://renkulab.io/gitlab/learn-renku/teaching-on-renku/autograde/raw/master/images/toVNC.png)
+  
+Now we can access the interactive interface running at http://127.0.0.1:5000/   
+![image](https://renkulab.io/gitlab/learn-renku/teaching-on-renku/autograde/raw/master/images/audit_interface_480.gif)
+  
+**Preview** results of automtic grading.  
+![image](https://renkulab.io/gitlab/learn-renku/teaching-on-renku/autograde/raw/master/images/preview_results_480.gif)
 
-Project options can be found in `.renku/renku.ini`. In this
-project there is currently only one option, which specifies
-the default type of environment to open, in this case `/lab` for
-JupyterLab. You may also choose `/tree` to get to the "classic" Jupyter
-interface.
+**Edit** and **comment** the results of automatic grading.  
+![image](https://renkulab.io/gitlab/learn-renku/teaching-on-renku/autograde/raw/master/images/edit_results_480.gif)
 
-## Moving forward
+### Generate Reports
+Let's generate a human readable HTML report for grading results using `autograde report`, the command below generates a report inside the results archive `grading/results_\[Bar\,Foo\]_598b9a96.zip`.
+```bash
+autograde report grading/
+```
+
+Let's uncompress the results
+```bash
+pushd grading/
+unzip results_\[Bar\,Foo\]_598b9a96.zip
+popd
+```
+  
+Here is how the generated `grading/report.html` looks like:  
+![image](https://renkulab.io/gitlab/learn-renku/teaching-on-renku/autograde/raw/master/images/report_html.gif)
+You can display the full contents of this report by opening it either from interactive session or locally.
+
+Let's summarise results and generate inside subfolder `grading/` the follwoing 2 files:
+- `summary.csv` with aggregated results
+- `summary.html` with human readable summary report
+```bash
+autograde summary grading/
+```
+  
+Here is how the generated summary `grading/summary.html` looks like:  
+![image](https://renkulab.io/gitlab/learn-renku/teaching-on-renku/autograde/raw/master/images/summary_html.gif)
 
-Once you feel at home with your project, we recommend that you replace
-this README file with your own project documentation! Happy data wrangling!
\ No newline at end of file

grading/raw.csv

+student_id,last_name,first_name,notebook_id,test_id,score,max_score,archive
+12345,Foo,Alice,598b9a96d5c04cf88f5b26771572a62544e80369535855044fb4583c3bb78878,239eadbc01bcd3892b5c01a86b007dca86eeb0b2daa51ad5a580a1ddc079e040,1.0,1.0,"results_[Bar,Foo]_598b9a96.zip"
+12345,Foo,Alice,598b9a96d5c04cf88f5b26771572a62544e80369535855044fb4583c3bb78878,f03691ea9f44d6018fd43ef2a106e527843f5984f9eb44c49aeecb48d1266454,2.5,2.5,"results_[Bar,Foo]_598b9a96.zip"
+12345,Foo,Alice,598b9a96d5c04cf88f5b26771572a62544e80369535855044fb4583c3bb78878,35797fb23f082ed01f8a12050bfd0343e28ceca91cd00ef3be38a144563ec0a2,2.0,3.0,"results_[Bar,Foo]_598b9a96.zip"
+12345,Foo,Alice,598b9a96d5c04cf88f5b26771572a62544e80369535855044fb4583c3bb78878,7e0e6ff3545a0510d5ab9a1d7ac6ab2c14609d3843d57cc04638f321423ac3c5,1.0,2.0,"results_[Bar,Foo]_598b9a96.zip"
+12345,Foo,Alice,598b9a96d5c04cf88f5b26771572a62544e80369535855044fb4583c3bb78878,90f0231eef0abe9f9b28e3be1315d27a1ef89297c6009d92d7b1f24a165f4673,1.0,1.0,"results_[Bar,Foo]_598b9a96.zip"
+12345,Foo,Alice,598b9a96d5c04cf88f5b26771572a62544e80369535855044fb4583c3bb78878,85f2e61341ca4bfbdda86b328c3704ff3983d49c786b51b176dd558979e59122,0.0,1.5,"results_[Bar,Foo]_598b9a96.zip"
+12345,Foo,Alice,598b9a96d5c04cf88f5b26771572a62544e80369535855044fb4583c3bb78878,95cf4e22dcc910d0384c63a600fe79d886e3f23f4c6bb6eb8fdc616180793d9d,-0.5,1.0,"results_[Bar,Foo]_598b9a96.zip"
+12345,Foo,Alice,598b9a96d5c04cf88f5b26771572a62544e80369535855044fb4583c3bb78878,511d0ac725daadf807d17cfbe7fa0da1a4a446ad85f737f5ca94fb51bc393c4b,0.5,0.5,"results_[Bar,Foo]_598b9a96.zip"
+12345,Foo,Alice,598b9a96d5c04cf88f5b26771572a62544e80369535855044fb4583c3bb78878,53bca0ce4cd4c8c8c4e1792086e8bd79bc049eb7f61e660bebf485dd8668e84c,0.0,1.0,"results_[Bar,Foo]_598b9a96.zip"
+12345,Foo,Alice,598b9a96d5c04cf88f5b26771572a62544e80369535855044fb4583c3bb78878,6d1546e07fd32d5d0eab09ea89abc6b5de7f3660236b5486ca83d7c0d44364ee,0.0,1.0,"results_[Bar,Foo]_598b9a96.zip"
+12345,Foo,Alice,598b9a96d5c04cf88f5b26771572a62544e80369535855044fb4583c3bb78878,ce7ad25b6af8ed05fc10dd726b893fd12abe5408a20d5e2b5aaa36977bbb8e18,1.0,1.0,"results_[Bar,Foo]_598b9a96.zip"
+12345,Foo,Alice,598b9a96d5c04cf88f5b26771572a62544e80369535855044fb4583c3bb78878,81b637d8fcd2c6da6359e6963113a1170de795e4b725b84d1e0b4cfd9ec58ce9,,4.0,"results_[Bar,Foo]_598b9a96.zip"
+12345,Foo,Alice,598b9a96d5c04cf88f5b26771572a62544e80369535855044fb4583c3bb78878,a0ec927b1044a5e945fbd9cf4370b4be1bc1ebc93bca51bc37f02a8814196bde,,1.0,"results_[Bar,Foo]_598b9a96.zip"
+12345,Foo,Alice,598b9a96d5c04cf88f5b26771572a62544e80369535855044fb4583c3bb78878,a03b221c6c6eae7122ca51695d456d5222e524889136394944b2f9763b483615,0.0,2.5,"results_[Bar,Foo]_598b9a96.zip"
+12345,Foo,Alice,598b9a96d5c04cf88f5b26771572a62544e80369535855044fb4583c3bb78878,f81f712816b4a576cd12ecf67c0e85267a22109f5ffef8c50576ac85ccf4e048,,1.0,"results_[Bar,Foo]_598b9a96.zip"
+12345,Foo,Alice,598b9a96d5c04cf88f5b26771572a62544e80369535855044fb4583c3bb78878,affcf4c8d293c17fcb845deb7f4bb1aff6697426d75bf28a2bf7403067646673,0.0,1.0,"results_[Bar,Foo]_598b9a96.zip"
+12345,Foo,Alice,598b9a96d5c04cf88f5b26771572a62544e80369535855044fb4583c3bb78878,d376512f3901b0a8347d0221155af8c527795b23b9b884d8d25484149ea0da47,1.0,1.0,"results_[Bar,Foo]_598b9a96.zip"
+12345,Foo,Alice,598b9a96d5c04cf88f5b26771572a62544e80369535855044fb4583c3bb78878,a814e91177667827d91ab4d1c8abe0e3a11471d8299afb274eafdc5b99355309,1.0,1.0,"results_[Bar,Foo]_598b9a96.zip"
+54321,Bar,Bob,598b9a96d5c04cf88f5b26771572a62544e80369535855044fb4583c3bb78878,239eadbc01bcd3892b5c01a86b007dca86eeb0b2daa51ad5a580a1ddc079e040,1.0,1.0,"results_[Bar,Foo]_598b9a96.zip"
+54321,Bar,Bob,598b9a96d5c04cf88f5b26771572a62544e80369535855044fb4583c3bb78878,f03691ea9f44d6018fd43ef2a106e527843f5984f9eb44c49aeecb48d1266454,2.5,2.5,"results_[Bar,Foo]_598b9a96.zip"
+54321,Bar,Bob,598b9a96d5c04cf88f5b26771572a62544e80369535855044fb4583c3bb78878,35797fb23f082ed01f8a12050bfd0343e28ceca91cd00ef3be38a144563ec0a2,2.0,3.0,"results_[Bar,Foo]_598b9a96.zip"
+54321,Bar,Bob,598b9a96d5c04cf88f5b26771572a62544e80369535855044fb4583c3bb78878,7e0e6ff3545a0510d5ab9a1d7ac6ab2c14609d3843d57cc04638f321423ac3c5,1.0,2.0,"results_[Bar,Foo]_598b9a96.zip"
+54321,Bar,Bob,598b9a96d5c04cf88f5b26771572a62544e80369535855044fb4583c3bb78878,90f0231eef0abe9f9b28e3be1315d27a1ef89297c6009d92d7b1f24a165f4673,1.0,1.0,"results_[Bar,Foo]_598b9a96.zip"
+54321,Bar,Bob,598b9a96d5c04cf88f5b26771572a62544e80369535855044fb4583c3bb78878,85f2e61341ca4bfbdda86b328c3704ff3983d49c786b51b176dd558979e59122,0.0,1.5,"results_[Bar,Foo]_598b9a96.zip"
+54321,Bar,Bob,598b9a96d5c04cf88f5b26771572a62544e80369535855044fb4583c3bb78878,95cf4e22dcc910d0384c63a600fe79d886e3f23f4c6bb6eb8fdc616180793d9d,-0.5,1.0,"results_[Bar,Foo]_598b9a96.zip"
+54321,Bar,Bob,598b9a96d5c04cf88f5b26771572a62544e80369535855044fb4583c3bb78878,511d0ac725daadf807d17cfbe7fa0da1a4a446ad85f737f5ca94fb51bc393c4b,0.5,0.5,"results_[Bar,Foo]_598b9a96.zip"
+54321,Bar,Bob,598b9a96d5c04cf88f5b26771572a62544e80369535855044fb4583c3bb78878,53bca0ce4cd4c8c8c4e1792086e8bd79bc049eb7f61e660bebf485dd8668e84c,0.0,1.0,"results_[Bar,Foo]_598b9a96.zip"
+54321,Bar,Bob,598b9a96d5c04cf88f5b26771572a62544e80369535855044fb4583c3bb78878,6d1546e07fd32d5d0eab09ea89abc6b5de7f3660236b5486ca83d7c0d44364ee,0.0,1.0,"results_[Bar,Foo]_598b9a96.zip"
+54321,Bar,Bob,598b9a96d5c04cf88f5b26771572a62544e80369535855044fb4583c3bb78878,ce7ad25b6af8ed05fc10dd726b893fd12abe5408a20d5e2b5aaa36977bbb8e18,1.0,1.0,"results_[Bar,Foo]_598b9a96.zip"
+54321,Bar,Bob,598b9a96d5c04cf88f5b26771572a62544e80369535855044fb4583c3bb78878,81b637d8fcd2c6da6359e6963113a1170de795e4b725b84d1e0b4cfd9ec58ce9,,4.0,"results_[Bar,Foo]_598b9a96.zip"
+54321,Bar,Bob,598b9a96d5c04cf88f5b26771572a62544e80369535855044fb4583c3bb78878,a0ec927b1044a5e945fbd9cf4370b4be1bc1ebc93bca51bc37f02a8814196bde,,1.0,"results_[Bar,Foo]_598b9a96.zip"
+54321,Bar,Bob,598b9a96d5c04cf88f5b26771572a62544e80369535855044fb4583c3bb78878,a03b221c6c6eae7122ca51695d456d5222e524889136394944b2f9763b483615,0.0,2.5,"results_[Bar,Foo]_598b9a96.zip"
+54321,Bar,Bob,598b9a96d5c04cf88f5b26771572a62544e80369535855044fb4583c3bb78878,f81f712816b4a576cd12ecf67c0e85267a22109f5ffef8c50576ac85ccf4e048,,1.0,"results_[Bar,Foo]_598b9a96.zip"
+54321,Bar,Bob,598b9a96d5c04cf88f5b26771572a62544e80369535855044fb4583c3bb78878,affcf4c8d293c17fcb845deb7f4bb1aff6697426d75bf28a2bf7403067646673,0.0,1.0,"results_[Bar,Foo]_598b9a96.zip"
+54321,Bar,Bob,598b9a96d5c04cf88f5b26771572a62544e80369535855044fb4583c3bb78878,d376512f3901b0a8347d0221155af8c527795b23b9b884d8d25484149ea0da47,1.0,1.0,"results_[Bar,Foo]_598b9a96.zip"
+54321,Bar,Bob,598b9a96d5c04cf88f5b26771572a62544e80369535855044fb4583c3bb78878,a814e91177667827d91ab4d1c8abe0e3a11471d8299afb274eafdc5b99355309,1.0,1.0,"results_[Bar,Foo]_598b9a96.zip"

grading/summary.csv

+student_id,last_name,first_name,notebook_id,score,max_score,archive,duplicate
+54321,Bar,Bob,598b9a96d5c04cf88f5b26771572a62544e80369535855044fb4583c3bb78878,10.5,27.0,"results_[Bar,Foo]_598b9a96.zip",False
+12345,Foo,Alice,598b9a96d5c04cf88f5b26771572a62544e80369535855044fb4583c3bb78878,10.5,27.0,"results_[Bar,Foo]_598b9a96.zip",False