Autograde assignments

See also

nbgrader autograde
Command line options for nbgrader autograde
How to structure course files
Details about how the directory hierarchy is structured
Configuration options
Details on nbgrader_config.py

After assignments have been submitted by students, you will want to save them into a submitted directory. As described in How to structure course files, you need to organize your files in a particular way. For autograding assignments, you should have the submitted versions of students’ assignments organized as follows:

submitted/{student_id}/{assignment_id}/{notebook_id}.ipynb

After running nbgrader autograde, the autograded version of the notebooks will be:

autograded/{student_id}/{assignment_id}/{notebook_id}.ipynb

Workflow example: Instructor autograding assignments

In the following example, we have an assignment with two notebooks. There are two submissions of the assignment:

Submission 1:

Submission 2:

Before we can actually start grading, we need to actually specify who the students are. We can do this through the nbgrader_config.py file, where we also specify the list of assignments:

%%file nbgrader_config.py

c = get_config()
c.NbGrader.db_assignments = [dict(name="Problem Set 1")]
c.NbGrader.db_students = [
    dict(id="Bitdiddle", first_name="Ben", last_name="Bitdiddle"),
    dict(id="Hacker", first_name="Alyssa", last_name="Hacker"),
    dict(id="Reasoner", first_name="Louis", last_name="Reasoner")
]

Overwriting nbgrader_config.py

Once the config file has been set up with the students, we can run the autograder (and as with the other nbgrader commands for instructors, this must be run from the root of the course directory):

%%bash

nbgrader autograde "Problem Set 1"

[AutogradeApp | INFO] Copying /Users/jhamrick/project/tools/nbgrader/docs/source/user_guide/submitted/Bitdiddle/Problem Set 1/jupyter.png -> /Users/jhamrick/project/tools/nbgrader/docs/source/user_guide/autograded/Bitdiddle/Problem Set 1/jupyter.png
[AutogradeApp | INFO] Copying /Users/jhamrick/project/tools/nbgrader/docs/source/user_guide/submitted/Bitdiddle/Problem Set 1/timestamp.txt -> /Users/jhamrick/project/tools/nbgrader/docs/source/user_guide/autograded/Bitdiddle/Problem Set 1/timestamp.txt
[AutogradeApp | INFO] Creating/updating student with ID 'Bitdiddle': {'first_name': 'Ben', 'last_name': 'Bitdiddle'}
[AutogradeApp | INFO] SubmittedAssignment<Problem Set 1 for Bitdiddle> submitted at 2015-02-02 14:58:23.948203
[AutogradeApp | INFO] Overwriting files with master versions from the source directory
[AutogradeApp | INFO] Copying /Users/jhamrick/project/tools/nbgrader/docs/source/user_guide/source/./Problem Set 1/jupyter.png -> /Users/jhamrick/project/tools/nbgrader/docs/source/user_guide/autograded/Bitdiddle/Problem Set 1/jupyter.png
[AutogradeApp | INFO] Sanitizing /Users/jhamrick/project/tools/nbgrader/docs/source/user_guide/submitted/Bitdiddle/Problem Set 1/Problem 1.ipynb
[AutogradeApp | INFO] Converting notebook /Users/jhamrick/project/tools/nbgrader/docs/source/user_guide/submitted/Bitdiddle/Problem Set 1/Problem 1.ipynb to notebook
[AutogradeApp | WARNING] Attribute 'checksum' for cell correct_squares has changed! (should be: 8e029652317e6c6a37a72710dc8d2429, got: 226c4a10e053c37fdb3a14a105c53fbb)
[AutogradeApp | WARNING] Attribute 'checksum' for cell squares_invalid_input has changed! (should be: c6ff383fa27ce1c2eb97789816c93069, got: 123394e73f33a622ec057e2eae51a54a)
[AutogradeApp | INFO] Writing 8228 bytes to /Users/jhamrick/project/tools/nbgrader/docs/source/user_guide/autograded/Bitdiddle/Problem Set 1/Problem 1.ipynb
[AutogradeApp | INFO] Autograding /Users/jhamrick/project/tools/nbgrader/docs/source/user_guide/autograded/Bitdiddle/Problem Set 1/Problem 1.ipynb
[AutogradeApp | INFO] Converting notebook /Users/jhamrick/project/tools/nbgrader/docs/source/user_guide/autograded/Bitdiddle/Problem Set 1/Problem 1.ipynb to notebook
[AutogradeApp | INFO] Executing notebook with kernel: python
[AutogradeApp | INFO] Writing 21478 bytes to /Users/jhamrick/project/tools/nbgrader/docs/source/user_guide/autograded/Bitdiddle/Problem Set 1/Problem 1.ipynb
[AutogradeApp | INFO] Sanitizing /Users/jhamrick/project/tools/nbgrader/docs/source/user_guide/submitted/Bitdiddle/Problem Set 1/Problem 2.ipynb
[AutogradeApp | INFO] Converting notebook /Users/jhamrick/project/tools/nbgrader/docs/source/user_guide/submitted/Bitdiddle/Problem Set 1/Problem 2.ipynb to notebook
[AutogradeApp | INFO] Writing 2556 bytes to /Users/jhamrick/project/tools/nbgrader/docs/source/user_guide/autograded/Bitdiddle/Problem Set 1/Problem 2.ipynb
[AutogradeApp | INFO] Autograding /Users/jhamrick/project/tools/nbgrader/docs/source/user_guide/autograded/Bitdiddle/Problem Set 1/Problem 2.ipynb
[AutogradeApp | INFO] Converting notebook /Users/jhamrick/project/tools/nbgrader/docs/source/user_guide/autograded/Bitdiddle/Problem Set 1/Problem 2.ipynb to notebook
[AutogradeApp | INFO] Executing notebook with kernel: python
[AutogradeApp | INFO] Writing 2553 bytes to /Users/jhamrick/project/tools/nbgrader/docs/source/user_guide/autograded/Bitdiddle/Problem Set 1/Problem 2.ipynb
[AutogradeApp | INFO] Setting destination file permissions to 444
[AutogradeApp | INFO] Copying /Users/jhamrick/project/tools/nbgrader/docs/source/user_guide/submitted/Hacker/Problem Set 1/jupyter.png -> /Users/jhamrick/project/tools/nbgrader/docs/source/user_guide/autograded/Hacker/Problem Set 1/jupyter.png
[AutogradeApp | INFO] Copying /Users/jhamrick/project/tools/nbgrader/docs/source/user_guide/submitted/Hacker/Problem Set 1/timestamp.txt -> /Users/jhamrick/project/tools/nbgrader/docs/source/user_guide/autograded/Hacker/Problem Set 1/timestamp.txt
[AutogradeApp | INFO] Creating/updating student with ID 'Hacker': {'first_name': 'Alyssa', 'last_name': 'Hacker'}
[AutogradeApp | INFO] SubmittedAssignment<Problem Set 1 for Hacker> submitted at 2015-02-01 09:28:58.749302
[AutogradeApp | INFO] Overwriting files with master versions from the source directory
[AutogradeApp | INFO] Copying /Users/jhamrick/project/tools/nbgrader/docs/source/user_guide/source/./Problem Set 1/jupyter.png -> /Users/jhamrick/project/tools/nbgrader/docs/source/user_guide/autograded/Hacker/Problem Set 1/jupyter.png
[AutogradeApp | INFO] Sanitizing /Users/jhamrick/project/tools/nbgrader/docs/source/user_guide/submitted/Hacker/Problem Set 1/Problem 1.ipynb
[AutogradeApp | INFO] Converting notebook /Users/jhamrick/project/tools/nbgrader/docs/source/user_guide/submitted/Hacker/Problem Set 1/Problem 1.ipynb to notebook
[AutogradeApp | INFO] Writing 8965 bytes to /Users/jhamrick/project/tools/nbgrader/docs/source/user_guide/autograded/Hacker/Problem Set 1/Problem 1.ipynb
[AutogradeApp | INFO] Autograding /Users/jhamrick/project/tools/nbgrader/docs/source/user_guide/autograded/Hacker/Problem Set 1/Problem 1.ipynb
[AutogradeApp | INFO] Converting notebook /Users/jhamrick/project/tools/nbgrader/docs/source/user_guide/autograded/Hacker/Problem Set 1/Problem 1.ipynb to notebook
[AutogradeApp | INFO] Executing notebook with kernel: python
[AutogradeApp | INFO] Writing 9595 bytes to /Users/jhamrick/project/tools/nbgrader/docs/source/user_guide/autograded/Hacker/Problem Set 1/Problem 1.ipynb
[AutogradeApp | INFO] Sanitizing /Users/jhamrick/project/tools/nbgrader/docs/source/user_guide/submitted/Hacker/Problem Set 1/Problem 2.ipynb
[AutogradeApp | INFO] Converting notebook /Users/jhamrick/project/tools/nbgrader/docs/source/user_guide/submitted/Hacker/Problem Set 1/Problem 2.ipynb to notebook
[AutogradeApp | INFO] Writing 2686 bytes to /Users/jhamrick/project/tools/nbgrader/docs/source/user_guide/autograded/Hacker/Problem Set 1/Problem 2.ipynb
[AutogradeApp | INFO] Autograding /Users/jhamrick/project/tools/nbgrader/docs/source/user_guide/autograded/Hacker/Problem Set 1/Problem 2.ipynb
[AutogradeApp | INFO] Converting notebook /Users/jhamrick/project/tools/nbgrader/docs/source/user_guide/autograded/Hacker/Problem Set 1/Problem 2.ipynb to notebook
[AutogradeApp | INFO] Executing notebook with kernel: python
[AutogradeApp | INFO] Writing 2683 bytes to /Users/jhamrick/project/tools/nbgrader/docs/source/user_guide/autograded/Hacker/Problem Set 1/Problem 2.ipynb
[AutogradeApp | INFO] Setting destination file permissions to 444

When grading the submission for Bitdiddle, you’ll see some warnings that look like “Checksum for grade cell correct_squares has changed!”. What’s happening here is that nbgrader has recorded what the original contents of the grade cell correct_squares (when nbgrader assign was run), and is checking the submitted version against this original version. It has found that the submitted version changed (perhaps this student tried to cheat by commenting out the failing tests), and has therefore overwritten the submitted version of the tests with the original version of the tests.

You may also notice that there is a note saying “Problem Set 1 for Bitdiddle is 86303.948203 seconds late”. What is happening here is that nbgrader is detecting a file in Bitdiddle’s submission called timestamp.txt, reading in that timestamp, and saving it into the database. From there, it can compare the timestamp to the duedate of the problem set, and compute whether the submission is at all late.

Once the autograding is complete, there will be new directories for the autograded versions of the submissions:

Autograded submission 1:

Autograded submission 2: