CodeRunner is a Moodle question type that allows teachers to run a program in order to grade a student's answer. By far the most common use of CodeRunner is in programming courses where students are asked to write program code to some specification and that code is then graded by running it in a series of tests. CodeRunner questions have also been used in other areas of computer science and engineering to grade questions in which many different correct answers are possible and a program must be used to assess correctness. 

Regardless of the behaviour chosen for a quiz, CodeRunner questions always run in an adaptive mode, in which students can click a Check button to see if their code passes the tests defined in the question. If not, students can  resubmit, typically for a small penalty. In the typical 'all-or-nothing' mode, all test cases must pass if the submission is to be awarded any marks. The mark for a set of questions in a quiz is then determined primarily by which questions the student is able to solve successfully and then secondarily by how many submissions the student makes on each question. However, it is also possible to configure CodeRunner questions so that the mark is determined by how many of the tests the code successfully passes.

CodeRunner has been in use at the University of Canterbury for over twelve years, running millions of student quiz question submissions in Python, C , JavaScript, PHP, and Octave. Laboratory work, assignment work and mid-semester tests in the introductory first year Python programming courses (COSC121 and COSC131), which together have around 1400 students in the first semester and 700 in the second, are all assessed using CodeRunner questions. The final exams for COSC121 and COSC131 have also been run using Moodle/CodeRunner since November 2014. Other courses at the University of Canterbury using CodeRunner include:

1. ENCE260 Computer Systems
2. ENCN305 Programming, Statistics and Optimisation
3. SENG02 Software Engineering I
4. COSC261 Formal Languages and Compilers
5. COSC 262 Algorithms
6. COSC367 Computational Intelligence
7. ENCE360 Operating Systems
8. SENG365 Web Computing Architectures

CodeRunner is also being used at over 3500 other sites worldwide.

CodeRunner currently supports Python2 (considered obsolescent), Python3, C, C++, Java, PHP, JavaScript (NodeJS), Octave and SQL (sqlite3). The architecture allows easy extension to other languages.

CodeRunner can safely be used on an institutional Moodle server, provided that the sandbox software in which code is run ("Jobe") is installed on a separate machine with adequate security and firewalling. However, if CodeRunner-based quizzes are to be used for tests and final exams, a separate Moodle server is recommended, both for load reasons and so that various Moodle communication facilities, like chat and messaging, can be turned off without impacting other classes.

The CodeRunner question type can be installed on any modern Moodle system (version 4.0 or later), on Linux, Windows and Mac. For security reasons submitted jobs are run on a separate machine called the "Jobe server" or "Jobe sandbox machine". CodeRunner is intitially configured to use a small outward-facing Jobe server at the University of Canterbury, and this can be used for initial testing.  However, this is not suitable for production use, for which institutions will need to install their own Jobe server. Instructions for installing a Jobe server are given in the Jobe documentation. Once Jobe is installed, use the Moodle administrator interface for the CodeRunner plug-in to specify the Jobe host name and perhaps port number.

A single 8-core Moodle server paired with a similarly-configured Jobe server can handle a quiz question submission rate of over 10 questions per second while maintaining a response time of less than about 3 - 4 seconds, assuming the student code itself runs in a fraction of a second. We run CodeRunner-based exams in Python with up to 500 students on a 12-core nginx-based Moodle server with a pool of several Jobe servers. However, other languages, such as C++ and Java, are much slower than Python for a full compile-and-execute and may call for more Jobe servers and/or smaller classes. 

The full documentation for CodeRunner, together with forums, questions banks and other resources, is at http://coderunner.org.nz