Question types: CodeRunner

qtype_coderunner
Maintained by Picture of Richard Lobb Richard Lobb, Tim at Lone Pine Koala Sanctuary Tim Hunt
A question type that allows question authors to set programming questions in which the student answer is code in some programming language, which is graded by running it. More generally it can handle any question to which the answer is text that can be graded by a computer program written by the question author.
565 sites
860 downloads
30 fans
Moodle 3.0, 3.1, 3.2, 3.3, 3.4, 3.5

CodeRunner is a Moodle question type that allows teachers to run a program in order to grade a student's answer. By far the most common use of CodeRunner is in programming courses where students are asked to write program code to some specification and that code is then graded by running it in a series of tests. CodeRunner questions have also been used in other areas of computer science and engineering to grade questions in which many different correct answers are possible and a program must be used to assess correctness. However, the focus throughout most of this document will be on programming questions, where the students' code is being graded.

Regardless of the behaviour chosen for a quiz, CodeRunner questions always run in an adaptive mode, in which students can click a Checkbutton to see if their code passes the tests defined in the question. If not, students can  resubmit, typically for a small penalty. In the typical 'all-or-nothing' mode, all test cases must pass if the submission is to be awarded any marks. The mark for a set of questions in a quiz is then determined primarily by which questions the student is able to solve successfully and then secondarily by how many submissions the student makes on each question. However, it is also possible to configure CodeRunner questions so that the mark is determined by how many of the tests the code successfully passes.

CodeRunner and its predecessors pycode and ccode has been in use at the University of Canterbury for about six years, running over a million student quiz question submissions in Python, C , JavaScript, PHP, Octave and Matlab. Laboratory work, assignment work and mid-semester tests in the introductory first year Python programming course (COSC121), which has around 500 students in the first semester and 300 in the second, are all assessed using CodeRunner questions. The final exams for COSC121 have also been run using Moodle/CodeRunner since November 2014.

The second year C course (ENCE260) of around 200 students makes similar use of CodeRunner using C questions and a third year Civil Engineering course (ENCN305), taught in Matlab, uses CodeRunner for all labs and for the mid-semester programming exam. Other courses using Moodle/CodeRunner include:

  1. EMTH171 Mathematical Modelling and Computation
  2. SENG02 Software Engineering I
  3. COSC261 Formal Languages and Compilers
  4. COSC367 Computational Intelligence
  5. ENCE360 Operating Systems
  6. SENG365 Web Computing Architectures

CodeRunner currently supports Python2 (considered obsolescent), Python3, C, C++, Java, PHP, JavaScript (NodeJS), Octave and Matlab. The architecture allows easy extension to other languages.

CodeRunner can safely be used on an institutional Moodle server, provided that the sandbox software in which code is run ("Jobe") is installed on a separate machine with adequate security and firewalling. However, if CodeRunner-based quizzes are to be used for tests and final exams, a separate Moodle server is recommended, both for load reasons and so that various Moodle communication facilities, like chat and messaging, can be turned off without impacting other classes.

A single 4-core Moodle server can handle an average quiz question submission rate of about 60 quiz questions per minute while maintaining a response time of less than about 3 - 4 seconds, assuming the student code itself runs in a fraction of a second. We have run CodeRunner-based exams with nearly 300 students and experienced only light to moderate load factors on an 8-core Moodle server. The Jobe server, which runs student submissions (see below), is even more lightly loaded during such an exam.

The CodeRunner question type can be installed on any modern Moodle system (version 2.9 or later including version 3.2), on Linux, Windows and Mac. For security reasons submitted jobs are usually run on a separate machine called the "Jobe server" or "Jobe sandbox machine".

Potential privacy issues

None that I'm aware of.

Screenshots

Screenshot #0
Screenshot #1
Screenshot #2
Screenshot #3
Screenshot #4
Screenshot #5
Screenshot #6
Screenshot #7
Screenshot #8

Contributors

Picture of Richard Lobb
Richard Lobb (Lead maintainer)
Tim at Lone Pine Koala Sanctuary
Tim Hunt: Co-developer from December 2016
Please login to view contributors details and/or to contact them

Comments RSS

Show comments
  • Picture of Richard Lobb
    Wed, 11 Jan 2017, 9:45 AM
    This plugin requires the special behaviour "adaptive_adapted_for_coderunner", from https://github.com/trampgeek/moodle-qbehaviour_adaptive_adapted_for_coderunner. That plugin has been submitted separately.
  • Picture of Plugins bot
    Fri, 20 Jan 2017, 10:10 PM
    Approval issue created: CONTRIB-6701
  • Picture of Эмиль Зарипов
    Wed, 31 May 2017, 2:18 PM
    Please make HTML!
  • Picture of Diego C Martin
    Sat, 24 Feb 2018, 4:49 PM
    Hi! Good job, looks really interesting. I'd like to know if can also be used to teach HTML and CSS. Thanks!
  • Picture of Richard Lobb
    Thu, 1 Mar 2018, 4:17 PM
    In reply to Diego's question "I'd like to know if can also be used to teach HTML and CSS.", the short answer is "It depends". There are no built-in question types for HTML and CSS. But CodeRunner can be used to grade any textual question provided you can write the code to grade it yourself. So, for example, you could ask the student to write some HTML with a level 1 header containing the "My Header" followed by three bullet points "Bullet1", "Bullet2", "Bullet3". You could write a Python template (or use whatever other language you like) to parse the submitted answer and print OK if the parsed document contained the required elements, or a suitable error message otherwise.

    Some years ago, when teaching a web programming course, I got students to implement a web site (with database, form submission etc) and paste into the answer box the URL of their site. I then graded it by making a series of GETs and POSTs to it to see how it behaved. But it's a lot of work to write a grader for a question like that and you need a large class to justify the effort. [And as an aside, you also need to poke a hole or holes in the Jobe firewall to let the grader access their sites, which, to minimise the number of holes, should be on a server provided by the institution.]

    It always comes down to:

    1. What question do you want to ask?
    2. What answers are you prepared to accept?
    3. What feedback do you wish to present to the student?
    4. Can you bothered writing the program to check the answer?

    Richard
  • Picture of Diego C Martin
    Mon, 2 Apr 2018, 12:23 AM
    Thank you Richard!
  • Picture of Ania Polanska
    Fri, 11 May 2018, 9:13 PM
    And what about a Javascript question. How can we add it?
  • Picture of Richard Lobb
    Sat, 12 May 2018, 7:49 AM
    Hi Ania.

    Select *nodejs* as the CodeRunner question type (nodejs is the package used to run JavaScript on the server). If you want more details, please ask in the Question Authors' Forum on coderunner.org.nz.

    Richard
  • Picture of Ania Polanska
    Sun, 10 Jun 2018, 1:59 AM
    Thank you for your replay. I like code runner very muchsmile Great job! We work with python and now we want to extend for js.
Please login to post comments