Question types: CodeRunner

qtype_coderunner
Maintained by Picture of Richard LobbRichard Lobb, Tim at Lone Pine Koala SanctuaryTim Hunt
A question type that allows question authors to set programming questions in which the student answer is code in some programming language, which is graded by running it. More generally it can handle any question to which the answer is text that can be graded by a computer program written by the question author.
691 sites
886 downloads
46 fans
Current versions available: 3

CodeRunner is a Moodle question type that allows teachers to run a program in order to grade a student's answer. By far the most common use of CodeRunner is in programming courses where students are asked to write program code to some specification and that code is then graded by running it in a series of tests. CodeRunner questions have also been used in other areas of computer science and engineering to grade questions in which many different correct answers are possible and a program must be used to assess correctness. 

Regardless of the behaviour chosen for a quiz, CodeRunner questions always run in an adaptive mode, in which students can click a Check button to see if their code passes the tests defined in the question. If not, students can  resubmit, typically for a small penalty. In the typical 'all-or-nothing' mode, all test cases must pass if the submission is to be awarded any marks. The mark for a set of questions in a quiz is then determined primarily by which questions the student is able to solve successfully and then secondarily by how many submissions the student makes on each question. However, it is also possible to configure CodeRunner questions so that the mark is determined by how many of the tests the code successfully passes.

CodeRunner has been in use at the University of Canterbury for over seven years, running millions of student quiz question submissions in Python, C , JavaScript, PHP, Octave and Matlab. Laboratory work, assignment work and mid-semester tests in the introductory first year Python programming course (COSC121), which has around 650 students in the first semester and 350 in the second, are all assessed using CodeRunner questions. The final exams for COSC121 have also been run using Moodle/CodeRunner since November 2014. Other courses at the University of Canterbury using CodeRunner include:

  1. ENCE260 Computer Systems
  2. ENCN305 Programming, Statistics and Optimisation
  3. EMTH171 Mathematical Modelling and Computation
  4. SENG02 Software Engineering I
  5. COSC261 Formal Languages and Compilers
  6. COSC 262 Algorithms
  7. COSC367 Computational Intelligence
  8. ENCE360 Operating Systems
  9. SENG365 Web Computing Architectures

CodeRunner is also being used at over 600 other sites worldwide.

CodeRunner currently supports Python2 (considered obsolescent), Python3, C, C++, Java, PHP, JavaScript (NodeJS), Octave and Matlab. The architecture allows easy extension to other languages.

CodeRunner can safely be used on an institutional Moodle server, provided that the sandbox software in which code is run ("Jobe") is installed on a separate machine with adequate security and firewalling. However, if CodeRunner-based quizzes are to be used for tests and final exams, a separate Moodle server is recommended, both for load reasons and so that various Moodle communication facilities, like chat and messaging, can be turned off without impacting other classes.

The CodeRunner question type can be installed on any modern Moodle system (version 3.0 or later), on Linux, Windows and Mac. For security reasons submitted jobs are run on a separate machine called the "Jobe server" or "Jobe sandbox machine". CodeRunner is intitially configured to use a small outward-facing Jobe server at the University of Canterbury, and this can be used for initial testing.  However, this is not suitable for production use, for which institutions will need to install their own Jobe server. Instructions for installing a Jobe server are given in the Jobe documentation. Once Jobe is installed, use the Moodle administrator interface for the CodeRunner plug-in to specify the Jobe host name and perhaps port number.

A single 4-core Moodle server can handle an average quiz question submission rate of about 60 quiz questions per minute while maintaining a response time of less than about 3 - 4 seconds, assuming the student code itself runs in a fraction of a second. We have run CodeRunner-based exams with nearly 300 students and experienced only light to moderate load factors on an 8-core Moodle server. The Jobe server, which runs student submissions (see below), is even more lightly loaded during such an exam.

The full documentation for CodeRunner, together with forums, questions banks and other resources, is at http://coderunner.org.nz


Potential privacy issues

None that I'm aware of.

Screenshots

Screenshot #0
Screenshot #1
Screenshot #2
Screenshot #3
Screenshot #4
Screenshot #5
Screenshot #6
Screenshot #7
Screenshot #8

Contributors

Picture of Richard Lobb
Richard Lobb (Lead maintainer)
Tim at Lone Pine Koala Sanctuary
Tim Hunt: Co-developer from December 2016
Please login to view contributors details and/or to contact them

Comments RSS

Show comments
  • Picture of Richard Lobb
    Wed, Jan 11, 2017, 9:45 AM
    This plugin requires the special behaviour "adaptive_adapted_for_coderunner", from https://github.com/trampgeek/moodle-qbehaviour_adaptive_adapted_for_coderunner. That plugin has been submitted separately.
  • Picture of Plugins bot
    Fri, Jan 20, 2017, 10:10 PM
    Approval issue created: CONTRIB-6701
  • Picture of Plugins bot
    Thu, Feb 9, 2017, 8:33 PM
  • Picture of Эмиль Зарипов
    Wed, May 31, 2017, 2:18 PM
    Please make HTML!
  • Picture of Diego C Martin
    Sat, Feb 24, 2018, 4:49 PM
    Hi! Good job, looks really interesting. I'd like to know if can also be used to teach HTML and CSS. Thanks!
  • Picture of Richard Lobb
    Thu, Mar 1, 2018, 4:17 PM
    In reply to Diego's question "I'd like to know if can also be used to teach HTML and CSS.", the short answer is "It depends". There are no built-in question types for HTML and CSS. But CodeRunner can be used to grade any textual question provided you can write the code to grade it yourself. So, for example, you could ask the student to write some HTML with a level 1 header containing the "My Header" followed by three bullet points "Bullet1", "Bullet2", "Bullet3". You could write a Python template (or use whatever other language you like) to parse the submitted answer and print OK if the parsed document contained the required elements, or a suitable error message otherwise.

    Some years ago, when teaching a web programming course, I got students to implement a web site (with database, form submission etc) and paste into the answer box the URL of their site. I then graded it by making a series of GETs and POSTs to it to see how it behaved. But it's a lot of work to write a grader for a question like that and you need a large class to justify the effort. [And as an aside, you also need to poke a hole or holes in the Jobe firewall to let the grader access their sites, which, to minimise the number of holes, should be on a server provided by the institution.]

    It always comes down to:

    1. What question do you want to ask?
    2. What answers are you prepared to accept?
    3. What feedback do you wish to present to the student?
    4. Can you bothered writing the program to check the answer?

    Richard
  • Picture of Diego C Martin
    Mon, Apr 2, 2018, 12:23 AM
    Thank you Richard!
  • Picture of Ania Polanska
    Fri, May 11, 2018, 9:13 PM
    And what about a Javascript question. How can we add it?
  • Picture of Richard Lobb
    Sat, May 12, 2018, 7:49 AM
    Hi Ania.

    Select *nodejs* as the CodeRunner question type (nodejs is the package used to run JavaScript on the server). If you want more details, please ask in the Question Authors' Forum on coderunner.org.nz.

    Richard
  • Picture of Ania Polanska
    Sun, Jun 10, 2018, 1:59 AM
    Thank you for your replay. I like code runner very muchsmile Great job! We work with python and now we want to extend for js.
  • Picture of Oleg Kovalenko
    Mon, Feb 4, 2019, 4:19 PM
    There is versions incompatibility in new 3.6.0 release. Form method "hideIf", used in edit_coderunner_form.php, available only since Moodle 3.4, but compatible Moodle versions specified in release information is 3.0 and higher. Please, correct the release information.
  • Picture of Richard Lobb
    Tue, Feb 5, 2019, 4:36 AM
    Thanks for reporting this Oleg and my apologies for the inconvenience. I have removed Moodle 3.0 through 3.3 from the compatibility list as you suggest. It's a shame that just one line of code has made the latest version incompatible with the earlier versions.
  • Picture of Jean-Nicolas Gautier
    Fri, Feb 15, 2019, 6:13 PM
    Hi everyone,
    I'm a teacher in Belgium. I'm using coderunner in one of my courses during a part of the year for a few students.
    I can't afford a dedicated server on wich install a Jobe server. Does anyone know if there is a solution like a free Jobe server?
    Any help would be apreciated.
  • Tim at Lone Pine Koala Sanctuary
    Fri, Feb 15, 2019, 6:19 PM
    The best place to get help with CodeRunner is in the forums at https://coderunner.org.nz/mod/forum/view.php?id=51.
Please login to post comments