Question types: CodeRunner

qtype_coderunner
Maintained by Picture of Richard LobbRichard Lobb, Tim at Lone Pine Koala SanctuaryTim Hunt
A question type that allows question authors to set programming questions in which the student answer is code in some programming language, which is graded by running it. More generally it can handle any question to which the answer is text that can be graded by a computer program written by the question author.
1749 sites
1k downloads
80 fans
Moodle 3.0, 3.1, 3.2, 3.3, 3.4, 3.5, 3.6, 3.7, 3.8, 3.9

CodeRunner is a Moodle question type that allows teachers to run a program in order to grade a student's answer. By far the most common use of CodeRunner is in programming courses where students are asked to write program code to some specification and that code is then graded by running it in a series of tests. CodeRunner questions have also been used in other areas of computer science and engineering to grade questions in which many different correct answers are possible and a program must be used to assess correctness. 

Regardless of the behaviour chosen for a quiz, CodeRunner questions always run in an adaptive mode, in which students can click a Check button to see if their code passes the tests defined in the question. If not, students can  resubmit, typically for a small penalty. In the typical 'all-or-nothing' mode, all test cases must pass if the submission is to be awarded any marks. The mark for a set of questions in a quiz is then determined primarily by which questions the student is able to solve successfully and then secondarily by how many submissions the student makes on each question. However, it is also possible to configure CodeRunner questions so that the mark is determined by how many of the tests the code successfully passes.

CodeRunner has been in use at the University of Canterbury for over seven years, running millions of student quiz question submissions in Python, C , JavaScript, PHP, Octave and Matlab. Laboratory work, assignment work and mid-semester tests in the introductory first year Python programming course (COSC121), which has around 650 students in the first semester and 350 in the second, are all assessed using CodeRunner questions. The final exams for COSC121 have also been run using Moodle/CodeRunner since November 2014. Other courses at the University of Canterbury using CodeRunner include:

  1. ENCE260 Computer Systems
  2. ENCN305 Programming, Statistics and Optimisation
  3. EMTH171 Mathematical Modelling and Computation
  4. SENG02 Software Engineering I
  5. COSC261 Formal Languages and Compilers
  6. COSC 262 Algorithms
  7. COSC367 Computational Intelligence
  8. ENCE360 Operating Systems
  9. SENG365 Web Computing Architectures

CodeRunner is also being used at over 600 other sites worldwide.

CodeRunner currently supports Python2 (considered obsolescent), Python3, C, C++, Java, PHP, JavaScript (NodeJS), Octave and Matlab. The architecture allows easy extension to other languages.

CodeRunner can safely be used on an institutional Moodle server, provided that the sandbox software in which code is run ("Jobe") is installed on a separate machine with adequate security and firewalling. However, if CodeRunner-based quizzes are to be used for tests and final exams, a separate Moodle server is recommended, both for load reasons and so that various Moodle communication facilities, like chat and messaging, can be turned off without impacting other classes.

The CodeRunner question type can be installed on any modern Moodle system (version 3.0 or later), on Linux, Windows and Mac. For security reasons submitted jobs are run on a separate machine called the "Jobe server" or "Jobe sandbox machine". CodeRunner is intitially configured to use a small outward-facing Jobe server at the University of Canterbury, and this can be used for initial testing.  However, this is not suitable for production use, for which institutions will need to install their own Jobe server. Instructions for installing a Jobe server are given in the Jobe documentation. Once Jobe is installed, use the Moodle administrator interface for the CodeRunner plug-in to specify the Jobe host name and perhaps port number.

A single 4-core Moodle server can handle an average quiz question submission rate of about 60 quiz questions per minute while maintaining a response time of less than about 3 - 4 seconds, assuming the student code itself runs in a fraction of a second. We have run CodeRunner-based exams with nearly 300 students and experienced only light to moderate load factors on an 8-core Moodle server. The Jobe server, which runs student submissions (see below), is even more lightly loaded during such an exam.

The full documentation for CodeRunner, together with forums, questions banks and other resources, is at http://coderunner.org.nz


Potential privacy issues

None that I'm aware of.

Screenshots

Screenshot #0
Screenshot #1
Screenshot #2
Screenshot #3
Screenshot #4
Screenshot #5
Screenshot #6
Screenshot #7
Screenshot #8

Contributors

Picture of Richard Lobb
Richard Lobb (Lead maintainer)
Tim at Lone Pine Koala Sanctuary
Tim Hunt: Co-developer from December 2016
Please login to view contributors details and/or to contact them

Comments RSS

Show comments
  • Picture of Abhinav Dayal
    Sat, Jun 20, 2020, 1:25 PM
    Again, same error:
    Upgrading to new version
    qtype_coderunner
    Error writing to database

    More information about this error

    ×Debug info: Table 'moodle.mdl_question_coderunner_options' doesn't exist
    DELETE FROM mdl_question_coderunner_options WHERE questionid = ?
    [array (
    0 => '38844',
    )]
    Error code: dmlwriteexception
    ×Stack trace:
    line 489 of /lib/dml/moodle_database.php: dml_write_exception thrown
    line 1686 of /lib/dml/mysqli_native_moodle_database.php: call to moodle_database->query_end()
    line 1977 of /lib/dml/moodle_database.php: call to mysqli_native_moodle_database->delete_records_select()
    line 67 of /question/type/coderunner/db/upgradelib.php: call to moodle_database->delete_records()
    line 43 of /question/type/coderunner/db/upgradelib.php: call to delete_existing_prototypes()
    line 342 of /question/type/coderunner/db/upgrade.php: call to update_question_types()
    line 692 of /lib/upgradelib.php: call to xmldb_qtype_coderunner_upgrade()
    line 1917 of /lib/upgradelib.php: call to upgrade_plugins()
    line 711 of /admin/index.php: call to upgrade_noncore()
  • Picture of Richard Lobb
    Sat, Jun 20, 2020, 2:06 PM
    I'd like to be able to replicate this problem. Earlier you said "I upgraded to 3.9 and then installed coderunner". But I have the impression you already had CodeRunner installed, as you were posting to the CodeRunner forum a year or two ago. That suggests you are not installing CodeRunner from scratch. However, if you are installing CodeRunner from scratch you must be getting it from github as the version in the Moodle plugin repository doesn't officially support Moodle 3.9.

    Can you explain in some detail please:

    1. What was your starting point? i.e. which version of Moodle did you have installed, what OS was it on, and what version of CodeRunner was installed within it (if any)?

    2. What upgrade sequence did you follow to get to the above situation?

  • Picture of Abhinav Dayal
    Sat, Jun 20, 2020, 5:08 PM
    I installed coderunner in Moodle 3.4.
    I upgraded to 3.8 with no issues
    However when I upgraded to 3.9 It showed coderunner along with several other plugins to be missing on disk errors and would not continue the installation. At that point of time i thought on uninstalling coderunner and later to reinstall. So I uninstalled coderunner and then the upgrade went fine. Later when I tried to reinstall, it gave that error.
  • Picture of Richard Lobb
    Sat, Jun 20, 2020, 5:34 PM
    Can you clarify how you uninstalled coderunner, please? CodeRunner doesn't have an Uninstall button, so how did you do it?
  • Picture of Abhinav Dayal
    Sat, Jun 20, 2020, 5:43 PM
    I removed the coderunner and adaptive folders by moving them elsewhere. That is how I uninstalled. They was no other way. I did not edit anything in the database.
  • Picture of Abhinav Dayal
    Sat, Jun 20, 2020, 6:16 PM
    Doing it again. I reverted to 3.8 backup and with coderunner already installed in 3.8 version i checked out the 3.9 stable branch of moodle. I then restarted the site and it asked to upgrade to 3.9 which I clicked on continue. After that it said:

    [pluginname,qtype_coderunner]
    /question/type/coderunner
    2020012500 Additional Missing from disk!

    Similar errors are there for many other plugins including Attendance etc. whiever were not supported for 3.9.

    I clicked then on "Upgrade Moodle Database Now"

    it stppoed at
    Plugin "mod_attendance" is defective or outdated, can not continue, sorry.

    More information about this error

    ×Debug info: Missing version.php
    Error code: detectedbrokenplugin
    ×Stack trace:
    line 750 of /lib/upgradelib.php: plugin_defective_exception thrown
    line 565 of /lib/upgradelib.php: call to upgrade_plugins_modules()
    line 1917 of /lib/upgradelib.php: call to upgrade_plugins()
    line 711 of /admin/index.php: call to upgrade_noncore()

    I removed attendance by removing its older and so on for every plugin including coderunner and then it installed fine.

    Later when i try to reinstall coderunner i get the error I sent earlier.
  • Picture of Abhinav Dayal
    Sat, Jun 20, 2020, 6:31 PM
    This time i installed from github as you said instead of from Moodle plugins. It worked!
  • Picture of Michael Wiemers
    Thu, Jul 9, 2020, 9:30 PM
    Hello Richard,

    Do you know of anybody that has used the Coderunner plugin with Stata or could you tell me whether it is possible to set it up for Stata?

    Many thanks
    Michael
  • Picture of Richard Lobb
    Sat, Jul 11, 2020, 1:59 PM
    No, sorry Michael. In fact I'd never even heard of Stata until you posted. You could try a posting to the question authors' forum on coderunner.org.nz. There aren't a lot of permanent subscribers but it does get lots of Google hits so in time someone might answer. To be honest, though, most Moodle/CodeRunner users want software that's free and open source, so I wouldn't expect a lot of enthusiasm. As to whether it's possible it set CodeRunner up to use it: I'd be pretty confident the answer is yes but it would require a high level of CodeRunner expertise, particularly if you want to be able to deal with charts and graphs. You could start by reading how to interface to new languages at https://github.com/trampgeek/moodle-qtype_coderunner#supporting-or-implementing-new-languages.
    If you want to get back images, see https://coderunner.org.nz/mod/forum/discuss.php?d=167 (though that doesn't let you grade them - just look at them).
  • Picture of Adrian Adlaon
    Wed, Jul 15, 2020, 6:42 PM
    Hello fellow educators,

    Can anyone help me fix the CodeRunner Question Type? it does have the drop down choices of programming languages.
    it will only display --> Undefined

    Thank you in advance!
  • Picture of Richard Lobb
    Thu, Jul 16, 2020, 5:51 AM
    Could you post your question to the question authors' forum on https://coderunner.org.nz, please, detailing the steps required to replicate the problem.
  • Sudeep
    Mon, Jul 20, 2020, 10:52 PM
    Can we add another programming language in this plugin like Sphere Engine?
  • Picture of Richard Lobb
    Tue, Jul 21, 2020, 6:43 AM
    Please see https://github.com/trampgeek/moodle-qtype_coderunner#supporting-or-implementing-new-languages. If you have further questions, please post them on one of the forums at https://coderunner.org.nz.
  • Picture of Kashyap Kr
    Sat, Sep 19, 2020, 12:04 AM
    Hello, It's a great plugin and so helpful. How can i add other programming languages to run in CodeRunner?? Like ideone has many other programming languages support.
  • Picture of Richard Lobb
    Sat, Sep 19, 2020, 9:51 AM
    Please see https://github.com/trampgeek/moodle-qtype_coderunner#supporting-or-implementing-new-languages. If you have further questions, please post them on one of the forums at https://coderunner.org.nz.
1 2 3
Please login to post comments