For automated grading of computer programs, https://github.com/trampgeek/CodeRunner is another Moodle add-on that you might want to look at. (It is a question type.)
Re: New Virtual Programming Lab (VPL) module
Hi Antonio,
You can find more information about automatic evaluation in VPL at the following links:
http://vpl.dis.ulpgc.es/index.php/en/documentation/76-general-documentation
Section "Creating automated tests"
and
http://vpl.dis.ulpgc.es/index.php/en/documentation/58-automatic-evaluation-of-students-submissions
Best regards
Having problem creating user vpl_evaluate.sh script
Dear Ruan Carlos,
First, thanks for putting together such a useful and sophisticated module for Moodle.
I teach computer science and am interested in setting VPL up for my class next semester.
I have setup two virtual servers, one running LAMP+Moodle+VPL module, the other one running the vpl-jail-system.
It took me a while, but everything seems to run fine. I have a simply python 3 program called test2.py that I am currently using as test:
x = float( input( "> " ) )
y = float( input( "> " ) )
print( min( x, y ) )
And I have a vpl_evaluate.cases file to test it:
case = Test 2a
input = 3
5
output = 3.0
case = Test 2b
input = 3
3
output = 3.0
case = Test 2c
input = 1000
-1000
output = -1000.0
When I run or evaluate the script with the default vpl_run.sh and vpl_evaluate.sh, i.e. without specifying them, I get the console, and I can enter 2 numbers and get the smallest one back, and the evaluate gives me a "0/3 tests fail" which is good.
When I create my own vpl_run.sh script, it also works fine and runs the program in the console.
#! /bin/bash
echo "#! /bin/bash" > vpl_execution
echo python3.4 test2.py >> vpl_execution
chmod a+x vpl_execution
jail/home/p15913# ls -l
total 16
-rwx------ 1 15913 15913 1559 Jun 7 21:28 common_script.sh
-rw------- 1 15913 15913 77 Jun 7 21:28 test2.py
-rwx------ 1 15913 15913 360 Jun 7 21:28 vpl_environment.sh
-rwxr-xr-x 1 15913 15913 32 Jun 7 21:28 vpl_execution
Re: Having problem creating user vpl_evaluate.sh script
Hello Dominique,
Thank for the clear exposition of your problem.
All types of executions (run/debug/evaluate) take two stages: The first generates vpl_execution or vpl_wexecution, the second executes vpl_?execution. But before start the second, the system remove all the execution files (vpl_evaluate.sh, vpl_evaluate.cases, etc.) that you set in "ADVANCED SETTINGS ► EXECUTION FILES".
Your 'ls' may be showing that you are in the second stage.
You can go to " ADVANCED SETTINGS / FILES TO KEEP WHEN RUNNING" and set the files that you miss as to be kept. Also you can rename the files in vpl_evaluation.sh and the system will not be able to delete them.
Best regards
Re: Having problem creating user vpl_evaluate.sh script
Thanks for the quick response, Juan Carlos.
Unfortunately, setting the files to keep to all the vpl_run/evaluate/cases does not solve my problem.
However, I discovered that I get an error on the jail server when I attempt to use my own vpl_evaluate.sh:
ERROR: apport (pid 8308) Tue Jun 10 10:16:51 2014: Unhandled exception:
Traceback (most recent call last):
File "/usr/share/apport/apport", line 298, in <module>
cwd = os.readlink('/proc/' + pid + '/cwd')
FileNotFoundError: [Errno 2] No such file or directory: '/proc/8307/cwd'
ERROR: apport (pid 8308) Tue Jun 10 10:16:51 2014: pid: 8308, uid: 0, gid: 0, euid: 0, egid: 0
ERROR: apport (pid 8308) Tue Jun 10 10:16:51 2014: environment: environ({})
ERROR: apport (pid 8310) Tue Jun 10 10:16:51 2014: another apport instance is already running, aborting
Re: Having problem creating user vpl_evaluate.sh script
Hi Dominique,
the error you show is not due to VPL and I don't know the reason of the problem. Nevertheless, I found that apport is logging a normal action in vpl-jail-server as an error. This is my fault I wrote a call to abort() instead of exit(). This problem (the logging) of apport will be solved in vpl-jail-server 2.0.2.
Returning to your question, I think that the better way to help you to resolve your problem with vpl_evaluation.sh is that you show the details of the activity you are developing.
A backup file with the activity migth be great! if you are using "based on" don't forget to also add this activity
Best regards.
Juan Carlos
Re: Having problem creating user vpl_evaluate.sh script
Sure, here's a description of the activity:
Test Python 1
Requested files: test1.py (Download)
Type of work: Individual work
Grade settings: Maximum grade: 100
Run: Yes Evaluate: Yes
Automatic grade: Yes
This is the second trial...
Execution files
vpl_run.sh
1 #! /bin/bash 2 echo "#! /bin/bash" > vpl_execution 3 echo "python3.4 test1.py" >> vpl_execution 4 chmod a+x vpl_execution 5
vpl_debug.sh
vpl_evaluate.sh
1 #! /bin/bash 2 echo "#! /bin/bash" > vpl_execution 3 echo "python3.4 test1.py" >> vpl_execution 4 chmod a+x vpl_execution 5
vpl_evaluate.cases
When I test the activity in the VPL administration module, I go to Edit and enter this simple code:
print( "Hello World!" )
which I save. I click on Run and get the correct output in the console.
Now, for some reason I don't understand, when I click on Evaluate, I do not get the counter shown seconds elapsing (I haven't changed anything in my setup), but instead I get the output of the program but no evaluation of its output and no grade.
I am forging ahead and learning how to setup VPL activities without having a custom vpl_evaluate.sh file, but I imagine that I will need to do so in the future, so any help on this would be much appreciated.
By the way, I am documenting my progress and putting sample VPL activities on my wiki, mostly so that I can remember how to set this up when the time comes next semester. Others may find them useful too: http://cs.smith.edu/dftwiki/index.php/Tutorials
Thanks again for the superb tool and support!
Dominique
Re: Having problem creating user vpl_evaluate.sh script
Ah! I did find what I think is the problem. I just created a new VPL activity and created my own vpl_evaluate.sh, which is now stuck running and counting seconds as I originally described in my first post.
When I look on the jail server, I see that the files tested are located in /jail/home/p19526:
root@VPLServer:/jail/home/p19526# ls -l
total 32
-rwx------ 1 19526 19526 1559 Jun 11 10:23 common_script.sh
-rw------- 1 19526 19526 233 Jun 11 10:23 gremlin.py
-rw------- 1 19526 19526 97 Jun 11 10:23 sample.txt
-rwx------ 1 19526 19526 408 Jun 11 10:23 vpl_environment.sh
-rw------- 1 19526 19526 175 Jun 11 10:23 vpl_evaluate.cases
-rwx------ 1 19526 19526 162 Jun 11 10:23 vpl_evaluate.sh
-rwxr-xr-x 1 19526 19526 49 Jun 11 10:23 vpl_execution
-rwx------ 1 19526 19526 162 Jun 11 10:23 vpl_run.sh
However, when I check the vpl-related processes running, I see this:
root@VPLServer:/jail/home/p19526# ps auxw | grep vpl
root 2242 0.0 0.0 23520 1000 ? S Jun04 5:01 /etc/vpl/vpl-jail-server
root 22910 0.6 0.0 32232 1856 ? S 10:23 0:00 /etc/vpl/vpl-jail-server
19526 22911 0.0 0.0 0 0 ? Zs 10:23 0:00 [vpl_evaluate.sh] <defunct>
19526 22913 0.0 0.0 17960 1436 pts/3 Ss+ 10:23 0:00 /bin/bash /home/p19526/vpl_execution
root 22915 1.0 0.0 23912 1724 ? S 10:23 0:01 /etc/vpl/vpl-jail-server
root 22963 0.0 0.0 11744 924 pts/0 S+ 10:25 0:00 grep --color=auto vpl
Note that the vpl_execution file was executed from "/home/p19526/", which does not exist. It should have been "/jail/home/p19526"...
I will try to figure out where/how this setting can be modified...
Dominique
Re: Having problem creating user vpl_evaluate.sh script
The ps output is normal. Due to chroot, the path
/home/p19526/vpl_execution
is correct.
Best regards
Re: Having problem creating user vpl_evaluate.sh script
Hi Juan,
i have a totally unrelated question, is it possible to install Moodle+VPL PLugin + Jail service install in one virtual machine?
i need to give this logic to one of our client who is unable to understand why we need to have two machines dedicated to moodle.
Re: Having problem creating user vpl_evaluate.sh script
Yes, it's possible, but not recommended. The load of execution in the jail server can interfere in the execution of Moodle. The security is another problem that can arise.
If you have enough hardware assigned to the moodle server you can install Virtualbox in your virtual machine (yes, a virtual machine into a virtual machine) and install there the jail server.
You must be aware of three things:
1) Your jail server (service) must have assigned two ports differents than defaults (80 and 443) for example 52000 and 52001. Change properly the configuration file.
2) Your jail server ports must be accessible from internet (not only from your Moodle server). You can use NAT port forwarding if using in VirtualBox
3) Use the correct URL (eg http://servername:port/path) in the general plugin configuration in the Moodle server
Best regards
Juan Carlos
Re: Having problem creating user vpl_evaluate.sh script
Thank you very much for detailed response. really appreciate your help.
Cheers,
A
Re: Having problem creating user vpl_evaluate.sh script
Hi Dominique,
Thanks for your tutorial, any help is appreciated.
Again your clear report of the issue makes me to point out the problem easily.
The problem is that vpl_evaluation.sh is the script to generate the program/script that evaluates the student's program and NOT the student's program that is going to be evaluated.
The default vpl_evaluate.sh uses the result of vpl_run.sh (vpl_execute) as the program to be evaluated.
Notice that the output of the execution of vpl_execute, generated by vpl_evaluate.sh, is the result of the evaluation. You can see the format of the output of this execution in the documentation .
Best regards
Question: How to make output of program evaluated become the grade
Thank you so much for all the help, Juan Carlos.
I have another question, if you don't mind.
I would like to have students submit assembly language programs, and one of the requirements for one of the tests would be to generate code that as small an executable as possible. In other words they would have to pay close attention to the size of their code and of their data.
I have figured out how to test/evaluate an assembly language program. That's fine. (By the way, I documented the VPL activity here for those interested: http://cs.smith.edu/dftwiki/index.php/Tutorial_Moodle_VPL_Assembly_Language_with_Output)
Now, I know I could add a new program to the "Execution Files" section of the VPL administration block that could get the size of the executable version of the student assembly program. This second program could easily output 100, 90, 80, 70, etc... depending on whether the size of the student program is within 10%, 20%, 30%, 40%, etc., of the best size.
I would like the output of this second program (with will be a number between 100 and 0) to become the grade assigned to the student program.
I'm still very confused about how to write a vpl_evaluate.sh script that would take this number output by the program and make it the grade.
Any suggestion or hint?
Merci!
Dominique
Re: Question: How to make output of program evaluated become the grade
In this case I think that an example will be better. Copy and paste the following script into vpl_evaluate.sh and push evaluate in the test activity.
-------------------------------------------
#!/bin/bash
cat >vpl_execution <<END_SCRIPT
#!/bin/bash
# This is an example of evaluation output
#
# One line comment
echo "Comment :=>>This is a one line comment"
# Multiline comment
#start tag
echo "<|--"
echo "This is a multi line comment (line 1)"
echo "This is a multi line comment (line 2)"
echo "This is a multi line comment (line 3)"
echo "This is a multi line comment (line 4)"
#end tag
echo "--|>"
#This is a grade set
echo "Grade :=>>1Value"
# Comments may have some format
echo "<|--"
echo "-Titles start lines with '-'"
echo "Normal text"
echo ">preformated text"
echo "--|>"
#Last grade is taken
echo "Grade :=>>Value"
END_SCRIPT
chmod +x vpl_execution
--------------------------------
Best regards
Re: Question: How to make output of program evaluated become the grade
Once again, thanks for the quick answer. This was what I needed to figure out how to make it work. I can now test a program and assign a grade inversely proportional to the size of the executable version of the program.
I have documented it here for those interested: http://cs.smith.edu/dftwiki/index.php/Tutorial:_Moodle_VPL_--_Testing_the_Size_of_A_Program%27s_Executable
Thanks much!
Dominique