Grading Submissions on Moodle VPL

Grading Submissions on Moodle VPL

by Ricardo Mizos -
Number of replies: 14

Hello, I am in Moodle and I have several execution files that I execute in a VPL that use Python, and I am having trouble trying to get VPL to evaluate some submission files.

A student account submits several submission files, and when I go to execution files, and I click on "Evaluate", it turns out that those files that the program is calling for under "customEval.py" are not even inside the temporary directory that the Moodle server makes for the purpose of this evaluation. Only files like customEval.py, vpl_run.sh, vpl_evaluate.sh, etc.

Are those student submission files located somewhere else in the server? If so, where? What other directory? How can I get to this other directory? Would I have to use the grep command in the code, or the Python equivalent of that? If the student submission files are in there somewhere, I could type in code to copy them over to the temporary directory for the evaluation process.

Average of ratings: -
In reply to Ricardo Mizos

Re: Grading Submissions on Moodle VPL

by Juan Carlos Rodríguez-del-Pino -
Picture of Particularly helpful Moodlers Picture of Plugin developers
Dear Ricardo,
I don't know what are you trying to do but I will try to explain how the evaluatión is carried out.
1) A user pushes the "evaluate" button related to a submission.
2) The system package the submitted files plus the "execution files" (set of files defined in VPL activity) and the custimized or default scripts for "run" and "evaluate" actions.
3) The system selects an execution server.
4) The system sends the package to the execution server
5) The execution server creates a home dir for a temp user and unpackages the files there.
6) Compilation stage: The execution server runs the vpl_evaluate.sh script.
7) if vpl_evaluate.sh generate the vpl_execution program goes next stage.
8) The system removes all "execution files" (set of files defined in VPL activity) but not the files in "Files to keep when running" (VPL activity advanced setting)
9) Run stage: The system runs vpl_execution (The test program) and processes the output to generate a report and grade.
10) The systems remove the home directory and the temp user.

As you can see, the system only uses the home directory of the temp user. Subdirectories used in VPL (submission and execution files) are appended to this home dir.

I hope that helps.

Best regards,
Juan Carlos.
Average of ratings: -
In reply to Juan Carlos Rodríguez-del-Pino

Re: Grading Submissions on Moodle VPL

by Ricardo Mizos -

Thank you. I have a few more questions.

1. Some of the files defined in the VPL Activity do not get packaged for use in evaluation (such as example submission files that can be used for testing purposes), which is what I am trying to do. So far I had to use a separate student account to test it, and the test worked, but now I need to know how to get more new files in besides the main ones and other ones that do get in.

2. Follow up of question 1, it seems like when a student submits a file, and a file of the exact same name is defined in VPL activity, the student's file(s) that are in VPL activity are not packaged for the evaluation and seem to be ignored. Why is this? This test was even done with both files (student's and in the VPL) have the same contents., so is this supposed to be some sort of anti-plagiarism measure?

3. Under what privileges do the scripts execute and the submissions get evaluated? A student can run the evaluation script to auto check their submissions' output, but that student should not be able to do anything that compromises security, such as have access to the execution files in VPL activity.

Average of ratings: -
In reply to Ricardo Mizos

Re: Grading Submissions on Moodle VPL

by Juan Carlos Rodríguez-del-Pino -
Picture of Particularly helpful Moodlers Picture of Plugin developers

Dear Ricardo,
A1) Please read the answer to the Raju post
A2) The security is first. Yes if the student submits files with the same name that files in the "Execution files" the one in the "execution files" is used.
A3) All executions in the VPL execution server are done as a temp regular user without privileges. To avoid the students' access to the testing code see the answer to the Raju post.

Best regards,
Juan Carlos.

Average of ratings: -
In reply to Juan Carlos Rodríguez-del-Pino

Re: Grading Submissions on Moodle VPL

by Raju Pandey -

Dear Dr. Carlos,

This question is a follow up to Ricardo's question. Here is what we are trying to do.

Instructor:

  * Instructor defines a programming homework in the VPL.

 * Instructor defines a number of files (about 10) that will be used for testing the homework. These files can be packaged in a zip file, or can be uploaded individually. 

* Instructors loads these files in VPL using "execution files" upload mechanism. The instructor can see those files. Indeed, when zipped files are uploaded, the VPL automatically unzips them and shows them.

Student:

  * Implements the programming homework

 * Presses the evaluate button, which should cause the evaluation to occur.

What we have found is the following:

After the evaluate button is pressed, the evaluation script runs, and:

 * The student's test programs gets copied.

* Only some of the files in the "execution files" are copied. Not all files from execution files are copied

* Evaluation fails because not all of the instructor's testing files are there. 

We checked the directory by running a python script when an evaluation takes place. The script iterates over the files of the home directory. It shows that only some files are copied, and most of the instructor's test files are not. Is there something that we are missing?

We truly appreciate your help in this regard.

Regards,

Raju Pandey


 

Average of ratings: -
In reply to Raju Pandey

Re: Grading Submissions on Moodle VPL

by Juan Carlos Rodríguez-del-Pino -
Picture of Particularly helpful Moodlers Picture of Plugin developers

Dear Ricardo and Raju,

Have you set these files at "Files to keep when running"? else the system removes these files before running the code (Step 8)

IMPORTANT (if you allow the students to run their code on this activity): As you are using these files to test the students' submissions the presence of these files while the students run its code allows students to view the test code and may lead to cheats.

To remove these files from the run action add the file "pre_vpl_run.sh" to the executions files. This script is run before the vpl_run.sh, vpl_debug.sh and vpl_evaluate.sh. The vpl_run.sh may look like:

#!/bin/bash
if [ ! -f vpl_evaluate.sh ] ; then
   rm test_file1.py
   rm test_file2.py
   ...
fi
Average of ratings: -
In reply to Juan Carlos Rodríguez-del-Pino

Re: Grading Submissions on Moodle VPL

by Raju Pandey -
Dear Dr. Carlos,

Your suggestion worked. We had not set the "files to keep when running" properly.

Thank you again for your quick reply and help. 

Regards,
Raju
Average of ratings: -
In reply to Juan Carlos Rodríguez-del-Pino

Re: Grading Submissions on Moodle VPL

by Ricardo Mizos -
I tried creating the pre_vpl_run.sh file, even making sure to checkmark it under "Files to keep when running", but for some reason this file isn't running at all. The contents of the file don't matter, the VPL will not execute this file first, and I even checked to make sure the name of the file was right.

It seems I do have to get this file to run first before vpl_run.sh because otherwise the student's files would be deleted by the code meant for pre_vpl_run.sh before the program evaluates them. It seems that pre_vpl_run.sh would run first, deleting the sample files, before the student files are added into the temporary space, then the student files are added in and vpl_run.sh is run, and the sample files do not cancel out the student files because they (sample files) were deleted first.

How do I get this file to run before vpl_run.sh?
Average of ratings: -
In reply to Ricardo Mizos

Re: Grading Submissions on Moodle VPL

by Juan Carlos Rodríguez-del-Pino -
Picture of Particularly helpful Moodlers Picture of Plugin developers
Dear Ricardo,
It is very difficult to spot the problem without the details. Please, download the "requested files" and the "execution files" and post it to try to find the problem.

Best regards,
Juan Carlos.
Average of ratings: -
In reply to Juan Carlos Rodríguez-del-Pino

Re: Grading Submissions on Moodle VPL

by Ricardo Mizos -
I don't feel like I'm at liberty to share the files, but I also don't think that would help as the problem actually has to do with the VPL itself choosing which shell file to run first. Is there a way to change which exact file is run when Run, Debug, or Evaluate is clicked?
Average of ratings: -
In reply to Ricardo Mizos

Re: Grading Submissions on Moodle VPL

by Juan Carlos Rodríguez-del-Pino -
Picture of Particularly helpful Moodlers Picture of Plugin developers

Dear Ricardo,
To customize what program VPL runs when Run, Debug, or Evaluate is clicked you can write your own vpl_run.sh, vpl_debug.sh or vpl_evaluate.sh at "execution files". Notice that these files are the "compilation stage" and generate the program that finally is run that must be named vpl_execution.
You can see examples of these files at https://github.com/jcrodriguez-dis/moodle-mod_vpl/tree/v3.3.7/jail/default_scripts

Best regards,
Juan Carlos.

Average of ratings: -
In reply to Juan Carlos Rodríguez-del-Pino

Re: Grading Submissions on Moodle VPL

by Ricardo Mizos -
Thank you.

We decided to move on to using an AI-experience center (AIEC) called Jupyter for students to be able to store their files on that server, with each student/instructor having their own directory there.

We want to integrate that server to our VPL LMS-client, and we want students to be able to download the files they work on in Jupyter and then upload them to a VPL for evaluation. One issue we have is the security. When trying to download a file, someone should be presented with an authentication request (username/password).

Is it possible to use a client like Putty or WinSCP to SSH into a Jupyter server? If so, what would be the proper credentials? I tried this with WinSCP and failed, possibly because the server may not have FTP/SFTP installed.

The thing is, from the VPL I am able to use the wget command with the link to the file to download it onto the temporary drive in the VPL server during the execution. This should fail and/or ask for authentication credentials. We do not want students to be able to see or get files they're not supposed to.

How can I implement this kind of security to any of the servers? I believe our main goal is to unify all 3 servers (Jupyter, VPL, LMS)
Average of ratings: -
In reply to Ricardo Mizos

Re: Grading Submissions on Moodle VPL

by Juan Carlos Rodríguez-del-Pino -
Picture of Particularly helpful Moodlers Picture of Plugin developers
Dear Ricardo,
Your question does not have an immediate or easy answer. The authentication of VPL is under Moodle control. There is no easy way to access VPL without Moodle authentication. VPL provides (under Moodle control) a web services API you can try to use.

Best regards,
Juan Carlos.
Average of ratings: -
In reply to Juan Carlos Rodríguez-del-Pino

Re: Grading Submissions on Moodle VPL

by Ricardo Mizos -

Dear Juan,

Thank you for your response, but what we are trying to do now is find a way for VPL to be able to access a student's files on their directory on a Jupyter server and grade them.

For the time being we log in and download them manually, then submit them to Moodle, but we want the execution of the VPL to get the files automatically so that when a student decides to have their files evaluate, the VPL will get them from Jupyter and grade them. I understand what you explained to me about how there's no easy answer, but I came up with another idea based on what I know works, and I explain it below.

Since I am capable of using wget to do this, I thought if there was a way, with the use of strings and variables, to run a wget that gets the correct files based on the name of the student on Moodle and the name of the student on Jupyter. I would assume these have to match, but is there a line of code that gets the name of the student running the execution which would be stored as a string during the execution itself? This would help in the creation of the file path for wget, which uses variables and strings to construct it in the program.

One last question, I would like to know if and how it's possible for a student to run the VPL's execution without providing their own file manually in Moodle since this would be done automatically by the VPL's execution itself. 

Thank you!


Average of ratings: -