Hi all!
I've just commited a whole lot of changes to HEAD, related to centralising all the file upload stuff, and integration with clam av.
I've so far only integrated it with: assignment, exercise, glossary, forum, resource and scorm (still going with other modules), but if people could begin testing it that would be grand.. clam av can be got from http://www.clamav.net/ and a test virus file (not actually a virus, but contains a virus signature that clam av will pick up) is included, or can be got from
http://nzvle.eduforge.org/clam/testvirus.zip
Stuff to test:
~ all file upload forms should look the same
~ all functionality that previously existed should still be in place (eg. checking upload file size against $CFG->maxbytes, course max file size and module max file size, old files being deleted (if that's what happened before - only some of the modules do this)
~ all uploads should be logged in mdl_log with the userid and the full path to the uploaded file
~ the four new configuration parameters (runclamonupload, pathtoclam, quarantinedir, clamfailureonupload) should behave as advertised (this includes handling of infected files)
~ on-upload scanning should work with both clamscan and clamdscan (which behave slightly differently) also, note that clamdscan needs to be told to scan archives otherwise it will report testvirus.zip as ok even though it hasn't scanned it.
I've also added a file, handlevirus.php in admin/ which takes the output of a command line based clamscan (or clamdscan) and resolves each infected file back to the user that uploaded it (using the log table) and notifies the administrators and the user, and does something with the infected file according to the new config params.
the command line based clamscan should be run like so:
clamscan -r --infected --no-summary [filelist] 2>&1 | php [pathtomoodle]/admin/handlevirus.php
And the filelist should be an absolute path to your dataroot (this is important, because if you don't provide an absolute path the output won't print absolute paths of the files and then handlevirus.php won't be able to find them - I'm going to rework this slightly so it tries harder by adding $CFG->dataroot on the beginning but it's still better to use an absolute path).
And it should be run as root, or whatever your webserver is running so that it has the appropriate permissions to the quarantine directory.
If this can be tested too, that would be fandangle.
Happy testing! Good luck!
Penny
Also, I just added workshop integration.
and now also with files/index.php
.. and now also with lib/editor/coursefiles.php and courseimages.php
Will this be included in the next release? Thanks you very much, this is exciting...
In reply to Penny Leach
Re: centralised upload code with virus scanning
Hi Penny,
great work!! Centralizing things is always good!! And the AV functionality, amazing!!
Ciao
PS: Perhaps user/edit.php should use it too?
great work!! Centralizing things is always good!! And the AV functionality, amazing!!
Ciao
PS: Perhaps user/edit.php should use it too?
In reply to Eloy Lafuente (stronk7)
Re: centralised upload code with virus scanning
by Martin Dougiamas -
Yes, it would be nice to use this new centralised upload function there (even though the virus checker is not needed on that particular upload).
One thing this new central upload functionality will allow is the FTP workaround for safe-mode.
One thing this new central upload functionality will allow is the FTP workaround for safe-mode.
Hey,
yup, user/edit.php is on the list of places to integrate, I just haven't quite got there yet
yup, user/edit.php is on the list of places to integrate, I just haven't quite got there yet
In reply to Penny Leach
Re: centralised upload code with virus scanning
Cool, I'll wait... 
Hi Martin!
Does VirusScanning deserve a special Moodle home (Forum) of its own??
WP1
Does VirusScanning deserve a special Moodle home (Forum) of its own??
WP1
In reply to Eloy Lafuente (stronk7)
Re: centralised upload code with virus scanning
by Penny Leach -
just committed user/edit.php, user/edit.html, course/group.php, course/group-edit.html and (associated change) lib/gdlib.php
Just downloaded the new version of the Workshop module which references lib/uploadlib.php. Running that on 1.4 Moodle fell over because that file is not in the 1.4 distribution. I think we need to ramp up the "requires" version number in all the modules that have been modified to use the new upload routine. Not sure what the correct version number is, something quite recent I suspect.
Or, are we thinking of back-pedelling the upload routine into 1.4.x?
Or, are we thinking of back-pedelling the upload routine into 1.4.x?
You're right, Ray. I hadn't thought of that.
I don't _think_ MD is planning on porting to 1.4, but I've been developing against 1.4 and porting my patches to 1.5 to commit to HEAD.
FYI, We're maintaining a public arch repository where you can see all this at http://lists.eduforge.org/cgi-bin/archzoom.cgi/arch-eduforge@catalyst.net.nz--2004-MIRROR/moodle--eduforge--1.3.3
I will be providing a full patch against 1.4 for all the virus stuff but I think yes, we need to bump the requires version in HEAD.
I don't _think_ MD is planning on porting to 1.4, but I've been developing against 1.4 and porting my patches to 1.5 to commit to HEAD.
FYI, We're maintaining a public arch repository where you can see all this at http://lists.eduforge.org/cgi-bin/archzoom.cgi/arch-eduforge@catalyst.net.nz--2004-MIRROR/moodle--eduforge--1.3.3
I will be providing a full patch against 1.4 for all the virus stuff but I think yes, we need to bump the requires version in HEAD.
I'm not planning on porting this to 1.4, no, and yes, all the requires in the different modules should be updated to match. I've just updated the main version.php to 2004091700 so please use that that ... thanks for all your work on this Penny!
No problemo, I'm glad to help
Should I bump the module versions as well as the module requires?
Should I bump the module versions as well as the module requires?
Oh yes, bump both of them ... thanks for checking!
bumped!
cheers
cheers
I've just made a complete patch against 1.4.1
Well, actually I say complete but it's not quite, it's missing full integration with the wiki module, I'm waiting on the module maintainers for that. (I've integrated the wikifiles part, which is where you choose or upload an initial file for the wiki)
The patch is at http://nzvle.eduforge.org/clam/moodle141stableupload.diff
to apply it, change into the moodle working directory, and:
patch -p1 < /path/to/downloaded/patch.diff
it will create two files:
admin/handlevirus.php
lib/uploadlib.php
and patch the following:
admin/config.html
admin/handlevirus.php
admin/uploaduser.php
backup/lib.php
course/group-edit.html
course/group.php
course/lib.php
files/index.php
lang/en/glossary.php
lang/en/moodle.php
lib/editor/coursefiles.php
lib/editor/courseimages.php
lib/gdlib.php
lib/moodlelib.php
lib/weblib.php
mod/assignment/lib.php
mod/assignment/upload.php
mod/exercise/locallib.php
mod/exercise/upload.php
mod/forum/lib.php
mod/forum/post.html
mod/forum/post.php
mod/glossary/edit.html
mod/glossary/edit.php
mod/glossary/import.html
mod/glossary/import.php
mod/glossary/lib.php
mod/lesson/import.php
mod/quiz/format/coursetestmanager/format.php
mod/quiz/import.php
mod/resource/coursefiles.php
mod/scorm/coursefiles.php
mod/wiki/wikifiles.php
mod/workshop/locallib.php
mod/workshop/upload.php
user/edit.html
user/edit.php
please back up your files first.
it doesn't require any database changes. you will need to configure it, there are four new parameters in the configure variables screen. also if you want to use cron based clamscan, see the instructions at the top of admin/handlevirus.php
also, bear in mind this will be in the 1.5 release.
Well, actually I say complete but it's not quite, it's missing full integration with the wiki module, I'm waiting on the module maintainers for that. (I've integrated the wikifiles part, which is where you choose or upload an initial file for the wiki)
The patch is at http://nzvle.eduforge.org/clam/moodle141stableupload.diff
to apply it, change into the moodle working directory, and:
patch -p1 < /path/to/downloaded/patch.diff
it will create two files:
admin/handlevirus.php
lib/uploadlib.php
and patch the following:
admin/config.html
admin/handlevirus.php
admin/uploaduser.php
backup/lib.php
course/group-edit.html
course/group.php
course/lib.php
files/index.php
lang/en/glossary.php
lang/en/moodle.php
lib/editor/coursefiles.php
lib/editor/courseimages.php
lib/gdlib.php
lib/moodlelib.php
lib/weblib.php
mod/assignment/lib.php
mod/assignment/upload.php
mod/exercise/locallib.php
mod/exercise/upload.php
mod/forum/lib.php
mod/forum/post.html
mod/forum/post.php
mod/glossary/edit.html
mod/glossary/edit.php
mod/glossary/import.html
mod/glossary/import.php
mod/glossary/lib.php
mod/lesson/import.php
mod/quiz/format/coursetestmanager/format.php
mod/quiz/import.php
mod/resource/coursefiles.php
mod/scorm/coursefiles.php
mod/wiki/wikifiles.php
mod/workshop/locallib.php
mod/workshop/upload.php
user/edit.html
user/edit.php
please back up your files first.
it doesn't require any database changes. you will need to configure it, there are four new parameters in the configure variables screen. also if you want to use cron based clamscan, see the instructions at the top of admin/handlevirus.php
also, bear in mind this will be in the 1.5 release.
Hey Penny,
Thanks for all of your work on this. This is really fantastic.
I have one small request. If it doesn't take too much time, could you post a 1.4.3+ diff? The 1.4.1 doesn't work so well (many of the modules fail). I'm doing some patching by hand, but it's a very difficult process.
We won't be able to use 1.5 this semester, but I'd like to be able to tell faculty the files are being scanned.
Thanks again.
Jason
Thanks for all of your work on this. This is really fantastic.
I have one small request. If it doesn't take too much time, could you post a 1.4.3+ diff? The 1.4.1 doesn't work so well (many of the modules fail). I'm doing some patching by hand, but it's a very difficult process.
We won't be able to use 1.5 this semester, but I'd like to be able to tell faculty the files are being scanned.
Thanks again.
Jason
Actually that's not as easy as it sounds. When I developed it, it was against what was then stable, 1.4.1.
The only way to get the diff would be to diff our version against the 1.4.3+ from CVS, but we're running a pretty customised version of moodle, so as well as the file upload/virus scanning stuff, there would be an awful lot of noise to filter out.
What you're asking would probably mean a few hours work for me
I know that dealing with .rej files from failed patches is annoying, but something that I found to be really useful was emacs diff mode - http://wiki.gnuarch.org/moin.cgi/Process_20_2a_2erej_20files
Also you can always take a look at our arch repository and grab the files you need and have a look at them - you'll find a full set of files with the latest patches applied here:
http://lists.eduforge.org/cgi-bin/archzoom.cgi/arch-eduforge@catalyst.net.nz--2004/moodle--eduforge--1.3.3--patch-466
For the modules that are failing, diffing your version against ours will probably yeild what you want - there might be some excess noise with some customisations but you should be able to see straight away which changes are to do with upload/virus stuff.
Good luck! Hope that helps.
The only way to get the diff would be to diff our version against the 1.4.3+ from CVS, but we're running a pretty customised version of moodle, so as well as the file upload/virus scanning stuff, there would be an awful lot of noise to filter out.
What you're asking would probably mean a few hours work for me
I know that dealing with .rej files from failed patches is annoying, but something that I found to be really useful was emacs diff mode - http://wiki.gnuarch.org/moin.cgi/Process_20_2a_2erej_20files
Also you can always take a look at our arch repository and grab the files you need and have a look at them - you'll find a full set of files with the latest patches applied here:
http://lists.eduforge.org/cgi-bin/archzoom.cgi/arch-eduforge@catalyst.net.nz--2004/moodle--eduforge--1.3.3--patch-466
For the modules that are failing, diffing your version against ours will probably yeild what you want - there might be some excess noise with some customisations but you should be able to see straight away which changes are to do with upload/virus stuff.
Good luck! Hope that helps.
That emacs mode just rocks.