WebDAV for "Course Files" in 2.0

WebDAV for "Course Files" in 2.0

by Martín Langhoff -
Number of replies: 28

Everybody wants to know more about WebDAV on 2.0! And it seems to be the right time too. Looking at the gapps 'planning document' and the "meta" tracker MDL-14589 it seems that the core API is complete and stable, and that "course files" and "site files" are working.

(Petr could confirm that I am reading it right; if he says "still needs a rewrite" I can wait some more... smile )

So, what we have now, in summary, is the following:

A 1.9 implementation that is known to work with earlier 1.9.x. I haven't personally tested it on the latest _STABLE, but I see no reason for it to have problems (other than minor merge conflicts).

The 1.9 implementation is known to be good, safe to use, and generally with outvices.

  • Upload 7GB DVD images? No problem with Moodle! (your ISP might charge some extras smile )
  • Upload 10k little files? It works too.
  • Unicode chars in the course name? Works (if your client side does proper unicode, most do).
  • Rename a "course directory" -- forbidden, that'd break moodledata!
  • Delve into moddata directories? Naughty boy that's forbidden!
  • Create files or dirs with names that Moodle wouldn't approve of. Well, forbidden.

Tested and used widely with MacOSX, Windows XP, Vista and Gnome and KDE on Linux. Known issues:

  • Early MacOSX have some issues (IIRC, with https). Very early OSX is unusable in general w WebDAV.
  • Some clients cannot handle the nice "Course name" -- (I think I may have fixed this). Check whether your 'client' handles "DisplayNames".
  • Windows XP implements WebDAV via a DLL that various versions of Office upgrade/downgrade. Depending on which version of the DLL you have, there are different subtle bugs. None are the end of the world. There is a page on the 'net tracking all the versions and bugs...
  • WebDAV has to do username/password authentication. so it won't play well with unconventional auth plugins. First casualty here is the the ActiveDirectory "transparent SSO" feature of auth/ldap (which I reworked/merged from Dan Marsden's original code). Maybe someone with access to Windows clients and understanding of the subtleties involved can make those 2 interact better. It is a "little project from hell" but I think it can be made to work.

Where is the 1.9 code? The branch in the git repo is here: http://git.catalyst.net.nz/gitweb?p=moodle-r2.git;a=shortlog;h=mdl19-webdav

It has seen several merges from 1.9.x so it's not trivial to get a clean diff from gitweb directly. If you use git, the command is (note the 3 dots):

git diff origin/MOODLE_19_STABLE...origin/mdl19-webdav

The resulting patch is here: http://dev.laptop.org/~martin/mdl19-webdav-g385bbff.diff

In the coming 2 weeks, if Petr says the Files stuff is in good shape, I can rebase this code onto current CVS HEAD, so we have it for 2.0.

Interest? wink

Average of ratings: -
In reply to Martín Langhoff

Re: WebDAV for "Course Files" in 2.0

by Petr Skoda -
Picture of Core developers Picture of Documentation writers Picture of Peer reviewers Picture of Plugin developers
First of all there are no "site files" and "course files" as you remember them in 2.0, in fact the last step in 2.0 migration is to disable these completely in new installs wink

Instead we have the file_browser class that is used for virtual browsing of the context tree and plugin file areas. The access control when browsing is pretty simple now, each folder knows if writing allowed or not.

I looked at the WebDAV some time ago, the major problem was all libraries expected the files to actually exist in the file system which is not our case.

In 2.0 we can implement several things:
* webdav access to some real OS directory which is then accessed through some repository plug-in
* implement webdav access to user's personal file area - no access control problems, we could use low level file API
* implement full webdav access to the who file_browser tree - all file access as to go through the plugins, we can not access the files table directly

I think that the code from 1.9.x is not much useable in the HEAD, most of the code will need to be rewritten, but maybe with your expertise it would not be that difficult.

Anyway thanks a lot the long list of bugs in client software.
Average of ratings: -
In reply to Petr Skoda

Re: WebDAV for "Course Files" in 2.0

by Martín Langhoff -

Hmmm, interesting...

One of the key questions then is "what files do we provide access to via WebDAV?".

On 1.x, teachers could upload files to the "course files" area. My WebDAV patches allow access to that area specifically, which "works the same" from a user PoV as the non-WebDAV access.

In Moodledata (in 1.x) we store a lot of other things, session, tmp, and moddata inside each course dir. My patch does not offer access there.

Questions:

  • From a user PoV, how do I use the "2.0" version of 'course files'?
  • From a user PoV, what do I do with "personal file area"? We can sure provide WebDAV access to it.

To address one of your concerns -- the WebDAV code we have does not need things to be in a real directory. You subclass the base class and override the different methods to hook into the "directory listing" and "path lookup" methods.

So it is perfectly OK for our purpose smile we just have to figure out what we want to show, and based on what data structures.

Average of ratings: -
In reply to Martín Langhoff

Re: WebDAV for "Course Files" in 2.0

by Petr Skoda -
Picture of Core developers Picture of Documentation writers Picture of Peer reviewers Picture of Plugin developers
"From a user PoV, how do I use the "2.0" version of 'course files'?" - there are no course files, each file is in some area which usually belongs to some module, moodle core must not touch those files directly, all file access (download, browsing, link printing) goes through the module code. Course maintains separate sets of files for course introduction, course sections, etc (in this case core is the owner of files).

You can provide "low level" access to Personal files, because the files are "owned" by the core - these files are avaialble in the file picker (exposed as repository) and teachers or students may used them in any file picker (upload homework, add blog image, add attachmetns, create resource, etc.).

In files/index.php you can see a temporary file manager that is using the file_browser class to browse the virtual file tree.
Average of ratings: -
In reply to Petr Skoda

Re: WebDAV for "Course Files" in 2.0

by Martín Langhoff -

Course maintains separate sets of files

So that's is no longer 'core' but is the equivalent (from a user's perspective). Right? Is that handled by code under /course ?

In files/index.php you can see a temporary file manager

Is that complete enough that I can learn the right APIs? I am a monkey-see, monkey-do kind of programmer...

Average of ratings: -
In reply to Martín Langhoff

Re: WebDAV for "Course Files" in 2.0

by Petr Skoda -
Picture of Core developers Picture of Documentation writers Picture of Peer reviewers Picture of Plugin developers
You can find some leads to the hidden bananas in lib/file/file_browser.php smile
Average of ratings: -
In reply to Martín Langhoff

Re: WebDAV for "Course Files" in 2.0

by Martín Langhoff -

Right, so following the orangutan, I got an initially-working WevDAV implementation.

Right now it passes the 'basic' part of the litmust test -- browsing, reading, file creation, directory creation and deletion work.

What is missing? COPY/MOVE and PROPFIND.

Took quite a bit longer than I expected... it is here http://dev.laptop.org/git/users/martin/moodle.git/log/?h=mdl20-webdav

This is my first serious splash into 2.0... so I have some notes.

  • The new DB stuff is great (yay!). No, really. Outrageous! Fantastic!
  • The File API is good... but... I really wish it was more unixy, and followed the same function names as we use for... files and directories.
  • The error reporting / debugging stuff gives me a ton more trouble than it reports....

When things go wrong... I don't have a theme, or a $PAGE, or anything. The path of errorhandling should not rely on more libraries, globals and stuff that may well be borked.

Average of ratings: -
In reply to Martín Langhoff

Re: WebDAV for "Course Files" in 2.0

by Petr Skoda -
Picture of Core developers Picture of Documentation writers Picture of Peer reviewers Picture of Plugin developers
It is still not too late to improve the FIle API a bit smile
Average of ratings: -
In reply to Petr Skoda

Re: WebDAV for "Course Files" in 2.0

by Martín Langhoff -

Some nice things could come of that smile

  • Have the same function names as for files
  • Not require that directory paths are terminated with a slash( ie: find the directory either way).
  • In the API, use the "full path" instead of 'path' and 'name' separately. Right now the handling of files and directories is inconsistent.
  • Separate the permission checks to a separate function. The code needs to see if there are files regardless of the logged in user and applicable permissions.

That's what I have from my notes while I was hacking on this.

Average of ratings: -
In reply to Martín Langhoff

Re: WebDAV for "Course Files" in 2.0

by Martín Langhoff -

More updates pushed out to the git repo...

We now have a partially working MOVE/COPY, PROPFIND and LOCK/UNLOCK.

Right now I am mainly held back by File API mixed

  • How do I copy a file?
  • How do I copy a directory recursively? (If I can't, can I least have a "find" like System::Find?)
  • How do I move a file? A directory?
  • How do I update a file with a new version? (Edit: tried to create a new file with the same path, but I get a DB exception).
Average of ratings: -
In reply to Martín Langhoff

Re: WebDAV for "Course Files" in 2.0

by Petr Skoda -
Picture of Core developers Picture of Documentation writers Picture of Peer reviewers Picture of Plugin developers

When working on the low level (== working with own files), the code is using only file_storage class:

  • "copy file" - in the low level API create a new file using file_storage::create_file_from_storedfile()

  • "move file" - create a new and delete old

  • "overwrite file" - delete file, create new

  • finding of files is quit difficult, because if you use the complete file "address" no indexes are used because you hit the db index size limit sad this was the reason for adding of pathhash into the file table

  • "trailing slashes" - no, core is accepting only valid parameters, no guessing there

  • "directories" - they are not so important any more, most of the areas have flat file structure (personal files is one of the exceptions); all the directories are in fact emulated via a really ugly hack

Surprisingly we did not need these in modules. In fact all the file handling is done in draftareas and changes are then merged into the target file areas automatically. Please note there is no access control stored in the file_storage layer. Access control is always defined separately in file_browser classes and file serving scripts (file.php, pluginfile.php).

General file browsing (such as in repository) and file manager must not access the files directly via file_storage. Instead you have to use the file_browser class. This class is not finished yet, somebody has to implement all the high level commands such as copy, move, overwrite. The file browser uses similar addressing system as the file storage, but the mapping is NOT 1:1.

I have quickly looked at your webdav class, there is a major problem, you must not mix file_browser and file_storage at the same time, sorry. In the case of course files (which will be disabled in all new installs) you need to use only file_browser and add any missing methods into file_info and file_info_coursefile classes.

Average of ratings: -
In reply to Petr Skoda

Re: WebDAV for "Course Files" in 2.0

by Martín Langhoff -

course files... will be disabled in all new installs

Petr,

what will users use when they want to host files as part of their course?

You cannot require that every file is referenced from somewhere. Users will want to be able to saw "hey, I dropped a bunch of PDFs there that may be interesting -- look at the further reading directory".

Average of ratings: -
In reply to Martín Langhoff

Re: WebDAV for "Course Files" in 2.0

by Petr Skoda -
Picture of Core developers Picture of Documentation writers Picture of Peer reviewers Picture of Plugin developers

If you are going to just "drop a bunch of PDFs" you will use the folder module. There is always some better place that the old course files wink Benefits are:

  • no more copying of all course files when restoring just one activity
  • no more hacking with site files
  • easy duplication of module instances
  • fewer file renaming issues
  • 100% correct file access control - no guessing
Average of ratings: -
In reply to Petr Skoda

Re: WebDAV for "Course Files" in 2.0

by Martín Langhoff -

Ooooook.

The WebDAV code for 1.9.x only manages stuff in 'course files' (and by extension, 'site files' which are 'course files' from the sitecourse).

So my guess then is that the WebDAV thing only makes sense to provide access to content managed by mod/folder. So the plan would be to display

  • Top level dir is course listing ('my courses')
  • Next level is each mod/folder modinstance
  • Next level is whatever is in the folders

Right now mod/folder barfs at me (stacktrace below). Let me know if you get a chance to work on it. Otherwise I'll look into it next time I get a chance to work on this smile

Thanks for the hints you've given me that I should skip the browser object and use the storage object directly -- this also means that my code gets to do the access control.

I don't agree with your position that the core "doesn't do guessing" on file vs directory. There are valid situations where you are given a path and you just don't know whether it is a file or a directory. I can query the backend twice, or the backend can DTRT and issue a single SQL query with a 'path in ('/var/foo', '/var/foo/');

This also means that when we create objects we must fail with a 'file exists' when we get a file or dir creation that has a name conflict.

In other words, there is a good case for storing all paths without the trailing slash -- the implementation gets a lot simpler (when performing a lookup, trim the trailing slash if present), and the unique index handles the 'conflict' case.

All filesystems and fs-like backends work this way, and I strongly believe we should as well. Otherwise, we'll write a lot of code papering over the impedance mismatch.

And -- the stacktrace -- which has ignored my $CFG->debugdisplay=0 sad --

Coding error detected, it must be fixed by a programmer: You cannot call $PAGE->set_activity_record until after $PAGE->cm has been set.

Stack trace:

* line 653 of /lib/pagelib.php: coding_exception thrown
* line 55 of /mod/folder/view.php: call to moodle_page->set_activity_record()
Average of ratings: -
In reply to Martín Langhoff

Re: WebDAV for "Course Files" in 2.0

by Petr Skoda -
Picture of Core developers Picture of Documentation writers Picture of Peer reviewers Picture of Plugin developers

hmm, there is still some misunderstanding here - you are not supposed to access mod files in most cases, of course there are always exceptions. We should not encourage teachers to access the files directly if there is a high danger that they will break stuff. I do not think it would be wise to give users direct webdav access to files in course descriptions, forum introductions or posts, etc.

I think the webdav should allow things like: * access personal files area (read/write) - you can rename, delete, move files around there without causing and link breakage * access Folder module files - read/write for teacher * download student assignments, maybe upload marked assignments - teachers only

In general all users should be encouraged to use the repositories and all code should be using the forms+draftareas for management of files.

How does the webdav fit in? It is not practical to create repository on your local PC and link it from moodle, webdav allows you to upload files into moodle repositories and manage the files there (such as your personal files area).

Hmm, I am starting to think that Webdav is being killed by programs like Dropbox. I would prefer to launch a small deamon on my computer which syncs content of one of my directories with my personal files area in Moodle. Why should we mess with slow and buggy Webdav clients that you can use only when on-line?

Average of ratings: -
In reply to Petr Skoda

Re: WebDAV for "Course Files" in 2.0

by Martín Langhoff -

Petr,

Perhaps my post was unclear: I am implementing access only to the mod/folder area. And it is only for teachers & coursecreators.

So in agreement with you smile

The mod/folder area is also an important collaboration place for teachers. Now that this is in mod/folder, it can be a good facility for such collaboration using a 'hidden' instance for example.

Average of ratings: -
In reply to Martín Langhoff

Re: WebDAV for "Course Files" in 2.0

by Martin Dougiamas -
Picture of Core developers Picture of Documentation writers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers
I think the best way forward for WebDAV is just to look at the personal user fileareas only.

In this case WebDAV is just a way to easily dump files into Moodle via drag and drop (perhaps lots of them at once). They end up in a private "repository" that only you can access.

Then when you want to share files or use them in Moodle, you then go to those activities/courses and use the filepicker to bring the files from your private filearea into those more public contexts (forums, resources, etc etc).

I think this workflow would work well for a lot of teachers.

It'll be mostly useful for people who don't already have another type of supported repository that they would use instead (like Dropbox, Box.net, SAMBA etc)
Average of ratings: -
In reply to Martin Dougiamas

Re: WebDAV for "Course Files" in 2.0

by Martín Langhoff -

Ok. I will make sure personal areas are done as well.

My experience with various webdav uses in the past (using apache's mod_dav on moodledata before I had the php code) is that institutions have teams of coursecreators, who want to work quickly preparing and exporting the materials to a series of courses.

These are the kinds of users that asked for it. The closest match in 2.0 is the course folder. And I agree w Petr that it clears up all the access control issues we had by defining a clear context for it.

Average of ratings: -
In reply to Martín Langhoff

Re: WebDAV for "Course Files" in 2.0

by Pedro Jiménez -
Hi, I have seen that is available for the 1.9.x Webdav although the problem is that
the (http://git.catalyst.net.nz/gitweb?p=moodle-r2.git a = shortlog; h = mdl19-webdav)
not really like using it.
I wanted to test the webdav to be in 2.0 in 1.9.6 since we webdav created by Toni Mas, but we would be testing the new modules
and utilities that will appear in 2.0.


thanks
Average of ratings: -
In reply to Pedro Jiménez

Re: WebDAV for "Course Files" in 2.0

by Martín Langhoff -
I didn't know Toni Mas had prepared a 1.9 webdav implementation. Interesting!

I have a draft 2.0 implementation, but I have to re-implement it a bit after comments from Petr and also a bit of thinking of my own.

Right now I am tangled with work on XO-1 and XO-1.5 OS images. Trying to find the time to get this done.
Average of ratings: -
In reply to Martin Dougiamas

Re: WebDAV for "Course Files" in 2.0

by Lilian Austin -

I am new to Moodle after years using WebDav and Dreamweaver to publish to a range of Blackboard shells.  At the moment we are looking at how we can republish lots of course content into Moodle as the migration process has resulted in pages and pages of scrolling which is horrible.  So we have decided to start from scratch with some subjects.  For example i currently have lots of learning modules set up in dreamweaver which were placed into the BB Learning Module too,l which unfortunately isn't replicated in Moodle.  Now I want to upload these html files together with images/swf files etc in one go. Is this possible?  can you point me to some documentation that describes the process?

Cheers Lilian

 

Average of ratings: -
In reply to Lilian Austin

Re: WebDAV for "Course Files" in 2.0

by Mary Cooch -
Picture of Documentation writers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Testers Picture of Translators

Are you using Moodle 2.0? Or 1.9? Either way you can put all your Dreamweaver files into one folder ( like a website) ,zip/compress the folder, upload the zipped folder and unzip it and then on the course page link to the index.html file. How you do this depends on whether you are using 1.9 or 2.0

Average of ratings: -
In reply to Mary Cooch

Re: WebDAV for "Course Files" in 2.0

by Lilian Austin -
Hi thanks forvyour reply We are about to go to moodle 2 at La Trobe. I think I am struggling to understand the moodle way of doing things? In BB i set up a webdav connection and hit the 'put' button in dreamweaver and all files/folders where put up instantly. Then those files could be used in multiple places within the subject. I hope I don't have to zip and unzip. That would be like stepping back 10 years and would be incredibly time consuming.
Average of ratings: -
In reply to Martín Langhoff

Re: WebDAV for "Course Files" in 2.0

by Joseph Rézeau -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers Picture of Translators

Any re-assuring news about WebDav in Moodle 2.0?

Years ago when we were using WebCT at my university I found the WebDav feature an indispensable feature and am very much looking forward to having it in Moodle at long last.

Joseph

Average of ratings: -
In reply to Joseph Rézeau

Re: WebDAV for "Course Files" in 2.0

by Simon Hanmer -
To echo Joseph's question - any updates on webdav for Moodle 2.0?

I've tried configuring a repository using the existing setup and had no luck - I don't know if this is just because it isn't fully implemented yet or if there's a problem.

Thanks
Average of ratings: -
In reply to Simon Hanmer

Re: WebDAV for "Course Files" in 2.0

by K P -
I have had the same problem. I can connect to WebDAV through both my Mac and my Windows XP laptop, but Moodle 2.0 WebDAV repository shows nothing.

I have even tired the "/#" trick at the end of the URL.
Average of ratings: -
In reply to K P

Re: WebDAV for "Course Files" in 2.0

by K P -
Has anyone found a solution to the webdav problem?? Can anyone confirm if webdav repositories are actually working right now in the HEAD?
Average of ratings: -
In reply to K P

Re: WebDAV for "Course Files" in 2.0

by K P -
Does anyone have a clue? It is still not working for me in 2.0 RC1

Here is what I am doing:

Admin -> Plugins -> Repositories -> Manage Plugins

I have WebDAV enabled and visible. I set up a new instance and entered the URL just as I have for my iMac AND my WinXP laptop (both of which can connect to my WebDAV with no problems at all). I entered my username and password.

When I go to insert an image when writing a question, I can see my WebDAV in the file picker but there is nothing in it, which there in fact is. As I said, I can see it just fine on two different computers connecting to the same WedDAV.
Average of ratings: -
In reply to K P

Re: WebDAV for "Course Files" in 2.0

by Simon Hanmer -
If anyone has anymore evidence or wants to add their voice - I do have a raised issue on this @ MDL-22663
Average of ratings: -