General developer forum

Some background for those following at home...

• Here we have some some background to Eloy, Helen and Tom's code for IMS repositories. Seems to be (all?) in HEAD.
• Matt Oquist is working on a full blown repo API in contrib based on some discussions with MartinD. Actually, there's good documentation for it
• OSLOR project implementation by Jun -- based on a draftish spec (pdf, attached). This is on a branch very closely related to HEAD, we are keeping it in GIT, here is the gitweb page for the oslor branch - you can click on any commit, get a tarball (snapshot link) or look at the files (tree link).

A bit more background on the OSLOR implementation -- which I know hasn't been discussed much in the public space

• MartinD tells me that repo is not a good directory name. Shall be renamed to repository
• When designing it, we planned to support a range of backends with different features and limitations. Our thinking was in supporting Hive-like "smart" backends as well as Debian-repository-like "stupid" backends.
• It provides infrastructure for repo plugins to maintain their own db tables if needed (similar to modules).
• Each repo plugin defines its config page (similar to auth and enrol plugins), all reachable under /admin.
• A single Moodle install can have many backend repos configured (as a debian system can have many archives in sources.list)
• It is up to the plugin how tight/loose the integration with the backend is. "Stupid-backend" plugins don't really need the backend to be there.
• Once the file/resource is found, we copy it to the course directory in moodledata and pass the ball to the module. This means that the existing modules don't need much rework to take advantage of it. MartinD tells me this is a significant difference with Matt's work. In any case, smartish backends like Hive sometimes provides just a URL rather than the actual file. Not sure if all modules will like this without some changes

The OSLOR work on Moodle has a complementary part in work we have done on GNU EPrints, which is a great (Learning) Digital Object Repo, with metadata support and other niceties. We have taught EPrints to create a published archive that looks a lot like a Debian archive (possibly the most popular Digital Object Repository today by any standard). The main differences are that we don't care about 'architecture' and Packages is now an XML file full of IMSPackage/SCORM-ish metadata.

The plugin for that is a bit like apt-cache for those familiar with the Debian infrastructure. It fetches the index and performs all the searching locally -- optionally it can fetch all the objects too, so it is a boon for disconnected Moodle setups.

If you want to fetch the OSLOR code, you can get a snapshot in tar.gz or tar.bz format from the gitweb page or (even better) install a recent GIT and Cogito and do

  cg-clone http://locke.catalyst.net.nz/git/moodle.git#mdl-oslor

Feel free to ask any questions about it. I have surely forgotten something...

END_OF_SPAM

Wow - this certainly is exciting.

How ready it is -- I think it's close to being a workably complete API, but nobody (including me! (to my knowledge)) has carefully reviewed the entire API, and I'm sure it has some weaknesses that need to be changed. I changed a few things when I modified my portfolio module to use my repository, but a good, careful review of the API would be a very good thing.

DEMO: You can check out my repository work already running at http://portfolio.spdc.org/portfolio/. Just log in and poke around the "File keeper" block. If you want admin access just email me after you create your own account (and if I don't already know who you are, I might ask to know a little about you first). Admins have the additional abilities to create arbitrary files/folders in the top-level repository folder (just like root access to the root directory) and change resource ownership.

Whether you can help us fit it in our plans instead of the simple repo API -- Yes, I can do that. I agree that starting with the simple API and evolving into a more-capable one may be the best approach. (I just spoke with my sponsors, and I have funding to work on this. Yay!) Here's the only cramp on my involvement -- I'm presenting the portfolio module to Massachusetts educators on March 20th, and I definitely need to move that effort significantly forward by that time. So if I want to be working like an insane person, I can surely do both!

I did try to make relatively few assumptions about the backends, but I'm sure some unintentional assumptions snuck in, too. Firstly, I (like you, I believe) started with a base class that can be extended by plugins for each type of supported backend.

• filesystem-ish: Yes, that tallies with my primary assumption about which API methods each plugin will implement. Ahh - in my in-code documentation about the API, I didn't make a distinction between APIs each backend should implement and APIs each may implement. I've added appropriate comments now, and I'll commit to CVS soon so it's public. But here's what I was thinking each backend should implement:
  • new_resource($type=0, $path='', $resourceid=0, $interactive=false, $new_resource_data='', $new_version=false, $inherit_access=false);
    • $type: REPO_TYPEFILE, REPO_TYPEFOLDER, REPO_TYPEURL, etc. Right here we see an assumption that backends won't be adding additional resource-types that the base doesn't know about... but at the moment I can't think of any problems that would arise if a plugin did add its own custom resource-type. Interesting.
      
    • $path: optional -- /filesystem/like/path within the repository; i.e., each repository is like a DOS drive, with its own "root directory". This is a path to an existing non-folder resource (if the backend supports versioning or overwriting old resources) or else the parent folder of the new resource. $path always takes precedence over ID in every API that has them both.
      
    • $resourceid: optional -- specifies a resource by ID instead. The ID is stored in the object as $blah->nativeid, and should be set by the repository plugin to any ID that the backend may associate with the resource. Hmmm. Possibly bad assumption: All backends will have IDs for resources. (Removing this assumption should be feasible since almost all the APIs take in $path and an ID.)
      
    • $interactive: bool -- Whether to display errors, warnings, etc. interactively.
    • $new_resource_data: assoc. array of data for a new resource (name and content for a file, name and URL for a URL, just name for a folder)
      
    • $new_version: bool -- helps to handle some simple checking about filetypes. This parameter should probably be dropped; it won't hurt anything for plugins that don't support versioning, but it's probably just unnecessary/confusing paranoid overkill.
    • $inherit_access: bool -- whether or not to inherit the ownership and access_controls of the parent folder. Backends without ownership and ACLs can just ignore this, of course.
  • new_file($resource='', $new_file='', $interactive=false, $source_path='', $inherit_access=false)
    • $resource: object -- parent folder (or old file version)
    • $new_file: optional, object/array -- new file data with displayname and content
    • $interactive: bool -- same as above
    • $source_path: optional, for copying legacy Moodle files in from $CFG->dataroot
    • $inherit_access: bool -- same as above
  • create_folder($folder='', $new_folder_name='', $interactive=false, $make_parents=true, $preexisting_ok=false, $inherit_access=false)
    • $folder: object -- parent folder
      
    • $new_folder_name: string
      
    • $interactive: bool -- same as above
      
    • $make_parents: bool -- if necessary, create parent folders
      
    • $preexisting_ok: bool -- return "true" if folder already exists
      
    • $inherit_access: bool -- same as above
  • read($fileobj='', $expected_type=0, $add_parents=true, $interactive=false)
    • $fileobj: object -- resourse we're going to read (should probably be "$resourceobj" or just "$resource")
      
    • $expected_type: optional -- expected REPO_TYPE* of $fileobj
      
    • $add_parents: bool -- whether to add a runtime-only Parent folder when reading a folder's contents
    • $interactive: bool -- same as above
  • delete_resource($path='', $fileid=0)
    • $path: optional -- in-repository /path/to/doomed/resource (takes precedence over $fileid)
      
    • $fileid: optional -- $blah->nativeid of the doomed resource. This should probably be $resourceid instead.
      
  • get_file_object($path='', $nativeid=0, $expected_type=0)
    • $path: optional -- in-repository /path/to/resource (takes precedence over $nativeid)
      
    • $nativeid: optional -- $blah->nativeid of resource
      
    • $expected_type: optional -- REPO_TYPE* that we expect to get (return false if specified and non-matching)
  • get_file_path($nativeid=0, $crumbs=false)
    • $nativeid: $blah->nativeid of resource whose path we want
      
    • $crumbs: bool -- generate path as a breadcrumbs list for the header
    • The native plugin I wrote has an additional parameter, but that parameter is specific to that plugin.
• few required calls -- The preceeding list is relatively short, I think. One thing I have not done is go through my repository_base class carefully to be absolutely sure I never assume that other methods are also implemented. Let me be more specific: There are ~10 other methods that must exist for each plugin, but they can be empty and simply return appropriate values. Since each plugin provides its own controls to the user interface, these unimplemented methods should only be called when the APIs are used by other code in Moodle.
• perfectly happy with repositories/backends that are not always up, or reliable -- The plugin itself is assumed to be responsible for establishing a connection to the (possibly remote) backend, so if the connection isn't up or can't be established, the plugin will have to deal with that (and probably just return "false", etc.)
• perfectly happy with repositories/backends that are not always...browseable -- same as above, but I should note that the repository_base class provides only an architecture within which each plugin can implement its own browsing methods. The repository_base method is list_index(), and the plugin methods follow:
  • list_index($path='', $ids=array(), $recurse_levels=0, $form_action='', $hiddens=array(), $return=false, $headers=true, $columns=array(), $menu_names='', $exclusive_type=0)
    • $path: in-repository /path/to/resource
      
    • $ids: array -- list of resource IDs of resources to display
      
    • $recurse_levels: int -- number of levels to recurse down under folder(s)
      
    • $form_action: <form action="$form_action">
      
    • $hiddens: assoc. array -- print hidden form inputs
      
    • $return: bool - return the index or print immediately
      
    • $headers: bool - print headers
      
    • $columns: array -- ordered list of names of columns to display in the index table
      
    • $menu_names: array -- ordered list of menus to display for the index
      
    • $exclusive_type: REPO_TYPE* of a single resource type if we only want to display files, folders, URLs, etc.
      
  • file_display_index($table='', $file='', $baseurl='', $formname='', $columns='', $recurse=0)
    • Used to add one or more rows to an "index" table that displays a group of resources (primarily used for browsing through folders). This should probably be named "resource_display_index()".
      
    • $table: object -- for print_table()
      
    • $file: object -- file for which to display index row(s)
      
    • $baseurl: URL to which the form will submit
      
    • $formname: form in which table will be printed
      
    • $columns: ordered array of names of columns in the index table
      
    • $recurse: int -- how many levels down to recurse if $file is a folder
  • function index_menus($formname='', $menu_names='', $return=false)
    • Used to create menus that apply to the entire index table.
      
    • $formname: form in which the menus will be included
      
    • $menu_names: ordered array of names of menus to create (right now I'm using 'groupaction', 'add', and 'search' in the native plugin, but this is entirely plugin-dependent)
      
    • $return: bool -- return the menu data instead of printing immediately
  • file_action_menu($file='', $formname='', $return=true)
    • Used to create an action menu that applies to a single resource. (This should probably be "resource_action_menu()".)
      
    • $file: object -- file for which we're creating a menu
      
    • $formname: name of the form this will submit to
      
    • $return: bool -- return the menu data instead of printing immediately

I hope this gives a better picture of what I've done and what I've had in mind.

If we want to head toward an evolutionary approach from the simple toward the complex, then we'd best

1. review the APIs at either end of the spectrum to be sure of our starting and ending points, and then
2. craft our starting set to be extensible to the ending set by only adding parameters to existing methods and adding new methods.

It's possible that we could then use a simplified(?) version of the repository_base class to start with, and grow toward a more complex ending repository_base.

I have a paper due tomorrow (I'm a grad student) that I really need to start on, so I should stop with this for now. But tomorrow I could possibly take more of a look at the simplified API...nah, I'll look now.

The following are the class methods you list in the above-attached PDF:

• cron: Great idea to fetch updated package/resource lists regularly. (The native plugin can just "return true".)
  
• simple_search: Also good idea, more mature than what I slapped together for searching. Hmm. Alternatively, perhaps the search form could provide separate text boxes for each field? Then the backend doesn't have to parse the text. Is there a reason to support the complex text parsing? (I know lots of repositories use things like au:foo ti:"bar baz".) If we have a class method that returns a list of supported search fields, then the generic/simple search form can simply disable (or disappear) the unsupported ones.
  
• advanced_search_form: great. Perhaps this method simply prints out additional search fields after the simple search form is include()d?
  
• advanced_search_results: This might be doable with the current way I have columns defined and displayed in the index tables, if a tabular format is sufficient here. repository_base->list_index() will call down into the plugin's index-displaying methods, and those methods put whatever is appropriate into the index table. Perhaps if we add a parameter to repository_base->list_index(), then we can specify a "level" or "type" of display that plugin->file_display_index() should insert into the table. If a tabular format is inappropriate here, then a separate method is appropriate.
  
• fetch_object: Yep. I called it get_file_object(). ("get_resource_object()" would be a better name.)
  
• object_path: Yep. I called it get_file_path(). ("get_resource_path()" would be better.) We might be making different assumptions about what these paths are; as I mentioned above I'm picturing something like DOS disks, where each repository has its own independent filesystem tree, and all the paths that get passed around (or that I expected to be passed around) in the repository API are already associated with a particular repository.

With all that said, the only significant things I know my repository implementation needs are paging and backup. Right now it attempts to return/display everything under a folder every time, and that really breaks when you have 10,000 things in a folder. If we get these taken care of and we do some serious testing (and it holds up sufficiently under testing), I think we could consider using this implementation now and integrating the other repository work with it. Here are further thoughts along those lines:

• It shouldn't be too difficult to fit what you've already done in OSLOR into the my repository infrastructure. Based on my reading of your PDF, anyway, the simple APIs should fit right in.
  
• Just to note again -- paging must be added. I didn't even think of it, which is a huge oversight. This will involve changes to the repository_base class and my "native" plugin.
  
• Just to note again -- backup must be added. At least the native plugin stores all its folders and files on the actual HDD, so those will get backed up by normal system backups. But the metadata and ACL stuff has no backup yet. This will involve changes to the repository_base class and my native plugin.
  
• Just to note again -- testing must be done. I've obviously tested things myself, and I think a couple other people have poked around at things casually, but this needs to be seriously hammered at. I've done just a tiny bit of performance testing.
  • I added 10,000 files to a folder and immediately removed them, because what was the point? Without paging, the repository was useless to display that folder so I could do anything with it.
  • I shared a file to 3,000+ individual users, and then un-shared from all of them at once, through the user interface. IIRC, this completed, but it took a few minutes. (Only 50(?) users at a time are displayed in the right-hand column, just like the create-new-admins interface, so sharing to 3000+ users simultaneously -- through the user interface -- is not possible.)
• Performance tuning (of my "native" plugin) -- I have not made the slightest attempt to make this effecient. I wanted to make it work first, and increase complexity for the sake of efficiency after I had it working. But I think there is quite a bit of low-hanging fruit here; there are several places we could just add an array parameter to a method and get it (and the database) to process lots of resources all at once. This functionality would replace PHP loops that do database lookups, call other methods, and do database writes for multiple resources. Sloooooooow.
• We must keep in mind that adopting the "native" plugin for my repository architecture requires us also to adopt the access_control class that I've implemented. Of course, I think it's a fantastic piece of work , but others may disagree or at least have reservations. It attempts to provide completely generic ACLs that can be applied to absolutely anything (even concepts, roles, etc.) in Moodle. There is some slightly out-of-date documentation of the access_control class API here. There is also a flowchart of how it does access-checking here. (If the flowchart isn't perfectly up-to-date, it's close.)
  • At the moment, the access_control class supports ownership and access_control (of whatever type -- read, write, rename, etc., plugins can add their own) by userid, by groupid, and by courseid.  We're planning to add a system by which expiring access keys can be created, so that non-Moodle users can be sent URLs containing access keys that grant them a specific access for a given amount of time.  ("Click here to view my portfolio.  You will have access until March 25th...")
  • I'm planning to add configurability so the administrator can grant/deny access for students/teachers to publish/share (etc.) by userid, groupid, or courseid.  Put more simply, and by example, I'm thinking that a school might not always want kids to be able to share their files to one another, or to entire groups or courses, or by key.  Also, some schools may want to disable (especially) access-by-key altogether, to make sure that only Moodle users can access data on the site.
    

OK, now I need to go work on that paper.

Let me just make it very clear to everyone what we are facing here before this gets too excited and confusing.

For Moodle 1.6 release, a very small modification to rationalise two repository-related things that are already in Moodle 1.6 dev. It's too late to do anything major for this release, especially with many features still incomplete. In fact, if it can't be done by the end of this week then I'm happy to leave it as is.

For Moodle 2.0, a complete and radical new repository API that allows a number of repository plugins available from ALL file points in Moodle (resources, SCORM, assignments, forum attachments etc)

• read-write access to a DMS (defaulting to the server filesystem) for all users (not just teachers) with Moodle holding only links to that DMS.
  
• read-only access (with optional local caching) to external repositories like MERLOT, NLN, eprints etc.

The API is actually very thin, just providing the hooks, really.  Most of the code will be in the plugins.

Here's more info about the APIs I've implemented in the Moodle Native repository plugin, which I was intending to become a sufficient replacement for "Site files" and all the file.php stuff.

For the most part this is not carefully ordered.

• change_access($ac='', $specify=true, $hiddens='', $recursive=true)
  • Used to specify or unspecify an access control, which may be a grant or a denial.
  • $ac: an access_control object or an assoc. array that defines one
  • $specify: whether we're specifying or unspecifying a control
    
  • $recursive: whether to recurse down through folders
• set_owner($resourceid=0, $ownerid=0, $owner_type=0, $safe=false)
  • $resourceid: ID of the resource whose ownership is being set
    
  • $ownerid: ID of the new owner
    
  • $owner_type: is the owner a user, a group, or a course?
    
  • $safe: if true do NOT perform access checking
• set_access_a_like_b($aid=0, $bid=0, $safe=false, $setowner=true, $setaccess=true, $yesno=-1)
  • $aid: ID of the item whose access we're setting
    
  • $bid: ID of the item whose access we're copying
    
  • $safe: same as above
    
  • $setowner: set the ownership of a like b
    
  • $setaccess: set the access of a like b
    
  • $yesno: if 0 or 1, set only access grants or denials, else set all
    • I added this because I wanted new files to inherit the access controls of the parent directory, but users are denied 'delete' and 'rename' access to their home folders, so all the files they created inherited these denials. Now I can specify '1' here when the users home folder is the parent directory, and only granted access controls are inherited in that case.
• versions_index($path='', $resourceid=0, $return=false)
  • Create a table of all the previous versions of a specified resource.
    
  • $path: path to the resource (takes precedence over ID)
    
  • $resourceid: ID of the resource (precedence given to $path)
    
  • $return: whether to return the table instead of printing it now
• delete_resource($path='', $fileid=0)
  • Heh, I forgot to list this earlier even though it's basic.
  • This is assumed to be recursive -- if $path or $fileid specifies a folder, that folder and everything under it will be affected.
    
  • The Moodle Native plugin has a "trashcan" into which all deleted resources go, so this routine actually only moves all the resources and sets their 'trashed' field in the DB.
  • $path & $fileid behave like all the other APIs
    
• empty_trash()
  • Implemented but not accessible from the UI yet.
• find($path='', $fileid=0, $ids='', $follow_links=false)
  • Recursively 'find' all the files/folders/etc. under the specified file/folder.
  • $path & $fileid -- typical
  • $ids: used by recursive calls to construct the list of IDs and protect against loops
    
  • $follow_links: I haven't yet implemented file linking, but this will be useful once I have.
• rename_resource($path='', $fileid=0, $new_file_name='', $interactive=false)
  • $path & $fileid -- typical
  • $new_file_name: string
    
  • $interactive: whether to display errors, etc.
• does_name_collide($path='', $folderid=0, $new_file_name='', $interactive=false)
  • Check for a name collision in the specified folder.
    
  • $path & $folderid -- typical
  • $new_file_name: string
    
  • $interactive -- typical
• copy_a_to_b($spath='', $sresourceid='', $dpath='', $dresourceid='', $newname='', $recursive=true, $interactive=false)
  • Copy resource 'a' to location 'b'.
  • $spath: source resource path (takes precedence)
    
  • $sresourceid: source resource ID ($spath takes precedence)
    
  • $dpath: destination resource path (takes precedence)
    
  • $dresourceid: destination resource ID ($dpath takes precedence)
  • $newname: string, new resource name, if any
    
  • $recursive: recurse through folders, etc.
• change_owner_form($path='', $fileids=array(), $recurse=true)
  • Present the user with a form to change the owner of one or more resources.
  • $path: path to a single resource (takes precedence)
    
  • $fileids: array of one or more resource IDs ($path takes precedence)
    
  • $recurse: recurse down through folders
• change_access_form($path='', $fileids=array(), $access_type='')
  • Present the user with a form to change the access granted or denied to the specified resource.
  • $path: path to resource (takes precedence)
  • $fileids: array of one or more resource IDs ($path takes precedence)
  • $access_type: REPO_ACTION_SHARE, ...SHARER (recursive), ...PUBLISH, ...PUBLISHR
    
• can_i_access($accesstype=0, $nativeid=0)
  • Test whether the current user has the specified access to the specified resource.
  • $accesstype: REPO_CHMOD, ...CHOWN ...COPY ...DELETE ...LINK ...MOVE ...READ ...RENAME ...WRITE ...etc.
    
  • $nativeid: ID of the resource. (Hmm. Why isn't this taking in a $path as well?)
• change_owner($path='', $fileid=0, $ownerid=0, $ownertype=0, $recursive=true)
  • $path & $fileid -- typical
  • $ownerid: same as above
    
  • $ownertype: same as above - can be a user, a group, or a course (or whatever else the access_control module supports)
• download_file($path='', $fileid=0)
  • This replaces file.php. It performs all the necessary access checking and then sends the file if everything's legit.
• find_user_shares($access_type=0, $userid=0, $how='')
  • Find all the resources shared to the specified user with the specified access.
  • $access_type: only REPO_READ, REPO_WRITE are in the switch(){} ATM, because I didn't think other types would be especially useful.
    
  • $userid: obvious
    
  • $how: user, group, course. This lets us find only access granted by virtue of my userid, my group memberships, or my course enrollments, if that's what we want to find.
• user_shares_index()
  • Present an index table of all the resources shared to a user.
• delete_access($ac='', $ids='')
  • Delete all access_control records (but not ownership) for a specified(set of) resource(s).
  • $ac: access_control object
    
  • $ids: list of resource IDs
• search($search_criteria='', $exclusive=true)
  • I should change this to support the simple search strings you're doing in OSLOR.

That's it ATM. I have dummies in there for things like move_a_to_b(), link_a_to_b() and lock(), but those are low-priority for me so I'm leaving them alone for now.

Hi,

Please excuse my post if it does not make sense, I am relatively new to moodle.

From what I understand each of us is making/solving a different problem space but are using the same term - repository.

Matt's repository is about a central repository that moodle can use, very much like a file manager.  So it deals with access, organizing objects, etc.  While Eloy's IMS CP is about getting an IMS CP and deploying the package.  It is not a remote repository, but more of a user/consumer of a repository (remote or local).  The oslor repository is about finding and getting objects from a remote repository, which is similar what the current hive/(resources/type/repository) is doing.

The way I understand it, there is no consolidation work that needs to be done other than hive/(resources/type/repository) and the oslor repository.  These are the 2 things that I see that deals with remote objects, rather than managing the objects in moodle.  Matt's repository is a complementary part, wherein it manages the objects once its been fetched/linked from a remote repository.

Does this make sense?

Hi,

Just posting a link to an interesting discussion in a forum you developers may not be subscribed to: RFC ::: Open University Repository System Proposal.