General developer forum

Using Google Cloud Storage on Moodle

 
Picture of Jorge Castro
Using Google Cloud Storage on Moodle
 
Hello, 


I was trying to connect Google Cloud as a repository for Moodle, for large files like videos, doing some doc research i found I can use Cloud Storage FUSE to use a bucket as a file system, and used vim to edit the config.php, but it doesn't seems to work. 

Config.php line:

$CFG->alternative_file_system_class = '\\local_myfilestorage\\file_system';

where I used the generated FUSE local:


$ gcsfuse example-bucket /local_myfilestorage/


I used the Bitnami Moodle in Cloud Laucher.


Has anyone tried this?


Thanks in advance for the responses!

 
Average of ratings: -
Picture of Ken Task
Re: Using Google Cloud Storage on Moodle
Group Particularly helpful Moodlers

Do have a Google Bucket on a CentOS 7 server running multiple versions of Moodle.

Google documentation page : https://cloud.google.com/storage/docs/gcs-fuse
Github repo : https://github.com/GoogleCloudPlatform/gcsfuse

I'd have to go back and look at my cli history and try to figure out what worked and what didn't as well as edit a text file I had for note taking .... some stuff can't share with anyone.

Can tell you that it took two days. :\

Am using a mount point in moodledata/repository/ - 2 of them as a matter of fact:

sossrv                               1.0P     0  1.0P   0% /mnt/gbucket
sossrv                               1.0P     0  1.0P   0% /mnt/gb

Uhhhhh .... that's 1 Pentabyte!!!!   The two above are really in different Moodles but same google buicket by a different named mount point.

Of course that will cost ... and I've yet to actually find a tool that would help calculate the cost .... yep, it's not free.

Was thinking of using it for autobackups ... keep all ... for several sites.

Since it was command line, there will be some commands that will require you to copy a url ... go to it so you can get an access key which you need to plugin to the initial access to the bucket.

'spirit of sharing', Ken



 
Average of ratings: -
Picture of Jorge Castro
Re: Using Google Cloud Storage on Moodle
 

First of all, thanks for your response!

I did not have any trouble while installing and mounting the bucket as a filesystem with FUSE, the problem occurs when editing the config.php file, i save it and when I enter the site to try to make some testing i get error:

Exception - Class '\[dataroot]/repository\file_system' not found

Could you please help me with the correct syntax?


Do you have to create the repository in the web UI? 


Thanks :D


 
Average of ratings: -
Picture of Ken Task
Re: Using Google Cloud Storage on Moodle
Group Particularly helpful Moodlers

What do you see when issuing:

df -h

Any file system in Moodle has to be created manually inside /pathtomoodledata/repository/ or one could use a symlink as I have done to mount points.

In a Moodle 33 that's using a Google Bucket I have a partition called home that is the largest and use it for various versions of Moodle data dirs:

/home/moodle33data/repository

contents of above

lrwxrwxrwx. 1 apache apache 13 Sep 24 09:29 gbucket -> /mnt/gbucket/
lrwxrwxrwx. 1 root   apache  7 Oct  2 15:29 google -> /mnt/gb
lrwxrwxrwx. 1 root   apache 14 Sep 24 09:43 sosfiles -> /mnt/sosfiles/
lrwxrwxrwx. 1 apache apache 17 May 15 10:31 unirepo -> /var/www/unirepo/

You can see I have one physical ... ie, on the drive of the server ... unirepo

Moodle sees that as 'unirepo'

One to another backend server in that network /mnt/sosfiles/ - sosfiles that's a sshfs

Moodle sees that as 'sosfiles'

And two that are mount points for the Google Bucket ...

the 'alias' for 'google' is really the mount point /mnt/gb/ which was mounted via gcsfuse part of the google SDK linked /info provided previous post.

That's not the same as other fuse's extensions to os.

here's what I had to use to mount one of the buckets .... edited for security reasons:

export GOOGLE_APPLICATION_CREDENTIALS=/root/gcloudinfo/api-project-##ll##ll##ll###.json;
mount -t gcsfuse -o rw,allow_other,nonempty localname /mnt/themountpointname

Google docs explain options bu that 'localname' is the name given in Google config of the bucket.

I don't add anything to config.php files of the various moodles.

However, whatever you called the mount point example_xxx is what you should see when setting it up in moodle ... the box you type in is really meta data.  If you don't see it in the pick list think it means Apache can't see it ... and if apache can't see it, Moodle certainly can't. :\

Not at all sure one could use a Google bucket as the location of Moodledata ... matter of fact ... think that would lead to some issues.   Besides, fees for Google are by usage .... not only size but traffic.  Even if those look extremely cheap ... shown in pennies ... on any Moodle used on a regular basis, that could rack up a rather large variable bill I think.    Can't budget for that!

'spirit of sharing', Ken


 
Average of ratings: -
Picture of Jorge Castro
Re: Using Google Cloud Storage on Moodle
 

Ken,

Thanks for your patience i'm a bit newbie.

I used df -h command i didn't saw anything about the mount i thought I "successfully did", so I tried to replicate your steps.

I mounted my google bucket which localname is 'moodle-jorge' and used the docs & your help to do this.

I run 

$ mount -t gcsfuse -o rw,allow_other,nonempty moodle-jorge /mnt/gb <- (actually made sense for (google bucket)

And then run df -h, so it finally shown:

..... some data ....

tmpfs           1.9G     0  1.9G   0% /dev/shm

tmpfs           5.0M     0  5.0M   0% /run/lock

tmpfs           1.9G     0  1.9G   0% /sys/fs/cgroup

moodle-jorge    1.0P     0  1.0P   0% /mnt/gb <- Finally!


In the web interface i stopped received the class error (I didn't mess with the config.php this time!)

And when I tried to make a repository, it asked me to create a directory in /var/www/moodledata/ so I created a directory called Videos. How can i get the files from the mounted bucket (/mnt/gb/) shown in the UI in videos directory (/var/www/moodledata/videos/)? 

I manually added a video to the Google Storage Bucket (with public views) but it doesn't show in the moodle web UI. 

What seems to be the problem? I want to use videos as material for moodle, Upload them on the bucket and then to the moodle, so I don't have to enhance the instance memory on regular pasis, and make it a GS Bucket thing.

One thing i did notice is that if you stop the Google Cloud instance and re run it, the mounted bucket is destroyed. So I need to figure how to make this mount persistent, I suppose this is possible with a start-up script added to instance metadata, so it's not a big deal tho (I wish).

Also i appreciate your concerns about the billing in GCP. I'm a local Google partner :p so it's not a big deal, and you know the phrase: "Whatever the customer asks for." 

HAVE A NICE DAY!



 
Average of ratings: -
Picture of Ken Task
Re: Using Google Cloud Storage on Moodle
Group Particularly helpful Moodlers

First, am new at this also ... and fumbled around quite a bit.    So don't consider me any where near a google expert! ;)

In order to get the file system repo to show to Moodle ... I created symlinks in /moodledata/repository/ for a google bucket that pointed to /mnt/nameofbuket.

lrwxrwxrwx. 1 apache apache 13 Sep 24 09:29 gbucket -> /mnt/gbucket/

command as root user:

in /moodledata/repository/

ln -fs /mnt/gbucket/ gbucket

What I call the bucket on the system like that is only for Moodle ... cause the real name in the setup of the bucket on Google .... your's is moodle-jorge

Uhhhh ... that might not be a good name to use ... hate to say this but have been seeing bot scan's on all servers looking for phpmyadmins and the agent is jorgee .... hmmm ... that's not you is it? ;)

Anyhoo ... Moodle sees 'gbucket'.  But, since it is a file system when setting up Moodle to use it, one has a check box for something about being allowed to use 'aliases'.   Haven't tried that as am not sure that videos or anything as a resource in Moodle that could change (dynamic nature ... let's say video01-10212017.mp4 needs to have it's named changed to video01-10312017.mp4 ... will it work?)

Haven't tried anything like that.  Not sure the bucket can be accessed like that ... however, having said that ... hit the bucket with your browser and you'll see there is additional meta data that you can add to each file ... mimetype stuff ... as per example, mimetype info so that IF a browser can actually see/use the file browser knows it's a video/mp4.

If there were a way for the bucket to auto do that then I think I could be used like that ... other wise, I don't have time to enter meta data for every file ... do you?

I was thinking in terms of autobackups ... keep all! ;)  in something that could also be used on another moodle instance even on different IP/network to restore courses on another Moodle ... that sort of thing.

We're fumbling around together on this one. smile

'spirit of sharing', Ken


 
Average of ratings: Useful (1)